Color Grading of Content Based on Similarity to a Sample

Abstract

Systems and methods for color grading of images and videos based on similarity to a sample. As preparation for color grading of new content, a sample frame related to the expected new content may be obtained, and color grading parameters for the sample frame may be obtained. When new content is generated or received, the similarity between the frame of the new content and the sample frame can be determined in order to color grade it. The similarity between the frame of the new content and the sample frame can be combined with the color grading parameters obtained from the sample frame in order to determine the appropriate color grading parameters to apply to the new content. The new content can then be color graded using the determined color grading parameters.

Description

Technical Field

[0001] This application generally relates to systems and methods for image processing, image display, and image reproduction. Embodiments of the present invention provide methods and apparatuses for processing image data to convert colors and / or tones for display or reproduction on a local or downstream device. Some example embodiments are displays configured to receive and process image data and display it for viewing the processed image data.

Summary of the Invention

[0002] The present invention is defined by the independent claims. The dependent claims relate to any features of some embodiments of the present invention. The embodiments described herein relate to systems and methods for color grading of images and videos based on similarity to a sample. As preparation for color grading of new content, a sample frame related to the expected new content may be obtained, and color grading parameters for the sample frame may be obtained. When new content is generated or received, the similarity between the frame of the new content and the sample frame can be determined to color grade the new content. The similarity between the frame of the new content and the sample frame can be combined with the color grading parameters obtained from the sample frame to determine the appropriate color grading parameters to apply to the new content. The new content can then be color graded using the determined color grading parameters.

[0003] In an aspect that serves as an example of the present disclosure, a method for color grading a source image by a plurality of color grading operations is provided. The method includes obtaining a source image for color grading, obtaining a plurality of color grading parameter settings associated with each sample frame among a plurality of sample frames, where each color grading parameter setting serves as an input parameter for an associated color grading operation among the plurality of color grading operations, obtaining a plurality of similarity measures, where each similarity measure indicates a level of similarity between the source image and each different sample frame among the plurality of sample frames, obtaining a plurality of weighted averages based on the plurality of similarity measures and the plurality of color grading parameter settings, where each weighted average is associated with a different one of the plurality of color grading operations, applying the plurality of color grading operations to the source image by using each of the plurality of weighted averages as an input parameter to its associated color grading operation so as to generate a color-graded image, and supplying the color-graded image.

[0004] In another exemplary aspect of the present disclosure, a color grading system including at least one controller is provided. The at least one controller is configured to obtain a source image for color grading, obtain a plurality of color grading parameter settings, where each color grading parameter setting serves as an input parameter for a color grading operation and is associated with a different respective sample frame among a plurality of sample frames, calculate a plurality of similarity measures, where each similarity measure indicates a level of similarity between the source image and a different respective sample frame among the plurality of sample frames, calculate a weighted average based on the plurality of similarity measures and the plurality of color grading parameter settings, apply the color grading operation to the source image using the weighted average as an input parameter for the color grading operation to generate a color graded image, and supply the color graded image.

[0005] In another exemplary aspect of the present disclosure, a method is provided that includes obtaining a sequence of image frames, obtaining a plurality of similarity measures, where each similarity measure indicates a level of similarity between different respective image frame pairs in the sequence of image frames, and selecting a set of sample frames from the sequence of image frames by (i) adding a first image in the sequence of image frames to the set of sample frames, (ii) identifying which image frame among the image frames not yet added to the set of sample frames is least similar to the image frames in the set of sample frames according to the similarity measures, (iii) adding the frame identified in (ii) to the set of sample frames, and (iv) repeating (ii) and (iii) until a completion condition is met.

[0006] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as, but not limited to, random access memory (RAM) devices, read-only memory (ROM) devices, and the like, as described herein. Accordingly, some innovative aspects of the subject matter disclosed herein may be implemented by one or more non-transitory media storing software.

[0007] At least some aspects of the present disclosure may be implemented by an apparatus. For example, one or more devices may be capable of at least partially performing the methods disclosed herein. In some implementations, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof.

[0008] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will be apparent from the description, the drawings, and the claims. Note that the relative dimensions in the following figures may not be to scale.

[0009] These and other more detailed and specific features of the various embodiments are fully disclosed in the following description with reference to the accompanying drawings.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Mode for Carrying Out the Invention

[0011] The present disclosure and aspects thereof can be embodied in various forms including hardware, devices, or circuits controlled by a method implemented by a computer, computer program products, computer systems and networks, user interfaces, and application programming interfaces, as well as methods implemented by hardware, signal processing circuits, memory arrays, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and the like. The above is only intended to give a general idea of various aspects of the present disclosure and is in no way intended to limit the scope of the present disclosure.

[0012] In the following description, numerous details such as optical device configurations, timings, operations, etc. are described to provide an understanding of one or more aspects of the present disclosure. As will be apparent to those skilled in the art, these specific details are merely illustrative and are not intended to limit the scope of the present application.

[0013] FIG. 1 represents a process example of an image distribution pipeline showing various stages from image capture to image content display. An image (102) that may include a sequence of video frames (102) is captured or generated using an image generation block (105). The image (102) may be digitally captured (e.g., by a digital camera) or generated by a computer (e.g., using computer animation) to supply image data (107). Alternatively, the image (102) may be captured on film by a film camera. The film is converted to a digital format to supply image data (107). In the production phase (110), the image data (107) is edited to supply an image production stream (112).

[0014] The image data of the production stream (112) is then supplied to a processor (or one or more processors such as a central processing unit (CPU)) in a block (115) for sample-based color grading (e.g., real-time production editing). The sample-based color grading of the block (115) may include adjusting or changing the color or brightness of the image in order to enhance the image quality or realize the specific appearance of the image according to the creative intention of the image creator. This is sometimes referred to as "color timing" or "color grading". The image data from the sample-based color grading (115) results in a final version (117) of the product for distribution.

[0015] The sample-based color grading (115) and pipeline (100) of FIG. 1 can be configured to deliver live content in which images that are unpredictable due to dynamic situations, such as live broadcasts, game content, virtual reality content, etc., are generated. When delivering such live content, generally, it is impossible for an original artist or other user to manually color grade each image frame (in order to enhance the image quality or realize the specific appearance of the image according to the creative intention of the image creator). For example, an original artist may provide a set of color grading parameters or adjustments to apply to all image frames, but fixed color grading adjustments are not suitable for the changing situations of live content.

[0016] Sample-based color grading (115) enables color grading of live content by using predetermined color grading parameters (126) of a sample frame over a range of unknown future views and conditions. As an example, an image creator can obtain past live content (e.g., previous live broadcasts, clips of game content, clips of virtual reality content, etc.) as part of a sample color grading (124), identify sample frames from the past live content, and supply their artistic inputs using a color grading station (125) to manually color grade the sample frames. Then, when new live content is being processed through the sample-based color grading (115) and pipeline, the frames of the live content can be compared for similarity to the manually color graded sample frames. The new frames can then be automatically color graded by transferring the color grading of the sample frames by the image creator to the new frames based on the similarity between the new frames and the color graded sample frames. In this way, the sample-based color grading (115) and pipeline (100) of FIG. 1 can color grade live content in real time or near real time despite the unpredictable nature of live content.

[0017] In some embodiments, the color grading operations applied to the sample-based color grading (115) and the sample color grading (124) may include operations where the impact on the image output is substantially monotonic. A color grading operation is substantially monotonic when, if the settings of the operation are changed in a particular direction, the output also moves in a particular direction without returning to its original state. As a specific example, brightness control with a setting range from 0 to 1 does not make the image brightest when the setting is 0.5. Instead, the brightness always increases as the setting increases from 0 to 1. Examples of color grading operations that may be applied to the sample-based color grading (115) and the sample color grading (124) include, but are not limited to, ambient color temperature, contrast, luminance, hue, color-saturation, highlights, shadows, white clipping, black clipping, gradients (multiplication of the brightness channel and / or one or more chromaticity channels by a coefficient), offsets (addition of an offset to the brightness channel and / or one or more chromaticity channels), and the like.

[0018] Following the sample-based color grading (115), the image data of the final product (117) can be supplied to an encoding block (120) for downstream distribution to a decoding and playback device such as a computer monitor, a television receiver, a set-top box, a movie theater, etc. In some embodiments, the encoding block (120) can include audio and video encoders such as those defined by ATSC, DVB, DVD, Blue-Ray, and other distribution formats to generate an encoded bitstream (122). At the receiver, the encoded bitstream (122) is decoded by a decoding unit (130) to generate a decoded signal (132) that represents the same or nearly the same as the signal (117). The receiver can be attached to a target display (140) that can have unique characteristics. In that case, a display management block (135) can be used to map the dynamic range of the decoded signal (132) to the characteristics of the target display (140) by generating a display mapping signal (137). Further methods described herein may be performed by the decoding unit (130) or the display management block (135). Both the decoding unit (130) and the display management block (135) may include their own processors or may be integrated into a single processing unit. Although the present disclosure refers to a target display (140), it will be understood that this is merely an example. Further, the target display (140) can include any device configured to display or project light, such as a computer display, a television, an OLED display, an LCD display, a quantum dot display, a movie, a consumer, and other commercial projection systems, a head-up display, a virtual reality display, etc.

[0019] Figure 2 provides a method (200) that enables color grading based on the similarity to a sample frame. The method (200) may be executed by a processor, for example, as part of block (115) and / or the encoding block (120).

[0020] In step (202), raw video (or image) data is acquired. The raw video (or image) data may be, for example, live content such as live broadcasts, game content, virtual reality content, etc.

[0021] In step (204), a sample frame and predetermined color grading parameters are acquired. The sample frame may be a frame from past live content. In some embodiments, the sample frame is selected according to the method (300) of FIG. 3. In some embodiments, the predetermined color grading parameters are acquired through a color grading station such as the station (125) of FIG. 1 and as part of the method (300) of FIG. 3. Step (204) may include acquiring a plurality of sample frames and, for each sample frame, acquiring one or more color grading parameters (which may also be referred to as color grading parameter settings or color grading settings). The color grading parameters acquired in step (204) may include any color grading parameters that can be adjusted by a color grading station (125) or other color grading device.

[0022] In step (206), the method may include determining a similarity between the frame of the raw video (or image) obtained in step (202) and the sample frame obtained in step (204). In some embodiments, a feature vector may be computed for each image frame obtained in step (202) and for each sample frame obtained in step (204). The feature vector of the raw video data may be computed in real time, or, when dealing with recorded content rather than live content, may be computed at any point prior to step (206). The feature vector of the sample frame may be computed, for example, during step (204) or as part of the operation of the method (300) of FIG. 3. The feature vector may be used in determining the similarity between the frame of the raw video (or image) and the sample frame obtained in step (204). The feature vector of each frame may include, for example, the non-zero bin counts within a population of 3D histogram binning values in a preferred color space, and the mean and standard deviation of the brightness within a predetermined number of horizontal intervals divided from top to bottom within the frame.

[0023] Step (206) may include computing a 3D histogram of a part or each of the image frames obtained from steps (204) and (206). Each 3D histogram may include bins that divide the initial luminance range into a downsampled range and divide the chromaticity parameters (e.g., the u' parameter and v' parameter of the International Commission on Illumination, abbreviated as CIE) into a downsampled range. As an example, an image frame encoded by a perceptual quantizer based on Society of Motion Picture and Television Engineers ST.2084 (SMPTE 2084) is initially encoded at 10 - 16 bits for 0 - 10,000 cd / m 2It can have a brightness range and can have a pair of chromaticity parameters respectively encoded in 10 to 16 bits. As part of the calculation of the characteristic vector, the initial brightness range can be divided into a finite number of ranges, for example, 16 brightness bins (for example, 4-bit depth), and each chromaticity parameter can similarly be divided into a finite number of ranges, for example, 8 chromaticity parameter ranges (for example, 3-bit depth). The divided brightness ranges can be sized such that each covers a relatively similar perceptual difference. In other words, if the brightest value within a given divided brightness range has a perceptual brightness approximately twice that of the darkest value within the same brightness range, it is desirable that the remaining brightness ranges also have the brightest value with a perceptual brightness approximately twice that of their darkest value. As an example, step (206) may include populating a 3D histogram for some or all of the image frames obtained from steps (204) and (206), where the first dimension of the histogram is related to brightness, the second dimension is related to the first chromaticity parameter (for example, u'), and the third dimension is related to the second chromaticity parameter (for example, v'). Each dimension can include any desired number of bins. As an example, there may be 16 bins (for example, 16 different brightness ranges) in the brightness dimension, 8 bins (for example, 8 different ranges of u') in the first chromaticity dimension, and 8 bins (for example, 8 different ranges of v') in the second chromaticity dimension.

[0024] Computing a 3D histogram for a given image frame, for example, one of the image frames obtained from steps (204) and (206), can include counting the number of pixels in the given image associated with each bin of the 3D histogram. In other words, if a given bin in the 3D histogram covers a brightness range, a chromaticity range in the first chromaticity dimension, and a given chromaticity range in the second chromaticity dimension, and the image frame in question contains X pixels within all three independent ranges of the given bin, then the given bin can be assigned a value of X.

[0025] In some embodiments, the resolution of the image frames may be downsampled as part of the calculation of their feature vectors, thereby reducing the processing load of the calculation of the feature vectors. As an example, the image frames may be downsampled to a resolution of 100×100 pixels, 150×150 pixels, 180×180 pixels, 200×200 pixels, or 250×250 pixels, or other desired resolutions. The downsampling of the image frames may be performed regardless of whether the initial aspect ratio of the image frames is maintained. In such embodiments, the pixel counts in the 3D histogram may be based on the downsampled version of the image frames.

[0026] Step (206) may also include computing the mean and standard deviation of the brightness in a predetermined number of horizontal intervals divided from top to bottom within the target frame. As an example, computing the feature vector of a given image frame may include dividing the given image frame into a predetermined number of horizontal intervals and then computing the mean brightness of each horizontal interval and the standard deviation of the brightness of each horizontal interval. For example, each of the horizontal intervals may span the entire width of the target frame and may simultaneously span a part of each of the vertical dimensions. The horizontal intervals may have similar or identical sizes (e.g., each horizontal interval spans a similar or identical part of the vertical dimension.) if necessary, or may have different sizes. As an example, the central horizontal interval may have a larger or smaller vertical span than other horizontal intervals that are below or above the central horizontal interval. The horizontal intervals may partially overlap, may be adjacent to each other without overlap or vertical gaps, or may be separated by vertical gaps. As an example, the predetermined number of horizontal intervals may be three horizontal intervals, four horizontal intervals, five horizontal intervals, six horizontal intervals, seven horizontal intervals, eight horizontal intervals, nine horizontal intervals, ten horizontal intervals, or more than ten horizontal intervals. If necessary, the calculation of the mean and standard deviation of the brightness in the predetermined number of horizontal intervals may be performed after downsampling the original image (as discussed above).

[0027] Step (206) may also include comparing the characteristic vectors of the raw video frames with the characteristic vectors of the sample frames. As an example, the characteristic vectors of some or each of the raw video frames may be compared with the characteristic vectors of the sample frames. Determining the similarity of the characteristic vectors of the raw video frames with the characteristic vectors of the sample frames may include comparing the 3D histogram bin populations of the two image frames, comparing the average and standard deviation of the brightness over the horizontal intervals described above, and then combining these two comparisons.

[0028] Comparing the 3D histogram bin populations of two image frames may, as an example, include calculating the total of the overlaps within the 3D histogram bin populations of the two image frames. As a specific example, when a given 3D histogram bin has a non-zero value of X for a given raw video frame and a non-zero value of Y for a given sample frame, the smaller of the values of X and Y may be added to the total indicating the similarity between the frames. Such a calculation may be repeated for some or all of the 3D histogram bins until the final total of the overlaps within the 3D histogram is obtained. If the two 3D histograms are the same, the total is equal to 100% (1.0). If there is no overlap between the two 3D histograms, the total is 0% (0.0). Since most frames have some pixels in common, they are between these two extreme values, and the final normalized similarity in the 3D histogram bin population ranges from 0.0 to 1.0.

[0029] Comparing the average and standard deviation of brightness over the above horizontal intervals may, for example, involve computing the overlap of the normal distributions between the average and standard deviation of each of the horizontal intervals of the two image frames being compared. The overlap of the normal distributions between the matching horizontal intervals of the two image frames will be a real value within the range of 0.0 to 1.0. In some embodiments, the overlap of the normal distributions of different horizontal intervals may be averaged such that the overall match value is also within the range of 0.0 to 1.0. Optionally, the lowest horizontal interval may have a first weighting (W1), the second lowest horizontal interval may have a second weighting (W2), the middle horizontal interval may have a third weighting (W3), the second horizontal interval from the top may have a fourth weighting (W4), the topmost horizontal interval may have a fifth weighting (W5), and some or all of the weightings may be unique. In some embodiments, the weighting (W3) of the middle horizontal interval may be greater than the weightings of the other horizontal intervals. Computing the overlap of the normal distributions of the horizontal intervals of the two image frames may, for example, involve calculating a weighted average of the overlaps of the normal distributions of each of the horizontal intervals.

[0030] The output of the 3D histogram bin comparison and the overlap of the normal distributions of the horizontal intervals may be combined using any desired weighting factor. For example, the two measures may be combined with equal 1 / 2 weightings. In some embodiments, the two measures may be combined with unequal weightings (e.g., the 3D histogram bin comparison may be given a higher or lower weighting than the overlap of the normal distributions). In some embodiments, the output of the 3D histogram bin comparison includes a similarity value within the range of 0.0 to 1.0, and the overlap of the normal distributions of the horizontal intervals also includes a similarity value within the range of 0.0 to 1.0. If these two comparison metrics are aggregated with equal 1 / 2 weightings, the final similarity value between a given raw frame and a given sample frame will also have a range of 0.0 to 1.0.

[0031] In step (208), the similarity value between one or more raw frames and the sample frame, and the predetermined color grading parameters obtained in step (204) associated with the sample frame may be combined to determine the color grading parameters for the raw frames. In some embodiments, the similarity value obtained in step (206) may be used to assign weights to the predetermined color grading parameters associated with the sample frame. If the raw frame is exactly the same as the sample frame (e.g., the similarity value is 1.0), the set of predetermined color grading parameters for that sample frame will have the maximum weighting. In contrast, if the raw frame and the sample frame have a similarity value of 0.0, the set of predetermined color grading parameters for that sample frame will have a weighting of 0. In some embodiments, the weight of the predetermined color grading parameters from sample frame j applied to raw frame i can be calculated using Equation (1): w(i,j)=s(i,j) / [K S -s(i,j)] (1)

[0032] In Equation (1), w(i,j) can represent the weight of raw frame i with respect to sample frame j, s(i,j) can represent the similarity between raw frame i and sample frame j, and K S can be a predetermined constant value and be greater than 1.0. In some embodiments, K S can have a value of 1.05, which results in a maximum weighting of 20. Other equations for converting the similarity value to a weight may be used instead of Equation (1).

[0033] As discussed above, there may be M characteristic vectors (one for each of the M image sample frames), and each characteristic vector may have an associated set of 1 to N tone mapping parameter settings. Some characteristic vectors (e.g., some sample images) may be associated with tone mapping parameter settings for all N possible tone mapping operations, while other characteristic vectors (e.g., other sample images) may be associated with tone mapping parameter settings for fewer than N tone mapping operations and may even be associated with only one tone mapping parameter setting for one of the N possible tone mapping operations. When a new raw image frame arrives (e.g., at step (302)), the new raw image frame is optionally downsampled and its characteristic vector is computed. The characteristic vector of the new raw image frame is then compared to each of the M characteristic vectors of the sample frames to obtain M similarity values. The M similarity values may be converted to M weights (e.g., using equation (1) above) if desired. Then, as part of step (208), the N color grading parameters for the N tone mapping operations for the new raw image frame may be determined using the M weights (or the M similarity values). In some embodiments, each of the N tone mapping parameters for the new raw image frame may be calculated using a weighted average. As an example, the N tone mapping parameters for the new raw image frame may be calculated using the weighted average of equation (2): p(k,i)=[Σ j p(k,j)w(i,j)] / Σ j w(i,j) (2)

[0034] In Equation (2), p(k,i) represents the parameter settings for tone mapping operation k to be applied to the new frame i, p(k,j) represents the parameter settings for tone mapping operation k applied to the sample frame j, and w(i,j) represents the weight of the raw frame i with respect to the sample frame j (discussed with respect to Equation (1)). In some embodiments, only samples for which settings for the k-th tone mapping operation have been supplied or associated are used in the weighted average of Equation (2). In situations where there are no non-zero weights for a specified tone mapping operation (e.g., no samples with both supplied settings for the k-th tone mapping operation and non-zero weighting for the new frame), the method may instead assign global default values to the k-th tone mapping operation. The content creator may supply global default values for any desired number of the N possible tone mapping operations.

[0035] In step (210), each of the N tone mapping operations may be applied to the new video frame, and each tone mapping operation is applied using its associated parameter settings determined in step (208). In this way, the tone mapping operations can be applied to the raw video frame in real time without the need for manual review or input from a color grader. In step (212), the color-graded video frame may be allocated for use and / or storage.

[0036] Figure 3 provides a method (300) for identifying sample frames and obtaining color grading parameters for those sample frames used in the method (200) of Figure 2. The method (300) may be performed by a processor, for example, as part of block (124) and / or block (126). The content creator may have access to past live broadcasts or sample game play videos (e.g., sets of possible sample frames), but it may not be desirable for the content creator to color grade each possible sample frame individually. The method (300) is an example of a way to narrow down a set of possible sample frames to a more limited set of sample frames that are still suitable for individual color grading. As an example, the individually color graded sample frames are desirably a manageable number and are as distinct as possible to cover the widest range of possible future raw video inputs.

[0037] In step (302), raw sample video data may be obtained, where the raw sample video includes X possible sample frames. The raw sample video data may be associated with future raw video that is to be color graded. As an example, if the raw video to be color graded is a future broadcast of a sports event, step (302) may include obtaining a past broadcast of a similar sports event (e.g., a previous game at the same location, same date, etc.). As another example, if the raw video to be color graded is game content from a specific game, step (302) may include obtaining sample content from that specific game.

[0038] In step (304), characteristic vectors for some or all of the raw sample video frames may be calculated, where the characteristic vectors may be calculated as described in step (206) of method (200).

[0039] In step (306), similarity values between the feature vectors and their associated raw sample video frames can be calculated (as described, for example, in step (206)). Optionally, step (306) may include populating a symmetric matrix with a 1.0 diagonal (e.g., the similarity of each frame to itself), otherwise indicating the similarity values between the feature vectors and their associated raw sample video frames.

[0040] In step (308), a mode sample frame is identified and selected as the first sample frame in a set of M final sample frames. As an example, step (308) may include finding which row of the symmetric matrix has the largest sum, which indicates the most average frame (e.g., the mode) of the raw sample video frames.

[0041] In step (310), a frame having the smallest maximum similarity to the sample frames in the set of final sample frames is identified and can then be added to the set of final sample frames. As an example, step (310) may include identifying which row of the symmetric matrix has the smallest maximum value (excluding the 1.0 diagonal), and the sample frame associated with that row can be added to the set of final sample frames.

[0042] As shown by loop (312), step (310) can be repeated until an end condition is reached. Examples of potential end conditions include, but are not limited to, the final sample frame reaching a predetermined number (M), or the candidates identified in step (310) having a similarity exceeding a predetermined threshold with respect to the previous entry in the set of final sample frames. As an example, if the frame identified as having the smallest maximum similarity with the sample frames in the set of final sample frames in step (310) has a maximum similarity of at least 0.7 with respect to a frame already in the set of final sample frames, method (300) stops loop (312), and the finally identified frame may or may not be added to the set of final sample frames.

[0043] In step (314), color grading parameters for the sample frames identified in step (310) can be obtained. The color grading parameters can be obtained, for example, by a color grader using a color grading station (125). In particular, the color grader can supply an input via the control interface of the color grading station. Optionally, step (310) and / or step (314) can also include receiving user input to remove one or more sample frames from the set of final sample frames identified in step (310). In particular, method (300) can identify some sample frames that are not particularly useful for color grading, such as frames during a fade transition, frames that are completely black, etc. Generally, it should be easier to remove such frames from the smaller set of sample frames identified in step (310) than from the original sample frames obtained in step (302).

[0044] The above-described system and method can provide color grading of images and videos based on similarity to samples. The system, method, and device according to the present disclosure can take any one or more of the following configurations.

[0045] (1) A method for color grading a source image by a plurality of color grading operations, comprising: obtaining a source image for color grading; obtaining a plurality of color grading parameter settings associated with each sample frame among a plurality of sample frames, each color grading parameter setting being an input parameter for an associated color grading operation among the plurality of color grading operations; obtaining a plurality of similarity measures, each similarity measure indicating a level of similarity between the source image and each different sample frame among the plurality of sample frames; obtaining a plurality of weighted averages based on the plurality of similarity measures and the plurality of color grading parameter settings, each weighted average being associated with a different one of the plurality of color grading operations; applying the plurality of color grading operations to the source image by using each of the plurality of weighted averages as an input parameter for its associated color grading operation so as to generate a color-graded image; and supplying the color-graded image. A method having the above steps.

[0046] (2) The method according to (1), wherein: the plurality of color grading operations include a brightness adjustment operation, a hue adjustment operation, and a saturation adjustment operation. A method.

[0047] (3) The method according to (1) or (2), wherein: the plurality of color grading operations include a global brightness adjustment operation, a global hue adjustment operation, and a global saturation adjustment operation. A method.

[0048] (4) The method according to any one of (1) to (3), wherein the plurality of color grading operations include at least one local brightness adjustment operation, at least one local hue adjustment operation, and at least one local saturation adjustment operation; Method.

[0049] (5) The method according to any one of (1) to (4), wherein obtaining the plurality of similarity measures includes calculating a plurality of data structures; each data structure is associated with a different one of the source image and the sample frames among the plurality of sample frames; each data structure has a plurality of bins, each bin is associated with a unique range of chromaticity values and a unique range of luminance values, and each bin includes a count of the number of pixels of the image associated with the data structure that have chromaticity values within the range of chromaticity values associated with the bin and also have luminance values within the range of luminance values associated with the bin; Method.

[0050] (6) The method according to (5), wherein each of the plurality of sample frames and the source image among the plurality of sample frames is encoded with a transfer function that covers a luminance range of X nits with a Y-bit depth, and the bins of the data structure cover the luminance range of the X nits with a Z-bit depth, and Z is less than or equal to half of Y; Method.

[0051] (7) The method according to (5) or (6), wherein each of the plurality of sample frames and the source image among the plurality of sample frames is encoded with a transfer function that includes chromaticity values that cover a given color space with a Y-bit depth, and the bins of the data structure cover the given color space with a Z-bit depth, and Z is greater than half of Y; Method.

[0052] (8) The method according to any one of (5) to (7), Calculating the plurality of data structures further includes downsampling the source image before computing the data structure associated with the source image, and downsampling a sample frame among the plurality of sample frames before computing the data structure associated with the source image. Method.

[0053] (9) The method according to any one of (5) to (8), Obtaining the plurality of similarity measures further includes identifying an overlap between the data structure associated with the source image and the data structure associated with the sample frame. Method.

[0054] (10) The method according to any one of (1) to (9), Obtaining the plurality of similarity measures includes dividing each sample frame among the plurality of sample frames and the source image into a plurality of horizontal bands, obtaining an average and a standard deviation of luminance within each horizontal band of the horizontal bands, computing an overlap of a normal distribution of the obtained average and standard deviation of luminance within each horizontal band of the horizontal bands and further includes. Method.

[0055] (11) The method according to any one of (5) to (9), Obtaining the plurality of similarity measures includes identifying an overlap between the data structure associated with the source image and the data structure associated with the sample frame, dividing each sample frame among the plurality of sample frames and the source image into a plurality of horizontal bands, obtaining an average and a standard deviation of luminance within each horizontal band of the horizontal bands, Computing an overlap of a normal distribution of the obtained mean and standard deviation of luminance within each horizontal band of the horizontal bands; Calculating the similarity measure by combining a first metric based on the identified overlap between a data structure associated with the source image and a data structure associated with the sample frame with a second metric based on the computed overlap of the normal distribution; further comprising; Method.

[0056] (12) A color grading system having at least one controller, wherein the at least one controller is configured to: obtain a source image for color grading; obtain a plurality of color grading parameter settings, each color grading parameter setting being an input parameter to a color grading operation and each color grading parameter setting being associated with a different respective one of a plurality of sample frames; calculate a plurality of similarity measures, each similarity measure indicating a level of similarity between the source image and a different respective one of the plurality of sample frames; calculate a weighted average based on the plurality of similarity measures and the plurality of color grading parameter settings; apply the color grading operation to the source image using the weighted average as an input parameter to the color grading operation to generate a color graded image; and supply the color graded image. A color grading system configured to perform the above.

[0057] (13) The color grading system according to (12), further comprising a display, the controller is configured to supply the color-graded image to the display, the display is configured to display the color-graded image, a color grading system.

[0058] The color grading system according to (14)(13), wherein the source image includes live content, the at least one controller is configured to supply the color-graded image to the display within 200 milliseconds after acquiring the source image. a color grading system.

[0059] The color grading system according to any one of (15)(12) to (14), wherein the at least one controller is configured to acquire the plurality of color grading parameter settings before generation of the source image. a color grading system.

[0060] A method according to (16), acquiring a sequence of image frames, acquiring a plurality of similarity measures, each similarity measure indicating a level of similarity between different respective pairs of image frames in the sequence of image frames, (i) adding a first image in the sequence of image frames to a set of sample frames, (ii) identifying, from among the image frames not yet added to the set of sample frames, which image frame is least similar to the image frames in the set of sample frames according to the similarity measure, (iii) adding the image frame identified in (ii) to the set of sample frames, and (iv) Repeating (ii) and (iii) until the completion condition is satisfied selecting a set of the sample frames from the sequence of the image frames; and A method having the above.

[0061] (17) The method according to (16), wherein repeating (ii) and (iii) until the completion condition is satisfied includes repeating (ii) and (iii) until the number of sample frames added to the set of the sample frames is at least a predetermined number. A method.

[0062] (18) The method according to (16) or (17), wherein repeating (ii) and (iii) until the completion condition is satisfied includes repeating (ii) and (iii) until a similarity measure between the image frame specified in (ii) and an image frame already added to the set of the sample frames exceeds a predetermined threshold value. A method.

[0063] (19) The method according to any one of (16) to (18), wherein adding the first image in the sequence of the image frames to the set of the sample frames includes adding the most average frame in the sequence of the image frames to the set of the sample frames. A method.

[0064] (20) The method according to any one of (16) to (19), wherein adding the first image in the sequence of the image frames to the set of the sample frames includes adding a randomly selected frame in the sequence of the image frames to the set of the sample frames. A method.

[0065] (21) The method according to any one of (16) to (20), wherein (ii) Identifying which image frame among the image frames not yet added to the set of sample frames is least similar to the image frames in the set of sample frames according to the similarity measure includes identifying which image frame among the image frames not yet added to the set of sample frames has the smallest maximum similarity degree with any of the image frames in the set of sample frames. Method.

[0066] (22) The method according to any one of (16) to (21), Obtaining the plurality of similarity measures includes computing a feature vector for each image frame in the sequence of image frames. Method.

[0067] (23) The method according to (22), Computing the feature vector for a given image frame includes: Downsampling the given image frame to generate a downsampled image having a lower resolution than the given image frame; and Computing the average brightness and standard deviation of the brightness of a plurality of horizontal intervals of the downsampled image frame. Including Method.

[0068] (24) The method according to (22), The image frame is encoded with a transfer function having a luminance range encoded in X bits and at least two chrominance parameters each encoded in Y bits. Computing the feature vector for a given image frame includes: Downsampling the given image frame to generate a downsampled image frame having a lower resolution than the given image frame; and Computing the average brightness and standard deviation of the brightness of a plurality of horizontal sections of the downsampled image frame; Computing a data structure from the downsampled image frame; and having; the data structure includes a plurality of bins, the luminance range of the transfer function is encoded in A bits in the data structure, the at least two chromaticity parameters are encoded in B bits in the data structure, A is smaller than X, and B is smaller than Y; computing the data structure includes counting the number of pixels from the downsampled image frame associated with each bin; A method.

[0069] (25)(16) to any one of (24), the method comprising: after the completion condition is satisfied, further obtaining at least one color grading parameter setting for each sample frame in the set of sample frames; each of the color grading parameter settings serves as an input to a color grading operation; A method.

[0070] (26)(25) The method described above, comprising: obtaining a source image for color grading; calculating a plurality of further similarity measures, each further similarity measure indicating a level of similarity between the source image and each different sample frame in the set of sample frames; calculating a weighted average based on the plurality of further similarity measures and the color grading parameter settings associated with the sample frames in the set of sample frames; Using the weighted average as an input parameter to the color grading operation to generate a color-graded image and applying the color grading operation to the source image, supplying the color-graded image, and a method further comprising.

[0071] (27) A computer program product having instructions, wherein the instructions, when executed by a computing system, cause the computing system to perform the method according to any one of (1) to (11) or (16) to (26).

[0072] Regarding the processes, systems, methods, heuristics, etc. described herein, although the steps of such processes, etc. are described as occurring according to a particular ordered sequence, it should be understood that such processes may be implemented by the described steps being performed in an order other than the order described herein. Furthermore, it should be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the description of the processes in this specification is provided for the purpose of explaining particular embodiments and should in no way be construed as limiting the scope of the claims.

[0073] Accordingly, it should be understood that the foregoing description is intended to be illustrative and not restrictive. Many embodiments and applications other than the given examples will be apparent to those reading the above description. The scope of application should not be determined with reference to the above description, but rather should be determined with reference to the appended claims along with the full scope of equivalents to which those claims are entitled. The technology described herein is expected to develop in the future, and the disclosed systems and methods are intended to be incorporated into such future embodiments. In short, it should be understood that this application is capable of modification and variation.

[0074] All terms used in the claims are intended to be given their broadest reasonable interpretation and their ordinary meaning as understood by those skilled in the art of the technology described herein, unless explicitly indicated to the contrary herein. In particular, the use of singular articles such as "a," "the," "said," etc. should be interpreted to recite one or more of the indicated elements unless a contrary limitation is explicitly recited in the claim.

[0075] The summary of the disclosure is provided to enable the reader to quickly ascertain the nature of the technical disclosure. The summary is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Further, in the foregoing detailed description, for purposes of simplifying the disclosure, it can be seen that various features are grouped together in various embodiments. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, the subject matter of the invention lies in less than all of the features of a single disclosed embodiment. Accordingly, the following claims are incorporated into the detailed description, with each claim standing on its own as a separately claimed subject matter.

[0076] [Cross - Reference to Related Applications] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 391,528, filed Jul. 22, 2022, and European Patent Application No. 22197590.7, filed Sep. 23, 2022, the entire contents of which are hereby incorporated by reference into this application.

Claims

1. A computer-implemented method for color grading a source image by a plurality of color grading operations, comprising: obtaining a source image for color grading; obtaining a plurality of color grading parameter settings associated with each sample frame among a plurality of sample frames, each color grading parameter setting being an input parameter for an associated color grading operation among the plurality of color grading operations; obtaining a plurality of similarity measures, each similarity measure indicating a level of similarity between the source image and each different sample frame among the plurality of sample frames; obtaining a plurality of weighted averages based on the plurality of similarity measures and the plurality of color grading parameter settings, each weighted average being associated with a different one of the plurality of color grading operations; applying the plurality of color grading operations to the source image by using each of the plurality of weighted averages as an input parameter for its associated color grading operation so as to generate a color-graded image; and supplying the color-graded image. A method having the above steps.

2. The plurality of color grading operations include a brightness adjustment operation, a hue adjustment operation, and a saturation adjustment operation. The method according to claim 1.

3. The plurality of color grading operations include a global brightness adjustment operation, a global hue adjustment operation, and a global saturation adjustment operation. The method according to claim 1 or 2.

4. The plurality of color grading operations include at least one local brightness adjustment operation, at least one local hue adjustment operation, and at least one local saturation adjustment operation. The method according to any one of claims 1 to 3.

5. Obtaining the plurality of similarity measures includes calculating a plurality of data structures. Each data structure is associated with the source image and a different one of the sample frames among the plurality of sample frames. Each data structure has a plurality of bins, each bin being associated with a unique range of chrominance values and a unique range of luminance values, and each bin including a count of the number of pixels in an image associated with the data structure having a chrominance value within the range of chrominance values associated with the bin and also having a luminance value within the range of luminance values associated with the bin. The method according to any one of claims 1 to 4.

6. Each sample frame among the plurality of sample frames and the source image are encoded with a transfer function covering a luminance range of X nits with a Y-bit depth, and the bins of the data structure cover the luminance range of X nits with a Z-bit depth, where Z is less than or equal to half of Y. The method according to claim 5.

7. Each sample frame among the plurality of sample frames and the source image are encoded with a transfer function including chrominance values covering a given color space with a Y-bit depth, and the bins of the data structure cover the given color space with a Z-bit depth, where Z is greater than half of Y. The method according to claim 5 or 6.

8. Calculating the plurality of data structures further includes downsampling the source image before computing the data structure associated with the source image and downsampling the sample frames among the plurality of sample frames before computing the data structure associated with the source image. The method according to any one of claims 5 to 7.

9. Obtaining the plurality of similarity measures further includes identifying an overlap between the data structure associated with the source image and the data structure associated with the sample frame. The method according to any one of claims 5 to 8.

10. Obtaining the plurality of similarity measures includes dividing each sample frame among the plurality of sample frames and the source image into a plurality of horizontal bands, obtaining the average and standard deviation of luminance within each horizontal band of the horizontal bands, computing an overlap of normal distributions of the obtained average and standard deviation of luminance within each horizontal band of the horizontal bands and further includes. The method according to any one of claims 1 to 9.

11. Obtaining the plurality of similarity measures includes Identifying an overlap between a data structure associated with the source image and a data structure associated with the sample frame; Dividing each sample frame among the plurality of sample frames and the source image into a plurality of horizontal bands; Obtaining an average and a standard deviation of luminance within each of the horizontal bands; Computing an overlap of a normal distribution of the obtained average and standard deviation of luminance within each of the horizontal bands; Calculating the similarity measure by combining a first metric based on the identified overlap between the data structure associated with the source image and the data structure associated with the sample frame with a second metric based on the computed overlap of the normal distribution; Further comprising; The method according to any one of claims 1 to 9.

12. Having at least one controller, the at least one controller Obtaining a source image for color grading; Obtaining a plurality of color grading parameter settings, each color grading parameter setting being an input parameter to a color grading operation, and each color grading parameter setting being associated with a different respective sample frame among the plurality of sample frames; Calculating a plurality of similarity measures, each similarity measure indicating a level of similarity between the source image and a different respective sample frame among the plurality of sample frames; Calculating a weighted average based on the plurality of similarity measures and the plurality of color grading parameter settings; Applying the color grading operation to the source image using the weighted average as an input parameter to the color grading operation so as to generate a color graded image; Supplying the color graded image. A color grading system configured to perform the above.

13. Further having a display, The controller is configured to supply the color graded image to the display, The display is configured to display the color graded image. The color grading system according to claim 12.

14. The source image includes live content, The at least one controller is configured to supply the color-graded image to the display within 200 milliseconds after acquiring the source image. The color grading system according to claim 13.

15. The at least one controller is configured to acquire the plurality of color grading parameter settings before generation of the source image. The color grading system according to any one of claims 12 to 14.

16. Obtaining a sequence of image frames, Obtaining a plurality of similarity measures, each similarity measure indicating a level of similarity between different respective image frame pairs in the sequence of image frames, (i) adding a first image in the sequence of image frames to a set of sample frames, (ii) identifying, from among the image frames not yet added to the set of sample frames, which image frame is least similar to the image frames in the set of sample frames according to the similarity measure, (iii) adding the image frame identified in (ii) to the set of sample frames, and (iv) repeating (ii) and (iii) until a completion condition is satisfied to select the set of sample frames from the sequence of image frames and a method having the same.

17. Repeating (ii) and (iii) until the completion condition is satisfied includes repeating (ii) and (iii) until the number of sample frames added to the set of sample frames is at least a predetermined number. The method according to claim 16.

18. Repeating (ii) and (iii) until the completion condition is satisfied includes repeating (ii) and (iii) until a similarity measure between the image frame identified in (ii) and an image frame already added to the set of sample frames exceeds a predetermined threshold. The method according to claim 16 or 17.

19. Adding the first image in the sequence of image frames to the set of sample frames includes adding the most average frame in the sequence of image frames to the set of sample frames. The method according to any one of claims 16 to 18.

20. Adding the first image in the sequence of the image frames to the set of the sample frames includes adding a randomly selected frame in the sequence of the image frames to the set of the sample frames. The method according to any one of claims 16 to 19.

21. (ii) Identifying which image frame among the image frames not yet added to the set of the sample frames is least similar to the image frames in the set of the sample frames according to the similarity measure includes identifying which image frame among the image frames not yet added to the set of the sample frames has the smallest maximum similarity degree with any one of the image frames in the set of the sample frames. The method according to any one of claims 16 to 20.

22. Obtaining the plurality of similarity measures includes computing a characteristic vector of each image frame in the sequence of the image frames. The method according to any one of claims 16 to 21.

23. Computing the characteristic vector of a given image frame includes downsampling the given image frame to generate a downsampled image having a lower resolution than the given image frame, and computing the average brightness and standard deviation of the brightness of a plurality of horizontal intervals of the downsampled image frame. The method according to claim 22.

24. The image frame is encoded with a transfer function having a luminance range encoded in X bits and at least two chrominance parameters each encoded in Y bits. Computing the characteristic vector of a given image frame includes downsampling the given image frame to generate a downsampled image frame having a lower resolution than the given image frame, computing the average brightness and standard deviation of the brightness of a plurality of horizontal intervals of the downsampled image frame, and computing a data structure from the downsampled image frame. ​ The data structure includes a plurality of bins, the luminance range of the transfer function is encoded in A bits in the data structure, the at least two chromaticity parameters are encoded in B bits in the data structure, A is smaller than X, and B is smaller than Y. Computing the data structure includes counting the number of pixels from the downsampled image frames associated with each bin. The method according to claim 22.

25. After the completion condition is satisfied, further including obtaining at least one color grading parameter setting for each sample frame in the set of sample frames. Each of the color grading parameter settings serves as an input to the color grading operation. The method according to any one of claims 16 to 24.

26. Obtaining a source image for color grading. Calculating a plurality of further similarity measures, each further similarity measure indicating a level of similarity between the source image and each different sample frame in the set of sample frames. Calculating a weighted average based on the plurality of further similarity measures and the color grading parameter settings associated with the sample frames in the set of sample frames. Applying the color grading operation to the source image by using the weighted average as an input parameter to the color grading operation to generate a color graded image. Supplying the color graded image. The method according to claim 25, further including the above steps.

27. A computer program product having instructions for causing a computing system to execute the method according to any one of claims 1 to 11 or claims 16 to 26 when executed by the computing system.

Images