Color grading of content based on similarity to samples

Abstract

Systems and methods for color grading of images and videos based on similarity to a sample. As preparation for color grading of new content, a sample frame related to the expected new content may be obtained, and color grading parameters for the sample frame may be obtained. When new content is generated or received, the similarity between the frame of the new content and the sample frame can be determined in order to color grade it. The similarity between the frame of the new content and the sample frame can be combined with the color grading parameters obtained from the sample frame in order to determine the appropriate color grading parameters to apply to the new content. The new content can then be color graded using the determined color grading parameters.

Description

Technical Field

[0001] This application generally relates to systems and methods for image processing, image display, and image reproduction. Embodiments of the invention provide methods and apparatuses for processing image data to convert colors and / or tones for display or reproduction on local or downstream devices. Some example embodiments are displays configured to receive and process image data and display it for viewing the processed image data.

Summary of the Invention

[0002] The present invention is defined by the independent claims. The dependent claims relate to any features of some embodiments of the present invention. The embodiments described herein relate to systems and methods for color grading of images and videos based on similarity to a sample. As preparation for color grading of new content, a sample frame related to the expected new content may be obtained, and color grading parameters for the sample frame may be obtained. When new content is generated or received, the similarity between the frame of the new content and the sample frame can be determined to color grade the new content. The similarity between the frame of the new content and the sample frame can be combined with the color grading parameters obtained from the sample frame to determine the appropriate color grading parameters to apply to the new content. Then, the new content can be color graded using the determined color grading parameters.

[0003] One aspect of this disclosure provides a method for color grading a source image using a set of color grading operations. The method includes: obtaining a source image for color grading; obtaining a set of color grading parameter settings associated with each sample frame in a set of sample frames, each of which is an input parameter for the associated color grading operation in the set of color grading operations; obtaining a set of similarity measures, each of which indicates a level of similarity between the source image and each different sample frame in the set of sample frames; obtaining a set of weighted averages based on the set of similarity measures and the set of color grading parameter settings, each of which is associated with each different color grading operation in the set of color grading operations; applying the set of color grading operations to the source image using each of the weighted averages as an input parameter for its associated color grading operation, and supplying the color graded image.

[0004] In another aspect of the present disclosure, a color grading system is provided which includes at least one controller, the at least one controller is configured to acquire a source image for color grading, acquire a plurality of color grading parameter settings, each of which is an input parameter to a color grading operation, and each of which is associated with a different sample frame in a plurality of sample frames, calculate a plurality of similarity measures, each of which indicates a level of similarity between the source image and a different sample frame in a plurality of sample frames, calculate a weighted average based on the plurality of similarity measures and the plurality of color grading parameter settings, apply the color grading operation to the source image using the weighted average as an input parameter to the color grading operation to produce a color-graded image, and supply the color-graded image.

[0005] Another aspect of the present disclosure provides a method for obtaining a sequence of image frames and obtaining a number of similarity measures, each of which indicates a level of similarity between different pairs of image frames in the sequence of image frames, and for selecting a set of sample frames from the sequence of image frames by (i) adding a first image from the sequence of image frames to a set of sample frames, (ii) identifying which image frame from among the image frames not yet added to the set of sample frames is least similar to any of the image frames in the set of sample frames according to the similarity measures, (iii) adding the frame identified in (ii) to the set of sample frames, and (iv) repeating (ii) and (iii) until a completion condition is met.

[0006] Some or all of the operations, functions, and / or methods described herein may be executed by one or more devices in accordance with instructions (e.g., software) stored on one or more non-temporary media. Such non-temporary media may include, but are not limited to, memory devices such as random-access memory (RAM) devices and read-only memory (ROM) devices, as described herein. Accordingly, some innovative aspects of the subject matter described herein may be implemented by one or more non-temporary media on which software is stored.

[0007] At least some aspects of this disclosure may be implemented by an apparatus. For example, one or more devices may be capable of performing at least partially the methods disclosed herein. In some implementations, the apparatus is or includes an audio processing system comprising an interface system and a control system. The control system may include one or more general-purpose single or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or a combination thereof.

[0008] Details of one or more embodiments of the subject matter described herein are illustrated in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the specification, drawings, and claims. Note that the relative dimensions in the following figures may not be to scale.

[0009] These and other further detailed and specific features of various embodiments are fully disclosed in the following description with reference to the attached drawings. [Brief explanation of the drawing]

[0010] [Figure 1] This illustrates an example of an image coding and decoding pipeline process. [Figure 2]This illustrates an example of an image coding pipeline process. [Figure 3] This illustrates an example of the process for identifying a sample frame. [Modes for carrying out the invention]

[0011] This disclosure and its aspects can be embodied in various forms, including hardware, devices, or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, and application programming interfaces, as well as hardware-implemented methods, signal processing circuits, memory arrays, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and the like. The foregoing is intended solely to give a general idea of ​​the various aspects of this disclosure and is not intended to limit the scope of this disclosure.

[0012] Numerous details, such as optical device configuration, timing, and operation, are described below to provide an understanding of one or more aspects of this disclosure. As will be apparent to those skilled in the art, these specific details are merely illustrative and are not intended to limit the scope of this application.

[0013] Figure 1 illustrates an example process of an image delivery pipeline, showing various stages from image capture to image content display. An image (102), which may contain a sequence of video frames (102), is captured or generated using an image generation block (105). The image (102) may be captured or generated digitally (e.g., by a digital camera) by a computer (e.g., using computer animation) to supply image data (107). Alternatively, the image (102) may be captured on film by a film camera. The film is converted to a digital format to supply image data (107). In the production phase (110), the image data (107) is edited to supply an image production stream (112).

[0014] Image data from the production stream (112) is then fed to a processor (or one or more processors such as a central processing unit (CPU)) in a block (115) for sample-based color grading (e.g., real-time production editing). Sample-based color grading in block (115) may involve adjusting or modifying the color or brightness of the image to enhance image quality or to achieve a specific appearance of the image according to the creative intent of the image creator. This is sometimes called "color timing" or "color grading". Image data from sample-based color grading (115) yields the final version (117) of the product for distribution.

[0015] The color grading (115) and pipeline (100) based on the sample in Figure 1 may be configured for delivering live content where unpredictable images are generated due to dynamic circumstances, such as live broadcasts, game content, and virtual reality content. When delivering such live content, it is generally impossible for the creative artist or other user to manually color grade each image frame (to enhance image quality or to achieve a specific appearance of the image according to the image creator's creative intent). For example, the creative artist may provide a set of color grading parameters or adjustments to apply to all image frames, but fixed color grading adjustments are unsuitable for the changing circumstances of live content.

[0016] Sample-based color grading (115) enables color grading of live content by utilizing predetermined color grading parameters (126) of sample frames to a range of unknown future views and conditions. As an example, an image creator can acquire past live content (e.g., previous live broadcasts, game content clips, virtual reality content clips, etc.) as part of sample color grading (124), identify sample frames from the past live content, and manually color grade the sample frames by supplying their artistic input using a color grading station (125). Then, as new live content is processed through sample-based color grading (115) and the pipeline, frames of the live content can be compared for similarity to manually color-graded sample frames. The new frames can then be automatically color-graded by transferring the image creator's color grading of the sample frames to the new frames based on the similarity between the new frames and the color-graded sample frames. In this way, the color grading (115) and pipeline (100) based on the sample in Figure 1 can color grade live content in real time or near real time, despite the unpredictable nature of live content.

[0017] In some embodiments, the color grading operations applied in sample-based color grading (115) and sample color grading (124) may include operations whose effect on the image output is nearly monotonous. A color grading operation is nearly monotonous if changing the settings of the operation in a particular direction causes the output to move in a particular direction without reverting. For example, a brightness control with a setting range from 0 to 1 does not make the image brightest when the setting is 0.5, but rather the brightness always increases as the setting increases from 0 to 1. Examples of color grading operations that may be applied in sample-based color grading (115) and sample color grading (124) include, but are not limited to, ambient color temperature, contrast, luminance, hue, color saturation, highlights, shadows, white clipping, black clipping, gradient (multiplication of a brightness channel and / or one or more chromaticity channels with a coefficient), and offset (addition of an offset to a brightness channel and / or one or more chromaticity channels).

[0018] Following sample-based color grading (115), the image data of the final product (117) may be fed to an encoding block (120) for downstream distribution to decoding and playback devices such as computer monitors, television receivers, set-top boxes, and cinemas. In some embodiments, the encoding block (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-ray, and other distribution formats, to generate an encoded bitstream (122). At the receiver, the encoded bitstream (122) is decoded by a decoding unit (130) to generate a decoded signal (132) that represents the same as or nearly the same as the signal (117). The receiver may be mounted on a target display (140) which may have unique characteristics. In that case, a display management block (135) may be used to map the dynamic range of the decoded signal (132) to the characteristics of the target display (140) by generating a display mapping signal (137). Further methods described herein may be carried out by a decoding unit (130) or a display management block (135). Both the decoding unit (130) and the display management block (135) may include their own processors or be integrated into a single processing unit. While this disclosure refers to a target display (140), it will be understood that this is only an example. Furthermore, it will be understood that the target display (140) may include any device configured to display or project light, such as computer displays, televisions, OLED displays, LCD displays, quantum dot displays, movie, consumer, and other commercial projection systems, heads-up displays, virtual reality displays, and the like.

[0019] Figure 2 provides a method (200) that enables color grading based on similarity with a sample frame. Method (200) may be executed by a processor, for example, as part of block (115) and / or block (120) of encoding.

[0020] In step (202), raw video (or image) data is obtained. This raw video (or image) data may be live content, such as a live broadcast, game content, or virtual reality content.

[0021] In step (204), a sample frame and predetermined color grading parameters are acquired. The sample frame may be a frame from past live content. In some embodiments, the sample frame is selected according to method (300) in Figure 3. In some embodiments, predetermined color grading parameters are acquired through a color grading station such as station (125) in Figure 1 and as part of method (300) in Figure 3. Step (204) may include acquiring multiple sample frames and, for each sample frame, acquiring one or more color grading parameters (which may also be called color grading parameter settings or color grading settings). The color grading parameters acquired in step (204) may include any color grading parameters that can be adjusted by the color grading station (125) or other color grading devices.

[0022] In step (206), the method may include determining the similarity between frames of raw video (or images) acquired in step (202) and sample frames acquired in step (204). In some embodiments, characteristic vectors may be computed for each image frame acquired in step (202) and for each sample frame acquired in step (204). The characteristic vectors of the raw video data may be computed in real time, or, if dealing with recorded content rather than live content, may be computed at any point prior to step (206). The characteristic vectors of the sample frames may be computed, for example, during step (204) or as part of the operation of method (300) in Figure 3. The characteristic vectors may be used to determine the similarity between frames of raw video (or images) and sample frames acquired in step (204). The characteristic vector for each frame may include, for example, the non-zero bin count in a 3D histogram binning value set in a preferred color space, and the mean and standard deviation of brightness in a predetermined number of horizontal intervals divided from top to bottom within the frame.

[0023] Step (206) may include computing a portion of the image frames obtained from steps (204) and (206), or their respective 3D histograms. Each 3D histogram may include bins that divide the initial luminance range into downsampled ranges and the chromaticity parameters (e.g., the u' and v' parameters of the International Commission on Illumination, abbreviated as CIE) into downsampled ranges. As an example, an image frame encoded by a perceptual quantizer based on Society of Motion Picture and Television Engineers ST.2084 (SMPTE 2084) is initially encoded in 10-16 bits, ranging from 0 to 10,000 cd / m². 2can have a brightness range and can have a pair of chromaticity parameters each encoded in 10 to 16 bits. As part of the calculation of the characteristic vector, the initial brightness range can be divided into a finite number of ranges, for example 16 brightness bins (e.g., 4-bit depth), and each chromaticity parameter can similarly be divided into a finite number of ranges, for example 8 chromaticity parameter ranges (e.g., 3-bit depth). The divided brightness ranges can be sized such that each has a relatively similar perceptual difference. In other words, if the brightest value within a given divided brightness range has a perceptual brightness approximately twice that of the darkest value within that same brightness range, it is desirable that the remaining brightness ranges also have the brightest value having a perceptual brightness approximately twice that of their darkest values. As an example, step (206) may include populating a 3D histogram for part or all of the image frames obtained from steps (204) and (206), where the first dimension of the histogram is related to brightness, the second dimension is related to the first chromaticity parameter (e.g., u’), and the third dimension is related to the second chromaticity parameter (e.g., v’). Each dimension can include any desired number of bins. As an example, there may be 16 bins (e.g., 16 different brightness ranges) in the brightness dimension, 8 bins (e.g., 8 different ranges of u’) in the first chromaticity dimension, and 8 bins (e.g., 8 different ranges of v’) in the second chromaticity dimension.

[0024] Computing a 3D histogram for a given image frame, e.g., one of the image frames obtained from steps (204) and (206), can include counting the number of pixels in the given image associated with each bin of the 3D histogram. In other words, if a given bin in the 3D histogram covers a brightness range, a chromaticity range in the first chromaticity dimension, and a given chromaticity range in the second chromaticity dimension, and the image frame in question includes X pixels within all three independent ranges of the given bin, the given bin can be assigned a value of X.

[0025] In some embodiments, the resolution of the image frames may be downsampled as part of the calculation of their feature vectors, thereby reducing the processing load of calculating the feature vectors. As an example, the image frames may be downsampled to a resolution of 100×100 pixels, 150×150 pixels, 180×180 pixels, 200×200 pixels, or 250×250 pixels, or other desired resolutions. The downsampling of the image frames may be performed regardless of whether the initial aspect ratio of the image frames is maintained. In such embodiments, the pixel counts in the 3D histogram may be based on the downsampled version of the image frames.

[0026] Step (206) may also include computing the average and standard deviation of the brightness in a predetermined number of horizontal intervals divided from top to bottom within the target frame. As an example, computing the feature vector of a given image frame may include dividing the given image frame into a predetermined number of horizontal intervals and then computing the average brightness of each horizontal interval and the standard deviation of the brightness of each horizontal interval. For example, each of the horizontal intervals may span the entire width of the target frame and may simultaneously span a part of each of the vertical dimensions. The horizontal intervals may have similar or identical sizes (e.g., each horizontal interval spans a similar or identical part of the vertical dimension) or different sizes as needed. As an example, the central horizontal interval may have a larger or smaller vertical span than other horizontal intervals that are below or above the central horizontal interval. The horizontal intervals may partially overlap, be adjacent to each other without overlap or vertical gaps, or be separated by vertical gaps. As an example, the predetermined number of horizontal intervals may be three horizontal intervals, four horizontal intervals, five horizontal intervals, six horizontal intervals, seven horizontal intervals, eight horizontal intervals, nine horizontal intervals, ten horizontal intervals, or more than ten horizontal intervals. As needed, the calculation of the average and standard deviation of the brightness in the predetermined number of horizontal intervals may be performed after downsampling the original image (as discussed above).

[0027] Step (206) may also include comparing the characteristic vectors of the raw video frames with the characteristic vectors of the sample frames. For example, the characteristic vectors of some or each of the raw video frames may be compared with the characteristic vectors of the sample frames. Determining the similarity of the characteristic vectors of the raw video frames with the characteristic vectors of the sample frames may include comparing the 3D histogram bin sets of the two image frames, comparing the mean and standard deviation of brightness over the horizontal intervals described above, and then combining the two comparisons.

[0028] Comparing 3D histogram bin sets of two image frames may, for example, involve calculating the sum of overlaps within the 3D histogram bin sets of the two image frames. Specifically, if a given 3D histogram bin has a non-zero value for X for a given raw video frame and a non-zero value for Y for a given sample frame, the smaller of X and Y may be added to the sum indicating the similarity between the frames. Such calculations may be repeated for some or all of the 3D histogram bins until a final sum of overlaps within the 3D histograms is obtained. If the two 3D histograms are identical, the sum will be equal to 100% (1.0). If the two 3D histograms have no overlap, the sum will be 0% (0.0). Since most frames share some pixels, they lie between these two extremes, and the final normalized similarity in the 3D histogram bin set will range from 0.0 to 1.0.

[0029] Comparing the mean and standard deviation of brightness across the above horizontal intervals may, for example, involve computing the overlap of normal distributions between the mean and standard deviation of the respective horizontal intervals of the two image frames being compared. The overlap of normal distributions between matching horizontal intervals of the two image frames will be a real value in the range of 0.0 to 1.0. In some embodiments, the overlap of normal distributions of different horizontal intervals may be averaged so that the overall matching value is also in the range of 0.0 to 1.0. If necessary, the lowest horizontal interval may have a first weighting (W1), the second lowest horizontal interval may have a second weighting (W2), the middle horizontal interval may have a third weighting (W3), the second highest horizontal interval may have a fourth weighting (W4), and the topmost horizontal interval may have a fifth weighting (W5), and some or all of the weights may be unique. In some embodiments, the weighting (W3) of the middle horizontal interval may be greater than the weights of the other horizontal intervals. Computing the overlap of normal distributions in horizontal intervals of two image frames may, for example, involve calculating the weighted average of the overlaps of the normal distributions for each horizontal interval.

[0030] The output of the 3D histogram bin comparison and the overlap of the horizontal interval normal distribution may be combined using any desired weighting coefficient. For example, the two measures may be combined with equal 1 / 2 weighting. In some embodiments, the two measures may be combined with unequal weighting (for example, the 3D histogram bin comparison may be given a higher or lower weighting than the overlap of the normal distribution). In some embodiments, the output of the 3D histogram bin comparison will include similarity values ​​in the range of 0.0 to 1.0, and the overlap of the horizontal interval normal distribution will also include similarity values ​​in the range of 0.0 to 1.0. When these two comparison metrics are aggregated with equal 1 / 2 weighting, the final similarity values ​​between a given raw frame and a given sample frame will also be in the range of 0.0 to 1.0.

[0031] In step (208), the similarity values ​​between one or more raw frames and sample frames, and the predetermined color grading parameters associated with the sample frame, obtained in step (204), may be combined to determine the color grading parameters for the raw frames. In some embodiments, the similarity values ​​obtained in step (206) may be used to assign weights to the predetermined color grading parameters associated with the sample frame. If the raw frame is exactly the same as the sample frame (e.g., the similarity value is 1.0), the predetermined set of color grading parameters for that sample frame will have the maximum weight. In contrast, if the raw frame and the sample frame have a similarity value of 0.0, the predetermined set of color grading parameters for that sample frame will have a weight of 0. In some embodiments, the weights of the predetermined color grading parameters from the sample frame j applied to the raw frame i may be calculated using equation (1): w(i,j)=s(i,j) / [K S -s(i,j)] (1)

[0032] In equation (1), w(i,j) can represent the weight of the raw frame i with respect to the sample frame j, and s(i,j) can represent the similarity between the raw frame i and the sample frame j, K S K can be a predetermined constant value and is greater than 1.0. In some embodiments, K S It can have a value of 1.05, which results in a maximum weight of 20. Other formulas that convert similarity values ​​into weights may be used instead of formula (1).

[0033] As discussed above, there may be M characteristic vectors (one for each of the M image sample frames), and each characteristic vector may have an associated set of 1 to N tone mapping parameter settings. Some characteristic vectors (e.g., some sample images) may be associated with tone mapping parameter settings for all N possible tone mapping operations, while other characteristic vectors (e.g., other sample images) may be associated with tone mapping parameter settings for fewer than N tone mapping operations, or with only one tone mapping parameter setting for one of the N possible tone mapping operations. When a new raw image frame arrives (e.g., in step (302)), the new raw image frame is optionally downsampled and its characteristic vector is computed. The characteristic vector of the new raw image frame is then compared with each of the M characteristic vectors of the sample frame to obtain M similarity values. The M similarity values ​​can be converted into M weights (e.g., using equation (1) above), if necessary. Next, as part of step (208), N color grading parameters for N tone mapping operations on the new raw image frame may be determined using M weights (or M similarity values). In some embodiments, each of the N tone mapping parameters for the new raw image frame may be calculated using a weighted average. As an example, the N tone mapping parameters for the new raw image frame may be calculated using the weighted average of equation (2): p(k,i)=[ Σ j p(k,j)w(i,j)] / Σ j w(i,j) (2)

[0034] In equation (2), p(k,i) represents the parameter setting for tone mapping operation k to be applied to a new frame i, p(k,j) represents the parameter setting for tone mapping operation k applied to sample frame j, and w(i,j) represents the weight of the raw frame i to sample frame j (discussed in relation to equation (1)). In some embodiments, only samples that have been supplied or associated with settings for the k-th tone mapping operation are used in the weighted average of equation (2). In situations where there are no non-zero weights for a given tone mapping operation (e.g., no samples have both supplied settings for the k-th tone mapping operation and non-zero weights for the new frame), the method may instead assign a global default value to the k-th tone mapping operation. The content creator may supply global default values ​​for any desired number of N possible tone mapping operations.

[0035] In step (210), each of the N tone mapping operations may be applied to a new video frame, each tone mapping operation being applied using its associated parameter settings determined in step (208). In this way, tone mapping operations can be applied to raw video frames in real time without requiring manual review or input from a color grader. In step (212), the color-graded video frames may be distributed for use and / or storage.

[0036] Figure 3 provides a method (300) for identifying sample frames and obtaining the color grading parameters for those sample frames used in method (200) of Figure 2. Method (300) may be performed by a processor, for example, as part of block (124) and / or block (126). A content creator may have access to past live broadcasts or sample gameplay footage (e.g., sets of possible sample frames), but it may not be desirable for the content creator to color grade each possible sample frame individually. Method (300) is an example of a method for narrowing down sets of possible sample frames to a more limited set of sample frames that are still suitable for individual color grading. As an example, it is desirable that the individually color-graded sample frames be a manageable number and, at the same time, as clearly distinguishable as possible to cover the widest range of possible future raw video inputs.

[0037] In step (302), raw sample video data may be obtained, the raw sample video containing X possible sample frames. The raw sample video data may be associated with future raw video to be color graded. For example, if the raw video to be color graded is a future broadcast of a sporting event, step (302) may include obtaining past broadcasts of similar sporting events (e.g., previous matches at the same location, same date and time, etc.). As another example, if the raw video to be color graded is game content from a particular game, step (302) may include obtaining sample content from that particular game.

[0038] In step (304), characteristic vectors for some or all of the raw sample video frames may be calculated, where the characteristic vectors may be calculated as described in step (206) of method (200).

[0039] In step (306), similarity values ​​between the feature vectors and their associated raw sample video frames may be calculated (for example, as described in step (206)). If necessary, step (306) may also include populating a symmetric matrix with a diagonal of 1.0 (e.g., the similarity of each frame itself), otherwise showing the similarity values ​​between the feature vectors and their associated raw sample video frames.

[0040] In step (308), the mode sample frame is identified and selected as the first sample frame in a set of M final sample frames. For example, step (308) may also include finding which row of the symmetric matrix has the largest sum, which represents the most average frame (e.g., mode) of the raw sample video frames.

[0041] In step (310), the frame having the smallest maximum similarity to the sample frame in the final set of sample frames may be identified and then added to the final set of sample frames. For example, step (310) may include identifying which row of the symmetric matrix has the smallest maximum value (excluding the 1.0 diagonal), and the sample frame associated with that row may be added to the final set of sample frames.

[0042] As shown by loop (312), step (310) may be repeated until a termination condition is reached. Potential termination conditions include, but are not limited to, the final number of sample frames reaching a predetermined number (M), or the candidate identified in step (310) having a similarity above a predetermined threshold to a previous entry in the final set of sample frames. For example, if the frame identified in step (310) as having the smallest maximum similarity to a sample frame in the final set of sample frames has a maximum similarity of at least 0.7 to a frame already in the final set of sample frames, method (300) terminates loop (312), and the finally identified frame may or may not be added to the final set of sample frames.

[0043] In step (314), color grading parameters may be obtained for the sample frames identified in step (310). The color grading parameters may be obtained by a color grader using a color grading station (125) as an example. In particular, the color grader may provide input via the control interface of the color grading station. If necessary, steps (310) and / or (314) may also include receiving user input to remove one or more sample frames from the final set of sample frames identified in step (310). In particular, method (300) may identify some sample frames that are not particularly useful for color grading, such as frames in fade transitions or frames that are completely black. In general, it should be easier to remove such frames from a smaller set of sample frames identified in step (310) than to remove them from the original set of sample frames obtained in step (302).

[0044] The systems and methods described above may provide color grading of images and videos based on similarity to a sample. The systems, methods, and devices relating to this disclosure may take one or more of the following configurations.

[0045] (1) A method for color grading a source image using multiple color grading operations, Obtaining source images for color grading, This involves obtaining multiple color grading parameter settings associated with each sample frame among multiple sample frames, where each color grading parameter setting becomes an input parameter for the relevant color grading operation among the multiple color grading operations. This involves obtaining multiple similarity measures, each of which indicates the level of similarity between the source image and each different sample frame among the multiple sample frames. This involves obtaining multiple weighted averages based on the aforementioned multiple similarity measures and the aforementioned multiple color grading parameter settings, wherein each weighted average is associated with each different color grading operation among the aforementioned multiple color grading operations. Applying the multiple color grading operations to the source image, using each of the multiple weighted averages as an input parameter to its associated color grading operation, in order to generate a color-graded image, To supply the aforementioned color-graded image A method of having.

[0046] (2) The method described in (1), The aforementioned multiple color grading operations include brightness adjustment operations, hue adjustment operations, and saturation adjustment operations. method.

[0047] (3) The method described in (1) or (2), The aforementioned multiple color grading operations include global luminance adjustment operations, global hue adjustment operations, and global saturation adjustment operations. method.

[0048] (4) A method according to any one of (1) to (3), The aforementioned multiple color grading operations include at least one local luminance adjustment operation, at least one local hue adjustment operation, and at least one local saturation adjustment operation. method.

[0049] (5) A method according to any one of (1) through (4), Obtaining the aforementioned multiple similarity measures involves computing multiple data structures, Each data structure is associated with the source image and one different image from among the sample frames in the plurality of sample frames. Each data structure has multiple bins, each bin is associated with an intrinsic range of chromaticity values ​​and an intrinsic range of luminance values, and each bin includes a count of the number of pixels in the image associated with the data structure, each having a chromaticity value within the range of chromaticity values ​​associated with that bin and a luminance value within the range of luminance values ​​associated with that bin. method.

[0050] The method described in (6)(5), Each sample frame in the plurality of sample frames and the source image are encoded with a transfer function that covers the luminance range of Xnit with a Y-bit depth, and the bins of the data structure cover the luminance range of Xnit with a Z-bit depth, where Z is less than or equal to half of Y. method.

[0051] The method described in (7), (5), or (6), Each sample frame in the plurality of sample frames and the source image are encoded with a transfer function that includes a chromaticity value covering a given color space with a Y-bit depth, and the bins of the data structure cover the given color space with a Z-bit depth, where Z is greater than half of Y. method.

[0052] (8)(5) through (7) The method described in any one of these, Computing the plurality of data structures further includes downsampling the source image before computing the data structures associated with the source image, and downsampling the sample frames among the plurality of sample frames before computing the data structures associated with the source image. method.

[0053] A method according to any one of (9), (5) through (8), Obtaining the aforementioned multiple similarity measures further includes identifying the overlap between the data structure associated with the source image and the data structure associated with the sample frame. method.

[0054] (10) A method according to any one of (1) through (9), Obtaining the aforementioned multiple similarity measures is Dividing each sample frame and the source image from the aforementioned plurality of sample frames into a plurality of horizontal bands, To obtain the average and standard deviation of luminance within each horizontal band of the aforementioned horizontal band, To compute the overlap of the normal distributions of the acquired mean and standard deviation of the luminance within each horizontal band of the aforementioned horizontal band. This also includes, method.

[0055] A method according to any one of (11)(5) to (9), Obtaining the aforementioned multiple similarity measures is Identifying the overlap between the data structure associated with the source image and the data structure associated with the sample frame, Dividing each sample frame and the source image from the aforementioned plurality of sample frames into a plurality of horizontal bands, To obtain the average and standard deviation of luminance within each horizontal band of the aforementioned horizontal band, Computing the overlap of the normal distributions of the acquired mean and standard deviation of the luminance within each horizontal band of the aforementioned horizontal band, The similarity measure is calculated by combining a first metric based on the identified overlap between the data structure associated with the source image and the data structure associated with the sample frame with a second metric based on the computed overlap of a normal distribution. This also includes, method.

[0056] (12) A color grading system having at least one controller, The aforementioned at least one controller is Obtaining source images for color grading, This involves obtaining multiple color grading parameter settings, where each color grading parameter setting becomes an input parameter for the color grading operation, and each color grading parameter setting is associated with a different sample frame among multiple sample frames. This involves calculating multiple similarity measures, each of which represents the level of similarity between the source image and each different sample frame among the multiple sample frames. Calculating a weighted average based on the aforementioned multiple similarity measures and the aforementioned multiple color grading parameter settings, Applying the color grading operation to the source image using the weighted average as an input parameter to the color grading operation in order to generate a color-graded image, To supply the aforementioned color-graded image A color grading system configured to perform the following actions.

[0057] The color grading system described in (13)(12), It also has a display, The controller is configured to supply the color-graded image to the display, The display is configured to display the color-graded image. Color grading system.

[0058] (14)(13) The color grading system described above, The aforementioned source image includes live content, The at least one controller is configured to supply the color-graded image to the display within 200 milliseconds of acquiring the source image. Color grading system.

[0059] A color grading system described in any one of (15)(12) to (14), The at least one controller is configured to obtain the multiple color grading parameter settings before generating the source image. Color grading system.

[0060] (16) A method, Obtaining a sequence of image frames, This involves obtaining multiple similarity measures, each of which indicates the level of similarity between different pairs of image frames in the sequence of image frames. (i) Adding the first image from the sequence of image frames to the set of sample frames, (ii) Identifying which image frame from among the image frames not yet added to the set of sample frames is least similar to the image frames in the set of sample frames, according to the similarity measure. (iii) Add the image frame identified in (ii) to the set of sample frames, and (iv) Repeat (ii) and (iii) until the completion condition is met. This involves selecting the set of sample frames from the sequence of image frames. A method of having.

[0061] The method described in (17)(16), Repeating (ii) and (iii) until the completion condition is met includes repeating (ii) and (iii) until the number of sample frames added to the set of sample frames is at least a predetermined number. method.

[0062] The method described in (18), (16), or (17), Repeating (ii) and (iii) until the completion condition is met includes repeating (ii) and (iii) until the similarity measure of the image frame identified in (ii) with an image frame already added to the set of sample frames exceeds a predetermined threshold. method.

[0063] A method according to any one of (19), (16) through (18), Adding the first image from the sequence of image frames to the set of sample frames includes adding the most average frame from the sequence of image frames to the set of sample frames. method.

[0064] A method according to any one of (20)(16) to (19), Adding the first image from the sequence of image frames to the set of sample frames includes adding a randomly selected frame from the sequence of image frames to the set of sample frames. method.

[0065] A method according to any one of (21)(16) to (20), (ii) Identifying which image frame from among the image frames not yet added to the set of sample frames is least similar to any of the image frames in the set of sample frames according to the similarity measure includes identifying which image frame from among the image frames not yet added to the set of sample frames has the smallest maximum similarity to any of the image frames in the set of sample frames. method.

[0066] A method according to any one of (22)(16) to (21), Obtaining the aforementioned multiple similarity measures includes computing the characteristic vector of each image frame in the sequence of image frames. method.

[0067] The method described in (23)(22), Computing the characteristic vector of a given image frame is Downsampling the given image frame to generate a downsampled image with a lower resolution than the given image frame, To compute the average brightness and standard deviation of the brightness of multiple horizontal intervals of the downsampled image frame. including, method.

[0068] The method described in (24)(22), The image frame is encoded with a transfer function having a luminance range encoded with X bits and at least two chromaticity parameters, each encoded with Y bits. Computing the characteristic vector of a given image frame is Downsampling the given image frame to generate a downsampled image frame with a lower resolution than the given image frame, Compute the average brightness and standard deviation of the brightness of multiple horizontal intervals of the downsampled image frame, Computing the data structure from the downsampled image frame It has, The data structure includes a plurality of bins, the luminance range of the transfer function is encoded in the data structure with A bits, and the at least two chromaticity parameters are encoded in the data structure with B bits, where A is less than X and B is less than Y. Computing the aforementioned data structure includes counting the number of pixels from the downsampled image frame associated with each bin. method.

[0069] A method according to any one of (25)(16) to (24), After the completion conditions are satisfied, the process further includes obtaining at least one color grading parameter setting for each sample frame in the set of sample frames. Each of the aforementioned color grading parameter settings becomes an input to the color grading operation. method.

[0070] The method described in (26)(25), Obtaining source images for color grading, This involves calculating multiple further similarity measures, each of which represents the level of similarity between the source image and each different sample frame in the set of sample frames. Calculating a weighted average based on the aforementioned multiple further similarity measures and the color grading parameter settings associated with the sample frames in the set of sample frames, Applying the color grading operation to the source image using the weighted average as an input parameter to the color grading operation in order to generate a color-graded image, To supply the aforementioned color-graded image Methods that further include the above.

[0071] (27) A computer program product having instructions, A computer program product in which, when the instruction is executed by a computing system, the computing system causes the computing system to perform any one of (1) to (11) or (16) to (26).

[0072] With respect to the processes, systems, methods, heuristics, etc., described herein, the steps of such processes, etc., are described as occurring in a specific ordered sequence, but it should be understood that such processes may be carried out by having the steps performed in an order other than that described herein. Furthermore, it should be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the descriptions of processes in this specification are given for the purpose of illustrating specific embodiments and should never be construed as limiting the scope of the claims.

[0073] Therefore, it should be understood that the above description is intended to be illustrative and not limiting. Many embodiments and applications other than those given will become apparent from reading the above description. The scope should not be determined by reference to the above description, but rather by reference to the appended claims and the entire scope of the equivalents to which those claims are entitled. The technology described herein is expected to develop in the future, and the disclosed systems and methods are intended to be incorporated into such future embodiments. In short, it should be understood that this application is subject to modification and alteration.

[0074] All terms used in the claims are intended to be given their broadest reasonable interpretation and their ordinary meaning as understood by a person familiar with the art described herein, unless otherwise expressly indicated herein. In particular, the use of singular articles such as “a,” “the,” and “said” should be interpreted as describing one or more of the elements shown, unless otherwise expressly indicated in the claims.

[0075] This summary of the disclosure is provided to allow readers to quickly confirm the nature of the technical disclosure. The summary is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, it is evident from the preceding detailed description that, for the purpose of simplifying the disclosure, various features are grouped into various embodiments. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments incorporate more features than those expressly described in each claim. Rather, as reflected in the subsequent claims, the subject matter of the invention consists of fewer features than all the features of a single disclosed embodiment combined. Accordingly, the subsequent claims are incorporated into the detailed description, and each claim stands independently as individually claimed subject matter.

[0076] [Cross-references to related applications] This application claims priority to U.S. Provisional Patent Application No. 63 / 391528, filed on 22 July 2022, and European Patent Application No. 22197590.7, filed on 23 September 2022, which are incorporated herein by reference in their entirety.

Claims

1. A computer-based method for color grading a source image using multiple color grading operations, Obtaining source images for color grading, Determining the set of sample frames, This involves obtaining multiple color grading parameter settings associated with each sample frame in the set of sample frames, where each color grading parameter setting becomes an input parameter for the relevant color grading operation among the multiple color grading operations. For each sample frame in the set of sample frames, a similarity measure is obtained that indicates the level of similarity between the source image and the sample frame. For each sample frame in the set of sample frames, a weighted color grading parameter setting is obtained by applying a weight based on the similarity measure of that sample frame to each color grading parameter setting of that sample frame. For each of the aforementioned color grading operations, the average of the weighted color grading parameter settings associated with that color grading operation across the set of sample frames is obtained. Applying the multiple color grading operations to the source image, using each of the weighted color grading parameter settings' averages as input parameters to the associated color grading operations, to generate a color-graded image. To supply the aforementioned color-graded image It has, Determining the set of sample frames is Obtaining a sequence of image frames, This involves obtaining multiple similarity measures, each of which indicates the level of similarity between different pairs of image frames in the sequence of image frames. (i) Adding the first image from the sequence of image frames to the set of sample frames, (ii) Identifying which image frame from among the image frames not yet added to the set of sample frames is the least similar to the image frames in the set of sample frames, according to the plurality of similarity measures. Adding the image frame identified in (iii)(ii) to the set of sample frames, and (iv) Repeat (ii) and (iii) until the completion condition is met. This involves selecting the set of sample frames from the sequence of image frames. It has, A method wherein the multiple color grading parameter settings for each sample frame in the set of sample frames are obtained after the completion condition is met.

2. The aforementioned multiple color grading operations include brightness adjustment operations, hue adjustment operations, and saturation adjustment operations. The method according to claim 1.

3. The aforementioned multiple color grading operations include global luminance adjustment operations, global hue adjustment operations, and global saturation adjustment operations. The method according to claim 1.

4. The aforementioned multiple color grading operations include at least one local luminance adjustment operation, at least one local hue adjustment operation, and at least one local saturation adjustment operation. The method according to claim 1.

5. Obtaining the aforementioned similarity measure involves computing multiple data structures, Each data structure is associated with one different image from the sample frames in the set of the source image and the sample frames. Each data structure has multiple bins, each bin is associated with an intrinsic range of chromaticity values ​​and an intrinsic range of luminance values, and each bin includes a count of the number of pixels in the image associated with the data structure, each having a chromaticity value within the range of chromaticity values ​​associated with that bin and a luminance value within the range of luminance values ​​associated with that bin. The method according to claim 1.

6. Each sample frame in the set of sample frames and the source image are encoded with a transfer function that covers the luminance range of Xnit with a Y-bit depth, and the bins of the data structure cover the luminance range of Xnit with a Z-bit depth, where Z is less than or equal to half of Y. The method according to claim 5.

7. Each sample frame in the set of sample frames and the source image are encoded with a transfer function that includes a chromaticity value covering a given color space with a Y-bit depth, and the bins of the data structure cover the given color space with a Z-bit depth, where Z is greater than half of Y. The method according to claim 5.

8. Computing the plurality of data structures further includes downsampling the source image before computing the data structures associated with the source image, and downsampling the sample frames in the set of sample frames before computing the data structures associated with the source image. The method according to claim 5.

9. Obtaining the similarity measure further includes identifying the overlap between the data structure associated with the source image and the data structure associated with the sample frame. The method according to claim 5.

10. Obtaining the aforementioned similarity measure means Dividing each sample frame in the set of sample frames and the source image into multiple horizontal bands, To obtain the average and standard deviation of luminance within each horizontal band of the aforementioned horizontal band, To compute the overlap of the normal distributions of the acquired mean and standard deviation of the luminance within each horizontal band of the aforementioned horizontal band. This also includes, The method according to claim 1.

11. Obtaining the aforementioned similarity measure means Identifying the overlap between the data structure associated with the source image and the data structure associated with the sample frame, Dividing each sample frame in the set of sample frames and the source image into multiple horizontal bands, To obtain the average and standard deviation of luminance within each horizontal band of the aforementioned horizontal band, Computing the overlap of the normal distributions of the acquired mean and standard deviation of the luminance within each horizontal band of the aforementioned horizontal band, The similarity measure is calculated by combining a first metric based on the identified overlap between the data structure associated with the source image and the data structure associated with the sample frame with a second metric based on the computed overlap of the normal distribution. This also includes, The method according to claim 5.

12. A color grading system having at least one controller configured to perform the method according to any one of claims 1 to 11.

13. It also has a display, The controller is configured to supply the color-graded image to the display, The display is configured to display the color-graded image. The color grading system according to claim 12.

14. The aforementioned source image includes live content, The at least one controller is configured to supply the color-graded image to the display within 200 milliseconds of acquiring the source image. The color grading system according to claim 13.

15. The at least one controller is configured to acquire the multiple color grading parameter settings before generating the source image. The color grading system according to claim 12.

16. Repeating (ii) and (iii) until the completion condition is met includes repeating (ii) and (iii) until the number of sample frames added to the set of sample frames is at least a predetermined number. The method according to claim 1.

17. Repeating (ii) and (iii) until the completion condition is met includes repeating (ii) and (iii) until the similarity measure of the image frame identified in (ii) with the image frames already added to the set of sample frames exceeds a predetermined threshold. The method according to claim 1.

18. Adding the first image from the sequence of image frames to the set of sample frames includes adding the most average frame from the sequence of image frames to the set of sample frames. The method according to claim 1.

19. Adding the first image from the sequence of image frames to the set of sample frames includes adding a randomly selected frame from the sequence of image frames to the set of sample frames. The method according to claim 1.

20. (ii) Identifying which image frame from among the image frames not yet added to the set of sample frames is least similar to any of the image frames in the set of sample frames according to the plurality of similarity measures includes identifying which image frame from among the image frames not yet added to the set of sample frames has the smallest maximum similarity to any of the image frames in the set of sample frames. The method according to claim 1.

21. Obtaining the aforementioned multiple similarity measures includes computing the characteristic vector of each image frame in the sequence of image frames. The method according to claim 1.

22. Computing the characteristic vector of a given image frame is Downsampling the given image frame to generate a downsampled image with a lower resolution than the given image frame, To compute the average brightness and standard deviation of the brightness of multiple horizontal intervals of the downsampled image frame. including, The method according to claim 21.

23. The image frame is encoded with a transfer function having a luminance range encoded with X bits and at least two chromaticity parameters, each encoded with Y bits. Computing the characteristic vector of a given image frame is Downsampling the given image frame to generate a downsampled image frame with a lower resolution than the given image frame, Compute the average brightness and standard deviation of the brightness of multiple horizontal intervals of the downsampled image frame, Computing the data structure from the downsampled image frame It has, The data structure includes a plurality of bins, the luminance range of the transfer function is encoded in the data structure with A bits, and the at least two chromaticity parameters are encoded in the data structure with B bits, where A is less than X and B is less than Y. Computing the aforementioned data structure includes counting the number of pixels from the downsampled image frame associated with each bin. The method according to claim 21.

24. Obtaining source images for color grading, This involves calculating multiple further similarity measures, each of which represents the level of similarity between the source image and each different sample frame in the set of sample frames. Calculating a weighted average based on the aforementioned multiple further similarity measures and the color grading parameter settings associated with the sample frames in the set of sample frames, Applying the color grading operation to the source image using the weighted average as an input parameter to the color grading operation in order to generate a color-graded image, To supply the aforementioned color-graded image The method according to claim 1, further comprising:

25. A computer program having instructions that, when executed by a computing system, cause the computing system to perform the method according to any one of claims 1 to 11 or 16 to 24.

26. A color grading system having at least one controller configured to perform the method described in any one of claims 1 and 16 to 24.

Images