Apparatus and method for adjusting digital image

By identifying and adjusting the rendering differences between image data and reference monitors, and automating color grading and quality control, the problems of low efficiency and poor consistency of digital image color grading in the prior art are solved, and efficient and accurate multi-format color grading is achieved.

CN118901095APending Publication Date: 2024-11-05BARCO NV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380023082.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-02-22
Filing Date
2023-02-21
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

The prior art has problems of inefficiency and inconsistent results in the process of color grading of digital images. Especially in different displays and environments, it is difficult to achieve efficient and accurate color grading, and multiple manual interventions and repeated operations are required.

Method used

Provides an apparatus and method to identify the difference between image data and reference monitor rendering through a data processor, generate differential data, and optimize the color grading process using automated steps and quality control measures, including simulating the operation of different monitors, Analyze image data, identify crop and crush pixels, automatically adjust tone mapping parameters, and generate reference masters that are suitable for different distribution formats.

Benefits of technology

It improves the efficiency and accuracy of color grading of digital images, reduces manual intervention, achieves color consistency between multiple formats, reduces cost and time overhead, and enhances the automation and intelligence of color grading.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118901095A_ABST
    Figure CN118901095A_ABST
Patent Text Reader

Abstract

Apparatus and methods for color grading may be implemented to improve the efficiency of color grading digital content. Features of the apparatus and method include automatic detection of differences between image data and rendered images, automatic suggestion of color rating, detection and communication of temporal effects, guidance of selection of frame order for rating, automatic QC process, and / or reference monitor simulation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present technology relates to adjusting digital images. Example implementations provide tools for color grading content such as movies, videos, and other digital images. Background Art

[0002] An important step in the production of visual digital content (e.g., movies, videos, digital images) is color grading. Color grading is the process by which a colorist (person) views digital content on a reference monitor. The colorist uses a grading workstation to adjust the digital content (e.g., by changing the color and / or luminosity of the pixels). The grading workstation is typically provided by a computer workstation running such computer software that provides controls that the colorist can use to adjust the digital content. When color grading is complete, the digital content is saved as a "reference master." A copy of the content for distribution can be obtained from the reference master and delivered to the end user for viewing.

[0003] Different distribution channels may require different reference masters. For example, different reference masters may be provided for viewing in a theater and for viewing at home. Different reference masters may differ in their color grading, aspect ratio, image data formatting, digital resolution, color space, and / or other factors. In some cases, different reference masters are provided for different projector types (e.g., projectors from different projector manufacturers, etc.).

[0004] There is a need for new tools that can facilitate color grading of digital content in a more efficient manner that produces improved results and / or new tools that facilitate color grading in a more informed manner. Summary of the invention

[0005] The present invention has various aspects, including but not limited to:

[0006] Method and apparatus for color grading of digital images;

[0007] ●Color grading suite; and

[0008] Method and apparatus for generating a reference master of digital content;

[0009] Methods and apparatus for automating the color grading process;

[0010] · Method and apparatus for quality control of digital content.

[0011] The following are various non-limiting example aspects of the present technology.

[0012] One aspect of the present technology provides an apparatus comprising: an input system operable to receive image data for display on a reference monitor; a data processor configured to process the image data received at a first input to identify a difference between an image specified by the image data and a rendering of the image by the reference monitor and to generate difference data representative of the difference; and an output system operable to present the difference data.

[0013] In some embodiments, the input system is operable to receive an amplitude signal from a reference monitor, the data processor being configured to process the amplitude signal to identify clipped and / or squashed pixels of the image data in the reference monitor's rendering of the image.

[0014] In some embodiments, the data processor is configured to calculate an analog amplitude signal for a reference monitor, and to process the analog amplitude signal to identify clipped and / or squashed pixels of the image data in a rendering of the image by the reference monitor.

[0015] In some embodiments, the data processor is configured to compare the image data to performance characteristics of a reference display defining a performance envelope and determine differences by identifying pixels in the image data having pixel values ​​outside the performance envelope of the reference monitor.

[0016] In some embodiments, the data processor is configured to simulate operation of a reference monitor to render the image data to produce simulated rendered image data, and to identify differences by comparing the image data to the simulated rendered image data.

[0017] In some embodiments, when simulating the operation of the reference monitor, the processor is configured to simulate one or more or all of the following:

[0018] The maximum brightness of each pixel of the reference monitor;

[0019] The dark level of each pixel of the reference monitor;

[0020] The processing performed on the image data by the reference monitor;

[0021] · Operation of the power supply of the reference monitor;

[0022] Thermal management of reference monitors;

[0023] The local contrast of the reference monitor;

[0024] The rate of change of the brightness of the reference monitor;

[0025] · Reference monitor's light redirecting components.

[0026] In some embodiments, the differences include clipping and / or crushing by a reference monitor.

[0027] In some embodiments, the apparatus is operable to output simulation rendering data.

[0028] In some embodiments, the apparatus is operable to store simulation rendering data.

[0029] In some embodiments, the apparatus is configured to display the difference data.

[0030] In some embodiments, the apparatus is configured to display the difference data as an image indicating locations of pixels in the reference monitor's rendering of the image that differ from the image specified by the image data.

[0031] In some embodiments, the apparatus is configured to display the difference data as a number and / or graphical indicator showing the number of pixels in the reference monitor's rendering of the image that differ from the image specified by the image data.

[0032] In some embodiments, the indicator indicates an amount of cropped pixels relative to a crop amount threshold.

[0033] In some embodiments, the indicator indicates the number of cropped pixels for which cropping exceeds a crop magnitude threshold.

[0034] In some embodiments, the device is integrated with a reference monitor.

[0035] In some embodiments, the device is integrated with a color grading console.

[0036] In some embodiments, the apparatus is combined with a reference monitor and a color grading console, and the color grading console is connected to provide image data to the input of the reference monitor and to an input system of the apparatus.

[0037] In some embodiments, the color grading console is configured to display a representation of the difference data on a user interface of the color grading console.

[0038] In some embodiments, the data processor is configured to identify, for each color channel of a plurality of color channels, a difference between an image specified by the image data and a rendering of the image by a reference monitor.

[0039] In some embodiments, the data processor is configured to, simultaneously or at different times: simulate the operation of one or more other monitors different from the reference monitor to render image data and generate simulated rendered image data for each of the one or more other monitors; and output the simulated rendered image data for one of the one or more other monitors for display on the reference monitor.

[0040] In some embodiments, in the operation of emulating a monitor of one or more other monitors, the processor is configured to emulate one or more or all of the following:

[0041] The maximum brightness of each pixel of other monitors;

[0042] The dark level of each pixel of other monitors;

[0043] The processing of image data by other monitors;

[0044] ·Operation of power supply of other monitors;

[0045] Thermal management of other monitors;

[0046] Local contrast of other monitors;

[0047] The rate of change of brightness of other monitors;

[0048] ·Light redirecting components for monitors.

[0049] In some embodiments, an apparatus includes a data analyzer configured to process image data to collect data about each frame in a set of frames defined by the image data, and a database connected to store the collected data.

[0050] In some embodiments, items of collected data are associated with frames from which the items of collected data were respectively collected.

[0051] In some embodiments, items of collected data are associated in a database with scenes or clips of image data.

[0052] In some embodiments, collecting data includes one or more of the following:

[0053] ●Minimum, maximum, average and median values ​​for each color channel;

[0054] Minimum, maximum, average and median brightness measurements;

[0055] Maximum red, green and blue;

[0056] The percentage or number of clipped pixels and squashed pixels;

[0057] Based on the proportion of the light budget required for each color channel;

[0058] The proportion of the required light budget based on the total light budget;

[0059] A mark or counter indicating the presence of a defect;

[0060] Image histogram; and

[0061] Noise level.

[0062] In some embodiments, the data analyzer is integrated with the data processor.

[0063] In some embodiments, the apparatus is configured to perform any one, or any combination of two or more of the following:

[0064] Detecting temporal effects, such as flicker, in one or more color channels;

[0065] Automatically identify frames that require priority attention from colorists;

[0066] generating and displaying a graphical indicator showing how the selected statistical data varies over time in content corresponding to the image data;

[0067] Calculate statistics on clips, shots or scenes;

[0068] Compare statistics across frames;

[0069] Calculate and / or compare average statistics for different groups of frames;

[0070] Detect individual clips;

[0071] · Associations correspond to clips of similar visual stimuli.

[0072] In some embodiments, the apparatus is further configured to present a display item indicating a time effect.

[0073] In some embodiments, the apparatus is configured to store in a database tags indicating one or more types of tagged frames, and the apparatus provides a control operable to jump to tagged frames of a selected type of the one or more types.

[0074] In some embodiments, the markings include priority markings that indicate frames that should receive priority attention from a colorist.

[0075] In some embodiments, the priority marking indicates frames that have been identified as having the greatest number of cropped pixels in the respective clips. The apparatus of claim 30, wherein the priority marking indicates frames that have been identified as having the greatest number of cropped pixels in the respective clips whose magnitude exceeds a threshold clip magnitude.

[0076] In some embodiments, the database comprises a sorting function that allows a set of frames that are the result of a search of the database to be sorted in a desired order.

[0077] In some embodiments, the apparatus includes controls that invoke a search function and a sort function to do one or more of the following:

[0078] Identify the N frames in each clip that have the largest number of cropped pixels (N can be 1, 2, 3, ...);

[0079] Identify the N frames in each clip that have the largest number of squashed pixels (N can be 1, 2, 3, ...);

[0080] Identify frames with suspected compression artifacts;

[0081] Identify frames with bad pixels;

[0082] Identify frames with suspected flash defects; and

[0083] Identify frames that are similar to each other or to the currently selected frame.

[0084] In some embodiments, the device includes a control operable by a user to jump between those frames that have been identified as having the greatest number of cropped pixels in the respective clips.

[0085] In some embodiments, the device includes controls operable by a user to jump between and / or compare frames of image data having similar characteristics.

[0086] In some embodiments, the device includes a control operable by a user to jump between frames including defective image data.

[0087] In some embodiments, the defects include one or more of bad pixels, flash defects, and compression artifacts.

[0088] In some embodiments, the apparatus is configured to display metrics for a current frame of image data and a series of other frames adjacent to the current frame.

[0089] In some embodiments, the apparatus is configured to compute and display high-level metrics aggregated from multiple frames.

[0090] In some embodiments, the apparatus is configured to display a timeline graphically illustrating the magnitude of one or more high-level metrics.

[0091] In some embodiments, the apparatus is configured to suggest or automatically select frames in each of a plurality of clips in the image data for initial grading of the respective clips.

[0092] In some embodiments, the apparatus is configured to generate a scaling factor or tone mapping parameter or tone mapping function that will enable the cropped pixels and / or the squashed pixels in the frame to be within range.

[0093] In some embodiments, the apparatus is configured to generate one or more scaling factors or tone mapping parameters or tone mapping functions selected to make two or more selected clips more consistent with each other.

[0094] In some embodiments, the apparatus is configured to suggest key frames for grading dynamic behavior of the image data.

[0095] In some embodiments, the apparatus is configured to automatically ask one or more QC questions for each of a plurality of frames defined by the image data, and to generate and present a QC analysis.

[0096] In some embodiments, the apparatus includes a feature recognition module comprising a trained machine learning system trained to recognize movie-making devices in frames of image data.

[0097] Another aspect of the present technology provides an apparatus comprising: an input system operable to receive image data for display on a reference monitor; a data processor configured to simulate operation of one or more other monitors different from the reference monitor to render the image data and generate simulated rendered image data for each of the one or more other monitors; and output the simulated rendered image data for one of the one or more other monitors for display on the reference monitor.

[0098] In some embodiments, the data processor is configured to emulate the operation of a plurality of other monitors.

[0099] In some embodiments, in simulating the operation of a monitor of the one or more other monitors, the processor is configured to simulate one or more or all of the following:

[0100] The maximum brightness of each pixel of other monitors;

[0101] The dark level of each pixel of other monitors;

[0102] The processing of image data by other monitors;

[0103] ·Operation of power supply of other monitors;

[0104] Thermal management of other monitors;

[0105] Local contrast of other monitors;

[0106] The rate of change of brightness of other monitors;

[0107] · Light redirecting components for other monitors.

[0108] Another aspect of the present technology provides an apparatus comprising: an input system operable to receive image data for display on a reference monitor; a data processor configured to process the image data received at a first input; a data analyzer configured to process the image data to collect data about each frame in a set of frames defined by the image data; a database connected to store the collected data and configured to associate items of the collected data with corresponding frames in the frames of the image data; and an output system operable.

[0109] In some embodiments, items of collected data are associated in a database with scenes or clips of image data.

[0110] In some embodiments, collecting data includes one or more of the following:

[0111] The minimum, maximum, average and median values ​​for each color channel;

[0112] Minimum, maximum, average and median brightness measurements;

[0113] Maximum red, green and blue;

[0114] The percentage or number of clipped pixels and squashed pixels;

[0115] Based on the proportion of the light budget required for each color channel;

[0116] The proportion of the required light budget based on the total light budget;

[0117] · a mark or counter to indicate the presence of a defect;

[0118] Image histogram; and

[0119] Noise level.

[0120] In some embodiments, the data analyzer is integrated with the data processor.

[0121] In some embodiments, the apparatus is configured to do any one, or any combination of two or more of the following:

[0122] Detecting temporal effects, such as flicker, in one or more color channels;

[0123] Automatically identify frames that require priority attention from colorists;

[0124] generating and displaying a graphical indicator showing how the selected statistical data varies over time in content corresponding to the image data;

[0125] Calculate statistics on clips, shots or scenes;

[0126] Compare statistics across frames;

[0127] Calculate and / or compare average statistics for different groups of frames;

[0128] Detect individual clips;

[0129] · Associations correspond to clips of similar visual stimuli.

[0130] In some embodiments, the apparatus is further configured to present a display item indicating a time effect.

[0131] In some embodiments, the apparatus is configured to store in a database tags indicating one or more types of tagged frames, and the apparatus provides a control operable to jump to tagged frames of a selected type of the one or more types.

[0132] In some embodiments, the markings include priority markings that indicate frames that should receive priority attention from a colorist.

[0133] In some embodiments, the priority marking indicates the frame in the respective clip that has been identified as having the greatest number of cropped pixels.

[0134] In some embodiments, the database comprises a sorting function that allows a set of frames that are the result of a search of the database to be sorted in a desired order.

[0135] In some embodiments, the apparatus includes controls that invoke a search function and a sort function to do one or more of the following:

[0136] Identify the N frames in each clip that have the largest number of cropped pixels (N can be 1, 2, 3, ...);

[0137] Identify the N frames in each clip that have the largest number of squashed pixels (N can be 1, 2, 3, ...);

[0138] Identify frames with suspected compression artifacts;

[0139] Identify frames with bad pixels;

[0140] Identify frames with suspected flash defects; and

[0141] Identify frames that are similar to each other or to the currently selected frame.

[0142] In some embodiments, the device includes a control operable by a user to jump between frames in respective clips that have been identified as having a maximum number of cropped pixels.

[0143] In some embodiments, the device includes controls operable by a user to jump between and / or compare frames of image data having similar characteristics.

[0144] In some embodiments, the device includes a control operable by a user to jump between frames including defective image data.

[0145] In some embodiments, the defects include one or more of bad pixels, flicker defects, and compression artifacts.

[0146] In some embodiments, the apparatus is configured to display metrics for a current frame of image data and a series of other frames adjacent to the current frame.

[0147] In some embodiments, the apparatus is configured to compute and display high-level metrics aggregated from multiple frames.

[0148] In some embodiments, the apparatus is configured to display a timeline graphically illustrating the magnitude of one or more high-level metrics.

[0149] Another aspect of the present technology provides an apparatus comprising: an input system operable to receive image data for display on a reference monitor; a data processor configured to process frames of the image data received at a first input to suggest or automatically select frames in each of a plurality of clips of the image data for initial grading of the respective clips; and an output system operable to output indications of the suggested or selected frames to a user interface.

[0150] In some embodiments, the data processor is configured to generate a scaling factor or tone mapping parameter or tone mapping function that will enable the cropped pixels and / or the squashed pixels in the proposed or selected frame to be within a range.

[0151] Another aspect of the present technology provides an apparatus comprising: an input system operable to receive image data for display on a reference monitor; a data processor configured to process the image data received at the input; a feature recognition module comprising a machine learning system trained to identify film production devices in frames of the image data and to mark frames in which film production devices are identified; and an output system operable to output identification of the marked frames.

[0152] In some embodiments, the apparatus is configured to generate one or more scaling factors or tone mapping parameters or tone mapping functions selected to make two or more selected clips more consistent with each other.

[0153] In some embodiments, the apparatus is configured to suggest key frames for grading dynamic behavior of the image data.

[0154] Another aspect of the present technology provides a computer program product comprising a tangible medium storing machine-readable, machine-executable instructions which, when executed by a data processor, cause the data processor to perform a method performed by any of the apparatus described herein.

[0155] Another aspect of the present technology provides a color grading kit comprising an apparatus as described herein.

[0156] Another aspect of the technology provides a device having any new and inventive feature, combination of features, or sub-combination of features as described herein.

[0157] Another aspect of the present technology provides methods having any new and inventive steps, actions, combinations of steps and / or actions, or sub-combinations of steps and / or actions as described herein.

[0158] Other aspects and example embodiments are shown in the accompanying drawings and / or described in the following description.

[0159] It is emphasized that the invention relates to all combinations of features mentioned above, even if these are recited in different claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0160] The accompanying drawings illustrate non-limiting example embodiments of the invention.

[0161] Figure 1 is a functional block diagram illustrating an example apparatus according to an example implementation of the present technology.

[0162] Figure 2 An example layout of a portion of a user interface that may be included in a device in accordance with the present technology is shown.

[0163] FIG. 3 is a schematic diagram of a prior art apparatus for color grading.

[0164] Figure 4 is a functional block diagram illustrating an apparatus for analyzing image data, identifying instances of cropping or squashing in the display of the image data on a reference monitor, and communicating the extent of the cropping / squeezing to a user.

[0165] Figure 5A , Figure 5B and Figure 5C is a diagram illustrating example display items provided for indicating cropping and / or crushing in a frame of digital content.

[0166] Figure 6 is a schematic diagram of an apparatus for color grading according to an example embodiment.

[0167] Figure 7 is a combined flow chart and data flow diagram illustrating an example method for color grading content in multiple formats.

[0168] Figure 8 is a graph showing example conversions between different formats.

[0169] 9A to 9D is a schematic diagram illustrating an apparatus for color grading according to several example embodiments.

[0170] FIG. 10A to FIG. 10D are example display items that may be provided in the present technology for conveying time information. DETAILED DESCRIPTION

[0171] In the following description, specific details are set forth in order to provide a more thorough understanding of the present invention. However, the present invention may be practiced without these details. In other cases, well-known elements are not shown or described in detail to avoid making the present invention unnecessarily difficult to understand. Therefore, the description and the accompanying drawings should be considered illustrative, rather than restrictive.

[0172] Figure 1 An example apparatus 10 including features for facilitating color grading of digital content is shown. The apparatus 10 includes a data processor 12 configured to perform methods as described herein and / or provide controls as described herein. For example, the data processor 12 may be configured by computer executable instructions 14 in a data store 15 accessible to the data processor 12 to perform such methods and / or provide such controls.

[0173] The data processor 12 is not limited to being a single discrete device. The functionality of the data processor 12 may be distributed across multiple devices (e.g., multiple microprocessors, multiple cores, etc.). The data processor 12 may be implemented, for example, using specially designed hardware, configurable hardware, a programmable data processor, and / or a combination of two or more of these, the programmable data processor being configured by providing software (which may optionally include "firmware") that can be executed on a data processor, a special purpose computer, or a data processor that is specially programmed, configured, or constructed to perform one or more steps in the method as explained in detail herein.

[0174] Examples of specially designed hardware are: logic circuits, application specific integrated circuits ("ASICs"), large scale integrated circuits ("LSIs"), very large scale integrated circuits ("VLSIs"), etc. Examples of configurable hardware are: one or more programmable logic devices, such as programmable array logic ("PAL"), programmable logic arrays ("PLAs"), and field programmable gate arrays ("FPGAs"). Examples of programmable data processors are: microprocessors, digital signal processors ("DSPs"), embedded processors, graphics processors ("GPUs"), mathematical coprocessors, general purpose computers, server computers, graphics workstations, cloud computers, mainframe computers, etc. Data processor 12 need not be exclusively dedicated to the functions described herein, but can also be configured to perform other functions. In some embodiments, the output of data processor 12 is directly connected to the input of a reference monitor, projector system, etc.

[0175] The apparatus 10 includes a color grading console 16 for providing a user interface 16A through which a user (e.g., a colorist) can do one or more of the following: control viewing of digital content 17 on a connected reference monitor 18; access and control tools for allowing the user to change pixel values ​​of the digital content 17 to produce graded digital content 17A; access and control tools for performing certain automated processing of the digital content 17 and / or graded digital content 17A and / or performing analysis of the digital content 17 and / or graded digital content 17A and presenting them to the user.

[0176] User interface 16A may include physical controls (eg, switches, sliders, knobs) and / or software-defined controls (eg, controls provided by a graphical interface of color grading console 16).

[0177] Figure 2 An example user interface 16A of the grading console 16 is shown. The user interface 16A may be displayed on a display of the grading console 16. The user interface 16A includes: controls 19A, 19B, and 19C, which allow the levels of the corresponding primary colors to be adjusted, respectively; a control 19D, which allows the user to manipulate a tone curve that relates input pixel values ​​to output pixel values; sliders 19E, 19F, and 19G, which may be assigned for specific processing; a timeline 19H, which indicates a clip in the content being processed and which may be used to select a frame in the content to be operated on (different clips may be, for example, represented by a plurality of sliders, such as a plurality of sliders); and a timeline 19H. Figure 219L; a preview image area 19J; and a process definition chart and control 19K that allows the user to select and modify the operation of a process in a sequence of processes that produce an output image signal from an input image signal. For example, the user can define a chain of process operations 19L by dragging, dropping, and connecting boxes that specify process operations. Figure 2 , chain 19L begins with box 19M representing input image data (e.g., digital content 17) and ends with box 19N representing output image data (e.g., graded digital content 17A). The illustrated example processing chain 19L includes processing boxes 19P, 19Q, and 19R. A processing box can be selected (e.g., by touching, pointing, etc.) to adjust its operation, for example, by setting available processing parameters.

[0178] Figure 2 The timeline 19H shown in FIG. 1 is an example for illustrative purposes. Each clip may include Figure 2 Many more frames than the ones shown.

[0179] The data processor 12 may be integrated with the color grading console 16, the reference monitor 18 and / or another device, or may be provided as a separate unit in data communication with the color grading console 16 and the reference monitor 18, or may be provided as a configured cloud computing resource.

[0180] Typical workflow for color grading

[0181] To achieve the desired rendering / artistic intent on screen, a colorist uses a grading workstation (e.g., color grading console 16) to modify the video signal transmitted to a reference display (e.g., reference monitor 18) by adjusting the color and luminosity of the content in question (e.g., digital content 17).

[0182] A typical task for a colorist is to adjust the color and brightness of a video content item, such as a movie. The video content consists of a large number of frames, each of which is a separate image.

[0183] Typical video content includes multiple "clips" (also called "shots" or "segments"). A clip is a group of temporally adjacent frames where no scene changes occur. A colorist typically performs color grading on each clip. For example, a colorist can use controls provided by user interface 16A of color grading console 16 to apply tone mapping to all pixels in a clip to increase or decrease the brightness of the image data and / or adjust the color balance of the clip.

[0184] Color grading tends to be an iterative process. Most of the time, multiple fine-tuning and iterations are required to achieve the desired result. Adjustments to pixel values ​​in one clip may solve problems in some frames of the clip but not others, or may solve problems in some frames of the clip and potentially create new problems in other frames of the clip. For example, at the beginning of grading, the image needs to be properly exposed for the current display before moving on to a more detailed grading process. This process is usually performed on the first frame of the clip until a satisfactory rendering is achieved. However, when the full clip is reviewed, other frames may still have problems due to the initial settings, so the same process has to be performed again, which can be quite time-consuming.

[0185] Furthermore, similar adjustments may need to be made to different frames depicting similar scenes (eg, similar or identical color grading may be applied to frames depicting the same or similar scenes).

[0186] Furthermore, the entire content project needs to be put together. Even if all clips have been adjusted in a satisfactory manner, further adjustments may be required to achieve the desired artistic effect for the entire content project. Therefore, color grading a content project can be very time-consuming.

[0187] When color grading has been completed, the content item can be reviewed in a quality control process. As part of quality control (QC), artistic and technical teams review the content to detect issues with the color grading as well as issues with other aspects of the content. If QC fails, the post-production process is repeated until full compliance. If the content passes the QC process (i.e., the color graded content is deemed acceptable), the color graded image content is delivered and / or stored as a reference master for distribution to end users.

[0188] FIG3 shows a typical prior art color grading apparatus, which includes a color grading workstation 1 and a reference monitor 2. A colorist operates the grading workstation 1 to modify content (not shown in FIG3 ) to achieve a desired effect. The workstation 1 outputs a video signal 3, which is rendered by the reference monitor 2. When the colorist is satisfied with the modified (graded) content, the colorist can control the color grading workstation 1 to output or transfer a reference master 4 that can be used to generate various distribution formats 5.

[0189] As described above, the inventors have discovered a number of problems with existing color grading tools and methods. Apparatus 10 may include features as described herein to address any, all, or any combination of these problems. For example, apparatus 10 may include features as described herein to address one or more of the following problems:

[0190] Reference monitor 18 may not faithfully represent the image data (e.g., due to cropping and / or squashing), such that the image data may not have the desired appearance (e.g., the appearance viewed on reference monitor 18) when viewed on a display different from reference monitor 18.

[0191] • Content may need to be color graded for viewing on a specific display that is "expensive" to access or use (e.g., a high-end cinema projector).

[0192] Content may need to be color graded for viewing on multiple types of displays. However, providing multiple different reference monitors requires a lot of space and expense.

[0193] Content may need to be color graded for viewing on a range of different display types, each with different characteristics and / or intended to be used in different ambient light conditions. However, this may require repeating the entire color grading process using different types of reference monitors, which is expensive and time consuming.

[0194] It is time consuming and requires considerable skill for a colorist to determine an efficient sequence for color grading, especially on larger content items such as a full movie, and

[0195] Identify and correct for temporal effects such as flicker.

[0196] It is inefficient when the quality control process detects problems that were not noticed during color grading.

[0197] Current color grading methods require multiple manual interventions by colorists, which makes color grading time-consuming and expensive.

[0198] The following sections describe techniques that may be included in the apparatus and methods described herein. These techniques may be applied individually, all together, or in any combination. Use of these techniques may facilitate more efficient and / or more accurate color grading.

[0199] Cutting / crushing

[0200] The apparatus 10 may include means to address the problem that light output by individual pixels displayed by the reference monitor 18 may have a brightness that is different from that specified by the image data provided to the reference monitor 18 .

[0201] "Clipping" refers to a situation where the brightness of a pixel provided by the display is less than the brightness specified by the image data.

[0202] Clipping may occur for each primary color produced by reference monitor 18 (e.g., for the red, green, and blue channels of reference monitor 18). For example, clipping may occur if the image data specifies a pixel intensity that is above the maximum intensity of reference monitor 18.

[0203] Undetected clipping or crushing can be a problem because a reference master comprising image data having a desired appearance when viewed on the reference monitor 18 may have a different (undesirable) appearance when viewed on a different display. Therefore, in case pixels are clipped or crushed by the reference monitor 18, cross-compatibility of the graded content with other monitors is not ensured. Indeed, visual elements or artifacts that have been tone mapped or clipped by the reference display may not necessarily be tone mapped or clipped on another display and therefore become visible. It is essential that the transmitted signal reflects what the creative team has verified on screen to achieve cross-compatibility.

[0204] Clipping of reference monitor 18 may occur via any one or any combination of a variety of mechanisms.

[0205] These mechanisms include:

[0206] each pixel of the reference monitor 18 has a maximum brightness (which may be different for each color channel of the pixel where clipping occurs), and the image data specifies a brightness that exceeds the maximum brightness;

[0207] The reference monitor 18 performs processing on the incoming image data including tone mapping and / or soft or hard clipping;

[0208] the reference monitor 18 has a power supply with a maximum output power level that is insufficient to drive all pixels at maximum intensity, and the image data defines an image that will require more power than the maximum output power level of the power supply;

[0209] The reference monitor 18 has thermal constraints that limit how much light can be output in a local area of ​​the display to an average level that is insufficient to drive all pixels in that area at maximum intensity, and the image data defines an image that would require more light in that area than the local maximum average luminosity;

[0210] The reference monitor 18 has a limited local contrast that is insufficient to render two adjacent pixels where, for example, a first pixel of the pixels requires a very high brightness and a second pixel requires a very low brightness;

[0211] The reference monitor 18 has a timing constraint which limits the ability of the reference monitor to change the brightness of a pixel quickly enough; and

[0212] • The reference monitor 18 includes a light redirecting component, and the incoming image data defines an image that exceeds the light budget of the light redirecting component.

[0213] For example, assume that reference monitor 18 can accurately display image pixel values ​​in the intensity range of A to B. If the image data includes pixel values ​​C, where C>B, then in the image displayed by reference monitor 18, pixel values ​​C may be effectively clipped to pixel values ​​that do not exceed B (which is the maximum value that can be displayed by reference monitor 18). If the image data of the reference master is subsequently displayed on a display capable of displaying pixel values ​​greater than B, the displayed image will not have the intended appearance.

[0214] "Crushing" occurs when a display provides a pixel with a brightness greater than that specified by the image data. For example, many displays have a "dark level," which is the minimum brightness a pixel can have when the display is operating. Where such a display is used to display an image that includes pixels having values ​​below the dark level, the pixels below the dark level are crushed.

[0215] As described above with respect to clipping, crushing may additionally or alternatively occur if the reference monitor 18 has limited local contrast that is insufficient to render two adjacent pixels, where, for example, a first of the pixels requires a very high brightness and a second requires a very low brightness.

[0216] In some embodiments, reference monitor 18 generates an amplitude signal (or an amplitude signal for each color channel) that can be directly interpreted to determine whether clipping or crushing is occurring. In some embodiments, the amplitude signal is obtained from reference monitor 18 and processed to provide visual feedback about the number and / or location of clipped pixels and / or crushed pixels. The amplitude signal can, for example, control the final stage of spatial amplitude modulation in reference monitor 18.

[0217] For example, the amplitude signal may be a number having values ​​that are (or when scaled are) in any of the following three ranges:

[0218] 1. The unclipped amplitude signal is in the range [0; 1], in which case the reference monitor 18 does not clip or crush;

[0219] 2. The unclipped amplitude signal is below 0, in which case the reference monitor 18 is crushed; and

[0220] 3. The unclipped amplitude signal is above 1, in which case the reference monitor 18 clips.

[0221] In some embodiments, the operation of reference monitor 18 is simulated by data processor 12 to produce a simulated unclipped amplitude signal, which can be applied as described above to identify clipped pixels and / or crushed pixels. The details of the simulation will depend on the design and construction of reference monitor 18. Described below are example features that various design simulations for reference monitor 18 may include.

[0222] Displays typically form images by driving a light modulating device that operates to produce a modulated light field. There are different types of light modulating devices. A light modulating device may, for example, selectively emit, block, or reflect light or modulate light by any combination of these. Example display architectures include the following architectures, where:

[0223] • A light modulation device modulates light provided by the fixed uniform illumination, the light modulation device being controlled by an amplitude signal to selectively block light on a pixel basis to produce an image defined by the image data.

[0224] • A light modulation device modulates light provided by a variable uniform illumination, the level of which may vary from time to time (e.g., for each frame or each clip), the light modulation device being controlled by an amplitude signal to selectively block light on a pixel basis to produce an image defined by the image data.

[0225] Dual modulation architectures, where a first light modulation device generates a spatially modulated light field that is further modulated by a second light modulation device. Dual modulation can enable control of backlighting on a local basis. Rendering an image on a dual modulation type display is typically produced by a combination of a backlight signal (typically a low resolution signal) used to control the light field provided by the first light modulation device and an amplitude signal used to control the second light modulation device to block additional light that is not needed to form the image. Dual modulation displays typically process image data to generate a backlight signal and an amplitude signal.

[0226] A direct-view architecture that emits light at the pixel level (OLED displays are an example).

[0227] Examples of display technologies where clipping and crushing can occur include:

[0228] All displays have a peak brightness, which is the maximum light intensity a pixel can reach.

[0229] If the input code value requires a luminosity above this peak brightness, clipping will occur.

[0230] All displays have a minimum dark level, which is the lowest amount of light the display can emit when operating. If the input code value requires a light level below this minimum dark level, crushing will occur.

[0231] Most displays (eg, except for direct view displays, for example) have a contrast limit, which is the ability to block light emitted by the backlight. If the local backlight illumination divided by the contrast is higher than the input code value, crushing will occur.

[0232] RGB displays (e.g., some OLED displays are RGB displays), which have the additional property that the peak brightness of white is higher than the peak brightness of the individual colors, so the peak brightness of each color is needed to identify which pixels are not implementable.

[0233] Displays with many individual light sources (e.g., OLED displays or displays with backlights with a large number of individually controllable light sources) may be limited by the maximum amount of light that can be produced over the entire screen area (which may be referred to as the "light budget" of the display). If the sum of all pixels is above the light budget (a fixed threshold), the display will not be able to produce the required light, so clipping will occur.

[0234] Dual modulation displays that use an algorithm to generate each image. The algorithm typically computes two signals, the backlight illumination (which is used to generate light) and the light attenuation (amplitude) signal (which is used to block the extra light at each pixel that is not needed to form the image). In such displays, crushing occurs when the amplitude signal is less than 0 (meaning that in order to achieve a correct image, more light needs to be subtracted than the display's amplitude modulator can attenuate). In such displays, clipping occurs when the amplitude signal is greater than 1 (meaning that the amplitude modulator would need to output more light for the pixel than was incident on the pixel, which is impossible).

[0235] In some embodiments, the apparatus 10 is configured to detect and indicate, on a pixel-by-pixel basis, the cropping and / or crushing applied to the image displayed by the reference monitor 18, and to provide visual feedback indicating to the user where the cropped pixels appear in the displayed image. The visual feedback may, for example, include a map displayed on the interface of the color grading console 16 showing how the cropped pixels are distributed in the image displayed by the reference monitor 18, and / or an overlay that may be selectively displayed by the reference monitor 18 highlighting the locations of the cropped pixels.

[0236] Visual feedback can provide separate crop maps for different color channels (e.g., red, green, and blue color channels). In some cases, two or more of the separate crop maps for different color channels can be superimposed on each other. The maps can be superimposed automatically or by an operator (e.g., a colorist).

[0237] In some embodiments, the positions of the cropped pixels and / or squashed pixels are recorded and / or passed as metadata for downstream processing. For example, such metadata can enable direct selection of cropped pixels and / or squashed pixels (e.g., using the metadata as an alpha mask to directly select cropped pixels and / or squashed pixels).

[0238] In some embodiments, apparatus 10 is configured to detect and indicate cropping applied to an image displayed by reference monitor 18 on a pixel-by-pixel basis, and to provide visual feedback indicating a number of cropped pixels to a user. The visual feedback may, for example, include a bar graph or meter or digital display indicating the number of cropped pixels in the image displayed by reference monitor 18. The bar graph or meter may, for example, be displayed on an interface of color grading console 16. In some embodiments, the bar graph includes a bar for each color channel (e.g., a red bar, a green bar, and a blue bar).

[0239] In some embodiments, the apparatus 10 is configured to identify the number and / or location of pixels that need to appropriately display more than a threshold amount of light. The apparatus 10 may include a user control operable to adjust the threshold.

[0240] In some embodiments, the number of cropped pixels in an image displayed by reference monitor 18 may be determined by simulating the operation of reference monitor 18 using a computational model of reference monitor 18. The simulation may take into account the manner in which reference monitor 18 handles pixel values ​​that are out of range or nearly out of range. For example, reference monitor 18 may resolve out of range pixel values ​​by:

[0241] • Hard Clipping: Signal code values ​​outside the capabilities of the display are limited to the closest code value within the range of the display.

[0242] Soft Clipping: (Using tone curves) Processing signal code values ​​not only outside the display capabilities but also at the edge of the display capabilities to fit the display capabilities.

[0243] Global tone mapping: Maps every pixel using the same tone map so that no or almost no pixels lie outside the capabilities of the display.

[0244] Local tone mapping: The decision to process a pixel depends on the context of each pixel. For example, the decision to tone map a pixel can be made based on its spatial neighborhood or temporal neighborhood, or based on the frequency characteristics of the pixel, etc. The resulting processing does not necessarily process all pixels in the same way or at all.

[0245] Figure 4A functional block diagram is schematically shown showing a possible configuration of the data processor 12 for analyzing cropping and / or crushing. Image data 21 is provided to the data processor 12 and the reference monitor 18. The image data 21 may be image data output by the color grading console 16, for example.

[0246] The image data 21 specifies the color and brightness for each pixel in the image using any suitable color coordinate system. In some embodiments, the image data 21 is red, green and blue data. In some embodiments, the image data 21 is uncompressed.

[0247] The image data 21 is input into a model 22 for simulating the operation of the reference monitor 18. Techniques for modeling the operation of displays are known and described, for example, in S. Forchhammer, J. Korhonen, C. Mantel, X. Shu, X. Wu, eds., HDR Display Characterization and Modeling, Chapter 13 - From Acquisition to Display and Applications of High Dynamic Range Video, Academic Press 2016, pp. 347-369; and Youngshin Kwak, Lindsay William MacDonald, Methods for Characterizing LCD Projection Displays, Photonics West 2001 - Electronic Imaging, San Jose, CA, USA, Proceedings Vol. 4294, Projection Displays VII; (2001) https: / / doi.org / 10.1117 / 12.420780 Model 22 may appropriately implement such techniques to model the operation of reference monitor 18 .

[0248] In the illustrated embodiment, model 22 includes data 23A including display performance values ​​including maximum light output (eg, maximum brightness) of pixels of reference monitor 18 (which may be measured, for example, in “nits” (ie, caderas per square meter)).

[0249] Block 23B compares the image data 21 to the display performance value 23A and outputs data 23C indicating any clipped pixels when the image data 21 is displayed on the reference monitor 18 .

[0250] For example, if the display performance data 23A indicates that the reference monitor 18 has a maximum brightness, the box 23 may mark any pixels whose brightness for any color channel specified by the image data 21 is greater than the maximum brightness value for that color channel specified by the display performance data 23 A. As an example, if the reference monitor 18 has a maximum brightness of approximately 300 nits for white light, the reference monitor 18 may have a maximum brightness of approximately 75 nits for the red channel, 210 nits for the green channel, and 15 nits for the blue channel.

[0251] The data 23C may include, for example, one or more of the following:

[0252] The number of cropped pixels;

[0253] The number of cropped pixels for each of the multiple color channels;

[0254] An array or other data structure indicating which pixels specified by the image data 21 are to be cropped.

[0255] Box 25 formats the data 23C for display as, for example, one or more of a digital display item 26A, a bar graph 26B, one or more maps 26C (i.e., a two-dimensional display item including markings showing the locations of the cropped pixels). In some embodiments, the model 22 generates a set of maps 26C, each map showing the locations of the cropped pixels of a corresponding color channel.

[0256] exist Figure 4 , model 22 also includes a module 27 for simulating the processing of image data 21 by reference monitor 18. Module 27 may, for example, include a portion 27A that simulates data processing including tone mapping, gamma correction, color space adjustment, clipping, or soft clipping of image data 21 by reference monitor 18. Module 27 may also include a portion 27B that simulates the operation of the hardware of reference monitor 18 (e.g., what is the brightness output by reference monitor 18 for a given pixel value / control signal output by portion 27A).

[0257] In some embodiments, module 27 includes or has access to display output data 27C for correlating pixel values ​​with pixel brightness of reference monitor 18. Data 27C may be obtained, for example, by measuring light output by pixels of reference monitor 18 for different inputs. Data 27C may be provided, for example, in the form of a lookup table or other suitable data structure.

[0258] Block 27D compares the simulated light output of reference monitor 18 to image data 21 and outputs data 27E indicating any clipped pixels of image data 21 when displayed on reference monitor 18. Data 27E may optionally be input to block 25. Block 25 may format data 27E for display as described elsewhere herein.

[0259] In some embodiments, reference monitor 18 is a dual modulation display (e.g., a local dimming display). Such a display typically includes a spatial amplitude modulator having controllable pixels and one or more light sources that deliver light for modulation by the spatial amplitude modulator. The dual modulation display includes some mechanism for controlling the amount of light incident on the pixels of the spatial amplitude modulator according to image data 21, typically causing more light to be incident on pixels for which image data 21 specifies a higher brightness and less light to be incident on pixels for which image data 21 specifies a lower brightness.

[0260] In the case where reference monitor 18 is a dual modulation display, module 27 may emulate one or both of the following algorithms: an algorithm performed by reference monitor 18 to control the amount of light incident on each pixel of the spatial amplitude monitor based on image data, and an algorithm performed by reference monitor 18 to control the pixels of the spatial amplitude monitor based on image data. Module 27 may also estimate the amount of light incident at each pixel of the spatial amplitude monitor. The results of these calculations may be combined to produce an estimate of the brightness of each pixel of the image displayed by reference monitor 18 for each color channel. The details of these calculations will depend on the specific design of reference monitor 18.

[0261] In some embodiments, reference monitor 18 is a light-steering display. Light steering is a technology that selectively deflects light from a light source to pixels of a display based on image data. The light-steering display may be implemented as a dual-modulation display.

[0262] Various mechanisms for light steering are possible. For example, light steering can be performed by controlling the pixels of a phase modulator to change the phase of light from a light source according to a phase pattern, such that the interaction of the phase-changed light causes the light to be concentrated (higher brightness) in certain areas of the image and reduced (lower brightness) in other areas of the image. For example, light steering technology is described in the following published PCT patent applications: WO2015 / 054797A1, entitled “Light Field Projector and Method Thereof”; WO2015 / 172236A1, entitled “Optimizing Drive Schemes for Multi-Projector Systems”; WO2015 / 184549A1, entitled “Efficient Dynamic High-Contrast Lens Action for Imaging, Illumination and Projection”; WO2016 / 015163A1, entitled “Numerical Method for Freeform Lens Construction: Area Parameterized Freeform Lens Construction”; WO2017 / 059537A1, entitled “Projection System and Method”; and WO2018 / 205036A1, entitled “High Brightness Projection System and Method”, and in the following papers: Gerwin Damberg, Ronan Boitard, Anders Ballestad, Wolfgang Heidrich, “Light-steering projection systems and properties for HDR displays,” SID2017 International Symposium, Vol. 48, No. 1, May 2017, pp. 87–90, https: / / doi.org / 10.1002 / sdtp.11574.

[0263] exist Figure 4 , module 27 includes a portion 27F for simulating the operation of the light steering system of reference monitor 18. Portion 27F may, for example, include the same processing for generating control signals for the light steering hardware of reference monitor 18, and passing the resulting control signals to a computational simulation of the operation of the light steering hardware to produce an estimated light field produced by the light steering hardware. The computational simulation may include separate simulations for different color channels.

[0264] For example, reference monitor 18 may include light steering hardware that includes a spatial phase modulator (e.g., a device that includes a 2D array of pixels, each pixel of the array being controllable to vary a phase shift applied to incident light). For example, the phase modulator may be provided by a liquid crystal on silicon (LCoS) phase modulator. Portion 27F may model the optical path of the light steering hardware including the phase modulator to generate an estimated light field.

[0265] The estimated light field may be used to estimate the brightness of pixels for each color channel in an image displayed by the reference monitor 18 .

[0266] In some embodiments, portion 27F includes a light steering light budget calculator. The light steering system is capable of providing very high brightness over a certain proportion of the image. However, the light steering system may not be able to deliver enough light to achieve the same very high brightness over a larger proportion of the image. The light steering light budget calculator can process the image data 21 (using any preprocessing) to estimate the required light output that the light steering system of the reference monitor 18 will need to deliver in order to accurately render the image specified by the image data 21.

[0267] Light steering simulation 27F may compare an estimated desired light output of the light steering system to a stored value 27G representing the light output that the light steering system is capable of providing, and may generate an output indicating whether the estimated desired light output exceeds the light output represented by value 27G.

[0268] In some embodiments, display simulation 22 provides an output that graphically and / or numerically compares the magnitude of the estimated desired light output to the magnitude of the light output represented by value 27G. This output may be displayed to a user of device 10.

[0269] In some embodiments, reference monitor 18 has a total light budget that is insufficient to drive all pixels at maximum intensity simultaneously. Depending on the architecture of reference monitor 18, the light budget may arise, for example, due to one or more of the following: the power supply of reference monitor 18 has a maximum output power level that is less than the power required to drive all pixels at maximum brightness; the reference monitor 18 has insufficient heat dissipation capabilities to allow operation above the light budget without overheating; or the light budget is intentionally limited to extend the life of components of reference monitor 18.

[0270] In case the reference monitor 18 has such a total light budget, the module 27 may include a portion 27J that processes the image data 21 to obtain an estimate of the total light output and / or power required to display the image defined by the image data 21. The portion 27J may compare the total light output with a maximum light budget 27K of the reference monitor 18 and / or compare the power estimate corresponding to the total light output with a stored value indicating a maximum output power level of the reference monitor 18. If the total light output exceeds the light budget and / or the power estimate equals or exceeds the maximum output power level, the simulation 22 may take this into account when estimating the light output of the pixels of the reference monitor 18.

[0271] Reference monitor 18 responds to the light budget exceeded condition by reducing the brightness of some or all pixels (e.g., because the power required to display the image defined by image data 21 is greater than the maximum output power level of reference monitor 18). Which pixels are affected and how much the brightness of those pixels is changed relative to the brightness specified by image data 21 will vary depending on the magnitude of the difference between the power required to display the image defined by image data 21 and the maximum output power level of reference monitor 18, and will also depend on the construction details of reference monitor 18.

[0272] In some embodiments, portion 27J includes a computational simulation that models how reference monitor 18 responds in such circumstances and outputs an estimate of the actual brightness of pixels displayed by reference monitor 18 rendering image data 21. The estimate of the actual brightness of the displayed pixels may be compared to image data 21 to identify clipping and / or crushing as described herein.

[0273] In some embodiments, display simulation 22 provides an output that graphically and / or numerically compares the magnitude of the estimated desired light output to the magnitude of the light output represented by value 27 K. The output may be displayed to a user of device 10 .

[0274] Any of the same methods applied above to detect and provide information about clipping can also be used to detect and provide information about crushing, but instead of comparing and determining that the brightness specified by the image data 21 is greater than the brightness being displayed by the reference monitor 18 (in terms of detecting clipping), a comparison is made to determine that the pixel brightness specified by the image data 21 is less than the pixel brightness being displayed by the reference monitor 18, thereby detecting crushing.

[0275] In some embodiments, module 27 is connected to receive as input data from a location internal to reference monitor 18, and reference monitor 18 is configured to provide this internal data to module 27. This may simplify the construction of module 27, and may be particularly convenient if reference module 27 is included in reference monitor 18. For example, if module 27 is able to access image data that has been modified by processing in reference monitor 18, then portion 27A may be unnecessary.

[0276] In some embodiments, the apparatus 10 is configured to do one or more of the following:

[0277] Providing a feedback loop to the color grading console 16 (eg, via metadata) to indicate how the reference monitor 18 rendered the pixels. The color grading console 16 may be configured to apply the feedback to the corrected image data 21 to reflect the image as rendered by the reference monitor 18.

[0278] • Creation and transfer of a reference master (optionally by modifying the image data 21 to reflect the image actually rendered on the reference modulator 18).

[0279] Provide metadata to the color grading console 16, such as: a map or mask (e.g., alpha mask) identifying problematic (e.g., crushed or clipped) pixels; metadata including input values ​​identified by the color grading console 16 for specialized actions / transformations. The map or mask may show the light distribution (e.g., the expenditure of light used relative to the light budget). The map or mask is preferably displayed without affecting the light budget.

[0280] Figure 5A , Figure 5B and Figure 5C Exemplary display items that may be provided by apparatus 10 for indicating cropping and / or crushing to a colorist are shown. Figure 5A is a map indicating the locations of clipped pixels (stars) and squashed pixels (triangles). In some embodiments, different symbols and / or colors are used to represent data corresponding to different color channels (e.g., a red asterisk indicates clipped red pixels, a green asterisk indicates clipped green pixels, and a blue asterisk indicates clipped blue pixels). In some embodiments, white symbols are used to represent clipped or squashed pixels in two or more color channels. Figure 5B is a bar graph indicating the number of clipped and squashed pixels in the image. In some embodiments, a bar graph is displayed for each color channel (e.g., a bar graph showing clipped or squashed red pixels, a bar graph showing clipped or squashed green pixels, and a bar graph showing clipped or squashed blue pixels). In some embodiments, the bar graphs corresponding to different color channels are superimposed on each other. Figure 5C is a digital display that indicates the number of clipped and squashed pixels in the image. In some embodiments, the digital display shows the number of clipped and / or squashed pixels for each color channel. Figure 5A , 5B Two or more of the exemplary data visualization forms in 5C and described herein may be displayed simultaneously. In some embodiments, device 10 includes controls that allow the colorist to select which information to display and / or how to display the information.

[0281] Figure 61 shows a possible workflow of the apparatus 10 as described herein. Signal 14A represents a modified version of the image data 21, which represents the actual image rendered by the reference monitor 18. Signal 14A can be used to provide a reference master 14 (which can be used to generate various distribution formats 9). Signal 29 conveys feedback information to the color grading console 16. The feedback information can be provided to the colorist on a display that is part of the console 16 or on another display. For example, signal 29 can carry information about cropping and / or crushing (e.g., can be as shown in FIG. 1 ). FIG. 5A to FIG. 5C ) or any other information that can be communicated for notifying a colorist as described herein.

[0282] In some embodiments, device 10 is configured to detect and identify to the user any pixels that can be made brighter without significantly affecting the amount of light used to display the image.

[0283] Realistic Rendering

[0284] As described above, for some content (e.g., a high-profile movie), two or more different reference masters (or "formats") are often generated. Each of the reference masters may be intended to be viewed in a particular type of viewing environment (e.g., a movie theater, a home living room, etc.) on a display with a particular performance level (e.g., with particular specifications for dark level, dynamic range, maximum pixel brightness, resolution, etc.). Each reference master may store image data in a suitable corresponding video format. For example, for a particular content item, each of the following may be created: a movie master, a home master, an HDR home master, a master for mobile viewing, and possibly other masters.

[0285] Different formats used to store image data in different reference masters may require different video signals because the standard way of transmitting pixel information is different (e.g., 8-bit versus 10-bit for film versus home; 4:2:0 chroma subsampling versus 4:4:4 chroma subsampling for high definition (HD) versus ultra high definition (UHD)) or because the assumed performance level of the display on which the content will be rendered is significantly different (e.g., HDR versus SDR).

[0286] The color grading of each reference master can be done as a separate project using a corresponding matching reference monitor. This approach has the benefit of optimizing the color grading of each reference master for the corresponding distribution channel. However, this approach has the problem that it is very labor intensive (expensive) and requires the use of multiple reference monitors (which are expensive, require space and introduce technical complexity), and also introduces the risk that different reference masters may not be consistent (i.e., the desired artistic effect may not be preserved between different reference masters for different decisions in the color grading of different reference masters).

[0287] Some color grading workflows use "trim passes" to generate some reference masters. In this approach, color grading is done first in a first format, and the colorist then proceeds to color grade other formats by modifying or "fine tuning" the color graded image data from the first format for display on a reference display corresponding to the other format. This approach can save time and can help achieve consistency between different formats. However, using trim passes can have disadvantages, including:

[0288] A linear approach is required. Reference masters for other formats cannot be prepared until the color grading of the first format is completed.

[0289] • Propagating changes from the first format to the other formats after the pruning step has been completed is complicated, any modifications to the first format will need to be revalidated and then propagated to all other formats.

[0290] Large amounts of data may be required to store all the different versions of the project and masters. Large amounts of stored data may make it difficult to track where the various formats are saved, complicating archiving, backup, and / or transfer of data, etc.

[0291] • The trim step version is usually somewhat limited by the creative choices made in color grading the first format.

[0292] • The trimmed step version usually cannot fully exploit the capabilities of the corresponding reference display.

[0293] In some embodiments, the apparatus 10 is configured to perform one or both of the following: primarily using a single reference monitor to facilitate color grading of multiple distribution channels (multiple formats); and color grading in a manner that allows multiple reference masters for different distribution channels to be based on the color grading performed for one distribution channel. Applying either or both of these features can save a lot of cost and time.

[0294] In some embodiments, data processor 12 is configured to predict the appearance of a content item when rendering image data of the content on a particular type of reference monitor (target reference monitor). For example, the prediction may be generated by computational simulation of a target reference monitor. For example, computational simulation may be performed as described elsewhere herein. The result of the computational simulation may be modified image data. The modified image data is displayed on reference monitor 18.

[0295] In the case where reference monitor 18 has capabilities at least equal to those of the target reference monitor, the appearance of an image displayed on reference monitor 18 using the modified image data will be very close to the appearance of an image displayed on the target reference monitor using the original image data.

[0296] In the event that reference monitor 18 has less capabilities than the target reference monitor, the appearance of an image displayed on reference monitor 18 using the modified image data may be very close to the appearance of an image displayed on the target reference monitor using the original image data, except for pixels having values ​​(e.g., luminance values) that are outside the capabilities of the reference monitor (i.e., outside the performance envelope of the reference monitor). In this case, the colorist may still view accurately displayed pixels while ignoring inaccurately displayed pixels (e.g., clipped pixels / squashed pixels). Inaccurately displayed pixels may be identified and / or displayed, for example, using a map, mask, or visualized in other ways as described elsewhere herein.

[0297] The data processor 12 may be configurable to perform simulations of several different target reference monitors. This allows the system 10 to always use the reference monitor 18 for at least preliminary color grading of several different reference masters. The reference monitor 18 may, for example, comprise a portable display.

[0298] For selected content, the corresponding target reference monitor can be used to perform final color grading and verification (QC) of each reference master.

[0299] Providing emulation of different reference monitors can allow the apparatus 10 to be used to complete, or at least complete a large portion of, color grading in two or more formats without requiring access to different reference monitors.

[0300] In some embodiments, the apparatus 10 is configured to perform automatic conversion of grades across multiple formats. This feature can be applied to initiate a single grading workflow for all reference masters.

[0301] Figure 7 is a combined flow chart and data flow diagram illustrating an automatic grade conversion method according to an example embodiment. In block S1 , color grading is performed for a base format. Block S1 processes image data 21 to generate base format data 71 .

[0302] The base format may, but does not necessarily, correspond to any reference master to be produced.Preferably, the base format is an HDR format, such that the HDR information in the image data color graded for the base format achieves the desired artistic intent.

[0303] Each of boxes S2A to boxes S2N corresponds to a format and respectively applies corresponding transformations 72A to 72N (collectively or generally referred to as transformations 72) to base format data 71 (i.e., image data that has been color graded for the base format) to produce corresponding versions of image data 73A to 73N that have been color graded for the corresponding format.

[0304] Steps S2A to S2N may be performed in any order or simultaneously. In addition, step S1 may be performed incrementally for different portions of the image data 21 (e.g., a colorist may color grade a base format one clip at a time). Steps S2A to S2N may be performed incrementally with step S1.

[0305] During or after color grading for the base format, the colorist can view the results of the transformations 72A to 72N on one or more reference monitors. Here, various options can be employed, these include:

[0306] Data processor 12 may perform a transformation from a base format to another format and cause reference monitor 18 to emulate a corresponding reference monitor as described herein. A colorist may view the resulting image on reference monitor 18. Apparatus 10 may include controls that a colorist may operate to switch between reference monitors of different formats / emulations.

[0307] The data processor 12 may perform a conversion from the base format to one or more other formats. The apparatus 10 may include a reference monitor corresponding to each of the one or more other formats. A colorist may view the image in each of the one or more other formats on the corresponding reference monitor.

[0308] The data processor 12 may output the image data in the base format to one or more reference monitors, and the one or more reference monitors may be configured to transform the image data from the base format to another format for viewing on the reference monitors. In some embodiments, a separate reference monitor is provided for each of two or more other formats, and each reference monitor is configured to transform the image data from the base format to the corresponding other format. In some embodiments, the reference monitors are configured to selectively transform the image data from the base format to one or two or more other formats.

[0309] The colorist may adjust the color grading in the base format and / or adjust the parameters of the transform 72 until the transformed image data in all other formats is acceptable.

[0310] In some embodiments, the apparatus 10 includes controls that allow the colorist to adjust the transformation from the base format to one or more other formats. Such controls can be used to fully exploit the capabilities of the reference display for any of the other formats while achieving the desired artistic effect.

[0311] For example, the transformation 72 may be defined by some or all of the following parameters:

[0312] Diffuse White: This is mapped to the brightness value "Diffuse White". "Diffuse White" means the brightness threshold at which an object appears to only reflect light, not emit it. In graphics applications, "Diffuse White" usually refers to a white point / luminosity where only the specular highlights have a higher brightness value. "Specular Highlights" means the bright spots of light that appear on shiny objects when illuminated. Specular highlights are usually, but not necessarily, small, while diffuse light can cover larger areas / surfaces.

[0313] Peak Brightness: The peak brightness of the target format.

[0314] Dark Level: The darkness level of the target format.

[0315] Change points in shadows and highlights: The thresholds where brightness values ​​gradually change to shadows and highlights.

[0316] Figure 8 81A, SDR Cinema - curve 81B, HDR Cinema - curve 81C and Dolby Digital TM Cinema - curve 81D) of a simple example transformation. In this example, linear scaling (diffuse white scaling 3 times) is used for ambient lighting compensation, and hard clipping is used to handle different display capabilities (dark levels and peak brightness). Transformations 81A to 81D do not include any gradients in shadows / highlights. Figure 8 An example gradient of highlights applied to transform 81B at 82 and an example gradient of shadows applied to transform 81B at 83 are shown. Figure 8 The transformations shown are examples only. The transformations may be linear or non-linear.

[0317] After color grading of the base format is completed, image data 73 for each of the other formats may be automatically generated using the corresponding transforms 72 and provided as reference masters.

[0318] The selected or used transforms may be communicated as (or stored in) metadata. The metadata may be communicated per frame, per clip, for the entire content item, etc. The metadata may be communicated for each of the different formats.

[0319] In some embodiments, the transformation 73 from the base format 71 to each of the other formats converts from the color space of the base format to the color space of the other formats, and also converts from the transfer function of the base format (i.e., the function that relates pixel values ​​or "code values" to brightness) to the transfer function of the other formats. Such transformations can be implemented, for example, as a 1D lookup table, a 3D lookup table, a deterministic equation, and / or a spatial processing algorithm.

[0320] Fig. 9A An example apparatus 10 is shown in which the reference monitor 18 functions natively as a reference monitor of format "A", and the processor 12 provides modified image data to allow the reference monitor 18 to emulate reference monitors of formats "B" and "C".

[0321] Fig. 9B Shows something like Fig. 9A An example apparatus 10 of an apparatus of the type shown in FIG. 1 is shown in FIG. 1 , except that separate reference monitors 18A, 18B and 18C are provided for viewing formats A, B and C, respectively.

[0322] Fig. 9C An example apparatus 10 is shown in which reference monitors 18A, 18B, and 18C each include or use together a conversion process to convert a base format video signal to other formats. A colorist can operate a color grading console 16 to grade the base format while viewing the resulting images in formats A, B, and C on reference monitors 18A, 18B, and 18C, respectively. A reference master can be created in the base format (and subsequently converted to other formats), and / or separate reference masters can be created for different formats.

[0323] Fig.9D Shown with Fig. 9C The apparatus of FIG. 1 is similar to that of FIG. 1 , except that the apparatus includes reference displays 18A, 18B, and 18C and reference display 18, which can be operated to emulate any or all of reference displays 18A, 18B, and 18C. Reference display 18 can be used to perform most of the color grading of formats A, B, and C. Reference displays 18A, 18B, and 18C can be introduced and used for final color grading of formats A, B, and C, respectively.

[0324] Time Tracking

[0325] Content items can contain thousands of individual frames. For example, at 24 frames per second, one hour of content includes 86,400 frames. At higher frame rates, more frames are required per hour of content.

[0326] Shaders need to focus not only on individual frames, but also on the experience created when those frames are played back. Temporal effects such as flickering can cause problems, even if each individual frame is "perfect" when considered in isolation.

[0327] In some embodiments, the apparatus 10 includes a data analyzer 50 (see, e.g. Figure 1 ), the data analyzer is configured to process the image data to collect data about each frame in a set of frames of the content item. The set of frames can be, for example, all frames in the content, a selected range of frames, a clip or all frames in a set of clips, etc.

[0328] The data collected for each frame is stored in a database 52. In the database 52, the collected data (which may include several types of data) is associated with the frame from which the data was collected. Individual frames may be identified, for example, by time code, frame index number, etc. The database 52 may also associate individual frames with scenes.

[0329] The collected data may include, for example, one or more of the following:

[0330] the minimum, maximum, mean, and median values ​​for each color channel (e.g., red, green, and blue), the minimum, maximum, mean, and median values ​​for brightness measurements (e.g., Y tristimulus values), and

[0331] Maximum RGB (e.g., the maximum value across all color channels);

[0332] The percentage or number of clipped pixels and squashed pixels;

[0333] The proportion of the light budget required (or a measure of the total light output) based on each color channel and / or the proportion of the light budget required (or a measure of the total light output) based on the total light budget (for displays with a light budget);

[0334] Flags or counters to indicate the presence of defects such as bad pixels, suspected flash defects, and / or suspected compression artifacts;

[0335] Histogram (number of pixels of a frame as a function of brightness or another color value);

[0336] Noise level;

[0337] ·etc.

[0338] If desired, data analyzer 50 and / or database 52 may be integrated with data processor 12 or other components of apparatus 10. In some embodiments, data analyzer 50 performs analysis of frames of content items according to its own schedule (eg, overnight).

[0339] The collected data may be processed for various purposes, including any one of the following, or any combination of two or more:

[0340] Detecting temporal effects, such as flicker, in one or more color channels;

[0341] Automatically identify frames that require priority attention from colorists;

[0342] Generate and display a graphical indicator (e.g., a range) that shows how a selected statistical data varies over time in a content item (e.g., follows the evolution of a given metric over time);

[0343] Calculate statistics on shots or scenes (e.g., mean, average, maximum, minimum of a value);

[0344] Compare statistics across frames;

[0345] Calculate and / or compare average statistics for different groups of frames;

[0346] Detect scene cuts;

[0347] • Associate clips that correspond to similar visual stimuli (e.g., outdoor scenes, head shots, etc.).

[0348] In some embodiments, database 52 also stores frame information and / or other information associated with frames generated by processing the collected data and associates them. This processing may be performed by data processor 12, a separate processor associated with database 52, and / or a processor of color grading console 16. The associated information may include, for example, one or more of the following:

[0349] · Marks the end of the clip;

[0350] · Flags indicating frames that should be given priority attention by the colorist;

[0351] Bookmarks set by users;

[0352] • The total amount of time that has been spent grading each frame / clip.

[0353] Database 52 may include a search function that enables a colorist to easily find frames that meet certain criteria (eg, frames with similar darkness levels, maximum brightness, average brightness, etc. in a particular clip).

[0354] The database 52 may include a sorting function that allows a set of frames (eg, a set of frames resulting from searching the database 52) to be sorted in a desired order.

[0355] Device 10 may include controls that invoke search functions and sort functions to identify frames that a colorist is particularly interested in. For example, device 10 may provide controls (e.g., via interface 16A of color grading console 16) that cause database 52 to do one or more of the following:

[0356] Identify for the colorist the N frames in each clip that have the largest number of cropped pixels (N can be 1, 2, 3, ...);

[0357] Identify for the colorist the N frames per clip with the maximum number of crushed pixels (N can be 1, 2, 3, ...);

[0358] Identify frames with suspected compression artifacts for colorists;

[0359] Identify frames with bad pixels for colorists;

[0360] Identify frames with suspected flash defects for colorists;

[0361] Identifying for the colorist frames that are similar to each other or to the currently selected frame in some way (e.g., they have similar darkness levels, similar average brightness, similar maximum brightness, etc.);

[0362] ·wait.

[0363] The device 10 may provide controls that allow a colorist to efficiently perform color grading functions. Such controls may be provided, for example, by the user interface 16A. For example, a control may allow a colorist to jump between frames that have been identified as having the greatest number of cropped pixels in various clips. The colorist may address the cropped pixels by applying tone mapping to reduce the brightness of pixels in frames in a clip, thereby reducing or eliminating clipping. Since tone mapping may be applied to all frames of a clip, addressing the cropping in the frames of the clip with the most cropped pixels may improve any cropping issues in all frames of the clip.

[0364] Another example control may be provided to allow the colorist to jump between and / or compare (by displaying simultaneously) frames that have similar characteristics (e.g., similar average brightness, similar contrast, etc.) The colorist may use this tool to check consistency between different clips that require a similar feel.

[0365] Another example control may be provided to allow the colorist to jump between frames that may be affected by certain types of defects (e.g., bad pixels, flicker defects, compression artifacts, etc.). In some embodiments, the control is configured with a selection tool that allows the colorist to select which defects to include. The colorist can use such a control to quickly find frames that need adjustment to correct various types of defects.

[0366] In some embodiments, the apparatus 10 is configured to display the metrics in a temporal context (e.g., displaying metrics for a series of adjacent frames and the current frame). This helps the colorist understand the evolution of the displayed metrics over time, as well as discover single frame errors. Such a display item may, for example, be provided as part of the user interface 16A.

[0367] In some embodiments, the device 10 is configured to display high-level metrics aggregated from multiple frames. Such high-level metrics may include (but are not limited to) average brightness in a shot, average contrast in a shot, maximum frame luminance over all or part of the image data, etc. These high-level metrics can help colorists detect complex problems in video signals that would otherwise be difficult to detect in real time. For example, a colorist using the device 10 can easily monitor the consistency of high-level metrics over a given time period, for example, by comparing several similar (e.g., same peak brightness or color gamut or dark level, etc.) frames to ensure consistency. For example, the high-level metrics can be calculated by the data analyzer 50 during analysis of frames of a content item or subsequently using data from the database 52.

[0368] In some embodiments, the apparatus 10 is configured to display a timeline that graphically illustrates the magnitude of one or more such high-level metrics for each frame or portion of the content item. By viewing the timeline, a colorist can quickly identify inconsistencies in one or more high-level metrics.

[0369] Figures 10A to 10D Examples of different ways that individual frame statistics and / or high level statistics may be indicated to a user are shown. Fig. 10A An example display item 101 is shown, where timelines 102A, 102B and 102C respectively show temporal changes in cropping, crushing and power. A marker 103 indicates the current frame. A time code display item 104 indicates the time code of the current frame.

[0370] Fig. 10B Another example display item 105 is shown that indicates a time change of a parameter (such as any parameter or metric described herein) by a change in color and / or brightness. Marking 106 indicates a clipping boundary. Different colors, symbols, etc. can be used to indicate the current value of a parameter or metric compared to the capabilities of the display. For example, green can indicate that the brightness value is within the capabilities of the display, yellow can indicate that the brightness value is close to the maximum value of the display capabilities, and red can indicate that the brightness value has exceeded the maximum brightness that the display can support and clipping is occurring.

[0371] Fig. 10CAnother example display item 108 in the form of a timeline is shown, wherein each frame is indicated by a symbol 109 (a dot in this example). Symbol 109 may include components for indicating the characteristics of the corresponding frame. For example, symbol component 109A (a vertical line in this example) may indicate that the frame has some quality control issues. Other symbol components may be displayed to indicate other characteristics of the frame. For example, symbol component 109B may indicate that the frame includes more than a threshold amount of cropping, and symbol component 109C may indicate that the frame should be prioritized for color grading. As color grading progresses, fewer and fewer frames will be marked with symbol components that indicate problems with the frame.

[0372] Fig. 10D Another example display item 110 is shown in the form of a graph 111 showing the time variation of a metric (e.g., any metric or parameter value associated with a frame). The display 110 may include indicia 112 indicating one or more thresholds. Shading, coloring, or highlighting 113 may be provided to emphasize frames where the metric exceeds a threshold.

[0373] For example, FIG. 10A to FIG. 10D Such display items may be provided, for example, on user interface 16A or included in another display in apparatus 10. The display items may be dynamically updated as the hierarchy progresses.

[0374] Grading Assistant

[0375] In some embodiments, the apparatus 10 is configured to automatically generate information and suggestions that can help optimize the use of the colorist's time. Examples include:

[0376] • A content item such as a full movie may be analysed as described above and the apparatus 10 may suggest (or automatically select) the frame in each clip from which the clip should be started when rating it.

[0377] For a clip or other group of frames, the device 10 may be configured to generate and present suggested adjustments to correct one or more problems. For example, the device 10 may be configured to:

[0378] o Generate a scaling factor or tone mapping parameters or tone mapping function that will keep the clipped pixels and / or squashed pixels within range;

[0379] o Generate one or more scaling factors or tone mapping parameters or tone mapping functions selected to make two or more selected clips more consistent with each other (e.g., in terms of characteristics such as average brightness, color balance, contrast, etc.).

[0380] • The device 10 may suggest key frames for classifying dynamic behaviors.

[0381] The device 10 may generate the information and / or suggestions, for example, by processing the data in the database 52. For example, the device 10 may be configured to search the database 52 to identify which frame in each clip appears to require the most significant tone mapping (e.g., by finding the frame in each clip that has the largest number of cropped pixels and / or squashed pixels). In this case, the device 10 may also be configured to suggest a tone mapping function to bring each of the cropped pixels and / or squashed pixels within a range. The colorist may choose to start with the suggested tone mapping function and fine-tune the suggested tone mapping function as needed to achieve artistic intent.

[0382] In some embodiments, the device 10 is configured to record the changes made by the colorist while color grading the content item. The record of the changes can be stored as metadata in the database 52 or another data storage. For example, the metadata can be applied to automate future color grading of the same content or other content. For example, if the colorist has color graded two formats for a particular content item (e.g., HDR home and SDR movie), the device 10 can predict how the colorist will color grade another format of the same content.

[0383] For example, where one already-graded format has a higher dynamic range than another already-graded format, the device 10 may determine the degree to which the colorist could have used the higher dynamic range for a particular clip (e.g., by comparing statistics from the database 52, such as the maximum luminance, minimum luminance, and average luminance of the already-graded formats). In some embodiments, the device 10 determines to scale the degree of dynamic range that the colorist used in the already-graded format.

[0384] For example, if the statistics match (the colorist has not yet exploited the higher dynamic range available in either of the formats), device 10 may offer to color grade the corresponding clip for an additional HDR format that does not fully exploit the dynamic range of the additional HDR format.

[0385] As another example, if the analyzed statistics show that a grading in a previously graded higher dynamic range format has a significantly higher maximum brightness and / or a significantly higher dynamic range than a previously graded lower dynamic range format, device 10 may propose color grading the corresponding clip for an additional HDR format that does not fully utilize the dynamic range of the additional HDR format.

[0386] As another example, changes made by a colorist when color grading a content item from one format to another format may be recorded (e.g., as metadata). The recorded changes may, for example, include a transform or set of transforms applied to achieve the graded conversion from one format to another, including any adjustments to the transforms that may have been made by the colorist. When the same graded conversion is to be applied to a later content item, the recorded changes may be used by the device 10 to propose adjustments and / or a transform or transforms for the graded conversion of the later content.

[0387] Real-time quality control

[0388] QC involves checking the content against a range of questions. These can include questions such as:

[0389] Shooting issues: e.g. objects / people that shouldn’t be in the camera’s field of view (e.g. microphones), flickering, black borders around the picture;

[0390] Created / perceived artifacts: e.g. VFX inconsistencies, motion judder, wrong transitions;

[0391] Editorial issues: such as subtitles overlapping important narrative tools or being invisible due to the background;

[0392] Video signal issues: For example, cropping / crushing of content, out-of-gamut colors, bad pixels, etc.

[0393] The later in the production process a QC problem is detected, the more likely it is that correcting that QC problem will result in extensive rework. Video signal problems are particularly likely to be detected late because such problems may not be visible when rendering the content on one display, but may be apparent when rendering the same content on another display. Similarly, video signal problems may exist in some formats but not in others.

[0394] In some embodiments, the device 10 is configured to automatically review content for selected QC questions. The device 10 can analyze each frame for a series of selected QC questions, and can communicate these questions in any of a variety of ways. For example, the QC analysis can be presented to the colorist by one or more of the following:

[0395] Direct visual feedback to indicate frames with QC issues;

[0396] · Printed report describing the QC problem and the time code when it occurred;

[0397] Metadata files that enable importing markers indicating QC issues into grading software;

[0398] Cross-grade: Indicates that a grade different from the grade being audited may be problematic and should be reviewed to ensure it is not a problem;

[0399] • Once the content is encoded+decoded using the desired distribution codec, compress the image in real-time to assess if QC issues will occur.

[0400] Some QC issues associated with video signals (e.g., bad pixels, crushing, clipping, exceeding light budget, etc.) can be detected as described elsewhere herein. In some embodiments, the apparatus 10 includes a feature recognition module 54 (e.g., a trained machine learning system) (e.g., see Figure 1 ), feature recognition module 54 is trained to recognize objects or other features that may unexpectedly appear in a frame (e.g., microphones, cameras, filmmaking lights, etc.). Feature recognition module 54 may additionally or alternatively be trained to detect other problems with frames, such as sparkle defects, flare, and other identifiable problems that should be considered part of QC. Frames identified by feature recognition module 54 as including features of concern may be marked in database 52. Apparatus 10 may include controls that allow a colorist to review frames that have been marked by feature recognition module 54.

[0401] Various aspects of the technology described herein may also be provided in the form of a program product. A program product may include any non-transitory medium carrying a set of computer-readable, computer-executable instructions that, when executed by a data processor, cause the data processor to perform a method as described herein. A program product according to the present invention may be in any of a variety of forms. A program product may include, for example, non-transitory media, such as: magnetic data storage media including floppy disks, hard disk drives; optical data storage media including CD ROMs, DVDs; electronic data storage media including ROMs, flash RAMs, EPROMs, hard-wired or pre-programmed chips (e.g., EEPROM semiconductor chips); nanotechnology memories, etc. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0402] In some embodiments, the present invention may be implemented in software. For greater clarity, "software" includes any instructions executed on a processor, and may include (but not limited to) firmware, resident software, microcode, code for configuring configurable logic circuits, applications, apps, etc. As known to those skilled in the art, processing hardware and software may be centralized or distributed (or a combination thereof) in whole or in part. For example, software and other modules may be accessed via local memory, via a network, via a browser or other application in a distributed computing context, or via other devices suitable for the above purposes.

[0403] explain

[0404] "HDR" stands for High Dynamic Range. HDR video typically has 10 stops or more of dynamic range.

[0405] “SDR” means Standard Dynamic Range. SDR video typically has a dynamic range of about 6 stops (for 8-bit data) to about 10 stops (for 10-bit data).

[0406] "Format" means the version of the content designated for a particular distribution. For example, HDR Home, SDR Home, SDR Cinema (traditional cinema), Extended Dynamic Range (EDR) Cinema (e.g. Dolby Vision), TM (DVis)), HDR Cinema. Depending on the context, a format can also mean a specific way of storing or representing data.

[0407] "Quality Control" or "QC" means the process used to verify the compliance of content before delivering it.

[0408] "Reference Master" means image data in a delivery format that can be used to create a distribution master.

[0409] "Bit depth" refers to the number of bits that the pixel value is quantized to. For example, 8 bits are used for SDR families, 10 bits are used for UHD SDR families or HDR families, 12 bits are used for many film formats, and 16 bits can be used for reference masters.

[0410] "Chroma sampling" (or "chroma subsampling") is the number of samples used to describe a 2×2 pixel matrix. For example, 4:4:4 means 4 samples for each of the three channels. 4:2:2 means 4 samples for the first channel, 2 samples for the other 2 channels upstream, and 2 samples for the other 2 channels downstream, and 4:2:0 means 4 samples for the first channel, 2 samples for the other 2 channels upstream, and 0 samples for the other 2 channels downstream.

[0411] "Metadata" is information other than the image content. Metadata can be used for various purposes.

[0412] A "distribution master" is a version of content formatted for distribution in a specific distribution channel. A distribution master may have specific characteristics, such as color representation, bit depth, chroma sampling, data compression, bandwidth, etc. A distribution master may be generated by processing a corresponding reference master.

[0413] A "clip" / "shot" / "episode" is a set of temporally adjacent frames where no scene changes occur. The camera may pan / zoom / move during a clip.

[0414] "Keyframe" means a frame within a clip that is used to define a time-varying process. For example, a colorist may specify that a particular parameter should change from a first value at a first keyframe to a second value at a second keyframe. The associated software may then automatically interpolate the parameter value for each frame between the two keyframes.

[0415] Unless the context clearly requires otherwise, in the description and claims:

[0416] ·“comprise”, “comprising”, etc. should be interpreted as inclusive rather than exclusive or exhaustive; that is, as “including but not limited to”;

[0417] "Connected", "coupled" or any variation thereof means any connection or coupling between two or more elements, whether direct or indirect; the coupling or connection between elements may be physical, logical or a combination thereof;

[0418] When used to describe this specification, "herein," "above," "below," and words of similar meaning will refer to this specification as a whole rather than to any particular part of this specification;

[0419] · With respect to a list of two or more items, "or" includes all of the following interpretations of the word: any one of the items in the list, all of the items in the list, and any combination of the items in the list;

[0420] The singular forms "a", "an" and "the" also include the meaning of any appropriate plural forms. Unless otherwise specified, these terms ("a", "an" and "the")

[0421] means one or more;

[0422] "And / or" is used to indicate that one or both of the stated situations may occur, for example, A and / or B includes (A and B) and (A or B);

[0423] When applied to numerical values, “approximately” means ±10% of the value;

[0424] Where a feature is described as “optional” or “optionally” present or as present “in some embodiments”, it is intended that the disclosure covers embodiments in which the feature is present and other embodiments in which the feature is not necessarily present and other embodiments in which the feature is not present. Furthermore, where any combination of features is described in the application, this statement is intended to serve as antecedent basis for the use of exclusive terminology such as “solely”, “only”, etc., in relation to the combination of features, and the use of “negative” limitations to exclude the presence of other features; and

[0425] “First” and “second” are used for descriptive purposes and should not be understood as indicating or implying relative importance or indicating the quantity of the technical features indicated.

[0426] Words indicating directions, such as "vertical," "lateral," "horizontal," "upward," "downward," "forward," "backward," "inward," "outward," "left," "right," "front," "backward," "top," "bottom," "below," "above," "below," and the like, used in this specification and any appended claims (if any) depend on the particular orientation of the device being described and shown. The subject matter described herein can assume a variety of alternative orientations. Therefore, these directional terms are not strictly defined and should not be narrowly interpreted.

[0427] Where a range of values ​​is stated, the stated range includes all subranges of that range. It is intended that the statement of a range supports values ​​at the endpoints of the range and any intermediate values ​​of one-tenth of the unit to the lower limit of the range, as well as any subranges or groups of subranges of that range, unless the context clearly dictates otherwise or specifically excludes any portion of the stated range. Where a stated range includes one or both endpoints of the range, ranges excluding one or both of those included endpoints are also included in the invention.

[0428] Certain numerical values ​​described herein are preceded by "about". In this context, "about" provides literal support for the exact numerical value that follows it, the exact numerical value ±5%, and all other numerical values ​​that are close to or approximately equal to that numerical value. Unless otherwise indicated, a particular numerical value is included in the "about" specifically recited numerical value, where, in the context in which the specifically recited numerical value is presented, the particular numerical value provides a substantially equivalent numerical value to the specifically recited numerical value. For example, a statement that something has a numerical value of "about 10" would be interpreted as:

[0429] In some embodiments, the value is 10;

[0430] In some embodiments, the numerical value is in the range of 9.5 to 10.5; and if, based on the context, a person of ordinary skill in the art would understand that the value within the particular range is substantially equal to 10, because the value having the range would be understood to provide substantially the same result as the value 10, then "about 10" also includes:

[0431] In some embodiments, the value is in the range of C to D, where C and D are the lower and upper limits, respectively, of the range encompassing all those values ​​that provide substantially equal to the value 10.

[0432] Where reference is made herein to a component (e.g., software module, processor, assembly, device, circuit, etc.), unless otherwise specified, reference to the component (including reference to "means") should be interpreted to include any component that performs the function of the described component as an equivalent of the component (i.e., functionally equivalent), including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated exemplary embodiments of the invention.

[0433] For the purpose of illustration, specific examples of systems, methods, and devices have been described herein. These are examples only. The techniques provided herein may be applied to systems other than the example systems described above. Many changes, modifications, additions, omissions, and substitutions are possible in the practice of the present invention. The present invention includes variations of the described embodiments that are obvious to those skilled in the art, including variations obtained by: replacing features, elements, and / or actions with equivalent features, elements, and / or actions; mixing and matching features, elements, and / or actions from different embodiments; combining features, elements, and / or actions from embodiments as described herein with features, elements, and / or actions of other technologies; and / or omitting to combine features, elements, and / or actions from the described embodiments.

[0434] As will be apparent to those skilled in the art upon reading this disclosure, each of the various embodiments described and illustrated herein has discrete components and features that can be readily separated or combined with the features of any other described embodiment without departing from the scope of the invention.

[0435] Any aspect described above with respect to an apparatus may also be applied to a method, and vice versa.

[0436] Any of the methods described can be performed in the order of the events described or in any other order that is logically possible. For example, although the processes or blocks are presented in a given order, alternative examples can perform routines with steps or use systems with blocks in different orders, and some processes or blocks can be deleted, moved, added, subdivided, combined and / or modified to provide alternatives or sub-combinations. Each of these processes or blocks can be implemented in a variety of different ways. Moreover, although processes or blocks are sometimes shown as being performed in sequence, these processes or blocks can alternatively be performed in parallel, simultaneously or at different times.

[0437] Various features are described herein as being present in "some embodiments". Such features are not mandatory and may not be present in all embodiments. Embodiments of the present invention may include zero, any one, or any combination of two or more of such features. Even if such features are shown in different figures and / or described in different chapters or paragraphs, the present disclosure also contemplates all possible combinations of such features. This is limited only to the extent that some of such features are incompatible with other features of such features in the sense that it will be impossible for a person of ordinary skill in the art to construct an actual embodiment combining such incompatible features. Therefore, the description that "some embodiments" have feature A and "some embodiments" have feature B should be interpreted as a clear indication that the inventor also contemplates an embodiment combining feature A and feature B (unless the description states otherwise or feature A and feature B are not compatible at all). This is true even if feature A and feature B are shown in different figures and / or mentioned in different paragraphs, chapters or sentences.

[0438] It is therefore intended that the following appended claims and the claims introduced below be interpreted as including all such modifications, permutations, additions, omissions and sub-combinations that can be reasonably inferred. The scope of the claims should not be limited by the preferred embodiments set forth in the examples, but should be given the broadest interpretation consistent with the overall specification.

Claims

1. A device comprising: an input system operable to receive image data for display on a reference monitor; a data processor configured to process image data received at the first input to identify differences between an image specified by the image data and a rendering of the image by the reference monitor and to generate difference data representative of the differences; An output system is operable to present the difference data.

2. An apparatus according to claim 1 (or any other claim herein), wherein the input system is operable to receive an amplitude signal from the reference monitor, and the data processor is configured to process the amplitude signal to identify cropped and / or squashed pixels of the image data in the rendering of the image on the reference monitor.

3. An apparatus according to claim 1 (or any other claim herein), wherein the data processor is configured to calculate an analog amplitude signal for the reference monitor and process the analog amplitude signal to identify clipped and / or squashed pixels of the image data in the rendering of the image by the reference monitor.

4. An apparatus according to claim 1 (or any other claim herein), wherein the data processor is configured to compare the image data with performance characteristics of the reference display that define a performance envelope, and to determine the difference by identifying pixels in the image data having pixel values ​​outside the performance envelope of the reference monitor.

5. An apparatus according to claim 1 (or any other claim herein), wherein the data processor is configured to simulate operation of the reference monitor to render the image data to produce simulated rendered image data, and to identify the difference by comparing the image data with the simulated rendered image data.

6. The apparatus of claim 5 (or any other claim herein), wherein in simulating the operation of the reference monitor, the processor is configured to simulate one or more or all of the following: the maximum brightness of each pixel of the reference monitor; The dark level of each pixel of the reference monitor; the processing of the image data by the reference monitor; · operation of the power supply of the reference monitor; Thermal management of the reference monitor; the local contrast of the reference monitor; The brightness change rate of the reference monitor; - Light redirecting components of the reference monitor.

7. The apparatus of claim 5 (or any other claim herein), wherein the difference comprises clipping and / or crushing by the reference monitor.

8. An apparatus according to any one of claims 5 to 7 (or any other claim herein), wherein the apparatus is operable to output the simulation rendering data.

9. An apparatus as claimed in any one of claims 5 to 8 (or any other claim herein), wherein the apparatus is operable to store the simulation rendering data.

10. An apparatus as claimed in any preceding claim (or any other claim herein), wherein the apparatus is configured to display the difference data.

11. An apparatus according to claim 10 (or any other claim herein), wherein the apparatus is configured to display the difference data as an image indicating locations of pixels in the rendering of the image by the reference monitor that differ from the image specified by the image data.

12. An apparatus according to claim 10 or 11 (or any other claim herein), wherein the apparatus is configured to display the difference data as a number and / or a graphical indicator showing the number of pixels in the rendering of the image on the reference monitor that differ from the image specified by the image data.

13. The apparatus of claim 12 (or any other claim herein), wherein the indicator specifies an amount of clipped pixels relative to a clipping amount threshold.

14. The apparatus of claim 12 or 13 (or any other claim herein), wherein the indicator specifies a number of the cropped pixels for which cropping exceeds a cropping magnitude threshold.

15. The apparatus of any preceding claim (or any other claim herein), wherein the apparatus is integrated with the reference monitor.

16. An apparatus as claimed in any preceding claim (or any other claim herein), wherein the apparatus is integrated with a colour grading console.

17. A color grading suite comprising an apparatus according to any preceding claim (or any other claim herein) in combination with said reference monitor and a color grading console, wherein said color grading console is connected to provide said image data to an input of said reference monitor and to said input system of said apparatus.

18. The color grading suite of claim 17 (or any other claim herein), wherein the color grading console is configured to display a representation of the difference data on a user interface of the color grading console.

19. An apparatus according to any one of claims 1 to 18 (or any other claim herein), wherein the data processor is configured to identify, for each color channel of a plurality of color channels, a difference between the image specified by the image data and the rendering of the image by the reference monitor.

20. An apparatus according to any one of the preceding claims (or any other claim herein), wherein the data processor is configured to, simultaneously or at different times: simulating operation of one or more other monitors different from the reference monitor to render the image data and generating simulated rendered image data for each of the one or more other monitors; and The simulated rendered image data for one of the one or more other monitors is output for display on the reference monitor.

21. The apparatus of claim 20 (or any other claim herein), wherein in simulating the operation of one of the one or more other monitors, the processor is configured to simulate one or more or all of the following: the maximum brightness of each pixel of the other monitor; the dark level of each pixel of the other monitor; · the processing of the image data by the other monitors; · Operation of the power supply of said other monitors; Thermal management of said other monitors; the local contrast of the other monitor; The brightness change rate of the other monitors; - Light redirecting means for said other monitors.

22. An apparatus according to any preceding claim (or any other claim herein), comprising: A data analyzer configured to process the image data to collect data about each frame in a set of frames defined by the image data, and a database connected to store the collected data.

23. An apparatus according to claim 22 (or any other claim herein) wherein the items of collected data are associated with frames from which the items of collected data were respectively collected.

24. An apparatus according to claim 22 or 23 (or any other claim herein) wherein the items of collected data are associated in the database with scenes or clips of the image data.

25. An apparatus according to any one of claims 22 to 24 (or any other claim herein), wherein the collecting data comprises one or more of: The minimum, maximum, average and median values ​​for each color channel; Minimum, maximum, average and median brightness measurements; ●Maximum red, green and blue; The percentage or number of clipped pixels and squashed pixels; Based on the proportion of the light budget required for each color channel; The proportion of the required light budget based on the total light budget; A mark or counter indicating the presence of a defect; Image histogram; as well as Noise level.

26. An apparatus according to any one of claims 22 to 26 (or any other claim herein), wherein the data analyser is integrated with the data processor.

27. An apparatus according to any one of claims 22 to 26 (or any other claim herein), wherein the apparatus is configured to do any one, or any combination of two or more of the following: Detecting temporal effects, such as flicker, in one or more color channels; Automatically identify frames that require priority attention from colorists; generating and displaying a graphical indicator showing how the selected statistical data varies over time in content corresponding to the image data; Calculate statistics on clips, shots or scenes; Compare statistics across frames; Calculate and / or compare average statistics for different groups of frames; Detect individual clips; · Associations correspond to clips of similar visual stimuli.

28. The apparatus of claim 27 (or any other claim herein), wherein the apparatus is further configured to present a display item indicating the time effect.

29. An apparatus according to any one of claims 22 to 28 (or any other claim herein), wherein the apparatus is configured to store in the database tags indicating one or more types of tagged frames, and the apparatus provides controls operable to jump to tagged frames of a selected type of the one or more types.

30. The apparatus of claim 29 (or any other claim herein), wherein the markings include priority markings that indicate frames that should receive priority attention from a colorist.

31. An apparatus according to claim 30 (or any other claim herein), wherein the priority marking indicates the frame that has been identified as having the largest number of cropped pixels in each clip.

32. An apparatus according to claim 30 (or any other claim herein), wherein the priority tag indicates a frame that has been identified as having a maximum number of cropped pixels among each clip whose magnitude exceeds a threshold clip magnitude.

33. An apparatus according to any one of claims 22 to 32 (or any other claim herein), wherein the database includes a sorting function that allows a set of frames that are the result of a search of the database to be sorted in a desired order.

34. An apparatus according to any one of claims 22 to 33 (or any other claim herein), comprising controls that call the search function and the sort function to do one or more of the following: Identify the N frames in each clip that have the largest number of cropped pixels (N can be 1, 2, 3, ...); Identify the N frames in each clip that have the largest number of crushed pixels (N can be 1, 2, 3, ...); Identify frames with suspected compression artifacts; Identify frames with bad pixels; Identify frames with suspected flash defects; and Identify frames that are similar to each other or to the currently selected frame.

35. An apparatus as claimed in any one of claims 22 to 34 (or any other claim herein), comprising a control operable by a user to jump between frames in respective clips that have been identified as having the greatest number of cropped pixels.

36. An apparatus according to any one of claims 22 to 35 (or any other claim herein), comprising controls operable by a user to jump between and / or compare frames of the image data having similar characteristics.

37. An apparatus as claimed in any one of claims 22 to 36 (or any other claim herein), comprising a control operable by a user to jump between frames comprising the image data that is defective.

38. The apparatus of claim 37 (or any other claim herein), wherein the defects comprise one or more of bad pixels, flash defects, and compression artifacts.

39. An apparatus according to any one of claims 22 to 38 (or any other claim herein), wherein the apparatus is configured to display metrics for a current frame of the image data and a series of other frames adjacent to the current frame.

40. An apparatus as claimed in any one of claims 22 to 39 (or any other claim herein), wherein the apparatus is configured to calculate and display high level metrics aggregated from multiple frames.

41. An apparatus according to claim 40 (or any other claim herein), wherein the apparatus is configured to display a timeline graphically illustrating the magnitude of one or more of the high-level metrics.

42. An apparatus according to any of the preceding claims (or any other claim herein), wherein the apparatus is configured to suggest or automatically select frames in each of a plurality of clips in the image data for initial grading of the respective clips.

43. An apparatus according to any of the preceding claims (or any other claim herein), wherein the apparatus is configured to generate a scaling factor or tone mapping parameter or tone mapping function that will bring the cropped pixels and / or crushed pixels in a frame within a range.

44. An apparatus according to any of the preceding claims (or any other claim herein), wherein the apparatus is configured to generate one or more scaling factors or tone mapping parameters or tone mapping functions selected to make two or more selected clips more consistent with each other.

45. An apparatus as claimed in any preceding claim (or any other claim herein), wherein the apparatus is configured to suggest key frames for grading dynamic behaviour of the image data.

46. ​​An apparatus according to any of the preceding claims (or any other claim herein), wherein the apparatus is configured to automatically ask one or more QC questions for each of a plurality of frames defined by the image data, and to generate and present a QC analysis.

47. An apparatus according to any of the preceding claims (or any other claim herein), comprising a feature recognition module, the feature recognition module comprising a trained machine learning system trained to recognize film production devices in frames of the image data.

48. An apparatus comprising: an input system operable to receive image data for display on a reference monitor; a data processor configured to simulate operation of one or more other monitors different from the reference monitor to render the image data and to generate simulated rendered image data for each of the one or more other monitors; and outputting the simulated rendered image data for one of the one or more other monitors for display on the reference monitor.

49. An apparatus according to claim 48 (or any other claim herein) wherein the data processor is configured to emulate operation of a plurality of the other monitors.

50. The apparatus of claim 49 (or any other claim herein), wherein in simulating operation of a monitor of the one or more other monitors, the processor is configured to simulate one or more or all of the following: the maximum brightness of each pixel of the other monitor; the dark level of each pixel of the other monitor; · the processing of the image data by the other monitors; · Operation of the power supply of said other monitors; Thermal management of said other monitors; the local contrast of the other monitor; The brightness change rate of the other monitors; - Light redirecting means for said other monitors.

51. An apparatus comprising: an input system operable to receive image data for display on a reference monitor; a data processor configured to process image data received at the first input; a data analyzer configured to process the image data to collect data about each frame in a set of frames defined by the image data; a database connected to store collected data and configured to associate items of said collected data with corresponding ones of said frames of said image data; Output system, which is operational.

52. An apparatus according to claim 51 (or any other claim herein) wherein the items of collected data are associated in the database with scenes or clips of the image data.

53. An apparatus according to any one of claims 51 to 52 (or any other claim herein), wherein the collecting data comprises one or more of: The minimum, maximum, average and median values ​​for each color channel; Minimum, maximum, average and median brightness measurements; Maximum red, green and blue; The percentage or number of clipped pixels and squashed pixels; Based on the proportion of the light budget required for each color channel; The proportion of the required light budget based on the total light budget; · a mark or counter to indicate the presence of a defect; Image histogram; as well as Noise level.

54. An apparatus according to any one of claims 51 to 53, wherein the data analyser is integrated with the data processor.

55. An apparatus according to any one of claims 51 to 54 (or any other claim herein), wherein the apparatus is configured to do any one, or any combination of two or more of the following: Detecting temporal effects, such as flicker, in one or more color channels; Automatically identify frames that require priority attention from colorists; generating and displaying a graphical indicator showing how the selected statistical data varies over time in content corresponding to the image data; Calculate statistics on clips, shots or scenes; Compare statistics across frames; Calculate and / or compare average statistics for different groups of frames; Detect individual clips; · Associations correspond to clips of similar visual stimuli.

56. An apparatus according to claim 55 (or any other claim herein), wherein the apparatus is further configured to present a display item indicating the time effect.

57. An apparatus according to any one of claims 51 to 56 (or any other claim herein), wherein the apparatus is configured to store in the database tags indicating one or more types of tagged frames, and the apparatus provides controls operable to jump to tagged frames of a selected type among the one or more types.

58. The apparatus of claim 57 (or any other claim herein), wherein the markings include priority markings that indicate frames that should receive priority attention from a colorist.

59. An apparatus as described in claim 58 (or any other claim herein), wherein the priority marking indicates the frame that has been identified as having the largest number of cropped pixels in each clip.

60. The apparatus according to any one of claims 51 to 59, wherein the database comprises a sorting function allowing a set of frames resulting from the search of the database to be sorted in a desired order.

61. An apparatus according to any one of claims 51 to 60 (or any other claim herein), comprising controls for invoking the search function and the sort function to do one or more of the following: Identify the N frames in each clip that have the largest number of cropped pixels (N can be 1, 2, 3, ...); Identify the N frames in each clip that have the largest number of crushed pixels (N can be 1, 2, 3, ...); Identify frames with suspected compression artifacts; Identify frames with bad pixels; Identify frames with suspected flash defects; and Identify frames that are similar to each other or to the currently selected frame.

62. An apparatus according to any one of claims 51 to 61 (or any other claim herein) comprising a control operable by a user to jump between frames in respective clips that have been identified as having a maximum number of cropped pixels.

63. An apparatus according to any one of claims 51 to 62 (or any other claim herein), comprising controls operable by a user to jump between frames of the image data having similar characteristics and / or compare frames of the image data having similar characteristics.

64. An apparatus as claimed in any one of claims 51 to 63 (or any other claim herein) comprising a control operable by a user to jump between frames comprising defective image data.

65. The apparatus of claim 64 (or any other claim herein), wherein the defects comprise one or more of bad pixels, flicker defects, and compression artifacts.

66. An apparatus according to any one of claims 51 to 65 (or any other claim herein), wherein the apparatus is configured to display metrics for a current frame of the image data and a series of other frames adjacent to the current frame.

67. An apparatus as claimed in any one of claims 51 to 66 (or any other claim herein), wherein the apparatus is configured to calculate and display high level metrics aggregated from multiple frames.

68. An apparatus according to claim 67 (or any other claim herein), wherein the apparatus is configured to display a timeline graphically illustrating the magnitude of one or more of the high-level metrics.

69. An apparatus comprising: an input system operable to receive image data for display on a reference monitor; a data processor configured to process frames of image data received at said first input to suggest or automatically select frames in each of a plurality of clips of said image data for initial grading of respective said clips; An output system is operable to output an indication of the suggested or selected frame to a user interface.

70. An apparatus according to claim 69 (or any other claim herein), wherein the data processor is configured to generate a scaling factor or tone mapping parameter or tone mapping function that will enable the cropped pixels and / or crushed pixels in the proposed or selected frame to be within a range.

71. An apparatus comprising: an input system operable to receive image data for display on a reference monitor; a data processor configured to process image data received at the input; a feature recognition module comprising a machine learning system trained to recognize filmmaking equipment in frames of the image data and to label frames in which filmmaking equipment is recognized; An output system is operable to output identification of the marked frames.

72. An apparatus according to any of the preceding claims (or any other claim herein), wherein the apparatus is configured to generate one or more scaling factors or tone mapping parameters or tone mapping functions selected to make two or more selected clips more consistent with each other.

73. An apparatus as claimed in any preceding claim (or any other claim herein), wherein the apparatus is configured to suggest key frames for grading dynamic behaviour of the image data.

74. A computer program product comprising a tangible medium storing machine-readable, machine-executable instructions which, when executed by a data processor, cause the data processor to perform a method performed by any one of the apparatus recited in the present claims.

75. A colour grading kit comprising an apparatus according to any one of claims 1 to 74 (or any other claim herein) or as described herein.

76. A device having any new and inventive feature, combination of features, or sub-combination of features as herein described.

77. A method having any new and inventive step, action, combination of steps and / or actions, or sub-combination of steps and / or actions as herein described.