Method and apparatus for generating dynamic picture metadata

Abstract

Methods and systems for generating dynamic picture metadata are presented. Given an input picture generated on a master display, characteristics of a target display different from the master display, and a set of initial dynamic metadata for the input picture, the following iterative algorithm is performed: mapping the input image to a mapped image for the target display according to a display management process and image metadata, comparing the input image to the mapped image according to a visual appearance matching indicator, and updating the image metadata using an optimization technique until a value of a perceptibility difference between the input image and the mapped image according to the visual appearance matching indicator is below a threshold.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 066,663, filed August 17, 2020, and European Patent Application No. 20191269.8, filed August 17, 2020, each of which is hereby incorporated in its entirety by reference. Technical Field

[0003] This invention generally relates to images. More specifically, embodiments of the invention relate to generating dynamic metadata for still images and video sequences. Background Technology

[0004] As used herein, the term "dynamic range (DR)" can refer to the ability of the human visual system (HVS) to perceive a range of intensity (e.g., luminance, luma) in an image, such as from the darkest gray (black) to the brightest white (highlight). In this sense, DR relates to the intensity of a "scene-referred" intensity. DR can also refer to the ability of a display device to fully or approximately render a specific breadth of intensity range. In this sense, DR relates to the intensity of a "display-referred" intensity. Unless a particular meaning is explicitly specified to have a specific implication at any point in the description herein, it should be inferred that the terms can be used interchangeably in either sense.

[0005] As used herein, the term "high dynamic range (HDR)" refers to a DR width spanning 14 to 15 orders of magnitude across the human visual system (HVS). In practice, the DR, which humans can simultaneously perceive a wide range of intensity, may be slightly truncated relative to HDR. As used herein, the terms "visual dynamic range (VDR)" or "enhanced dynamic range (EDR)" can be associated, individually or interchangeably, with this type of DR: DR that can be perceived within a scene or image by the human visual system (HVS), including eye movements, allowing for some light-adaptive variations in the scene or image. As used herein, VDR can refer to a DR spanning 5 to 6 orders of magnitude. Therefore, while perhaps slightly narrower relative to HDR for a real-world scene reference, VDR or EDR can represent a wide DR width and can also be referred to as HDR.

[0006] In practice, an image comprises one or more color components (e.g., luminance Y and chrominance Cb and Cr), where each color component is represented by n bits per pixel with precision (e.g., n = 8). For example, using gamma luminance coding, images where n ≤ 8 (e.g., a color 24-bit JPEG image) are considered to have standard dynamic range, while images where n ≥ 10 can be considered to have enhanced dynamic range. HDR images can also be stored and distributed using high-precision (e.g., 16-bit) floating-point formats, such as the OpenEXR document format developed by Industrial Light & Magic.

[0007] Most consumer desktop monitors currently support 200 to 300 cd / m². 2 Or nits of brightness. Most consumer HDTVs range from 300 to 500 nits, with newer models reaching 1000 nits (cd / m²). 2 Therefore, such conventional displays represent the lower dynamic range (LDR) associated with HDR, also known as standard dynamic range (SDR). As the availability of HDR content has increased due to advancements in both capture devices (e.g., cameras) and HDR displays (e.g., Dolby Laboratories' PRM-4200 professional reference monitor), HDR content can be color-graded and displayed on HDR displays that support higher dynamic ranges (e.g., from 1,000 nits to 5,000 nits or higher).

[0008] As used herein, the term “metadata” refers to any auxiliary information transmitted as part of an encoded bitstream or sequence and used to assist the decoder in rendering the decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters as described herein. Metadata can be characterized as “static” or “dynamic.” Examples of static metadata include parameters associated with the master display, such as the primary color, white point, and luminance range of the display used to master video content (Reference [1]). Examples of dynamic metadata include minimum, average, and maximum luminance or RGB values ​​of image frames, trim-pass data, or tone mapping parameters used by the decoder to display the bitstream on the target display (Reference [2]). As the inventors understand herein, improved techniques are needed for generating image metadata for video sequences (especially but not limited to HDR video) in order to improve existing and future display solutions.

[0009] The methods described in this section are permissible but not necessarily methods that have been previously conceived or employed. Therefore, unless otherwise instructed, no method described in this section should be considered prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise instructed, any issues concerning one or more methods should not be considered to be in any prior art based on this section. Summary of the Invention

[0010] According to one aspect of this disclosure, a method for generating dynamic image metadata using a processor is provided, the method comprising: receiving an input image in a first dynamic range and dynamic image metadata, wherein the input image is mastered on a master display; a) applying a display mapping process to map the input image to a mapped image in a second dynamic range, wherein the display mapping process takes into account the dynamic image metadata and the display characteristics of a target display different from the master display; b) comparing the input image with the mapped image using an appearance matching metric to generate a visibility difference value; and if the visibility difference value is greater than a threshold, then: c) applying a metadata optimization method to the dynamic image metadata to reduce the visibility difference value and generating updated image metadata; d) replacing the dynamic image metadata with the updated image metadata; and returning to step a for another metadata update iteration until a termination criterion is met; otherwise generating an output including the input image and the dynamic image metadata.

[0011] According to another aspect of this disclosure, an apparatus for generating dynamic image metadata is also provided, the apparatus including a processor and configured to perform the method as described above.

[0012] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is also provided, having computer-executable instructions stored thereon for performing the methods described above using one or more processors.

[0013] According to another aspect of this disclosure, a computer program product is also provided, the computer program product including computer executable instructions for using one or more processors to perform the methods as described above. Attached Figure Description

[0014] Embodiments of the invention are illustrated in the accompanying drawings by way of example rather than limitation, and similar reference numerals refer to similar elements, and in the drawings:

[0015] Figure 1 An example process of a video transmission pipeline is described; and

[0016] Figure 2An example process flow for generating dynamic image metadata according to an embodiment of the present invention is described. Detailed Implementation

[0017] Methods and systems for generating dynamic image metadata are described. In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the invention. However, it will be apparent that the invention can be practiced without these specific details. In other instances, well-known structures and devices are not described in detail to avoid unnecessarily obscuring, obscuring, or confusing the invention.

[0018] Overview

[0019] The example embodiments described herein relate to methods and systems for generating dynamic metadata. A processor receives a sequence of video images in a first dynamic range (e.g., HDR or SDR) that has been mastered on a master display, and initial dynamic image metadata, wherein the image metadata includes syntax parameters for enabling the video images to be displayed on a target display that may be different from the master display.

[0020] The processor:

[0021] a) Apply a display mapping process to map an input image to a mapped image in a second dynamic range, wherein the display mapping process takes into account the dynamic image metadata and the display characteristics of the target display;

[0022] b) Compare the input image with the mapped image using an appearance matching metric to generate a visibility difference value; and

[0023] If the visibility difference value is greater than the threshold, then:

[0024] c) Apply a metadata optimization method to the dynamic image metadata to reduce the visibility difference value and generate updated image metadata;

[0025] d) Replace the dynamic image metadata with the updated image metadata; and

[0026] Return to step a for another metadata update iteration until the termination criteria are met;

[0027] Otherwise, an output including the input image and the dynamic image metadata is generated.

[0028] Metadata for high dynamic range video

[0029] Video encoding and decoding of signals

[0030] Figure 1An example process of a conventional video transmission pipeline (100) is depicted, illustrating the various stages from video capture to video content display. An image generation block (105) is used to capture or generate a sequence of video frames (102). The video frames (102) can be captured digitally (e.g., by a digital camera) or generated by a computer (e.g., using computer animation) to provide video data (107). Alternatively, the video frames (102) can be captured on film by a film camera. The film is converted to a digital format to provide video data (107). In the production stage (110), the video data (107) is edited to provide a video production stream (112).

[0031] The video data from the production stream (112) is then provided to the processor at block (115) for post-production editing. Post-production editing at block (115) may include adjusting or modifying the color or brightness in specific areas of the image to enhance image quality or achieve a specific look for the image according to the video creator's creative intent. This is sometimes referred to as "color timing" or "color grading". It may also include frame / image rate resampling. Other edits may be performed at block (115) (e.g., scene selection and sorting, image cropping, adding computer-generated visual effects, variable frame rate sorting, etc.) to produce a final version (117) of the work for release. During post-production editing (115), the video image is viewed on a reference monitor or master monitor (125).

[0032] After post-production (115), the video data of the final work (117) can be transferred to the encoding block (120) for downstream transmission to decoding and playback devices such as televisions, set-top boxes, and cinemas. In some embodiments, the encoding block (120) may include audio encoders and video encoders such as those defined by ATSC, DVB, DVD, Blu-ray, and other transmission formats to generate an encoded bitstream (122). In the receiver, the encoded bitstream (122) is decoded by the decoding unit (130) to generate a decoded signal (132) representing the same or nearly identical version of the signal (117). The receiver may be attached to a target display (140), which may have characteristics completely different from those of the reference display (125). In this case, the display management block (135) may be used to map the dynamic range or frame rate of the decoded signal (132) to the characteristics of the target display (140) by generating a display mapping signal (137).

[0033] In the case of a given ecosystem 100, image metadata is included to guide signal processing at various stages of the image processing pipeline, particularly the signal processing of the display management process (135). For example, and not limited to, as described in reference [2], using the terminology of the Dolby Vision high dynamic range pipeline, image metadata may include one or more of the following parameters.

[0034] L1 metadata can include quantities such as min, mid, and max, which represent the minimum (“crush”) luminance value, mid-tone (“mid”) luminance value, and maximum (“clip”) luminance value or RGB value for one or more scenes representing the source video data 122, respectively.

[0035] L2 metadata provides and / or describes information about adjustments to video characteristics made by directors, color graders, video professionals, etc., in the production studio using a reference monitor 125 with a reference dynamic range.

[0036] L3 metadata provides and / or describes information about adjustments to video characteristics made by directors, color graders, video professionals, etc., in the production studio using a second reference display (such as target display 140) with a second reference dynamic range different from that of reference display 125. L3 metadata may include, for example, offsets or adjustments relative to L1 metadata, such as Δmin, Δmid, and Δmax as offsets from the quantities min, mid, and max of (L1), respectively.

[0037] L4 metadata provides or describes information about global dimming operations. L4 metadata can be calculated by the encoder during preprocessing and can be calculated using RGB primary colors. In one example, L4 metadata may include data indicating the global backlight brightness level of the display panel frame by frame.

[0038] Other generated metadata (such as L11 metadata) can provide or describe information to be used to identify video data sources, such as movie content, computer game content, sports content, etc. This metadata can further provide or describe expected image settings, such as expected white point, sharpness, etc.

[0039] Typically, this metadata is generated in several ways, including:

[0040] a. Manually specify preset values ​​based on the content (e.g., "game" in L11).

[0041] b. Automatically, metadata (e.g., min, mid, and max values ​​of LI) is calculated by analyzing the input video frame by frame and based on statistics of image brightness and color.

[0042] c. Semi-manually or semi-automatically, for example, by adjusting metadata values ​​while observing results on one or more monitors (e.g., "trimming" L2 metadata).

[0043] When the decoder receives content 132, given a target display 140, the content is processed by a display management process 135, which adjusts the image characteristics specified by the incoming metadata to achieve optimized rendering on the target display. Examples of display management processes can be found in references [3-5] and may include various image processing operations, including:

[0044] • Preprocessing and post-processing color transformation (e.g., YCbCr or RGB to ICtCp)

[0045] Image scaling

[0046] • Tone mapping

[0047] Saturation control

[0048] Color volume mapping

[0049] • Finishing and processing control; and

[0050] Ambient light adjustment

[0051] The example embodiments improve the generation of dynamic image metadata to enhance the display rendering process. The embodiments described herein are primarily applicable to automated methods for generating metadata, but can also be applied to semi-manual or semi-automatic methods, for example, to adjust and trim metadata processing.

[0052] Figure 2 An example process (200) for generating dynamic image metadata according to an embodiment is described. Figure 2As described, given an input image or image sequence 205, step 210 generates a set of initial metadata based on known manual, automatic, or semi-automatic metadata generation techniques. Next, in step 215, given a target display, the input image (205), and associated metadata (e.g., the initial metadata from step 210 or the updated metadata (237) from step 235), a display management process (preferably similar to the display management process used by the downstream decoder) is applied to the input image to generate a mapped image (217). Next, in step 220, the original image (205) is compared to the mapped image (217) using an appropriate appearance matching metric or visual difference quality metric. The goal here is to simulate the comparison between the original image and the mapped image based on a simulated human observer. Generally, the smaller the error returned by the metric, the smaller the predicted visual difference, and the better the match. In a non-limiting manner, examples of such visual quality metrics include the Video Multi-Method Evaluation Fusion Metric (VMAF) (Reference [6]), the Structural Similarity Index Measure (SSIM) (Reference [7]), and the HDR Visual Difference Predictor (HDR-VDP) (Reference [8]).

[0053] Since the two images being compared may have different color volumes (e.g., dynamic range and color gamut) as well as different resolutions or frame rates, a suitable image quality metric should be able to predict the appearance match between the two images regardless of these differences.

[0054] In step 230, it is necessary to determine whether the iterative process needs to be terminated. For example, the process can be terminated if the matching difference is below a certain threshold or if the maximum amount of time or number of iterations has been reached. If a termination decision is made, the final metadata (240) is output; otherwise, updated metadata (237) can be generated in step 235.

[0055] Given a specific image quality metric, the metadata update step (237) can be implemented using any known optimization technique, such as gradient descent, Levenberg-Marquardt, etc. Applying a metadata optimization method to dynamic image metadata to reduce visibility difference values ​​and generate updated image metadata means that the dynamic image metadata is replaced with updated image metadata with reduced visibility difference values. Identifying updated metadata with reduced visibility difference values ​​is one type of metadata optimization method. Visibility difference values ​​are generated by comparing the input image with the mapped image using an appearance matching metric.

[0056] In this embodiment, for some metadata types (such as L2), different metadata can be computed for various target displays. In this scenario, the above process can be repeated for each target display, and the bitstreams destined for different display environments can include different metadata datasets.

[0057] In another embodiment, the mapped image (217) can be further processed by additional modeling operations on the target display, including power limiting operations, global dimming, image enhancement, etc., to produce a modified mapped image, which can then replace the mapped image (217) in the comparison step with the input image (step 220). The metadata generated by this process takes into account specific display operations in the target display that are not solely part of the display management process.

[0058] In another embodiment, the generated mapped image (217) can be rendered on a display device (e.g., a target display). The image shown on the display device can then be captured by a calibrated digital camera to produce a calibrated captured image, which can then replace the mapped image (217) in the comparison step with the input image (step 220). The metadata generated by this process is optimized to preserve the appearance between the display characteristics of the reference display and the target display, which could otherwise be difficult to simulate through a separate display management process or a modified mapped image.

[0059] In another embodiment, the output is transmitted via a network to an auxiliary system external to the main system, which generates the output. The auxiliary system may include a display on which an image is rendered using the input image and dynamic image metadata. The auxiliary system's display may have the same or different display characteristics as the target display. A potential advantage of providing the input image and dynamic image metadata as output is that the output can be sent to a wide variety of displays. Depending on the actual display, the decoder can then determine how to apply optimized metadata to the input image.

[0060] Each of these references is incorporated into this paper in its entirety by way of citation.

[0061] References

[0062] [1] "Mastering display color volume metadata supporting high luminance and wide color gamut images", SMPTE ST 2086:2014, Society of Motion Picture and Television Engineers, 2014.

[0063] [2]AKAChoudhuryd et al., “Tone curve optimization method and associated video”, PCT application PCT / US2018 / 049585 filed on September 5, 2018, has been published as WO 2019 / 050972 (March 14, 2019).

[0064] [3] R. Atkins, “Display management for high dynamic range video”, U.S. Patent 9,961,237.

[0065] [4] JAPytlarz et al., “Ambient light-adaptive display management”, U.S. Patent Application Publication 2019 / 0304379, October 3, 2019.

[0066] [5] R. Atkins et al., “Display management for high dynamic range images”, PCT application PCT / US2020 / 028552 filed on April 16, 2020.

[0067] [6] R. Rassool, “VMAF reproducibility: Validating a perceptual practical video quality metric” IEEE Broadband Multimedia Systems and Broadcasting (BMSB) International Workshop, 2017. IEEE, 2017.

[0068] [7] Z. Wang et al., “Image quality assessment: from error visibility to structural similarity”, IEEE Transactions on Image Processing, 13.4 (2004): 600-612.

[0069] [8] R. Mantiuk et al., “HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions”, ACM Transactions on Graphics (TOG) 30.4 (2011): 1-14.

[0070] Example computer system implementation

[0071] Embodiments of the present invention may be implemented using computer systems, systems configured with electronic circuits and components, integrated circuit (IC) devices (such as microcontrollers, field-programmable gate arrays (FPGAs) or other configurable or programmable logic devices (PLDs), discrete-time or digital signal processors (DSPs), application-specific integrated circuits (ASICs)), and / or means including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or carry out instructions related to the generation of moving image metadata, as described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to the generation of moving image metadata described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0072] Some embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the methods of the present invention. For example, one or more processors, such as those in a display, encoder, set-top box, transcoder, etc., can implement the methods related to generating dynamic image metadata as described above by executing software instructions in a program memory accessible to the processor. The present invention can also be provided in the form of a program product. The program product may include any tangible and non-transitory medium carrying a set of computer-readable signals, including instructions that, when executed by a data processor, cause the data processor to perform the methods of the present invention. The program product according to the present invention can take any of a variety of forms. The program product may include, for example, physical media, such as magnetic data storage media including floppy disks and hard disk drives, optical data storage media including CD-ROMs and DVDs, electronic data storage media including ROMs and flash RAMs, etc. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0073] In the case of the components mentioned above (e.g., software modules, processors, components, devices, circuits, etc.), unless otherwise specified, references to said components (including references to “devices”) should be interpreted as including any component that performs the function of the described component as an equivalent of said component (e.g., functionally equivalent), including components that are structurally different from those that perform the functions in the illustrated exemplary embodiments of the invention.

[0074] Equivalents, extensions, alternatives and miscellaneous

[0075] Therefore, example embodiments related to the generation of dynamic image metadata have been described. In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary depending on the implementation. Therefore, the sole and exclusive indication of the invention and the applicant's inventive intent is the set of claims issued in specific form according to this application, wherein such claims include any subsequent amendments. Any definitions expressly set forth herein with respect to terms contained in such claims shall govern the meaning of such terms as used in the claims. Therefore, any limitations, elements, properties, characteristics, advantages, or attributes not expressly referenced in the claims should not in any way limit the scope of such claims. Therefore, this specification and drawings should be viewed in an illustrative rather than restrictive sense.

[0076] Various aspects of the invention can be understood from the following enumerated example embodiments (EEE):

[0077] 1. A method for generating dynamic image metadata using a processor, the method comprising:

[0078] Receive an input image and dynamic image metadata in a first dynamic range, wherein the input image is mastered on a master display;

[0079] a) Apply a display mapping process to map the input image to a mapped image in a second dynamic range, wherein the display mapping process takes into account the dynamic image metadata and the display characteristics of the target display, which is different from the master production display;

[0080] b) Compare the input image with the mapped image using an appearance matching metric to generate a visibility difference value; and

[0081] If the visibility difference value is greater than the threshold, then:

[0082] c) Apply a metadata optimization method to the dynamic image metadata to reduce the visibility difference value and generate updated image metadata;

[0083] d) Replace the dynamic image metadata with the updated image metadata; and

[0084] Return to step a for another metadata update iteration until the termination criteria are met;

[0085] Otherwise, an output including the input image and the dynamic image metadata is generated.

[0086] 2. The method as described in EEE 1, wherein the first dynamic range is a high dynamic range and the second dynamic range is a standard dynamic range.

[0087] 3. The method as described in EEE 1 or 2, wherein the termination criterion includes limiting the total number of metadata update iterations to less than the maximum iteration count.

[0088] 4. The method of any one of EEE 1 to 3, wherein the appearance matching metric includes one of the Video Multi-Method Evaluation Fusion Metric (VMAF), the Structural Similarity Index Measure (SSIM), and the HDR-Visual Difference Predictor (HDR-VDP).

[0089] 5. The method as described in any of the preceding EEEs, wherein the metadata optimization method comprises either gradient descent or the Levenberg-Marquardt algorithm.

[0090] 6. The method as described in any one of the preceding EEEs, wherein the dynamic image metadata of the input image includes metadata parameters based on pixel value statistics in the input image.

[0091] 7. The method of any one of EEE 1 to 5, wherein the dynamic image metadata of the input image includes adjusted metadata parameters calculated when the input image is viewed on the target display.

[0092] 8. The method as described in any one of the preceding EEEs, further comprising:

[0093] Following step a, additional modeling operations for the target display are applied to the mapped image to generate a modified mapped image; and in step b,

[0094] The appearance matching metric is used to compare the input image with the modified mapped image to generate the visibility difference value.

[0095] 9. The method as described in EEE 8, wherein the additional modeling operations include power limiting, global dimming, or image enhancement operations of the target display.

[0096] 10. The method of any one of EEE 1 to 7, further comprising:

[0097] After step a, the mapped image is rendered on the target display to generate a rendered image, and the rendered image is captured by a camera to generate a captured image; and in step b,

[0098] The appearance matching metric is used to compare the input image with the captured image to generate the visibility difference value.

[0099] 11. An apparatus comprising a processor and configured to perform any one of the methods described in EEE 1 to 10.

[0100] 12. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon for performing a method according to any one of EEE 1 to 10 using one or more processors.

Claims

1. A method for generating dynamic image metadata using a processor, the method comprising: Receive an input image and dynamic image metadata in a first dynamic range, wherein the input image is mastered on a master display; a) Apply a display mapping process to map the input image to a mapped image in a second dynamic range, wherein the display mapping process takes into account the dynamic image metadata and the display characteristics of the target display, which is different from the master display, and uses a digital camera to capture the mapped image rendered on the target display; b) The input image is compared with the captured image of the mapped image using an appearance matching metric to generate a visibility difference value, wherein the appearance matching metric includes one of a video multi-method evaluation fusion metric, a structural similarity index measure, and an HDR-visual difference predictor; and If the visibility difference value is greater than the threshold, then: c) Apply a metadata optimization method to the dynamic image metadata to reduce the visibility difference value and generate updated image metadata, wherein the updated image metadata preserves the appearance between the display characteristics of the master display and the target display; d) Replace the dynamic image metadata with the updated image metadata; and Return to step a for another metadata update iteration until the termination criteria are met; Otherwise, an output including the input image and the dynamic image metadata is generated, wherein the output is transmitted via a network to an auxiliary system outside the main system for rendering the image on the display of the auxiliary system, and the main system is used to generate the output.

2. The method as described in claim 1, wherein, The auxiliary system includes a display that renders an image on the display using the input image and the dynamic image metadata.

3. The method as described in claim 2, wherein, The auxiliary system's display has the same or different display characteristics as the target display.

4. The method of claim 1, wherein, The first dynamic range is a high dynamic range, and the second dynamic range is a standard dynamic range.

5. The method of claim 1, wherein, The termination criteria include limiting the total number of metadata update iterations to less than the maximum iteration count.

6. The method of claim 1, wherein, The metadata optimization method includes either gradient descent or the Levenberg-Marquardt algorithm.

7. The method of claim 1, wherein, The dynamic image metadata of the input image includes metadata parameters based on pixel value statistics in the input image.

8. The method of claim 1, wherein, The dynamic image metadata of the input image includes adjusted metadata parameters calculated when the input image is viewed on the target display.

9. The method of claim 1, further comprising: Following step a, additional modeling operations for the target display are applied to the mapped image to generate a modified mapped image; And in step b, The appearance matching metric is used to compare the input image with the modified mapped image to generate the visibility difference value.

10. The method of claim 9, wherein, The additional modeling operations include power limiting, global dimming, or image enhancement operations for the target display.

11. The method of claim 1, further comprising: After step a, the mapped image is rendered on the target display to generate a rendered image, and the rendered image is captured by a camera to generate a captured image; And in step b, The appearance matching metric is used to compare the input image with the captured image to generate the visibility difference value.

12. An apparatus for image processing, the apparatus comprising a processor and configured to perform the method as claimed in any one of claims 1-11.

13. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon for performing the method according to any one of claims 1-11 using one or more processors.

Images