Multi-intent composite image encoding and rendering

Abstract

Techniques for multi-intent composite image encoding and rendering are disclosed. The techniques may include obtaining a set of component images of a composite image; determining a common rendering intent to be applied to the set of component images; adjusting one or more component images in the set of component images according to the common rendering intent to obtain an adjusted set of component images; creating a composite image based on the adjusted set of component images; generating metadata representing the common rendering intention; and encoding the composite image and the metadata to create an encoded multi-intent composite image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 368,766, filed on July 18, 2022, and European Application No. 22213582.4, filed on December 14, 2022, each of which is incorporated herein by reference in its entirety. Background Art

[0003] The present application relates generally to systems and methods for image encoding and decoding. Summary of the invention

[0004] At least some aspects of the present disclosure may be implemented via methods. Some methods may involve: obtaining a set of component images of a composite image; determining a common rendering intent to be applied to the set of component images; adjusting one or more component images in the set of component images according to the common rendering intent to obtain an adjusted set of component images; creating a composite image based on the adjusted set of component images; generating metadata characterizing the common rendering intent; and encoding the composite image and the metadata to create an encoded multi-intent composite image.

[0005] In some examples, adjusting one or more component images in the set of component images according to the common rendering intent may include converting the component images in the set of component images from a current rendering intent to the common rendering intent.

[0006] In some examples, converting the component image in the set of component images from the current rendering intent to the common rendering intent may include: reversing one or more source adjustments to the component image; and applying one or more common spatial adjustments to the component image.

[0007] In some examples, applying the one or more common spatial adjustments to the component images may include converting sensor values ​​to color values.

[0008] In some examples, applying the one or more common spatial adjustments to the component images may include: estimating capture environment ambient luminance and white point; and applying white point correction based on the estimated capture environment ambient luminance and white point.

[0009] In some examples, applying the one or more common spatial adjustments to the component images may include: estimating the capture environment ambient lighting; and applying an optical-optical transfer function (OOTF) based in part on the estimated capture environment ambient lighting to prepare the image for rendering on a reference display device.

[0010] In some examples, applying the one or more common spatial adjustments to the component images may include: applying one or more of: saturation enhancement; contrast enhancement; individual color saturation adjustment; slope-offset-power-Tmid enhancement; and tone curve fine-tuning.

[0011] In some examples, adjusting one or more component images in the set of component images according to the common rendering intent may include converting component images in the set of component images from a pre-processed state to the common rendering intent.

[0012] In some examples, adjusting one or more component images in the group of component images according to the common rendering intent may include: converting a first component image in the group of component images from a first current rendering intent to the common rendering intent; and converting a second component image in the group of component images from a second current rendering intent to the common rendering intent.

[0013] In some examples, determining the common rendering intent may include: selecting a current rendering intent of a component image in the group of component images as the common rendering intent.

[0014] In some examples, the current rendering intent of the component image can be selected as the common rendering intent based on the importance of the component image relative to other component images in the group of component images, wherein the importance of the component image is determined based on one or more of the following: the relative size of the component image in the rendering space of the composite image; the resolution of the component image; the centrality of the position of the component image in the rendering space of the composite image; and the point of focus of the viewer in the rendering space of the composite image.

[0015] In some examples, determining the common rendering intent may include identifying a preferred rendering intent as the common rendering intent.

[0016] In some examples, the preferred rendering intent can be identified based on one or both of: input received via a user interface and information in a configuration file.

[0017] Some methods may involve: receiving an encoded multi-intent composite image; decoding the encoded multi-intent composite image to obtain a composite image and metadata, the metadata describing one or more common spatial adjustments applied when creating the composite image; identifying the one or more common spatial adjustments based on the metadata; adjusting the composite image to reverse the one or more common spatial adjustments to obtain a real-rendered composite image; adjusting the real-rendered composite image according to a target rendering intent to obtain a target-adjusted composite image; and displaying the target-adjusted composite image.

[0018] In some examples, adjusting the reality-rendered composite image according to the target rendering intent may include applying one or more target adjustments to the reality-rendered composite image.

[0019] In some examples, applying the one or more target adjustments to the reality-rendered composite image may include converting sensor values ​​to color values.

[0020] In some examples, applying the one or more target adjustments to the reality-rendered composite image may include: estimating capture environment ambient luminance and white point; and applying white point correction based on the estimated capture environment ambient luminance and white point.

[0021] In some examples, applying the one or more target adjustments to the reality-rendered composite image can include: estimating the brightness of the capture environment ambient light; and applying an optical-optical transfer function (OOTF) based in part on the estimated brightness of the capture environment ambient light to prepare the image for rendering on a reference display device.

[0022] In some examples, applying the one or more target adjustments to the realistically rendered composite image may include: applying one or more of: saturation enhancement; contrast enhancement; individual color saturation adjustment; slope-offset-power-Tmid enhancement; and tone curve fine-tuning.

[0023] In some examples, the target rendering intent may be identified based on one or both of: input received via a user interface and information in a configuration file.

[0024] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

[0025] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may have the ability to at least partially perform the methods disclosed herein. In some embodiments, the apparatus is or includes an audio processing system with an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or a combination thereof.

[0026] The details of one or more embodiments of the subject matter described in this specification are set forth in the following drawings and description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative sizes of the following drawings may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] These and other more detailed and specific features of various embodiments are more fully disclosed in the following description with reference to the accompanying drawings, in which:

[0028] Figure 1 An example image transmission pipeline according to aspects of the present disclosure is depicted.

[0029] Figure 2 A first example process in accordance with aspects of the present disclosure is depicted.

[0030] Figure 3 A second example process in accordance with aspects of the present disclosure is depicted.

[0031] Figure 4 A third example process in accordance with aspects of the present disclosure is depicted.

[0032] Figure 5 An example operating environment is depicted in accordance with aspects of the present disclosure.

[0033] Figure 6 An example of a first method according to aspects of the present disclosure is depicted.

[0034] Figure 7 An example of a second method according to aspects of the present disclosure is depicted.

[0035] Figure 8 Examples of apparatuses according to aspects of the present disclosure are depicted. DETAILED DESCRIPTION

[0036] The present disclosure and its various aspects may be embodied in various forms, including: hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The foregoing is intended only to give a general idea of ​​the various aspects of the present disclosure and is not intended to limit the scope of the present disclosure in any way.

[0037] In the following description, many details such as optical device configuration, timing, operation, etc. are set forth to provide an understanding of one or more aspects of the present disclosure. It is obvious to those skilled in the art that these specific details are only exemplary and are not intended to limit the scope of the present application.

[0038] Figure 1 An example image transmission pipeline 100 is depicted, which shows various stages from image capture to image content display. An image 102 is captured or generated using an image generation block 105, which may be a video frame in a sequence of video frames, according to some embodiments. The image 102 may be captured digitally (e.g., by a digital camera) or generated by a computer (e.g., using computer animation) to provide image data 107. Alternatively, the image 102 may be captured on film by a film camera. The film is converted to a digital format to provide image data 107. In a production stage 110, the image data 107 is edited to provide an image production stream 112.

[0039] The image data of the production stream 112 is then provided to a processor (or one or more processors, such as a central processing unit (CPU)) at block 115 for post-production editing. Block 115 post-production editing may include adjusting or modifying the color or brightness in specific areas of the image to enhance the image quality or achieve a specific look of the image according to the creative intent of the image creator. This is sometimes referred to as "color timing" or "color grading." The methods described herein may be performed by a processor at block 115. Other edits (e.g., scene selection and sequencing, image cropping, adding computer-generated visual effects, etc.) may be performed at block 115 to produce a final version 117 for release. During post-production editing 115, the image is viewed on a reference display 125 (according to some embodiments, as part of a video image sequence). If desired, the reference display 125 may be a consumer-grade display or projector.

[0040] After post-production 115, the image data of the final production 117 may be transmitted to the encoding block 120 for downstream transmission to decoding and playback devices such as computer monitors, televisions, set-top boxes, movie theaters, etc. In some embodiments, the encoding block 120 may include an audio encoder and a video encoder such as those defined by ATSC, DVB, DVD, Blu-ray, and other transmission formats to generate an encoded bitstream 122. In the receiver, the encoded bitstream 122 is decoded by a decoding unit 130 to generate a decoded signal 132 representing the same or a close approximation of the signal 117. The receiver may be attached to a target display 140, which may have completely different characteristics from the reference display 125. In this case, the display management block 135 may be used to map the dynamic range of the decoded signal 132 to the characteristics of the target display 140 by generating a display mapping signal 137. The additional methods described herein may be performed by the decoding unit 130 or the display management block 135. Both the decoding unit 130 and the display management block 135 may include their own processors, or may be integrated into a single processing unit. Although the present disclosure refers to a target display 140, it should be understood that this is merely an example. It should be further understood that the target display 140 can include any device configured to display or project light; for example, a computer monitor, a television, an OLED display, an LCD display, a quantum dot display, a cinema, consumer and other commercial projection systems, a head-up display, a virtual reality display, etc.

[0041] When a scene is captured using a digital device, it is seldom that the true scene radiometry is directly converted to produce an image. Instead, it is common practice for the original electronics manufacturer (OEM) of the device or the software application designer to adjust the image, for example by adapting the image for viewing in a reference viewing environment such as dark environments and D65 lighting, and applying aesthetic adjustments such as enhancing contrast and color saturation. These and other adjustments produce a preferred realistic rendering that is considered pleasing to consumers.

[0042] Currently, these operations are lossy in two ways. First, no parameters are transmitted to apply these operations, and second, pixel operations can be lossy due to nonlinear clipping and quantization, non-reversible operations, unknown algorithms, or unknown order of operations.

[0043] Instead, it may be desirable to be able to transmit a raw captured / pre-processed image that represents "reality" as captured by the imaging sensor, and then apply these operations upon playback. This may enable multiple rendering intents: upon playback, the device may present the raw captured "reality" image, or alternatively, the device may create a "pleasing" image that is modified based on the raw captured "reality" image.

[0044] It may also be desirable to allow such content to be transmitted in a backwards compatible manner. In this approach, modifications to create a "pleasing" image may be applied during capture, and appropriate parameters transmitted to a playback device to allow the playback device to reverse the modifications, thereby restoring the originally captured "realistic" image.

[0045] Figure 2 An example process 200 is illustrated. Process 200 may allow for encoding and decoding of images with multiple intents using metadata. Process 200 may be performed by a processor, for example, as part of block 115 and / or block 120 for encoding and as part of block 130 and / or block 135 for decoding.

[0046] At step 202, an image is captured. In a digital capture device, the exposure scene is converted to sensor values ​​represented in a single channel. Through a process called demosaicing, the single channel image representation is expanded to a three channel image representation: red, green, and blue (RGB). There are numerous methods of demosaicing, any of which are sufficient in the embodiments disclosed herein.

[0047] As a non-limiting example, step 202 may include: reading a single channel value from a sensor; applying a demosaicing scheme to create a three-color channel (e.g., RGB) image; optionally applying a 3×3 transform to make the image sensitivity conform to the desired three primary colors (e.g., RGB) image sensitivity. Step 202 may also include: measuring the capture ambient light brightness (e.g., the level of ambient light in the capture environment).

[0048] To perfectly capture the chromaticity of a scene, the spectral sensitivity of the capture device should match the spectral sensitivity of the viewer. In practice, these will often not match exactly, but will be approximated using a 3×3 matrix transform to convert the sensor sensitivity to some set of expected RGB primaries. The approximation of sensor sensitivity is a source of loss. To avoid the loss caused by this, the applied camera spectral sensitivity can be transmitted along with the content along with the 3×3 matrix transform, allowing the playback device to apply or invert the transformation from the sensor output to the specified RGB primaries.

[0049] Once the desired RGB composition of the image is determined, these values ​​can be made to conform to a specified reference white point. The image can be made to conform to one of the standardized white points (D50, D65, etc.) by, for example, a Von Kries adaptive transform. The process involves: (a) estimating the capture environment ambient lighting and white point; and (b) applying corrections to the image to achieve color matching with an observer in a specified reference viewing environment (e.g., an environment with a known white point and ambient lighting). Methods that can be used to adjust an image to suit the observer's adaptation state in a chromatic ambient environment are outlined in PCT application number PCT / US2021 / 027826 filed on April 16, 2021 and PCT application number PCT / US2021 / 029476 filed on April 27, 2021, each of which is incorporated herein by reference in its entirety and for all purposes.

[0050] At step 204, one or more source appearance adjustments may be applied to the captured image, including but not limited to white balance adjustments, color correction adjustments, and optical-to-optical transfer function (OOTF) adjustments. Step 204 may include: calculating a nonlinear optical-to-optical transfer function (OOTF) to map from the measured captured ambient light brightness to a reference viewing environment. The order of the white point adjustment and the 3×3 matrix may vary. The calculation and application of the optical-to-optical transfer function (OOTF) may establish the rendering intent of the image on a standard display device. In practice, the OOTF is applied to map the image from the viewing environment at the time of capture to the display in the reference viewing environment. The application of the OOTF may be a lossy operation, which may make it difficult to invert the OOTF during playback. As with the white point adjustment, in a first step (a), the ambient lighting of the capture environment may be estimated, and in a second step (b), the image may be corrected to match an observer in the reference environment.

[0051] At step 206, one or more source preference adjustments may be applied to the captured image, including but not limited to contrast adjustment, color saturation adjustment including overall color saturation adjustment and / or individual color saturation adjustment, slope-offset-power-Tmid adjustment in the tone curve, and other tone curve fine-tuning and adjustments. As used herein, in a perceptual quantization (PQ) encoded image, "mid" refers to the average of the maximum RGB values ​​of the image, where each pixel has its own maximum RGB value, which is equal to the maximum color component value (R, G, or B) of the pixel. In other words, which color component of the pixel has the maximum value, which is the maximum RGB value of the pixel, and the average of the maximum RGB values ​​of the entire PQ encoded image is the "mid" of the image. "T-mid" may refer to "target mid", which may be the "mid" value desired by the user or content creator in the final image. In some embodiments, a single color saturation adjustment may include saturation adjustments for 6 different colors, which may be referred to as a "six-vector adjustment".

[0052] The modification of the captured image at steps 204 and 206 produces a source adjusted image 207. Source rendering intent 203 may specify source appearance adjustments to be applied at 204 and / or source preference adjustments to be applied at 206 to produce source adjusted image 207. Source rendering intent 203 may be indicated by a selection received from a user, wherein the selection of the intent specifies what source appearance adjustments and source preference adjustments to make, coefficients of such adjustments, which portions of the image to apply the adjustments to, etc. Alternatively, source rendering intent 203 may constitute a default rendering intent, or may represent a combination of the two, such that the source rendering intent reflects both user-specified adjustments and default adjustments.

[0053] It is common practice for an OEM or software application to apply source preference adjustments to the captured image. These changes are purely aesthetic and are typically in preparation for rendering an image with higher contrast and color saturation levels. In various embodiments of the present disclosure, these preference changes determined by the OEM are transmitted with the content as metadata and applied upon playback in the same manner as the source appearance metadata. In each case, there is a first step of (a) calculating or specifying the desired amount of correction to be applied, and (b) applying the correction using a parameterized function. Both (a) and (b) are transmitted as metadata, allowing the playback device complete flexibility in rendering a "pleasing" or "realistic" image, and allowing the capture device complete flexibility in transmitting a "pleasing" or "realistic" image.

[0054] As described herein, one benefit of the various embodiments disclosed herein is that all adjustments to the three-channel image can be encoded as metadata and sent with the content to the playback device for application. In one embodiment, the OEM or encoding device can decide not to apply adjustments to both appearance and preference in order to produce a "realistic" image.

[0055] At step 208, the source adjusted image may be encoded. Step 208 may include encoding the source adjusted image 207 for downstream transmission to a decoding and playback device such as a computer monitor, television, set-top box, movie theater, etc. In some embodiments, the encoding step 208 may include an audio encoder and a video encoder such as those defined by ATSC, DVB, DVD, Blu-ray, and other transmission formats to generate an encoded bitstream.

[0056] In addition to encoding the source adjusted image 207, step 208 may also include creating and / or encoding metadata characterizing the source appearance adjustment applied in step 204 and the source preference adjustment applied in step 206 (if any). The metadata may include metadata associated with the source appearance adjustment, such as the scene white point specified in x, y coordinates (or some other system); the scene ambient brightness specified in lux (or some other system) (e.g., information about the estimated capture environment); the coefficients of the applied white point adjustment matrix; the coefficients of the applied 3×3 color matrix, the coefficients of the applied parameterized OOTF; the spectral sensitivity of the sensor used to calculate the 3×3 matrix; and coefficients or other information of other enhancements applied in step 204. Additionally, the metadata may include metadata associated with the source preference adjustment, such as coefficients of contrast enhancement (such as slope-offset-power-Tmid contrast adjustment); coefficients of saturation enhancement; coefficients of individual color saturation adjustments; coefficients of tone curve fine-tuning; and coefficients of other enhancements applied in step 206.

[0057] If the metadata indicates source appearance adjustments and / or preference adjustments that were applied in steps 204 and / or 206, the metadata may include information that facilitates the inverse (or approximate inverse) of the functions applied in conjunction with those adjustments. The metadata may include information that indicates the order in which the source appearance adjustments and / or preference adjustments were applied in steps 204 and / or 206. If no adjustments were made at steps 204 and / or 206, the metadata may indicate that no source appearance adjustments and / or preference adjustments were made.

[0058] At step 210, the encoded source adjusted image 207 and metadata may be decoded. The source rendering intent 203 associated with the adjustments applied at steps 204 and 206 may then be compared to a target rendering intent 216. The target rendering intent 216 may specify a target appearance adjustment and / or a preference adjustment. The target rendering intent 216 may be indicated by a selection received from a user, wherein the selection of the intent specifies what target appearance adjustment and target preference adjustment to make, the coefficients of such adjustments, which portions of the image to apply the adjustments to, and the like. Alternatively, the target rendering intent 216 may constitute a default rendering intent or a desired rendering intent indicated by metadata. In some embodiments, the target rendering intent 216 may represent a combination of both, such that the target rendering intent reflects both user-specified adjustments and adjustments associated with the default rendering intent and / or the desired rendering intent indicated by metadata. In an example, the target rendering intent 216 may represent a combination of user-specified appearance adjustments and default preference adjustments.

[0059] If the target rendering intent 216 is different from the source rendering intent 203, that is, if the target appearance adjustments and / or preference adjustments are different from the source appearance adjustments and / or preference adjustments, a series of steps may be performed to convert the source adjusted image 207 from the source rendering intent 203 to the target rendering intent 216. These steps may generally involve converting the source adjusted image 207 to the reality rendered image 215, and then converting the reality rendered image 215 to the target adjusted image 219.

[0060] The source adjusted image 207 may be converted to a reality rendered image 215 via operations performed at steps 212 and / or 214 to undo the adjustments performed at steps 206 and / or 204 (in conjunction with the application of the source rendering intent 203). The reality rendered image 215 may correspond to the image captured at step 202, and may represent "reality" as captured by the imaging sensor. At step 212, an inverted source preference adjustment may be calculated using the metadata obtained at step 210. When applied to the source adjusted image 207, the inverted source preference adjustment of step 212 may undo some or all of the source preference adjustments of step 206. At step 214, an inverted source appearance adjustment may be calculated using the metadata obtained at step 210. When applied to the source adjusted image 207, the inverted source appearance adjustment of step 214 may undo some or all of the source appearance adjustments of step 204. Based on the metadata obtained at step 210, the order in which the source appearance adjustments and / or preference adjustments are applied in steps 204 and / or 206 may be determined. This order may then be reversed in conjunction with applying the inverted source preference adjustment and / or appearance adjustment at steps 212 and / or 214 .

[0061] The realistic rendered image 215 may be converted to a target-adjusted image 219 via operations performed at steps 217 and / or 218. At step 217, a target appearance adjustment may be calculated and applied. As a non-limiting example, the target appearance adjustment may include measuring the display ambient light brightness (e.g., the level of ambient light in the display environment), and then calculating and applying a nonlinear optical-optical transfer function (OOTF) to map from a reference viewing environment to the measured display ambient light brightness (e.g., the actual viewing environment). At step 218, a target preference adjustment may be calculated and applied. As a non-limiting example, the target preference adjustment may include contrast adjustment, color saturation adjustment, slope-offset-power-Tmid adjustment, single color saturation adjustment, and tone curve fine-tuning. In some embodiments, inverting the source adjustment and applying the target adjustment may be combined into a single processing step, and the adjustment may be calculated accordingly. In other words, some or all of steps 212, 214, 217, and 218 may be combined.

[0062] In some embodiments, the source rendering intent 203 may correspond to a realistic rendering, and steps 204 and 206 may be substantially bypassed. In some such embodiments, a realistic rendered image—the image captured at step 202—may be used as the source-adjusted image 207 for encoding at step 208, and the metadata encoded at step 208 may indicate that no source appearance adjustment and source preference adjustment were made. Based on the metadata, the decoding entity may recognize that the source-adjusted image 207 is a realistic rendering image, and steps 212 and 214 may be substantially bypassed. Similarly, in some embodiments, the target rendering intent 216 may correspond to a realistic rendering, and steps 217 and 218 may be substantially bypassed, such that the realistic rendered image is used as the target-adjusted image 219.

[0063] The target-adjusted image 219 may be rendered at step 220. As examples, the target-adjusted image 219 may be projected, displayed, saved to storage, transmitted to another device, or otherwise utilized.

[0064] Figure 3 An example process 300 is illustrated. According to process 300, a composite image can be created based on a set of component images that have been adjusted to apply different corresponding rendering intents. Figure 3 In the depicted example, the set of component images includes adjusted images 301-1, 301-2, 301-3, and 301-4. It should be appreciated that in various implementations, process 300 may involve creating a composite image based on a fewer or greater number of component images.

[0065] Adjusted images 301-1, 301-2, 301-3, and 301-4 represent captured images A, B, C, and D that have been adjusted (e.g., via appearance adjustment and / or preference adjustment, as in Figure 2 those adjustments at steps 204 and 206) to respectively implement rendering intent Figure 1 , 2 , 3 and 4. At steps 302-1, 302-2, 302-3 and 302-4, reverse adjustments may be applied to the adjusted images 301-1, 301-2, 301-3 and 301-4, respectively, to undo the adjustments made to their current rendering intents (rendering intents). Figure 1 , 2 , 3, and 4). These reverse adjustments may include reverse preference adjustments to undo previously applied preference adjustments, reverse appearance adjustments to undo previously applied appearance adjustments, or both. In the case where both types of reverse adjustments are performed, the reverse preference adjustments may be performed first, followed by the reverse source adjustments.

[0066] The reverse adjustments performed on the adjusted images 301-1, 301-2, 301-3, and 301-4 at 302-1, 302-2, 302-3, and 302-4 may produce corresponding real-life rendered images 303-1, 303-2, 303-3, and 303-4. The real-life rendered images 303-1, 303-2, 303-3, and 303-4 may then be converted to a common rendering intent 304. In some embodiments, the common rendering intent 304 to be applied may be user-specified, or may be a default rendering intent. In some embodiments, the common rendering intent 304 to be applied may be selected from a plurality of candidate rendering intents. In some embodiments, these candidate rendering intents may be statically pre-specified. In other embodiments, these candidate rendering intents may be dynamically identified during execution of process 300.

[0067] In some embodiments, the common rendering intent 304 can match the rendering intent of one or more of the group of component images (adjusted images 301-1, 301-2, 301-3, and 301-4). In some embodiments, the common rendering intent 304 can be selected from the rendering intents of the group of component images, such as based on the relative importance of each component image. In the example, the adjusted image 301-1 can be identified as the most important component image in the group of component images, and the rendering intent Figure 1 Therefore, it can be selected as the common rendering intent 304.

[0068] The relative importance of the various component images in the group may be determined based on one or more factors. In some embodiments, the factor(s) may include the relative sizes of the component images in the rendering space of the composite image to be created. For example, the largest component image may be identified as the most important component image among the component images. In some embodiments, the factor(s) may include the respective resolutions of the component images. For example, the highest resolution component image may be identified as the most important component image among the component images.

[0069] In some embodiments, (multiple) factors may include characteristics related to the relative prominence of the component images in the rendering space. For example, the most central component image may be identified as the most important component image among the component images. In another example, gaze detection may be used to identify the point of interest of the viewer within the rendering space, and the component image at which the point of interest resides may be identified as the most important component image among the component images. In some embodiments, any of the above factors may be considered in combination with respect to determining the relative importance of the component images. In some embodiments, (multiple) other factors may be considered in addition to or in place of any or all of the above factors with respect to determining the relative importance of the component images.

[0070] In some embodiments, the rendering intent represented by multiple component images may be selected as the common rendering intent 304. For example, if multiple, most, or all component images have the same rendering intent, then that rendering intent may be selected as the common rendering intent 304.

[0071] Via the common spatial adjustment performed at steps 305 and / or 306, a common rendering intent 304 may be applied to the real rendered images 303-1, 303-2, 303-3, and 303-4 to obtain corresponding adjusted images 307-1, 307-2, 307-3, and 307-4. At step 305, a common spatial appearance adjustment may be calculated and applied. As a non-limiting example, the common spatial appearance adjustment may include applying a nonlinear optical-optical transfer function (OOTF) to map from one reference viewing environment to another reference viewing environment or an actual viewing environment. At step 306, a common spatial preference adjustment may be calculated and applied. As a non-limiting example, the common spatial preference adjustment may include contrast adjustment, color saturation adjustment, slope-offset-power-Tmid adjustment, single color saturation adjustment, and tone curve fine-tuning.

[0072] At step 308, a compositing procedure may be utilized to create a composite image 309 based on the adjusted images 307-1, 307-2, 307-3, and 307-4. At step 310, the composite image 309 and metadata characterizing the common rendering intent 304 may be encoded to create an encoded multi-intent composite image 312.

[0073] Figure 4 An example process 400 is illustrated. According to the process 400, at step 402, Figure 3 The encoded multi-intent composite image 401 (created by the process 300 of ) is decoded to obtain a composite image 403 and metadata indicating a common rendering intent of a group of images based on which the composite image 403 is created. The metadata may describe the application of one or more common spatial adjustments to one or more of these images. The common rendering intent may be compared with a target rendering intent 408, which may specify a target appearance adjustment and / or a preference adjustment. The target rendering intent 408 may be indicated by a selection received from a user, wherein the selection of the intent specifies what target appearance adjustment and target preference adjustment to make, the coefficient of such adjustment, which parts of the image to apply the adjustment, etc. Alternatively, the target rendering intent 408 may constitute a default rendering intent or a desired rendering intent indicated by the metadata. In some embodiments, the target rendering intent 408 may represent a combination of the two, so that the target rendering intent simultaneously reflects the user-specified adjustment and the adjustment associated with the default rendering intent and / or the desired rendering intent indicated by the metadata. In an example, the target rendering intent 408 may represent a combination of the user-specified appearance adjustment and the default preference adjustment.

[0074] If the target rendering intent 408 is different from the common rendering intent, a series of steps may be performed to convert the composite image 403 from the common rendering intent 408. These steps may generally involve converting the composite image 403 to the real-rendered composite image 407, and then converting the real-rendered composite image 407 to the target-adjusted composite image 411.

[0075] The composite image 403 may be converted to a realistic rendered composite image 417 via operations performed at steps 404 and / or 406. At step 404, an inverted common spatial preference adjustment may be calculated using the metadata obtained at step 402. The inverted common spatial preference adjustment of step 404 may undo the creation of the composite image 403 (e.g., at step 406). Figure 3At step 406, the inverted common space appearance adjustment may be calculated using the metadata obtained at step 402. The inverted common space appearance adjustment of step 406 may undo the creation of the composite image 403 (e.g., at step 305). Figure 3 306 ) together with one or more common spatial appearance adjustments applied.

[0076] The realistic rendered synthetic image 407 may be converted to a target-adjusted synthetic image 411 via operations performed at steps 409 and / or 410. At step 409, a target appearance adjustment may be calculated and applied. As a non-limiting example, the target appearance adjustment may include measuring the display ambient light brightness (e.g., the level of ambient light in the display environment), and then calculating and applying a nonlinear optical-optical transfer function (OOTF) to map from a reference viewing environment to the measured display ambient light brightness (e.g., the actual viewing environment). At step 410, a target preference adjustment may be calculated and applied. As a non-limiting example, the target preference adjustment may include contrast adjustment, color saturation adjustment, slope-offset-power-Tmid adjustment, single color saturation adjustment, and tone curve fine-tuning. In some embodiments, inverting the common space adjustment and applying the target adjustment may be combined into a single processing step, and the adjustment may be calculated accordingly. In other words, some or all of steps 404, 406, 409, and 410 may be combined.

[0077] The target-adjusted composite image 411 may be rendered at step 412. As examples, the target-adjusted composite image 411 may be projected, displayed, saved to storage, transmitted to another device, or otherwise utilized.

[0078] Figure 5 An example operating environment 500 is illustrated in which multi-intent composite image encoding and rendering may be performed in accordance with aspects of the present disclosure. In the operating environment 500, a device 501A may (e.g., using Figure 3 300 or the like) to create an encoded multi-intent composite image 510 and transmit it to device 501B, which may (e.g., using Figure 4 The encoded multi-intent composite image 510 is processed by process 400 or the like to create a target-adjusted composite image 514.

[0079] Device 501A may obtain a set of component images 502 (e.g., Figure 3The device 501A may determine a common rendering intent to be applied to the group of component images 502. According to some embodiments, determining the common rendering intent may involve identifying a preferred rendering intent as the common rendering intent. In some embodiments, the preferred rendering intent may be identified based on input received via a user interface or information in a configuration file. In some embodiments, the preferred rendering intent may be identified based on a combination of the two. For example, an input identifying a component image may be received, the rendering intent of the component image constituting the preferred rendering intent, and a configuration file may be accessed to identify the rendering intent of the identified component image. In some embodiments, the device 501A may select a common rendering intent from a plurality of candidate rendering intents. In some embodiments, these candidate rendering intents may be statically pre-specified. In other embodiments, these candidate rendering intents may be dynamically identified.

[0080] In some embodiments, device 501A may select a common rendering intent that matches the rendering intent of one or more component images in the set of component images 502. In some embodiments, for example, device 501A may select the current rendering intent of a component image 502 as the common rendering intent (e.g., based on the relative importance of the component images 502). In some embodiments, device 501A may select a current rendering intent that is shared by multiple, most, or all of the component images 502 in the set of component images.

[0081] Device 501A may adjust one or more component images in the set of component images 502 according to the common rendering intent to obtain an adjusted set of component images 504. This may involve converting one or more component images in the set of component images 502 from a current rendering intent to a common rendering intent. Converting from a current rendering intent to a common rendering intent may include: reversing one or more source adjustments to obtain a real rendered image (e.g., Figure 3 303 - 1 , 303 - 2 , 303 - 3 , and 303 - 4 ) and applying one or more common spatial adjustments to the reality rendered images to obtain an adjusted set of component images 504 (eg, Figure 3 The adjusted images 307-1, 307-2, 307-3 and 307-4).

[0082] In some embodiments, applying one or more common spatial adjustments may include one or more of the following: converting sensor values ​​to color values; estimating the capture environment ambient luminance and white point and applying white point correction based on the estimated capture environment ambient luminance and white point; estimating the capture environment ambient luminance and applying an optical-optical transfer function (OOTF) based in part on the estimated capture environment ambient luminance to prepare the image for rendering on a reference display device. In some embodiments, applying one or more common spatial adjustments may include: applying one or more of the following: saturation enhancement; contrast enhancement; single color saturation adjustment; slope-offset-power-Tmid enhancement; and tone curve fine-tuning. In some embodiments, applying one or more common spatial adjustments may include: converting the component images 502 from a pre-processed state to a common rendering intent.

[0083] Device 501A may use a compositing program to create a composite image 506 based on the adjusted set of component images 504 (eg, Figure 3 Device 501A may then encode composite image 506 using metadata 508 to create encoded multi-intent composite image 510. Metadata 508 may indicate a common rendering intent applied to the set of component images 502 with respect to creating composite image 506.

[0084] Device 501B may receive encoded multi-intent composite image 510 from device 501A and may decode encoded multi-intent composite image 510 to obtain composite image 506 and metadata 508, which may describe one or more common spatial adjustments applied when creating composite image 506. Device 501B may adjust composite image 506 to reverse the one or more common spatial adjustments to obtain realistically rendered composite image 512 (e.g., Figure 4 Then, device 501B may adjust the realistic rendered synthetic image 512 according to the target rendering intent to obtain a target-adjusted synthetic image 514 (e.g., Figure 4 The target-adjusted composite image 411 may be displayed, projected, saved (to storage), transmitted (to another device), or otherwise utilized 514. In some embodiments, the target rendering intent may be identified based on one or both of: input received via a user interface and information in a configuration file.

[0085] In some embodiments, adjusting the realistic rendered composite image 512 according to the target rendering intent may include: applying one or more target adjustments to the realistic rendered composite image 512. In some embodiments, applying one or more target adjustments may include: estimating the ambient luminance and white point of the captured environment; and applying white point correction based on the estimated ambient luminance and white point of the captured environment. In some embodiments, applying one or more target adjustments may include: estimating the ambient luminance of the captured environment; and applying an optical-optical transfer function (OOTF) based in part on the estimated ambient luminance of the captured environment to prepare the image for rendering on a reference display device. In some embodiments, applying one or more target adjustments may include: applying one or more of the following: saturation enhancement; contrast enhancement; single color saturation adjustment; slope-offset-power-Tmid enhancement; and tone curve fine-tuning.

[0086] Figure 6 An example method 600 for encoding a composite image according to aspects of the present disclosure is illustrated. The method 600 may represent a method that may be performed by Figure 5 The device 501A in the operating environment 500 and / or the operations performed by the example apparatus 800 described below according to some embodiments.

[0087] According to method 600, a set of component images may be obtained at 602 for use in creating a composite image. Figure 5 In operating environment 500, device 501A may obtain component images 502. At 604, a common rendering intent to be applied to the group of component images may be determined. Figure 5 In the operating environment 500 of , device 501A can determine a common rendering intent to be applied to component images 502. At 606, one or more component images in the set of component images can be adjusted according to the common rendering intent to obtain an adjusted set of component images. Figure 5 In operating environment 500 , device 501A may adjust one or more component images in component images 502 to obtain adjusted component image 504 .

[0088] At 608, a composite image may be created based on the adjusted set of component images. Figure 5 In operating environment 500, device 501A may create composite image 506 based on adjusted component images 504. At 610, metadata characterizing the common rendering intent may be generated. Figure 5In the operating environment 500 of , device 501A can generate metadata 508, which can characterize the common rendering intent applied to component images 502. At 612, the composite image and the metadata can be encoded to create an encoded multi-intent composite image. Figure 5 In operating environment 500 , device 501A may encode a composite image 506 and metadata 508 to create an encoded multi-intent composite image 510 .

[0089] Figure 7 An example method 700 for rendering a composite image according to aspects of the present disclosure is illustrated. The method 700 may represent a method that may be performed by Figure 5 The device 501B in the operating environment 500 and / or the operations performed by the example apparatus 800 described below according to some embodiments.

[0090] According to method 700, an encoded multi-intent composite image may be received at 702. For example, Figure 5 In the operating environment 500 of , device 501B can receive the encoded multi-intent composite image 510 from device 501A. At 704, the encoded multi-intent composite image can be decoded to obtain a composite image and metadata describing one or more common spatial adjustments applied when creating the composite image. For example, in Figure 5 In the operating environment 500 of , device 501B can decode the encoded multi-intent composite image 510 to obtain a composite image 506 and metadata 508, and metadata 508 can describe one or more common spatial adjustments applied when creating the composite image 506. At 706, one or more common spatial adjustments can be identified based on the metadata. For example, Figure 5 In operating environment 500 , device 501B may identify, based on metadata 508 , one or more common spatial adjustments applied when creating composite image 506 .

[0091] At 708, the composite image may be adjusted to reverse the one or more common spatial adjustments identified at 706, resulting in a realistically rendered composite image. Figure 5 In the operating environment 500 of , the device 501B can adjust the synthetic image 506 to reverse the one or more common spatial adjustments identified based on the metadata 508, thereby obtaining the realistic rendered synthetic image 512. At 710, the realistic rendered synthetic image can be adjusted according to the target rendering intent to obtain a target adjusted synthetic image. For example, in Figure 5 In the operating environment 500 of FIG. 5 , the device 501B may adjust the reality rendering composite image 512 based on the target rendering intent to obtain a target adjusted composite image 514. At 712, the target adjusted composite image may be displayed. For example, in Figure 5 In the operating environment 500, device 501B can render the target-adjusted composite image on a display.

[0092] Figure 8 800 is a block diagram showing an example of components of an apparatus 800 capable of implementing various aspects of the present disclosure. According to some aspects of the present disclosure, the apparatus 800 may have a Figure 5 According to some aspects of the present disclosure, the apparatus 800 may have the ability to implement operations performed by one or both of the devices 501A and 501B in the operating environment 500. Figure 6 Method 600 and Figure 7 The ability to perform operations associated with one or both of the methods 700. Figure 1 Sample, Figure 8 The types and quantities of the elements shown in are provided as examples only. Other embodiments may include more, less and / or different types and quantities of elements. According to some examples, device 800 may be configured to perform at least some of the methods disclosed herein. In some embodiments, device 800 may be or may include one or more components of a television, audio, video or multimedia system, a mobile device (such as a cellular phone), a laptop computer, a tablet device, an intelligent speaker or another type of device.

[0093] According to some embodiments, the apparatus 800 may be or may include a server. In some such examples, the apparatus 800 may be or may include an encoder. In some embodiments, the apparatus 800 may be a device configured for use within an audio, video, or multimedia environment, such as a home audio, video, or multimedia environment, while in other instances, the apparatus 800 may be a device configured for use in the "cloud", such as a server.

[0094] In this example, the device 800 includes an interface system 805 and a control system 810. In some embodiments, the interface system 805 can be configured to communicate with one or more other devices of an audio, video or multimedia environment. In some examples, the audio, video or multimedia environment can be a home audio, video or multimedia environment. In other examples, the audio, video or multimedia environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some embodiments, the interface system 805 can be configured to exchange control information and associated data with the audio, video or multimedia device of the audio, video or multimedia environment. In some examples, the control information and associated data can be related to one or more software applications being executed by the device 800.

[0095] In some embodiments, the interface system 805 may be configured to receive a content stream or to provide a content stream. The content stream may include audio, video, or multimedia data. The audio, video, or multimedia data may include, but may not be limited to, audio, video, or multimedia signals. In some instances, the content stream may include audio data, which may include spatial data, such as channel data and / or spatial metadata. In some examples, the content stream may include video data and audio data corresponding to the video data.

[0096] The interface system 805 may include one or more network interfaces and / or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). According to some embodiments, the interface system 805 may include one or more wireless interfaces. The interface system 805 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 805 may include a control system 810 and a memory system (e.g., Figure 8 815). However, in some cases, the control system 810 may include a memory system. In some implementations, the interface system 805 may be configured to receive input from one or more microphones in the environment.

[0097] For example, control system 810 may include a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0098] In some embodiments, the control system 810 may reside in more than one device. For example, in some embodiments, a portion of the control system 810 may reside in a device within one of the environments described herein, and another portion of the control system 810 may reside in a device outside the environment, such as a server, a mobile device (e.g., a smart phone or a tablet computer), etc. In other examples, a portion of the control system 810 may reside in a device within an environment, and another portion of the control system 810 may reside in one or more other devices of the environment. For example, a portion of the control system 810 may reside in a device (e.g., a server) that implements a cloud-based service, and another portion of the control system 810 may reside in another device (e.g., another server, a memory device, etc.) that implements a cloud-based service. In some examples, the interface system 805 may also reside in more than one device.

[0099] In some embodiments, the control system 810 can be configured to at least partially perform the methods disclosed herein. According to some examples, the control system 810 can be configured to implement methods of utilizing aggressive control parameters when training a machine learning model, utilizing aggressive control parameters in post-processing, and the like.

[0100] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may reside, for example, on a computer. Figure 8 810 and / or resides in the control system 810. Thus, various innovative aspects of the subject matter described in the present disclosure may be implemented in one or more non-transitory media having software stored thereon. For example, the software may include instructions for utilizing aggressiveness control parameters when training a machine learning model, utilizing aggressiveness control parameters in post-processing, and the like. The software may, for example, be implemented by a control system (e.g., Figure 8 The control system 810 may be implemented by one or more components of the control system 810).

[0101] In some examples, apparatus 800 may include Figure 8 Optional microphone system 820 shown in . Optional microphone system 820 may include one or more microphones. In some embodiments, one or more microphones may be part of or associated with another device (such as a speaker of a speaker system, an intelligent audio, video or multimedia device, etc.). In some examples, device 800 may not include microphone system 820. However, in some such embodiments, device 800 may still be configured to receive microphone data of one or more microphones in an audio, video or multimedia environment via interface system 810. In some such embodiments, a cloud-based implementation of device 800 may be configured to receive microphone data or noise indicators corresponding at least in part to microphone data from one or more microphones in an audio, video or multimedia environment via interface system 810.

[0102] According to some embodiments, the apparatus 800 may include Figure 8Optional loudspeaker system 825 is shown in FIG. Optional loudspeaker system 825 may include one or more loudspeakers, which may also be referred to herein as "speakers," or more generally as "audio reproduction transducers." In some examples (e.g., cloud-based implementations), device 800 may not include loudspeaker system 825. In some implementations, device 800 may include headphones. The headphones may be connected or coupled to device 800 via a headphone jack or via a wireless connection (e.g., Bluetooth).

[0103] According to some embodiments, the device 800 may include or be coupled to a display 830. The display 830 may include a display capable of displaying visual information, content, and / or effects (e.g., Figure 4 Examples of display 830 may include, but are not limited to, a television, a monitor, a laptop screen, a projector, and a touch screen.

[0104] With respect to the processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of these processes, etc. have been described as being performed in a particular ordered order, these processes can be practiced with the described steps performed in an order different from that described herein. It should further be understood that certain steps can be performed simultaneously, other steps can be added, or certain steps described herein can be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should in no way be construed as limiting the claims.

[0105] Accordingly, it should be understood that the above description is intended to be illustrative rather than restrictive. By reading the above description, many embodiments and applications other than the examples provided will be apparent. The scope should not be determined with reference to the above description, but should be determined with reference to the full scope of the appended claims and the equivalents enjoyed by these claims. It is expected and hoped that the technology discussed herein will develop in the future, and the disclosed systems and methods will be incorporated into such future embodiments. In a word, it should be understood that the present application is capable of modification and change.

[0106] All terms used in the claims are intended to be given their broadest reasonable interpretations and ordinary meanings as understood by those skilled in the art described herein, unless an explicit indication to the contrary appears herein. In particular, use of singular articles such as "a," "an," "the," and "said" should be understood to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0107] The abstract of the present disclosure is provided to allow the reader to quickly determine the nature of the technical disclosure. This abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the aforementioned detailed description, it can be seen that various features are combined together into various embodiments for the purpose of connecting the present disclosure as a whole. The method of the present disclosure should not be interpreted as reflecting the intention that the claimed embodiments incorporate more features than those explicitly recited in each claim. On the contrary, as reflected in the attached claims, the inventive subject matter lies in less than all the features of a single disclosed embodiment. Therefore, the attached claims are hereby incorporated into the detailed description, and each claim stands alone as a separately claimed subject matter.

[0108] The above systems and methods can provide encoding and rendering of multi-intent composite images.

[0109] Various aspects of the present invention may be understood from the following enumerated example embodiments (EEE):

[0110] EEE(1) A method for encoding a composite image, the method comprising: obtaining a set of component images of the composite image; determining a common rendering intent to be applied to the set of component images; adjusting one or more component images in the set of component images according to the common rendering intent to obtain an adjusted set of component images; creating the composite image based on the adjusted set of component images; generating metadata representing the common rendering intent; and encoding the composite image and the metadata to create an encoded multi-intent composite image.

[0111] EEE(2) A method as described in EEE(1), wherein adjusting one or more component images in the group of component images according to the common rendering intent includes: converting the component images in the group of component images from a current rendering intent to the common rendering intent.

[0112] EEE(3) A method as described in EEE(2), wherein converting the component images in the group of component images from the current rendering intent to the common rendering intent includes: reversing one or more source adjustments to the component images; and applying one or more common spatial adjustments to the component images.

[0113] EEE(4) The method of EEE(3), wherein applying the one or more common spatial adjustments to the component images comprises converting sensor values ​​into color values.

[0114] EEE(5) A method as described in EEE(3) or EEE(4), wherein applying the one or more common spatial adjustments to the component images includes: estimating the ambient light brightness and white point of the capture environment; and applying white point correction based on the estimated ambient light brightness and white point of the capture environment.

[0115] EEE(6) A method as described in any one of EEE(3) to EEE(5), wherein applying the one or more common spatial adjustments to the component images comprises: estimating the ambient light brightness of a capture environment; and applying an optical-optical transfer function (OOTF) based in part on the estimated ambient light brightness of the capture environment to prepare the image for rendering on a reference display device.

[0116] EEE(7) A method as described in any one of EEE(3) to EEE(6), wherein applying the one or more common spatial adjustments to the component images includes: applying one or more of the following: saturation enhancement; contrast enhancement; individual color saturation adjustment; slope-offset-power-Tmid enhancement; and tone curve fine-tuning.

[0117] EEE(8) A method as described in any one of EEE(1) to EEE(7), wherein adjusting one or more component images in the group of component images according to the common rendering intent includes: converting the component images in the group of component images from a pre-processing state to the common rendering intent.

[0118] EEE(9) A method as described in any one of EEE(1) to EEE(8), wherein adjusting one or more component images in the group of component images according to the common rendering intent comprises: converting a first component image in the group of component images from a first current rendering intent to the common rendering intent; and converting a second component image in the group of component images from a second current rendering intent to the common rendering intent.

[0119] EEE(10) The method as described in any one of EEE(1) to EEE(9), wherein determining the common rendering intent comprises: selecting a current rendering intent of a component image in the group of component images as the common rendering intent.

[0120] EEE(11) The method as described in EEE(10) further includes: selecting the current rendering intent of the component image as the common rendering intent based on the importance of the component image relative to other component images in the group of component images, wherein the importance of the component image is determined based on one or more of the following: the relative size of the component image in the rendering space of the composite image; the resolution of the component image; the centrality of the position of the component image in the rendering space of the composite image; and the point of focus of the viewer in the rendering space of the composite image.

[0121] EEE(12) The method of any one of EEE(1) to EEE(9), wherein determining the common rendering intent comprises: identifying a preferred rendering intent as the common rendering intent.

[0122] EEE(13) The method of EEE(12), further comprising: identifying the preferred rendering intent based on one or both of the following: input received via a user interface and information in a configuration file.

[0123] EEE (14) A device configured to perform the method as described in any one of EEE (1) to EEE (13).

[0124] EEE (15) A system comprising: the device as described in EEE (14) and a display.

[0125] EEE (16) A computer-readable medium having instructions stored therein, which, when executed by one or more processors, cause the one or more processors to perform the method as described in any one of EEE (1) to EEE (13).

[0126] EEE(17) A method for rendering a composite image, the method comprising: receiving an encoded multi-intent composite image; decoding the encoded multi-intent composite image to obtain: the composite image and metadata, the metadata describing one or more common spatial adjustments applied when creating the composite image; identifying the one or more common spatial adjustments based on the metadata; adjusting the composite image to invert the one or more common spatial adjustments to obtain a real-rendered composite image; adjusting the real-rendered composite image according to a target rendering intent to obtain a target-adjusted composite image; and displaying the target-adjusted composite image.

[0127] EEE(18) The method of EEE(17), wherein adjusting the reality-rendered composite image according to the target rendering intent comprises: applying one or more target adjustments to the reality-rendered composite image.

[0128] EEE(19) A method as described in EEE(18), wherein applying the one or more target adjustments to the realistically rendered composite image includes: converting sensor values ​​into color values.

[0129] EEE(20) A method as described in EEE(18) or EEE(19), wherein applying the one or more target adjustments to the realistically rendered synthetic image comprises: estimating the ambient light brightness and white point of the capture environment; and applying white point correction based on the estimated ambient light brightness and white point of the capture environment.

[0130] EEE(21) ​​A method as described in any one of EEE(18) to EEE(20), wherein applying the one or more target adjustments to the realistically rendered synthetic image comprises: estimating the ambient light brightness of a capture environment; and applying an optical-optical transfer function (OOTF) based in part on the estimated ambient light brightness of the capture environment to prepare the image for rendering on a reference display device.

[0131] EEE(22) A method as described in any one of EEE(18) to EEE(21), wherein applying the one or more target adjustments to the realistically rendered composite image comprises: applying one or more of the following: saturation enhancement; contrast enhancement; individual color saturation adjustment; slope-offset-power-Tmid enhancement; and tone curve fine-tuning.

[0132] EEE(23) The method as described in any one of EEE(17) to EEE(22), comprising: identifying the target rendering intent based on one or both of the following: input received via a user interface and information in a configuration file.

[0133] EEE (24) A device configured to perform the method as described in any one of EEE (17) to EEE (23).

[0134] EEE (25) A system comprising: the device as described in EEE (24) and a display.

[0135] EEE (26) A computer-readable medium having instructions stored therein, which, when executed by one or more processors, cause the one or more processors to perform the method as described in any one of EEE (17) to EEE (23).

[0136] Various features and advantages of the embodiments described herein are set forth in the following claims.

Claims

1. A method for encoding a composite image, the method comprising: Obtaining a set of component images of the composite image, the set of component images having been adjusted to apply different corresponding rendering intents; applying an inverse adjustment to the set of component images to undo a previously applied adjustment and obtain a corresponding realistic rendered image; determining a common rendering intent to be applied to the set of realistic rendered images; Adjusting one or more realistic rendered images in the set of realistic rendered images according to the common rendering intent to obtain an adjusted set of component images; creating the composite image based on the adjusted set of component images; generating metadata representing the common rendering intent; as well as The composite image and the metadata are encoded to create an encoded multi-intent composite image.

2. The method of claim 1, wherein: Adjusting one or more of the set of real-rendered images according to the common rendering intent includes converting component images of the set of component images from a current rendering intent to the common rendering intent.

3. The method of claim 2, wherein: Converting the component images in the set of component images from the current rendering intent to the common rendering intent comprises: inverting one or more source adjustments to the component images; and One or more common spatial adjustments are applied to the component images.

4. The method of claim 3, wherein: Applying the one or more common spatial adjustments to the component images includes converting sensor values ​​to color values.

5. The method according to claim 3 or claim 4, wherein: Applying the one or more common spatial adjustments to the component images includes: estimating a capture environment ambient luminance and white point; and applying a white point correction based on the estimated capture environment ambient luminance and white point.

6. The method according to any one of claims 3 to 5, wherein: Applying the one or more common spatial adjustments to the component images includes estimating capture environment ambient illuminance; and applying an optical-to-optical transfer function (OOTF) based in part on the estimated capture environment ambient illuminance to prepare the images for rendering on a reference display device.

7. The method according to any one of claims 3 to 6, wherein: Applying the one or more common spatial adjustments to the component images comprises applying one or more of: Saturation enhancement; Contrast enhancement; Individual color saturation adjustment; Slope-Offset-Power-Tmid Enhancement; and Tone curve fine tuning.

8. The method according to any one of claims 1 to 7, wherein: Adjusting one or more component images in the set of component images according to the common rendering intent includes converting component images in the set of component images from a pre-processed state to the common rendering intent.

9. The method according to any one of claims 1 to 8, wherein: Adjusting one or more component images of the set of component images according to the common rendering intent comprises: converting a first component image of the set of component images from a first current rendering intent to the common rendering intent; and A second component image in the set of component images is converted from a second current rendering intent to the common rendering intent.

10. The method according to any one of claims 1 to 9, wherein: Determining the common rendering intent includes: selecting current rendering intents of component images in the group of component images as the common rendering intent.

11. The method of claim 10, further comprising: A current rendering intent of the component image is selected as the common rendering intent based on an importance of the component image relative to other component images in the set of component images, wherein the importance of the component image is determined based on one or more of the following: the relative sizes of the component images in the rendering space of the composite image; the resolution of the component images; centrality of the positions of the component images in the rendering space of the composite image; and The point at which a viewer focuses in the rendered space of the composite image.

12. The method according to any one of claims 1 to 9, wherein: Determining the common rendering intent includes identifying a preferred rendering intent as the common rendering intent.

13. The method of claim 12, further comprising identifying the preferred rendering intent based on one or both of the following: input received via a user interface; and Information in the configuration file.

14. An apparatus, the apparatus being configured to perform the method according to any one of claims 1 to 13.

15. A system comprising the device as claimed in claim 14 and a display.

16. A computer readable medium having stored therein instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 13.

17. A method for rendering a composite image, the method comprising: receiving an encoded multi-intent composite image; decoding the encoded multi-intent composite image to obtain the composite image and metadata describing one or more common spatial adjustments applied in creating the composite image; identifying the one or more common spatial adjustments based on the metadata; adjusting the composite image to reverse the one or more common spatial adjustments to produce a realistically rendered composite image; Adjusting the realistic rendered synthetic image according to a target rendering intent to obtain a target-adjusted synthetic image; as well as The target-adjusted composite image is displayed.

18. The method of claim 17, wherein: Adjusting the reality-rendered composite image according to the target rendering intent includes applying one or more target adjustments to the reality-rendered composite image.

19. The method of claim 18, wherein: Applying the one or more target adjustments to the reality-rendered composite image includes converting sensor values ​​to color values.

20. The method of claim 18 or 19, wherein: Applying the one or more target adjustments to the reality-rendered composite image includes: estimating capture environment ambient luminance and white point; and applying white point correction based on the estimated capture environment ambient luminance and white point.

21. The method according to any one of claims 18 to 20, wherein: Applying the one or more target adjustments to the reality-rendered synthetic image includes: estimating capture environment ambient illuminance; and applying an optical-to-optical transfer function (OOTF) based in part on the estimated capture environment ambient illuminance to prepare the image for rendering on a reference display device.

22. The method of any one of claims 18 to 21, wherein: Applying the one or more target adjustments to the realistically rendered composite image includes applying one or more of: saturation enhancement; contrast enhancement; individual color saturation adjustment; slope-offset-power-Tmid enhancement; and tone curve fine-tuning.

23. The method of any one of claims 17 to 22, comprising identifying the target rendering intent based on one or both of: input received via a user interface and information in a configuration file.

24. An apparatus configured to perform the method according to any one of claims 17 to 23.

25. A system comprising: The device and display as claimed in claim 24.

26. A computer readable medium having stored therein instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 17 to 23.

Images