Method and system for perceptually meaningful spatial content synthesis - Patents.com

Abstract

Techniques are described for generating metadata for content to be composited and rendered using the generated metadata. These techniques can be used in conjunction with the development and delivery of one or more web pages or other graphical user interfaces. For example, one can gather content (e.g., images, animations, text, and user interface elements) to be composited together into a web page and call a set of APIs to generate metadata for the content of the web page to be composited. A metadata generation system receives the calls through the APIs and generates the metadata. The web page can then be delivered with the generated metadata that can be used to create a representation of the web page with content that is perceptually modified based on metadata about individual elements on the web page and their spatial proximity.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of priority to European Patent Application No. 22165450.2, filed March 30, 2022, and U.S. Provisional Patent Application No. 63 / 362,170, filed March 30, 2022, the entire contents of each of which are incorporated by reference.

[0002] [Technical field] The present disclosure relates to the field of content creation and content presentation on data processing systems such as computers, smart phones, televisions and other electronic devices. [Background technology]

[0003] Content creation for film and video content has used techniques that utilize metadata to adjust the appearance of the content based on the metadata during presentation (e.g., display) of the content. Dolby Vision is an example of these techniques. In these techniques, content is created and presented for frames. The content can be considered linear content in that one frame follows another over time. The frames are displayed sequentially over time and are not composited into a single image or displayable unit. Any metadata used during presentation to modify the presentation of the content is created for a particular single frame or series of frames over time.

[0004] US 2015 / 143228 discloses a method for accelerating loading of a document. Resources of the document are requested from a first source and metadata of the document is requested from a second source different from the first source. The requested metadata is received from the second source and the requested resources are received from the first source. A first representation of the document based on the received metadata is provided for display. After providing the first representation, a second representation of the document is generated combining a portion of the first representation with a further portion of the document, and the second representation is provided for display.

[0005] US Patent Application Publication No. 2017 / 031878 discloses a method for displaying a web page. Metadata describing the web page is received. The metadata defines what the web page would look like without the content of the web page, the metadata defines a group of objects in the web page, the objects in the group of objects having a function to satisfy a policy of a political unit. Content required for the web page is identified based on the metadata. The content of the web page is obtained. The web page is created using the metadata and the content. The web page is displayed in a graphical user interface on a display system, enabling a reduction in resources in a web server used to display the web page, enabling a reduction in resources used to display the web page.

[0006] US 2021 / 051365 discloses an apparatus for improved rendering that receives multiple input content sources and includes multiple processing channels for processing the input content. A combiner combines the processed input content to generate a composite output signal. An output adaptation block adapts the composite output signal with dynamic metadata for display by a display device. Each processing channel includes a statistics generator and an input adaptation block. Summary of the Invention

[0007] The invention is defined by the independent claims. The dependent claims relate to optional features of some embodiments. Various techniques are described for generating metadata for content to be composited and for rendering the composited content using the generated metadata. These techniques can be used, for example, in conjunction with the development and delivery of one or more web pages. In one embodiment, a web page developer can collect content to be composited together in a web page (e.g., HDR or SDR images and HDR or SDR animations as well as text and user interface elements) and call a set of APIs to generate metadata for the content of the web page to be composited. A metadata generation system receives the calls through the APIs and generates the metadata. The web page can then be delivered (e.g., to a web browser) with the generated metadata, which can be used (e.g., by the web browser or other content delivery system at render time) to create a representation of the web page with perceptually modified content based on metadata about individual elements on the web page and their spatial proximity. These techniques can also be used for other types of displayable units.

[0008] In one embodiment, a method according to one aspect can include the acts of receiving a set of elements including content and data representing a position or positioning rule of each of the elements within the set of elements on a displayable unit, determining one or more types of content within each of the elements, generating a set of metadata from the set of elements, the set of metadata being used in creating composite content from the set of elements on the displayable unit when the composite content is displayed, the set of metadata including (1) spatial data about the elements and (2) image metadata about at least some of the elements, and storing the generated set of metadata with an association to each of the elements. In one embodiment, the displayable unit is one of: (a) a page, sheet, folio, or other unit of content, or (b) a web page, or (c) part or all of a screen of a display device, or (d) content at least partially generated by a computer program. In one embodiment, the set of metadata also describes temporal changes of the content over time. In one embodiment, the set of metadata is stored in a scalable vector graphics format, such as a format supporting vector graphics of images. In one embodiment, the display device may be a planar display device or a non-planar device (such as a display device in an augmented reality or virtual reality headset). In one embodiment, the method may further include an act of transmitting the set of elements and the generated set of metadata in response to a request for a web page, where data representing positions or positioning rules are included in a description of the displayable unit in a hypertext markup language. The positioning rules may include rules such as, for example, data related to a window of an application (e.g., a browser window), window scaling data, or data related to a display device, or instructions or other data defining how to determine a position of an element from a combination of all or a subset of such data.

[0009] In one embodiment, the content of the elements can be different types of content. For example, the one or more types of content can include at least one of: (a) high dynamic range (HDR) image content, (b) standard dynamic range (SDR) image content, (c) text content, or (d) user interface content for use in receiving input from a user. The image content can be specified by pixel data (a bitmap representation of an image) or by vector graphics data. In one embodiment, the spatial data can include vector-based spatial data that defines approximate boundaries on displayable units of each of the elements in the set of elements.

[0010] In one embodiment, the image metadata may include color volume characteristics or image statistics for at least some of the elements in the set of elements, hi one embodiment, the image statistics may be one or more of a maximum luminance of the image, a minimum luminance of the image, an average luminance of the image, or a median luminance of the image.

[0011] In one embodiment, the image metadata may include data regarding the detected glare or that glare may be detectable in at least one of the elements in the set of elements (depending on the viewing environment), and the detected glare may be detected using one or more glare models. In one embodiment, the detected glare may be classified as one of disability glare or discomfort glare.

[0012] In one embodiment, the image metadata may include a texture abstraction of at least one of the elements in the set of elements. The texture abstraction may be a mathematical representation of the image based on noise attributes and vector attributes. For example, the texture abstraction may be derived from a Fourier analysis based representation of at least one of the elements in the set of elements. In one embodiment, the image metadata may include a quantized representation of at least one of the elements in the set of elements.

[0013] In one embodiment, an application programming interface (API) is used to generate the set of metadata, the API linking a metadata generation component in a data processing system with web page creation software that can be called by the web page creation software to generate the set of metadata from a selected set of content that will be composited together to create a composite web page.

[0014] In one embodiment, a method according to the compositing aspect can include the acts of receiving a set of metadata for use in creating composite content from a set of elements to be composited together for display in a displayable unit, the set of metadata including (1) spatial data for the elements and (2) image metadata for at least some of the elements; processing the set of metadata to determine how to modify one or more of the elements based on the set of metadata; modifying one or more of the elements based on the set of metadata; and rendering the composite content having the modified one or more elements to display the displayable unit on a display device. In one embodiment, the receiving can be in response to a request for a web page, and the set of metadata is received by a web browser. In one embodiment, the modifying based on the set of metadata can be performed (a) before compositing the content, (b) after compositing the content, or (c) before and after compositing the content. For example, one set of elements can be modified before compositing and a subset of the modified elements can be modified again after compositing. The modifying can be based at least in part on the spatial data and the image metadata. In one embodiment, the spatial data may include vector-based spatial data defining approximate boundaries on a displayable unit of each of the elements in the set of elements, the approximate boundaries being used to calculate a distance between at least two elements in the set of elements. In one embodiment, the method may determine a distance on a displayable unit between a first element and a second element in the set of elements, and determine a difference in image metadata statistics (e.g., average luminance values) between the first element and the second element, and then the method may modify one or both of the first element or the second element based on the determined distance and the determined difference. The image metadata may include data regarding detected glare (or data that glare can be detected) that may cause the synthetic content to be modified to reduce the detected glare.

[0015] In one embodiment, the modification may take into account one or more of: (a) the on-screen and off-screen conditions of the content in the displayable unit, (b) the display device used to display the content in the displayable unit, (c) the surrounding viewing environment surrounding the display device displaying the displayable unit, or (d) the viewing distance of a viewer of the displayable unit. For example, a viewer's field of view based on viewing distance may be used to determine when an element is within the field of view and may need to be modified due to large differences in image metadata statistics within the same field of view (e.g., a dim image next to a bright image).

[0016] The aspects and embodiments described herein may include a non-transitory machine-readable medium capable of storing executable computer program instructions that, when executed, cause one or more data processing systems to perform the methods described herein. The instructions may be stored in a non-transitory machine-readable medium, such as dynamic random access memory (DRAM), which is a volatile memory, or a non-volatile memory, such as flash memory or other types of memory. The aspects and embodiments described herein may also be in the form of a data processing system that is constructed or programmed to perform these methods. For example, the data processing system may be constructed with hardware logic to perform these methods or may be programmed with a computer program to perform these methods.

[0017] The above summary does not include an exhaustive list of all embodiments and aspects of the present disclosure, and all systems, media and methods may be implemented from any suitable combination of the various aspects and embodiments summarized above, as well as those disclosed in the following detailed description. [Brief description of the drawings]

[0018] The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference symbols refer to similar elements and in which: [Figure 1A] 1 illustrates an example of a displayable unit (eg, a web page) that contains content that is composited or assembled from separate content elements such as separate images and text as well as other types of content. [Figure 1B] 1 illustrates an example of a content creation and delivery system, such as a system that creates and delivers a set of web pages from one or more web servers, according to one embodiment. [Figure 2A] 4 is a flow chart illustrating a method for creating metadata for content to be composited and rendered based on the created metadata, according to one embodiment. [Figure 2B] 1 is an example of a content creation system that creates metadata for content to be composited and rendered based on the created metadata, according to one embodiment. [Figure 2C] FIG. 2C illustrates an example method for creating metadata for a displayable unit (e.g., a web page) and then compositing and rendering the displayable unit according to one embodiment; the method in FIG. 2C may be performed by several different data processing systems or by a single data processing system. [Figure 3A] 1 is a flow chart illustrating a method for generating spatial metadata and image statistical metadata according to one embodiment. [Figure 3B] 1 is a flowchart illustrating a method for generating spatial metadata, according to one embodiment. [Figure 4A] 1 illustrates an example of creating a texture abstraction of an image (of a flower in front of a blurred background) according to one embodiment. [Figure 4B] 4 illustrates an example of quantization and segmented abstraction of an image according to another embodiment. [Figure 5A] 5 is a flowchart illustrating a method for compositing and rendering a displayable unit using generated metadata, according to one embodiment. [Figure 5B] FIG. 1 is a flow diagram illustrating a method for compositing and rendering a displayable unit using pre-generated metadata and metadata generated at rendering time by a device performing the compositing, according to one embodiment. [Figure 5C] 1 illustrates an example method for compositing and rendering a displayable unit that includes composite content, according to one embodiment. [Figure 6A] 13 illustrates an example for applying generated metadata when compositing, according to one embodiment. [Figure 6B] 13 illustrates an example for applying generated metadata to compensate for glare detected in a perceptually meaningful way during compositing, according to one embodiment. [Figure 6C] 13 illustrates another example of applying generated metadata to modify content during compositing, according to one embodiment. [Figure 6D] 13 illustrates an example of applying generated spatial and temporal metadata to modify content during composition of time-varying displayable units according to one embodiment. [Figure 7] 1 illustrates an example of a data processing system that can be used to implement one or more of the embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Various embodiments and aspects are described with reference to the details discussed below, and the accompanying drawings illustrate various embodiments. The following description and drawings are illustrative and should not be construed as limiting. Numerous specific details are set forth in order to provide a thorough understanding of the various embodiments. However, in certain instances, well-known or conventional details are not set forth in order to provide a concise discussion of the embodiments.

[0020] References herein to "one embodiment" or "embodiment" mean that a particular feature, structure, or characteristic described with an embodiment can be included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in this specification do not necessarily all refer to the same embodiment. The processes illustrated in the following figures are performed by processing logic including hardware (e.g., circuits, dedicated logic, etc.), software, or a combination of both. Although the processes are described below in terms of some sequential operations, it should be appreciated that some of the described operations may be performed in different orders. Additionally, some operations may be performed in parallel rather than sequentially.

[0021] The embodiments described herein can create metadata for content elements that are to be assembled or composited into a displayable unit, and when a data processing system composites content into the displayable unit, the data processing system can modify that content based on the created metadata such that the composite can be considered to adjust the appearance of at least some of the content of the entire displayable unit. A displayable unit can be, for example, a web page that is composited at display time (e.g., on a client web browser) from different types of content, often from different sources. The content is composited into a spatial arrangement on the displayable unit based on data (e.g., HTML or CSS in the web page) that defines the spatial arrangement. Content modification can take this spatial arrangement into account and modify the content based in part on the spatial arrangement and in part on image metadata or other content metadata. Figure 1A shows an example of a displayable unit that is a web page. Web page 10 includes image 12, image 14, text 18, and a user interface panel 16 that includes selectable commands that a user can select to cause the web page to perform an action. Image 12 can have different types of content. For example, these can be high dynamic range (HDR) or standard dynamic range (SDR) image content (or video content, which may be either HDR or SDR). Images may be stored or represented in a pixel data format (often in a compressed form) or may be represented by vector graphics data (and thus may be referred to as vector graphics-based images). Text is resizable.Although this disclosure often uses web pages as examples of displayable units, the embodiments described herein can work with other types of displayable units, such as (a) pages, sheets, folios, or other units of content, or (b) part or all of the screen of a display device, or (c) content that is generated at least in part by a computer program or electronic device. The display device can be a computer display or a smartphone display or a TV display, a display in a virtual reality or augmented reality headset, or other known and future display devices, and may use a flat display screen (e.g., LCD or OLED) or a curved or non-flat display.

[0022] In many embodiments, the displayable units are created by one or more content creators and then distributed to users operating data processing systems (desktop computers, other types of computers, smart phones, wearable devices, gaming systems, entertainment systems such as televisions, consumer electronic devices, etc.) to view and interact with the displayable units. FIG. 1B shows an example of such distribution that may be used for web pages. One or more content creators 30 may create one or more web pages to be distributed by one or more content delivery systems 32, which may be conventional web servers that transmit the web pages to web browsers (on client systems) that request the web pages. The one or more content delivery systems 32 may be coupled to one or more networks 34 and may receive requests for web pages from data processing systems 36 and 37 that are also coupled to the one or more networks 34. The data processing systems 36 and 37 may be computers or other devices that run web browsers (or other content consumption systems). These web browsers may request web pages through the network 34 and receive web pages from the network 34. A web page may include traditional content (e.g., images, text, user interface (UI) elements, etc.) and may also include metadata as described herein for use during composition of the web page. In one embodiment, one or more content creators 30 may create displayable units with such metadata using the method illustrated in Figure 2A and the system illustrated in Figure 2B.

[0023] The method shown in FIG. 2A may begin at operation 101 by receiving a collection of elements (e.g., images 12 and 14 and text 18 and UI 16 in FIG. 1A, or content elements 151, 153, 155, and 157 in FIG. 2C) to be composited into a single displayable unit, and by receiving data regarding their spatial arrangement within the displayable unit (e.g., two-dimensional positioning or arrangement / layout or other positioning information (e.g., positioning rules by which final positions are determined)). The spatial arrangement data may specify actual positions or positioning rules (e.g., CSS layout rules) that are dynamically used at display time (e.g., based on data regarding the application's window, scaling data, and display device data regarding the size of the display) to determine the actual positions. In one embodiment, the spatial arrangement may be described in HTML (Hypertext Markup Language) or a similar markup language format, or CSS (Cascading Style Sheets), or the like. In some embodiments, the method may include operation 103 for classifying different types of content. This may be necessary so that each different type of content can be processed according to its type. For example, an HDR movie may require the generation of temporal metadata (described below), while a displayable unit that contains only content that is static with respect to time would not require temporal metadata. Operation 103 may not be required if the type is specified by data associated with the displayable unit (e.g., the HTML of a web page specifies the type of content for different content elements).

[0024] In operation 105 in FIG. 2A, the method may determine one or more boundaries associated with each content element. For example, as described further below, the data processing system may trace the perimeter of one or more elements (or the content within each of the one or more elements) to generate spatial metadata. This spatial metadata may be vector-based metadata that defines location data using a vector format (e.g., a set of x,y Cartesian coordinates, such as a pair of Cartesian coordinates). In certain embodiments, it may be necessary for the perimeter of at least some content elements to be traced when the received data associated with the content elements (from operation 101) does not define a perimeter. In one embodiment, operation 105 may define a perimeter of a portion of content within a content element (such as a portion of an image within an image). These different portions may be classified by using a quantized threshold (e.g., a threshold on brightness values), as described further below, to segment the content within a content element, such as a single image.

[0025] In operation 107 in FIG. 2A, the method may generate image statistical metadata. In one embodiment, operation 107 is performed before operation 105 and may be used to segment different portions of each image (to segment different portions based on quantized threshold luminance values). The generated image statistical metadata may be used to determine how to modify each image based on the image statistical metadata and the spatial metadata, as described further below. Operation 107 may also optionally include generating one or more abstractions of the image. Examples of such abstractions (e.g., texture and quantized abstractions) are provided below. In one embodiment, the abstractions may be mathematical descriptions of the image that do not retain data for each pixel and have a significantly reduced storage size relative to the pixel data of the full image. These abstractions may also be used to determine how to modify each image based on the abstractions and the spatial metadata.

[0026] In operation 109 in FIG. 2A, the method can store the content elements and associated metadata (e.g., extracted from the elements and their spatial and temporal arrangement) for use during composition when rendering a displayable unit including all of the content elements is displayed on a client system. Then, in operation 111, the components of the displayable unit (including the content elements and associated metadata and other data such as HTML) can be transmitted to one or more other data processing systems. These components can then be used in such data processing systems to compose (e.g., assemble the components into a single web page) a displayable unit (e.g., a web page) and render it on a display device such as an organic light emitting diode (OLED) display device. The display device can be either a planar device or a non-planar device (e.g., a spherical screen).

[0027] An example of a system for performing the method in FIG. 2A is shown in FIG. 2B. The system in FIG. 2B is based on a web page embodiment where the displayable units are web pages. The system includes a web page creation system 125 coupled to a content storage 131. The content storage stores a collection of content items on the displayable units and the HTML of the web pages, and may also store extracted metadata to be used when the displayable units are composited. The web page creation system 125 may be based on a conventional web page authoring software package known in the art. The system also includes a metadata extraction processing system 129 that may perform one or more of the embodiments and methods described herein (e.g., such as the method shown in FIG. 2A) to generate metadata to be used during composition. The metadata extraction processing system 129 may operate with the web page creation system 125 through an application programming interface (API) 127 that allows the two different software components (web page creation system 125 and metadata extraction processing system 129) to operate and communicate to allow the metadata extraction processing system to generate metadata for each content element for use during composition. Once such metadata is generated, it may be stored as metadata 133. In one embodiment, the stored metadata 133 may be stored with a tag that associates each metadata component with an appropriate content element. In other words, the tag of a metadata component may specify the identity of the content element described by or associated with the metadata component. The metadata may also or alternatively be stored with the content (e.g., attached to its corresponding content).2B allows a content creator to call the metadata extraction processing system 129 through a call in an API, which can then generate metadata for each content element using the methods described herein, without requiring the content creator to manually generate the metadata. This simplifies the process of creating a web page that can be composited using the metadata described herein. A call through the API can enable the generation of the metadata once all of the content elements have been selected and placed / arranged on the web page.

[0028] A further example of a method for generating metadata during creation of a displayable unit is shown in FIG. 2C. This example includes four different types of content elements including PQ (perceptual quantization) encoded content element 151, Hybrid Log Gamma encoded HDR content element 153, SDR content element 155, and text element 157. Other types of content elements such as vector graphics or geometric shapes / regions and backgrounds may also be included in the displayable unit, and these other types of content elements may be processed as described herein. These content elements may be composited together to create a web page that is displayed in a web browser on a client data processing system after the client system downloads the components of the web page from one or more servers. In operation 159, a metadata extraction process generates metadata from each individual element of content using one or more embodiments described herein. The metadata extraction process may be performed by system 129 shown in FIG. 2B. In an exemplary embodiment, operation 159 analyzes each image using image statistics to generate image metadata for the image. This analysis creates metadata, although alternative embodiments may use metadata already associated with the image (so the metadata exists with the original image). The generated metadata may include vector-based spatial metadata for each content element and image statistics for each content element, and may also include spatial metadata with image statistics within each content element. Additionally, an abstraction of each image may also be created as further metadata for each image. The metadata extraction process in operation 159 generates extracted metadata 161 that can be stored with the content. In one embodiment, the metadata is tagged with an identifier that identifies the associated content described by the metadata. After the extracted metadata 161 is stored, a data processing system (e.g., a client data processing system having a web browser) may perform a compositing and rendering process 163 based on the extracted metadata 161.These compositing and rendering processes 163 may perform methods described herein to modify, composite, and render a collection of content elements based on metadata 161. These compositing and rendering processes 163 may then generate composited and rendered content elements in a displayable unit 165 that includes the modified content elements. The displayable unit may display the content elements after they have been modified based on the metadata 161 and viewing settings (e.g., number and type of monitors) and ambient viewing environment (e.g., dark or light viewing environment). Further disclosure of the aspects of FIG. 2C is provided below with respect to compositing and rendering operations.

[0029] An embodiment of a metadata generation method is shown in FIG. 3A. In this method, spatial metadata and image statistical metadata can be generated and encoded for each content element, such as an image (e.g., an SDR image), in a displayable unit (e.g., a web page). In operation 201, a data processing system can open an image or frame of content corresponding to the content element in the displayable unit. For example, a metadata processing component (system 129 in FIG. 2B) can open the content element and then process the content element to extract metadata. In operation 203, the data processing system can calculate image statistics for the image (or frame). For example, in one embodiment, the system can calculate luminance data (e.g., average luminance, median luminance, maximum luminance, minimum luminance, etc. based on the image metadata of the content element) and other image metadata statistics, and can calculate spatial data for regions within the content element as described herein. These calculated image statistics can be used to segment the image into different regions or segments based on the calculated statistics. In one embodiment, each segment may be based on a quantization level of luminance, such that consecutive pixels having a level of luminance or brightness (L) values ​​within each quantization bucket are classified within a segment (e.g., one quantization level for pixels having L less than 0.1 nit, another quantization level for pixels having L greater than 600 nit, and another quantization level for pixels having L between these two levels). The expression for L may also be expressed as cd / m 2 The seg- ment may be expressed in SI units of 100 kb / s. Each segment can be considered as being defined by a pixel threshold boundary. Numerous techniques for segmenting an image of pixels are known in the art. See, for example, published PCT application PCT / US2020 / 062942 (WO2021 / 113400).

[0030] Once the regions have been identified, their boundaries can be determined. For example, in operation 205, the data processing system can determine the boundaries of each segmented region in the image by tracing the contour or perimeter of the region in the image. The tracing can generate a set of vector-based spatial metadata that defines the contour or perimeter of each segmented area, and image statistics for each segment can also be included with this vector-based spatial metadata. Contours can be traced by identifying edges (that define the perimeter) and connecting them with lines. Points at each end of the lines describe the vectors. Alternatively, piecewise curve fitting according to pixel threshold boundaries can be used. Techniques for performing this piecewise curve fitting are known in the art. For example, the following literature describes curve fitting: Michael Plass & Maureen Stone. 1983. Curve fitting with piecewise parametric cubics. SIGGRAPH Comput. 17, 3 (July 1983), 229 239. DOI:https doi.org / 10.1145 / 964967.801153, Wenping Wang, Helmut Pottmann, and Yang Liu. 2006. Fitting B spline curves to point clouds by curvature based squared distance minimization. ACM Trans. Graph. 25, 2 (April 2006), 214-238. DOI:https doi.org / 10.1145 / 1138450.1138453, Gerald Farin. 1993. Curves and surfaces for computer aided geometric design (3 rded.): a practical guide. Academic Press Professional, Inc. In an alternative embodiment, at least some of the segments or regions can be selected manually. Once the contour or perimeter is defined, the method embodiment in FIG. 3A may attempt to reduce the size of the spatial metadata in operation 207. In one embodiment, this can be done by reducing the number of nodes along the perimeter (e.g., combining two lines into one line that approximates the two lines). Once the spatial metadata and image statistics are generated, in operation 209, they can be encoded and stored along with an association to the corresponding image for which the metadata was created. The association can be a tag that identifies the image. The metadata may be stored separately from the image and include the tag so that the compositing process can find the appropriate metadata for the image. In one embodiment, the metadata can be stored as a markup language or in a scalable vector graphics format (already supported by web browsers and GPU renderers).

[0031] 3B shows another example of how an image, which is one of the content elements to be synthesized, can be processed to generate metadata such as spatial metadata and image statistics. A source image is received in operation 221 and converted to a black and white image 223. The image can then be quantized in operation 225 to generate a set of segments based on the quantization (e.g., luminance quantization as described above). Then, in operation 227, the metadata processing system can trace each of the segments to generate a set of vectors that describe the boundary or perimeter of each segment. Then, in operation 229, the metadata processing system can reduce the spatial metadata and store the spatial metadata in operation 231.

[0032] Each image in the set of content elements may be described by an abstraction that can be processed when compositing displayable units. The abstraction can provide a statistical representation of the image for purposes of compositing and rendering (for perceptually meaningful content modification based on the composited displayable units) while requiring much less storage space than a stored version of the image. The abstraction can be a mathematical representation that describes the image without requiring individual pixel-by-pixel data. Figures 4A and 4B provide two examples of such abstractions for images that can be used in one or more embodiments.

[0033] Image abstraction can use texture or noise texture, which can successfully capture data that is sufficient for synthesis and still has a small storage (and transmission) "footprint". The example shown in FIG. 4A uses noise texture abstraction to generate an image abstraction that can be saved as metadata (called abstraction metadata) used during the synthesis and rendering process. This abstraction metadata can then be processed during synthesis without using the actual image to make decisions about how to modify content (within the image or other content around the image) during synthesis and rendering. In one embodiment, each segment or region of an image can be associated with a mathematically defined texture characteristic (e.g., defined by 1 / f noise, where f is the frequency of the pixel data). Examples of noise textures can be created based on techniques and methods described in the following published literature: Kunkel, T. and Daly, S. (2020), 57-1: Spatiotemporal Noise Targets Inspired by Natural Imagery Statistics. SID Symposium Digest of Technical Papers, 51:842-845. (see https: / / doi.org / 10.1002 / sdtp.14001). In one embodiment using these techniques, a random seed image is selected (or a varying random seed is selected), and then a process based on the selected alpha (α) generates frequency components that generate different two-dimensional noise texture images. In one embodiment, the metadata processing system may select a larger alpha value (e.g., α=4 or 5) for HDR images than for non-HDR images. Natural images (e.g., images of forests or river valleys) can benefit from an alpha value in the range of 2-3. The alpha value affects the frequency distribution of the noise.

[0034] As shown in FIG. 4A, an image 251, which is one of the content elements to be composited into a displayable unit such as a web page, can be processed into a final texture abstraction 269 using a noise texture abstraction process 257. The final texture abstraction 269 can provide a more refined representation of the image than the simple vector representation 255 produced by the simple vector abstraction process 253 using a threshold map as described above in connection with FIGs. 3A and 3B. The noise texture abstraction process 257, in one embodiment, can start with a vector-based mask (from the vector abstraction process 253) so that different parts of the image can be processed separately based on the vector-based mask using the noise texture abstraction process 257. The noise texture abstraction process 257 can start with an analysis 259 that derives a set of data used in the noise texture abstraction process 257. This data can include luminance data (e.g., minimum and maximum L), texture distribution data, object position data (e.g., positions of significant objects when positions are represented as vectors), data regarding dominant colors, and image boundary data. The noise texture abstraction process 257 may then generate a noise attribute list and a vector attribute list 267 in operation 261. The noise attribute list may identify noise textures, such as noise textures 263 and 265, that have been created as representing one or more portions of the image. The vector attribute list may identify respective locations of the noise textures to be used to represent one or more portions of the image. These noise textures may be based on the segments of the image that they are to replace, such that, for example, the minimum and maximum intensities of the noise textures are approximately the same as the minimum and maximum pixel values ​​in the segment of the image that will be represented by the noise textures in the generated metadata. Thus, the texture abstraction may include multiple different noise textures that are combined together to represent the entire image.The storage (and transmission) size of a texture abstraction of an image can be significantly smaller than the actual image, and can be processed more quickly during synthesis and rendering. Furthermore, the texture abstraction can be tuned to provide enough data (but not too much data) for the synthesis and rendering process.

[0035] Another approach to creating an image abstraction is to divide the image into segments and quantize each of the segments based on image metadata within each segment. For example, the average luminance value within each segment can be used along with a set of thresholds (e.g., three thresholds: less than 0.1 nit, greater than 0.1 nit and less than 600 nit, and greater than 600 nit) to classify each segment into one of the quantization levels. This approach is illustrated in FIG. 4B. A source image 301 is one of the content elements to be composited and rendered into a displayable unit such as a web page. The source image 301 is segmented into segments such as segments 305, 307, and 309, and then each segment is quantized to one of the different luminance levels. A noise abstraction texture can then be applied to each segment such as segments 305A, 307A, and 309A. These noise abstraction textures can be based on the approach described in connection with FIG. 4A. Although the segments in FIG. 4B are rectangular, in one or more embodiments, segments of other shapes can be used. For example, the shape of each segment may be defined by an arbitrary perimeter based on image statistics as described above (see, eg, the description of operation 203 above).

[0036] The examples shown in Figures 3A, 3B, 4A and 4B can create spatially local metadata in an image. This spatially local metadata can specify the location of each segment (and its boundaries) in the image and can also specify one or more image statistics describing each segment (e.g., average luminance, maximum luminance, minimum luminance, color-related properties such as dominant colors, color statistics, etc.). The spatially local metadata in an image can be helpful when a designer adjusts an image, for example, by cropping the image. Consider the case of an image that contains some dark content (e.g., a forest) and a sun that is normally very bright (e.g., 10,000 nits). For example, the image may be a photo of a forest with the sun above the forest on a bright, cloudless day. If the image is cropped to exclude the sun (e.g., either during the process of designing a displayable unit such as a web page, or during compositing when screen resizing or dynamic web layout or animation causes cropping), the spatially local metadata can enable a system (e.g., a compositing and rendering system) to use only the spatially local metadata in the cropped image (that excludes the sun). This selection of only spatially local metadata in the cropped image will likely change the maximum luminance value and cause a change in how the cropped image (or other images near the cropped image) are modified during composition and rendering. In one embodiment, metadata nodes (having such spatially local metadata) may be spatially distributed (e.g., in a grid) across the entire image to ensure proper coverage of the image. Such metadata may include a description of the spatial tone distribution within the image. In one embodiment, the method described in U.S. Patent No. 10,977,764 (assigned to Dolby Laboratories Licensing Corporation) may be used for such distributed metadata nodes.

[0037] Once the metadata for the compositing and rendering process has been created and stored (e.g., in act 109 of FIG. 2A), the content elements and metadata can be used by a data processing system that composes and renders a displayable unit. FIG. 5A provides a simplified version of a method that may be performed by such a data processing system. In act 351, the data processing system receives the content elements of the displayable unit and may also receive the generated metadata, such that the content elements can be composited in a perceptually meaningful manner based on the metadata. In act 353, the data processing system analyzes the metadata to determine how to modify one or more of the content elements. This analysis may include calculating distances between portions of the content elements and analyzing the metadata taking these distances into account. The calculated distances may be compared to a distance threshold based on a user's field of view. If the distance exceeds such field of view, no modification may be required. In act 355, the data processing system modifies one or more of the content elements. As described further below, the modification may be before the content elements are composited, or after they are composited, or both before and after. The modification may be based in part on the spatial arrangement of the content elements on the displayable unit, and may be based in part on image metadata and temporal metadata. Once the displayable unit has been composited and rendered to a frame buffer, it may be displayed in act 357.

[0038] 5B and 5C provide a more detailed example of a method and pipeline for compositing and rendering a displayable unit based on generated metadata (e.g., generated metadata stored in operation 109 in FIG. 2A). Referring now to FIG. 5B, for content elements that have previously generated associated metadata, the method reads the stored metadata in operation 402. The example in FIG. 5B includes a situation in which at least one of the content elements lacks previously generated metadata. That content element can be processed during the compositing process in operation 401 to generate metadata. Operation 401 can include any one or more of the above methods for generating metadata, such as the methods shown in any one of FIGs. 2A, 2C, 3A, 3B, 4A, or 4B. All of the metadata can then be provided to operation 403, which analyzes a set of inputs including the metadata (from 401 and 402), the content in the content elements, and a layout description 407 (e.g., HTML or CSS or other format describing the arrangement of various content elements on a displayable unit) to determine how to modify one or more content elements based on the inputs. The inputs can also include current data (e.g., real-time data) regarding the state of viewing conditions 405 (e.g., type of display, ambient light conditions surrounding one or more displays, etc.). In general, the appearance analysis in operation 403 attempts to determine how the physical arrangement and layout of the content affects the perceived appearance of the content elements. Operation 403 provides one or more outputs to appearance optimizers 409A and 409B, which modify one or more images based on the output from operation 403. Multiple appearance optimizers allow for parallel processing. In an alternative embodiment, there may be a single appearance optimizer. The adjusted pixel values ​​are then provided to a renderer 411 to create a composited page or other displayable unit 413 that is displayed 415 on one or more display devices (e.g., an OLED display or a VR or AR headset, etc.).

[0039] FIG. 5C shows further details regarding operation 403. As shown in the example of FIG. 5C, the data processing system may composite the spatial metadata (but not the content elements themselves) onto the displayable unit during operation 403 and then calculate distance and luminance difference values ​​between all of the elements (represented by the spatial metadata) for a given physical layout on the displayable unit. In fact, operation 403 may be considered a preliminary compositing operation to place the spatial metadata (but not the actual content elements) onto the displayable unit using the spatial metadata. This compositing occurs before compositing with the actual content elements in operation 411. Operation 403 may also apply one or more glare models known in the art. Operation 403 may then compute modifications such as adjustments or offsets. For example, the content elements may include mapping values ​​to ensure that all pixel values ​​are between preselected maximum and minimum luminance values, and these mapping values ​​may be adjusted by operation 403 based on the appearance analysis in operation 403. The modifications may be to the individual image content elements before compositing, or to the set of image content elements after compositing, or to both. Additionally, the appearance of the text may be adjusted, for example, if the contrast ratio of the text to its background is not within a desired range. In general, in one embodiment, the text should have a minimum contrast of about 4:1, and a maximum contrast should not exceed about 10:1. Text that contrasts too much with its background may cause visual discomfort, while text that contrasts too little may be difficult to read. Low-contrast text near an image with glare may also be difficult to read. Various examples of possible modifications are provided below in conjunction with Figures 6A, 6B, 6C, and 6D.

[0040] FIG. 6A illustrates an example of operation 403 and how an appearance optimizer (e.g., appearance optimizer 409A) modifies 427 one or more images based on analyzing 425 spatial metadata of one or more content elements. In this example, the dark image ("pixel / area 1") to the left of the image with detected glare ("pixel / area 2") is modified as shown in FIG. 6A to adjust it to account for the detected glare in the image on the right side of the displayable unit. Operation 403 determines the distance between two images that have a large difference in maximum luminance. Operation 403 can also receive input regarding viewing conditions (which may include the viewer's field of view), as shown in FIG. 6A. Based on the glare detected in the image on the right side, the maximum luminance of the dark image on the left is increased and the chroma or saturation is increased. These adjustments can be performed by the appearance optimizer, as shown in FIG. 5B. There are techniques known in the art for detecting glare, such as glare models known in the art. See, for example, https: / / www.lrc.rpi.edu / programs / Transportation / pdf / glareCalculation.pdf. These glare models can classify glare (from just noticeable to intolerable) using the deBoer scale. These glare models can allow designers of viewable pages to control the perceived impact of glare. For example, the perceived impact of glare can be creatively modified if desired by a content creator. For example, it may be desired to avoid any glare in the rendered image, to have some glare, or to intentionally desire glare (e.g., to create discomfort for a narrative).

[0041] FIG. 6B shows a flow chart illustrating a method similar to that shown in FIG. 6A, except that the method in FIG. 6B can be an optional trim pass method of adjusting an image. Optional trim pass metadata can be associated with a content element and used to further adjust the content element beyond what an initial analysis might suggest. In this case, a dark image is adjusted to account for glare in the vicinity of the dark image. A dark image is detected 451 near an image with detected glare, and a color volume mapping offset preference is defined 453 to modify the appearance of the dark image.

[0042] FIG. 6C shows an example of content element modification before compositing the five content elements 503, 504, 506, 508, and 502 shown in FIG. 6C. The top three content elements are each modified separately using tone mapping (or color volume mapping). In particular, tone mapping 505 modifies the appearance of content element 503, tone mapping 507 modifies the appearance of content element 504, and tone mapping 509 modifies the appearance of content element 506. In each of these modifications, pixel values ​​are modified according to the analysis of the metadata described above to improve the appearance in the final composite unit, which is HDR canvas 511 with custom dynamic range. SDR content elements 508 and 502 are mapped to perceptually quantized (PQ as described in ITU-R Rec. BT.2100) containers for display on the HDR canvas using processes known in the art. After the modified content elements are composited onto HDR canvas 511 (in a frame buffer), the displayable units are displayed on display devices 513 and 515. In another embodiment, the modification can occur after the content elements are composited onto the canvas of the displayable unit, for example, all of the content elements can be mapped after compositing, in yet another embodiment, the modification can occur both before and after compositing.

[0043] Composite content often includes video and animations composited with still images and text. Web pages often include video, text, and still images composited together (often from different sources) on the same web page. Video and animation can have images that change dramatically over time (e.g., a night scene in a movie followed by a daytime scene of a bright beach in the movie) or can include image elements that appear or disappear over time or based on a trigger. How the content around the movie on the page appears depends on both spatial and temporal metadata. Additionally, the content of the movie may be modified. Temporal metadata typically changes over time as the video and animation change, and can result in the modification of one or more content elements. Figure 6D shows an example 551 of how spatial and temporal metadata can be used to modify one or more content elements while two videos are playing on a displayable unit across three scenes 553, 555, and 557. The methods described for the static page can be used within each scene to modify the content based on the metadata within each scene. If the video content is Dolby Vision content that uses existing dynamic metadata based on Dolby Vision, this dynamic metadata can be compared to the spatial metadata and adjusted throughout playback.

[0044] Furthermore, the transition from the temporally local metadata of scene 553 to scenes 555 and 557 need not be abrupt. Instead, in one embodiment, the transition can be gradual or time-attenuated, for example, by following the time course of adaptation (based on photopic, scotopic and chromatic adaptation models known in the art). For example, metadata identified for mapping in scene 553 can be modified by 403 to gradually transition over time to metadata identified for mapping in scene 555. This can also help optimize and better use the available color volume of the output display, avoiding signal clipping or crashing or loss of contrast.

[0045] FIG. 7 illustrates an example of a data processing system 800 that may be used with one or more embodiments described herein. For example, the system 800 may be used to perform any of the methods described herein, such as those illustrated in FIGS. 2A, 2C, 3A, 3B, 4A, 4B, 5A, 5B, 5C, 6A, 6B, 6C, and 6D. Such a data processing system may create metadata and may be part of the system illustrated in FIG. 2B. Furthermore, such a data processing system may be any one of the systems illustrated in FIG. 1B. Such a system may include a web browser that uses the metadata described herein to compose and render web pages. It is noted that while FIG. 7 illustrates various components of a device, it is not intended to represent any particular architecture or manner of interconnecting the components, as such details are not germane to this disclosure. It will also be understood that network computers and other data processing systems or other consumer electronic devices having fewer or possibly more components may also be used with embodiments of the present disclosure. In one embodiment, the data processing system may be a smartphone or other mobile device.

[0046] As shown in Fig. 7, the device 800 in the form of a data processing system includes a microprocessor 805, a ROM (Read Only Memory) 807, and a bus 803 coupled to a volatile RAM 809 and a non-volatile memory 811. The microprocessor 805 may retrieve and execute instructions from the memories 807, 809, 811 to perform the operations described above. The microprocessor 805 may include one or more processing cores. The bus 803 interconnects these various components to each other and to the components 805, 807, 809, and 811 to a display controller and display device 813, as well as to peripheral devices such as input / output (I / O) devices 815, which may be a touch screen, a mouse, a keyboard, a modem, a network interface, a printer, one or more cameras, and other devices well known in the art. Typically, the input / output devices 815 are coupled to the system through an input / output controller 810. Volatile RAM (random access memory) 809 is typically implemented as dynamic RAM (DRAM) and requires constant power to refresh or maintain the data in the memory.

[0047] Non-volatile memory 811 is typically a magnetic hard drive or a magnetic optical drive, or an optical drive or a DVD RAM, or a flash memory, or other type of memory system that retains data (e.g., large amounts of data) even after power is removed from the system. Typically, non-volatile memory 811 is also a random access memory, although this is not required. While FIG. 7 illustrates non-volatile memory 811 as a local device directly coupled to the remaining components in the data processing system, it will be appreciated that embodiments of the present disclosure may utilize non-volatile memory off the system, such as a network storage device coupled to the data processing system through a network interface, such as a modem, an Ethernet interface, or a wireless network. Bus 803 may include one or more buses connected together through various bridges, controllers, and / or adapters, as is well known in the art.

[0048] Some of the above may be implemented using logic circuitry, such as special purpose logic circuitry, or a microcontroller, or other type of processing core that executes program code instructions. Thus, the processes taught by the above description may be performed using program code, such as machine-executable instructions that cause a machine that executes those instructions to perform a particular function. In this context, a "machine" may be a machine that translates intermediate form (or "abstract") instructions into processor-specific instructions (e.g., abstract execution environments such as "virtual machines" (e.g., Java Virtual Machine), interpreters, common language runtimes, high-level language virtual machines, etc.), and / or electronic circuitry (e.g., "logic circuitry" implemented with transistors) located on a semiconductor chip designed to execute instructions, such as general-purpose processors and / or special purpose processors. The processes taught by the above description may also be performed by (in place of or in combination with) electronic circuitry designed to execute the process (or a portion thereof) without executing program code.

[0049] The present disclosure also relates to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the required purposes or may include a general-purpose device selectively activated or reconfigured by a computer program stored in the device. Such a computer program may be stored in a non-transitory computer-readable storage medium, such as any type of disk, including but not limited to floppy disks, optical disks, CD-ROMs and magneto-optical disks, DRAM (volatile), flash memory, read-only memory (ROM), RAM, EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions, each coupled to a device bus.

[0050] A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, non-transitory machine-readable media include read-only memory ("ROM"), random access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, etc.

[0051] An article of manufacture for storing program code may be used. The article of manufacture for storing program code may be embodied as, but is not limited to, one or more non-transitory memories (e.g., one or more flash memories, random access memories (static, dynamic or other)), optical disks, CD-ROMs, DVD ROMs, EPROMs, EEPROMs, magnetic or optical cards, or other types of machine-readable media suitable for storing electronic instructions. The program code may also be downloaded from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by a data signal embodied in a propagation medium (e.g., via a communications link (e.g., a network connection)) and then stored in a non-transitory memory (e.g., DRAM or flash memory or both) in the client computer.

[0052] The above detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within the device memory. These algorithmic descriptions and representations are the tools used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations require physical manipulations of physical quantities. Commonly, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0053] It should be noted, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. As is apparent from the above description, unless otherwise indicated, it will be recognized that throughout the entire description, descriptions utilizing terms such as "receive," "determine," "send," "finish," "wait," "modify," and the like, refer to actions and processes of a device or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities in the device's registers and memories into other data similarly represented as physical quantities in the device's memory or registers or other such information storage, transmission, or display devices.

[0054] The processes and displays presented herein are not inherently related to any particular device or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the operations described. The required structure for a variety of these systems will appear from the description that follows. Further, the present disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages ​​may be used to implement the teachings of the present disclosure as described herein.

[0055] Exemplary embodiments The following text presents numbered embodiments in a claim-like format, with the understanding that these embodiments may be presented as claims in one or more future applications, such as one or more continuation or divisional applications.

[0056] Further embodiments are described in detail below, but it is recognized that these embodiments may be combined or modified, either in part or in whole, and each of these embodiments may be expressed as a machine-readable medium or a data processing system instead of a method.

[0057] Embodiment 1. 1. A method for processing data, comprising: receiving a set of elements including content and data representing positions or positioning rules for each of said elements within the set of elements on a displayable unit; determining one or more types of content within each of said elements; generating a set of metadata from the set of elements, the set of metadata being used in creating the composite content from the set of elements on the displayable unit when the composite content is displayed, the set of metadata including: (1) spatial data about the elements; and (2) image metadata about at least some of the elements; storing the generated set of metadata together with an association for each of the elements; The method includes:

[0058] Embodiment 2. 2. The method of claim 1, wherein the displayable unit is one of: (a) a page, sheet, folio, or other unit of content; (b) a web page; (c) part or all of the screen of a display device; or (d) content generated at least in part by a computer program.

[0059] Embodiment 3. transmitting the set of elements and the generated set of metadata in response to a request for the web page; 3. The method of embodiment 2, wherein the data representing a position or a positioning rule is included in a description of the displayable unit in a hypertext markup language.

[0060] Embodiment 4. The method of any one of embodiments 1 to 3, wherein the one or more types of content include at least one of: (a) high dynamic range (HDR) image content, (b) standard dynamic range (SDR) image content, (c) text content, or (d) user interface content for use in receiving input from a user.

[0061] Embodiment 5. A method according to any one of embodiments 1 to 4, wherein the spatial data comprises vector-based spatial data defining approximate boundaries on the displayable units of each of the elements in the set of elements.

[0062] Embodiment 6. 6. The method of any one of claims 1 to 5, wherein the image metadata includes color volume characteristics or image statistics for at least some of the elements in the set of elements.

[0063] Embodiment 7. 7. The method of any one of claims 1 to 6, wherein the image statistics include one or more of the maximum luminance of the image, the diffuse white level and white point of the image, the minimum luminance of the image, the average luminance of the image, or the median luminance of the image.

[0064] Embodiment 8. A method according to any one of embodiments 1 to 7, wherein the set of metadata also describes a temporal change of the content over time, the set of metadata describing a rate of change associated with the temporal change of the content over time, the rate of change indicating a rate at which metadata in the set of metadata changes from one scene to the next.

[0065] EMBODIMENT 9. A method according to any one of the preceding claims, wherein the image metadata includes data relating to detected glare or that glare is detected in at least one of the elements in the set of elements.

[0066] Embodiment 10. 10. A method according to any one of embodiments 1 to 9, wherein the image metadata includes a texture abstraction of at least one of the elements in the set of elements.

[0067] Embodiment 11. 11. The method of embodiment 10, wherein the texture abstraction is derived from a Fourier analysis based representation of at least one of the elements in the set of elements.

[0068] Embodiment 12. 12. A method according to any one of the preceding claims, wherein the image metadata includes a quantized representation of at least one of the elements in the set of elements.

[0069] Embodiment 13. A method according to any one of claims 1 to 12, wherein an application programming interface (API) is used to generate the set of metadata, the API linking a metadata generation component in a data processing system with web page creation software.

[0070] Embodiment 14. 14. The method of any one of claims 9 to 13, wherein the detected glare is classified as one of disabling glare or discomfort glare.

[0071] EMBODIMENT 15. 15. A method according to any one of embodiments 1 to 14, wherein the set of metadata is stored in a scalable vector graphics format.

[0072] EMBODIMENT 16. 1. A method for processing data, comprising: receiving a set of metadata for use in creating composite content from a set of elements to be composited together for display within a displayable unit, the set of metadata including: (1) spatial data for the elements; and (2) image metadata for at least some of the elements; processing the set of metadata to determine how to modify one or more of the elements based on the set of metadata; modifying one or more of the elements based on the set of metadata; rendering the composite content having the modified one or more elements to display the displayable unit on a display device; The method includes:

[0073] EMBODIMENT 17. 17. The method of claim 16, wherein the displayable unit is one of: (a) a page, sheet, folio or other unit of content, or (b) a web page, or (c) part or all of the screen of a display device, or (d) content generated at least in part by a computer program.

[0074] Embodiment 18. 18. The method of claim 17, wherein the receiving is in response to a request for the web page, the set of metadata is received by a web browser, and at least a portion of the modification is performed after the content is synthesized.

[0075] EMBODIMENT 19. A method as described in any one of embodiments 16 to 18, wherein the synthetic content includes at least one of (a) high dynamic range (HDR) image content, (b) standard dynamic range (SDR) image content, (c) text content, or (d) user interface content for use in receiving input from a user.

[0076] EMBODIMENT 20. 20. A method according to any one of embodiments 16 to 19, wherein the modification is based at least in part on the spatial data and the image metadata.

[0077] EMBODIMENT 21. A method according to any one of embodiments 16 to 20, wherein the spatial data includes vector-based spatial data defining approximate boundaries on the displayable units of each of the elements in the set of elements, the approximate boundaries being used to calculate a distance between at least two elements in the set of elements.

[0078] EMBODIMENT 22. 22. A method according to any one of embodiments 16 to 21, wherein the image metadata includes color volume characteristics or image statistics for at least some of the elements in the set of elements.

[0079] EMBODIMENT 23. 23. The method of any one of embodiments 16 to 22, wherein the image statistics include one or more of the maximum luminance of the image, the minimum luminance of the image, the average luminance of the image, or the median luminance of the image.

[0080] EMBODIMENT 24. A method according to any one of embodiments 16 to 23, wherein the set of metadata also describes temporal changes in the content over time, the set of metadata describing a rate of change associated with the temporal changes in the content over time, the rate of change indicating the rate at which the metadata in the set of metadata changes from one scene to the next.

[0081] 25. A method according to any one of embodiments 16 to 24, wherein the image metadata includes data relating to detected glare or data that glare is detected in at least one of the elements in the set of elements, and the modification reduces the detected glare.

[0082] 26. determining a distance between a first element and a second element in the set of elements on the displayable unit and determining a difference in image metadata statistics between the first element and the second element; modifying one or both of the first element or the second element based on the determined distance and the determined difference. 26. The method of any one of embodiments 16 to 25, further comprising:

[0083] EMBODIMENT 27. A method according to any one of embodiments 16 to 26, wherein the modification of the first element or the second element is performed either before combining the contents or after combining the contents.

[0084] 28. A method according to any one of embodiments 16 to 27, wherein the set of metadata is received in a scalable vector graphics format.

[0085] 29. A method according to any one of embodiments 16 to 28, wherein the image metadata includes a texture abstraction of at least one of the elements in the set of elements.

[0086] EMBODIMENT 30. A method as described in any one of embodiments 16 to 29, wherein the modification takes into account one or more of: (a) the on-screen and off-screen status of the content in the displayable unit; (b) the display device used to display the content in the displayable unit; (c) the surrounding viewing environment surrounding the display device displaying the displayable unit; or (d) the viewing distance of a viewer of the displayable unit.

[0087] EMBODIMENT 31. A non-transitory machine-readable medium storing executable computer program instructions, comprising: A non-transitory machine-readable medium, the computer program instructions, when executed by one or more data processing systems, causing the one or more data processing systems to perform a method according to any one of embodiments 1 to 30.

[0088] EMBODIMENT 32. A data processing system configured to perform the method according to any one of embodiments 1 to 30.

Claims

1. 1. A method for processing data, comprising: receiving a set of metadata for use in creating composite content from a set of elements to be composited together for display within a displayable unit, the set of metadata including: (1) spatial data for the elements; and (2) image metadata for at least some of the elements; processing the set of metadata to determine how to modify one or more of the elements based on the set of metadata; modifying one or more of the elements based on the set of metadata, the modification taking into account one or more of a surrounding viewing environment surrounding a display device displaying the displayable unit or a viewing distance of a viewer of the displayable unit; rendering the composite content having the modified one or more elements to display the displayable unit on a display device; The method includes:

2. 10. The method of claim 1, wherein the displayable unit is one of: (a) a page, sheet, folio, or other unit of content; (b) a web page; (c) part or all of a screen of a display device; or (d) content generated at least in part by a computer program.

3. The method of claim 2 , wherein the receiving is in response to a request for the web page, the set of metadata is received by a web browser, and at least a portion of the modifying occurs after the content is composed.

4. 2. The method of claim 1 , wherein the composite content includes at least one of: (a) high dynamic range (HDR) image content, (b) standard dynamic range (SDR) image content, (c) text content, or (d) user interface content for use in receiving input from a user.

5. The method of claim 1 , wherein the modification is based at least in part on the spatial data and the image metadata.

6. 2. The method of claim 1 , wherein the spatial data includes vector-based spatial data defining approximate boundaries on the displayable units of each of the elements in the set of elements, the approximate boundaries being used to calculate a distance between at least two elements in the set of elements.

7. 2. The method of claim 1 , wherein the image metadata includes color volume characteristics or image statistics for at least some of the elements in the set of elements, the image statistics including one or more of a maximum image luminance, a minimum image luminance, an average image luminance, or a median image luminance.

8. 10. The method of claim 1 , wherein the set of metadata also describes a temporal change of content over time, the set of metadata describing a rate of change associated with the temporal change of the content over time, the rate of change indicating a rate at which the metadata in the set of metadata changes from one scene to the next over time.

9. The method of claim 1 , wherein the image metadata includes data regarding detected glare or that glare is detected in at least one of the elements in the set of elements, and the modification reduces the detected glare.

10. determining a distance between a first element and a second element in the set of elements on the displayable unit and determining a difference in image metadata statistics between the first element and the second element; modifying one or both of the first element or the second element based on the determined distance and the determined difference. The method of claim 1 further comprising:

11. 2. The method of claim 1 , wherein the modification of a first element or a second element occurs either before compositing content or after compositing content, and the image metadata includes a texture abstraction of at least one of the elements in the set of elements.

12. The method of claim 1 , wherein the set of metadata is received in a Scalable Vector Graphics Format.

13. The amendment is: determining a field of view based on the viewing distance; Determine whether an element is within the field of view and needs to be corrected due to large differences in image metadata statistics in the same field of view.

2. The method of claim 1, further comprising taking into account a viewing distance of a viewer of the displayable unit by:

14. A non-transitory machine-readable medium storing executable computer program instructions, comprising: A non-transitory machine-readable medium, the computer program instructions, when executed by one or more data processing systems, causing the one or more data processing systems to perform a method according to any one of claims 1 to 13.

15. A data processing system configured to carry out a method according to any one of claims 1 to 13.

Images