Method and system for perceptually meaningful spatial content synthesis
By generating and applying metadata, the problem of poor appearance of different types of content during synthesis and rendering is solved, and the perceived optimization of content and glare reduction is achieved, and the display quality is improved.
Patent Information
- Application Number
- CN202380031349.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-30
- Filing Date
- 2023-03-28
- Publication Date
- 2024-11-15
AI Technical Summary
In the process of content creation and presentation, it is difficult to effectively handle synthesis and rendering between different types of content (such as HDR and SDR images, text, etc.), especially when considering differences in display devices and viewing environments, resulting in content. Poor appearance problem.
By generating metadata, including spatial data and image metadata, the content is adjusted and synthesized to ensure that the content is consistent with the characteristics of the display device and viewing environment when displayed. The API interface is used to combine the metadata generation system with web page creation software for web pages. Development and distribution to realize content perceived optimization.
It realizes dynamic adjustment and synthesis of content, improves the display quality of content in different devices and environments, reduces the impact of glare, and improves the user experience.
Smart Images

Figure CN118975259A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims the benefit of priority to European Patent Application No. 22165451.0, filed on March 30, 2022, and U.S. Provisional Patent Application No. 63 / 362,166, filed on March 30, 2022, both of which are incorporated herein by reference in their entirety. Background Art
[0003] The present disclosure relates to the field of content creation and content presentation on data processing systems such as computers, smartphones, televisions, and other electronic devices.
[0004] Content creation for movies and video content uses methods that utilize metadata to adjust the appearance of the content during presentation (e.g., display) of the content based on the metadata. Dolby Vision is an example of these methods. In these methods, content is created and presented for frames. The content can be viewed as linear content, that is, one frame follows another over a period of time. The frames are displayed sequentially over time; they do not composite into a single image or displayable unit. Any metadata used to modify the presentation of the content during presentation is created for a specific single frame or a series of frames over a period of time.
[0005] US2015 / 245004 A1 discloses a method and system for adaptively mixing video components with graphics / UI components, wherein the video components and graphics / UI components may be of different types, such as different dynamic ranges (such as HDR, SDR) and / or color gamuts (such as WCG). The mixing may produce frames optimized for the color space, environmental conditions, viewing distance and angle of the display device, etc., while taking into account the characteristics of the received data. The method includes receiving video and graphics / UI elements, converting the video to HDR and / or WCG, performing statistical analysis on the received data and any other applicable rendering information, and assembling the video frame with the received components based on the statistical analysis. The assembled video frame may be matched to the color space and displayed. The video data and graphics / UI data may have or be adjusted to have the same white point and / or color source. Summary of the invention
[0006] The invention is defined by the independent claims. The dependent claims relate to optional features of some embodiments. Various methods are described for generating metadata for content to be synthesized and rendering the synthesized content using the generated metadata. For example, these methods can be used for the development and distribution of one or more web pages. In one embodiment, a web page developer can collect content to be synthesized together into a web page (e.g., HDR or SDR images and HDR or SDR animations as well as text and user interface elements) and call a set of APIs to generate metadata for the content to be synthesized in the web page; the metadata generation system receives the call through the API and generates metadata. The web page can then be distributed (e.g., to a web browser) along with the generated metadata, which can be used (e.g., by a web browser or other content distribution system when rendering) to create a display of the web page, wherein the content can be perceptually modified based on metadata about individual elements on the web page and their spatial proximity. These methods can also be used for other types of displayable units.
[0007] In one embodiment, a method according to an aspect may include the following operations: receiving a set of elements, the set of elements including content and data representing a position or positioning rule of each element in the set of elements on a displayable unit; determining one or more types of content in each element; generating a set of metadata from the set of elements, the set of metadata being used to create the composite content from the set of elements on the displayable unit when displaying the composite content, the set of metadata including: (1) spatial data about the elements and (2) image metadata about at least some of the elements; and storing the generated set of metadata in association with each element. In one embodiment, the displayable unit is one of: (a) a page, sheet, folio, or other unit of content; or (b) a web page; or (c) a portion or all of a screen of a display device; or (d) content generated at least in part by a computer program. In one embodiment, the set of metadata also describes temporal changes in the content over time. In one embodiment, the set of metadata is stored in a scalable vector graphics format (such as a format that supports vector graphics for images). In one embodiment, the display device may be a flat display device or a non-flat device (such as a display device in an augmented reality or virtual reality headset). In one embodiment, the method may further include an operation of transmitting the set of elements and the generated set of metadata in response to a request for the web page, and wherein the data representing the position or positioning rules is included in the description of the displayable element in a hypertext markup language. The positioning rules may include rules, such as instructions or other data defining how to determine the position of an element based on data about, for example, an application window (e.g., a browser window), window scaling data, or data about a display device, or a combination of all or a subset of such data.
[0008] In one embodiment, the content of the element can be different types of content. For example, the one or more types of content may include at least one of the following: (a) high dynamic range (HDR) image content; (b) standard dynamic range (SDR) image content; (c) text content; or (d) user interface content for receiving input from a user. Image content can be specified by pixel data (a bitmap representation of an image) or vector graphics data. In one embodiment, the spatial data may include vector-based spatial data that defines the approximate boundaries of each element in the group of elements on the displayable unit.
[0009] In one embodiment, the image metadata may include color volume attributes or image statistics of at least some of the elements in the group of elements. In one embodiment, the image statistics may be one or more of the following: maximum brightness of the image; minimum brightness of the image; average brightness of the image; or median brightness of the image.
[0010] In one embodiment, the image metadata may include data about detected glare or data that can be used to detect glare in at least one of the set of elements (depending on the viewing environment), and the detected glare can be detected using one or more glare models. In one embodiment, the detected glare can be classified as one of the following: disabling glare or discomfort glare.
[0011] In one embodiment, the image metadata may include a texture abstraction of at least one of the set of elements. The texture abstraction may be a mathematical representation of the image derived based on noise properties and vector properties. For example, the texture abstraction may be derived from a Fourier analysis-based representation of at least one of the set of elements. In one embodiment, the image metadata may include a quantized representation of at least one of the set of elements.
[0012] In one embodiment, an application programming interface (API) is used to cause the generation of the set of metadata, the API linking the metadata generation component in the data processing system with the web page creation software. The API can be called by the web page creation software to generate the set of metadata from a selected set of content to be composited together to create a composite web page.
[0013] In one embodiment, a method according to a composition aspect may include the following operations: receiving a set of metadata for creating composite content based on a set of elements to be composited together for display on a displayable unit, the set of metadata including: (1) spatial data about the elements and (2) image metadata about at least some of the elements; processing the set of metadata to determine how to modify one or more elements based on the set of metadata; modifying the one or more elements based on the set of metadata; and rendering the composite content with the modified one or more elements to display the displayable unit on a display device. In one embodiment, the receiving may be in response to a request for a web page, and the set of metadata is received by a web browser. In one embodiment, the modification based on the set of metadata may be performed in the following circumstances: (a) before the content is composited; (b) after the content is composited; or (c) before and after the content is composited. For example, a set of elements may be modified before the composition, and a subset of the modified elements may be modified again after the composition. The modification may be based at least in part on the spatial data and the image metadata. In one embodiment, the spatial data may include vector-based spatial data that defines an approximate boundary of each element in the group of elements on the displayable unit, and the approximate boundary is used to calculate a distance between at least two elements in the group of elements. In one embodiment, the method may determine a distance between a first element and a second element in the group of elements on the displayable unit, and determine a difference in image metadata statistics (e.g., an average brightness value) between the first element and the second element; the method may then modify one or both of the first element or the second element based on the determined distance and the determined difference. The image metadata may include data about detected glare (or data that can be used to detect glare), which may cause the composite content to be modified to reduce the detected glare.
[0014] In one embodiment, the modification may take into account one or more of: (a) the on-screen and off-screen status of the content in the displayable unit; (b) the display device used to display the content in the displayable unit; (c) the surrounding viewing environment around the display device displaying the displayable unit; or (d) the viewing distance of the viewer of the displayable unit. For example, the viewer's field of view based on the viewing distance can be used to determine when an element is within the field of view, and modifications may be required due to large differences in image metadata statistics within the same field of view (e.g., dark images next to bright images).
[0015] Aspects and embodiments described herein may include non-transient machine-readable media that may store executable computer program instructions that, when executed, cause one or more data processing systems to perform methods described herein when executing computer program instructions. These instructions may be stored in non-transient machine-readable media, such as dynamic random access memory (DRAM) as a volatile memory, or non-volatile memory such as flash memory, or other forms of memory. Aspects and embodiments described herein may also take the form of a data processing system constructed or programmed to perform these methods. For example, a data processing system may be constructed to have hardware logic for performing these methods, or may be programmed to have a computer program for performing these methods.
[0016] The above summary does not include an exhaustive list of all embodiments and aspects of the present disclosure. All systems, media and methods can be practiced according to all suitable combinations of the various aspects and embodiments summarized above and the embodiments disclosed in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The present invention is illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.
[0018] Figure 1A Examples of displayable units (eg, web pages) are shown that include content that is composited or assembled from separate content elements (such as separate images and text, as well as other types of content).
[0019] Figure 1B An example of a content creation and distribution system according to one embodiment is shown, such as a system that creates and distributes a set of web pages from one or more web servers.
[0020] Figure 2A is a flow chart illustrating a method for creating metadata for content to be synthesized and rendered based on the created metadata according to one embodiment.
[0021] Figure 2B An example of a content creation system that creates metadata for content to be synthesized and rendered based on the created metadata is shown according to one embodiment.
[0022] Figure 2C An example of a method of creating metadata for a displayable unit (e.g., a web page) and then composing and rendering the displayable unit according to one embodiment is shown; Figure 2C The method may be performed by several different data processing systems or by a single data processing system.
[0023] Figure 3Ais a flow chart illustrating a method for generating spatial metadata and image statistics metadata according to one embodiment.
[0024] Figure 3B is a flow chart illustrating a method for generating spatial metadata according to one embodiment.
[0025] Figure 4A An example of creating a texture abstraction of an image (of flowers in front of a bokeh background) according to one embodiment is shown.
[0026] Figure 4B An example of a quantized and segmented abstraction of an image according to another embodiment is shown.
[0027] Figure 5A is a flow chart illustrating a method for compositing and rendering a displayable unit using generated metadata according to one embodiment.
[0028] Figure 5B is a flow chart illustrating a method for compositing and rendering a displayable unit using pre-generated metadata and metadata generated at rendering time by a device performing the compositing according to one embodiment.
[0029] Figure 5C An example of a method for compositing and rendering a displayable unit including composite content according to one embodiment is shown.
[0030] Fig. 6A An example of applying generated metadata at the time of synthesis according to one embodiment is shown.
[0031] Figure 6B An example is shown for applying generated metadata during compositing to compensate for detected glare in a perceptually meaningful manner, according to one embodiment.
[0032] Figure 6C Another example of applying generated metadata to modify content during composition according to one embodiment is shown.
[0033] Fig.6D An example of applying generated spatial and temporal metadata to modify content that varies over time during composition of displayable units is shown according to one embodiment.
[0034] Figure 7 An example of a data processing system is shown that may be used to implement one or more embodiments described herein. DETAILED DESCRIPTION
[0035] Various embodiments and aspects will be described with reference to the details discussed below, and the accompanying drawings will illustrate various embodiments. The following description and the accompanying drawings are illustrative and should not be construed as limiting. Many specific details are described to provide a full understanding of the various embodiments. However, in some cases, well-known or conventional details are not described in order to provide a concise discussion of the embodiments.
[0036] In this specification, reference to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment. In this specification, the phrase "in one embodiment" in various places does not necessarily refer to the same embodiment. The processes depicted in the following figure are performed by processing logic including hardware (e.g., circuits, dedicated logic, etc.), software, or a combination of the two. Although these processes are described below in terms of some sequential operations, it should be understood that some of the described operations may be performed in a different order. In addition, some operations may be performed in parallel rather than sequentially.
[0037] Embodiments described herein may create metadata for content elements that are to be assembled or synthesized into a displayable unit, and when a data processing system synthesizes the content into the displayable unit, the data processing system may modify the content based on the created metadata so that the synthesis may be considered to adjust the appearance of at least some of the content of the entire displayable unit. A displayable unit may be, for example, a web page that is synthesized from content, typically from different sources and of different types, when displayed (e.g., on a client web browser). The content is synthesized into a spatial arrangement on the displayable unit based on data that defines the spatial arrangement (e.g., HTML or CSS in a web page). Content modification may take into account this spatial arrangement so that the content is modified based in part on the spatial arrangement and in part on image metadata or other content metadata. Figure 1AAn example of a displayable unit is shown, namely a web page. The web page 10 includes an image 12, an image 14, text 18, and a user interface panel 16, which contains selectable commands that a user can select to cause the web page to perform an operation. The images 12 can have different types of content; for example, they can be high dynamic range (HDR) image content or standard dynamic range (SDR) image content (or can be HDR or SDR video content). The images can be stored or represented in a pixel data format (usually in compressed form), or can be represented by vector graphics data (and therefore can be referred to as vector graphics-based images). The size of the text can be adjusted. Although the present disclosure generally uses web pages as examples of displayable units, the embodiments described herein can also be used with other types of displayable units, such as: (a) pages, sheets of paper, folios, or other content units; or (b) part or all of the screen of a display device; or (c) content generated at least in part by a computer program or electronic device. The display device can be a computer display or a smartphone display or a TV display, a display in a virtual reality or augmented reality headset, or other known and future display devices, and can use a flat display screen (e.g., LCD or OLED) or a curved or non-flat display.
[0038] In many embodiments, displayable units are created by one or more content creators and then distributed to users operating data processing systems (such as desktop computers, other types of computers, smartphones, wearable devices, gaming systems, entertainment systems such as televisions, consumer electronic devices, etc.) to view and interact with the displayable units. Figure 1B An example of such distribution that may be used for web pages is shown. One or more content creators 30 may create one or more web pages for distribution by one or more content distribution systems 32, which may be conventional web servers that transmit web pages to web browsers (on client systems) that request the web pages. One or more content distribution systems 32 may be coupled to one or more networks 34 to receive web page requests from data processing systems 36 and 37 that are also coupled to one or more networks 34. Data processing systems 36 and 37 may be computers or other devices that execute web browsers (or other content consumption systems); these web browsers may request web pages over network 34 and receive web pages from network 34. Web pages may include conventional content (e.g., images, text, user interface (UI) elements, etc.) as well as metadata used during web page synthesis as described herein. In one embodiment, one or more content creators 30 may use Figure 2A The method shown and Figure 2B The system shown is used to create a displayable unit having such metadata.
[0039] Figure 2AThe illustrated method may begin at operation 101 by receiving a set of elements (eg, Figure 1A The images 12 and 14 and the text 18 and the UI 16 in Figure 2C 151, 153, 155, and 157 in the display unit) and receives data about their spatial arrangement in the display unit (e.g., two-dimensional positioning or arrangement / layout or other positioning information (such as positioning rules used to determine the final position)). The spatial arrangement data may specify the actual position or positioning rules (e.g., CSS layout rules) that are dynamically used when displayed (e.g., based on data about the application window and scaling data and display device data about the display size) to determine the actual position. In one embodiment, the spatial arrangement may be described in HTML (Hypertext Markup Language) or a similar markup language format or CSS (Cascading Style Sheets), etc. In some embodiments, the method may include an operation 103 for classifying different types of content; this may be necessary so that each different type of content can be processed according to type. For example, a movie in HDR may require the generation of temporal metadata (as described below), while a display unit that only contains content that is static over time does not require temporal metadata. Operation 103 may not be required in the case where the type is already specified by data associated with the display unit (e.g., the HTML of the web page specifies the type of content of different content elements).
[0040] exist Figure 2A In operation 105 of , the method may determine one or more boundaries associated with each content element. For example, as further described below, the data processing system may track the perimeter of one or more elements (or the content within each of the one or more elements) to generate spatial metadata. The spatial metadata may be vector-based metadata that defines position data using a vector format (e.g., a set of x, y Cartesian coordinates, such as a pair of Cartesian coordinates). In some embodiments, when the received data associated with the content element (from operation 101) does not define an outer perimeter, it may be necessary to track the outer perimeter of at least some content elements. In one embodiment, operation 105 may define the perimeter of portions of content within a content element (e.g., portions of an image within an image). These different portions may be classified using a quantization threshold (e.g., a threshold of brightness value) described further below to segment the content within a content element (such as a single image).
[0041] exist Figure 2AIn operation 107 of , the method may generate image statistics metadata. In one embodiment, operation 107 may be performed before operation 105 and used to segment different parts of each image (so as to segment different parts based on quantized threshold brightness values). As further described below, the generated image statistics metadata may be used to determine how to modify each image based on the image statistics metadata and spatial metadata. Operation 107 may also optionally include generating an abstraction of one or more images. Examples of such abstractions (e.g., texture and quantization abstractions) are provided below. In one embodiment, the abstraction may be a mathematical description of an image that does not retain data for each pixel and has a greatly reduced storage size relative to the pixel data of the complete image. These abstractions may also be used to determine how to modify each image based on the abstraction and spatial metadata.
[0042] exist Figure 2A In operation 109 of , the method may store the content elements and associated metadata (e.g., metadata extracted from the elements and their spatial and temporal arrangements) for use during composition when rendering a displayable unit containing all the content elements when displayed on a client system. Then, in operation 111, the components of the displayable unit (including the content elements and associated metadata and other data, such as HTML) may be transmitted to one or more other data processing systems. These components may then be used in such data processing systems to composite (e.g., assemble the components into a single web page) and render the displayable unit (e.g., web page) onto a display device such as an organic light emitting diode (OLED) display device; the display device may be a planar device or a non-planar device (e.g., a spherical screen).
[0043] Figure 2B The following table shows the method for executing Figure 2A An example of a system of the method in FIG. Figure 2B The system in is based on a web page embodiment, where the displayable unit is a web page. The system includes a web page creation system 125 coupled to a content storage 131; the content storage can store a collection of content items on the displayable unit, the HTML of the web page, and extracted metadata to be used when composing the displayable unit. The web page creation system 125 can be based on a conventional web page authoring software package known in the art. The system also includes a metadata extraction processing system 129, which can perform one or more embodiments and methods described herein (e.g., Figure 2AThe metadata extraction processing system 129 may operate in conjunction with the web page creation system 125 via an application programming interface (API) 127 that allows these two different software components (the web page creation system 125 and the metadata extraction processing system 129) to operate and communicate so that the metadata extraction processing system generates metadata for each content element for use during composition. Once such metadata is generated, it may be stored as metadata 133; in one embodiment, the stored metadata 133 may be stored along with a tag that associates each piece of metadata with the appropriate content element; in other words, the tag of the metadata component may specify the identity of the content element described by or associated with the metadata component. The metadata may also or alternatively be stored with the content (e.g., attached to its corresponding content). Figure 2B The system shown allows the content creator to call the metadata extraction processing system 129 through an API call, and then the metadata extraction processing system 129 can generate metadata for each content element using the methods described herein, without the need for the content creator to manually generate metadata. This simplifies the creation process of web pages that can use the metadata synthesis described herein. Once all content elements are selected and positioned / arranged on the web page, metadata can be generated through API calls.
[0044] Figure 2C Another example of a method for generating metadata during creation of a displayable unit is shown in . The example includes four different types of content elements, including PQ (perceptual quantizer) encoded content elements 151, hybrid log-gamma encoded HDR content elements 153, SDR content elements 155, and text elements 157. Other types of content elements may also be included in the displayable unit, such as vector graphics or geometric shapes / regions and backgrounds, and these other types of content elements may be processed as described herein. These content elements may be synthesized together to create a web page that is displayed on a web browser on a client data processing system after the client system downloads components of the web page from one or more servers. In operation 159, a metadata extraction process generates metadata from individual elements of each content using one or more embodiments described herein; the metadata extraction process may be performed by Figure 2BThe system 129 shown is performed. In a typical embodiment, operation 159 analyzes each image using image statistics to generate image metadata about the image; the analysis creates metadata, but alternative embodiments may also use metadata that already accompanies the image (so the metadata exists with the original image). The generated metadata may include vector-based spatial metadata about each content element and image statistics for each content element, as well as spatial metadata with image statistics within each content element; in addition, an abstraction of each image may be created as additional metadata for each image. The metadata extraction process in operation 159 produces extracted metadata 161 that can be stored with the content; in one embodiment, the metadata is tagged with an identifier that identifies the associated content described by the metadata. After storing the extracted metadata 161, a data processing system (e.g., a client data processing system with a web browser) can perform synthesis and rendering processes 163 based on the extracted metadata 161; these synthesis and rendering processes 163 can perform the methods described herein to modify, synthesize, and render a collection of content elements based on the metadata 161. These synthesis and rendering processes 163 can then produce synthesized and rendered content elements in a displayable unit 165, including the modified content elements. The display unit may display the modified content elements based on the metadata 161 and the viewing settings (e.g., the number and type of monitors) and the surrounding viewing environment (e.g., a dark viewing environment or a bright viewing environment). Figure 2C all aspects.
[0045] Figure 3A An embodiment of a metadata generation method is shown in FIG. In the method, spatial metadata and image statistics metadata may be generated and encoded for each content element (such as an image (e.g., an SDR image)) in a displayable unit (e.g., a web page). In operation 201, a data processing system may open an image or frame of content corresponding to a content element in a displayable unit. For example, a metadata processing component ( Figure 2BThe system 129 in operation 203 may open the content element and then process the content element to extract metadata. In operation 203, the data processing system may calculate image statistics for the image (or frame); for example, in one embodiment, the system may calculate brightness data [e.g., average brightness, median brightness, maximum brightness, minimum brightness, etc. based on the image metadata of the content element] and other image metadata statistics, and may calculate spatial data for regions in the content element, as described herein. These calculated image statistics may be used to segment the image into different regions or segments based on the calculated statistics. In one embodiment, each segment may be based on a quantization level of brightness, such that consecutive pixels having brightness values or brightness level (L) values within each quantization bucket are classified into a segment (e.g., one quantization level represents pixels with L less than 0.1 nits; another quantization level represents pixels with L greater than 600 nits; another quantization level represents pixels with L between these two levels). L may also be represented using the SI unit cd / m 2 Each segment can be considered to be defined by a pixel threshold boundary. A variety of techniques are known in the art to segment the pixels of an image; see, for example, published PCT application PCT / US2020 / 062942 (WO 2021 / 113400).
[0046] Once the regions are identified, their boundaries can be determined. For example, in operation 205, the data processing system can determine the boundaries of each segmented region in the image by tracing the outline or perimeter of the region in the image. The tracing can produce a set of vector-based spatial metadata that defines the outline or perimeter of each segmented region, and image statistics for each segmented region can also be included in the vector-based spatial metadata. The outline can be traced by identifying edges (defining the perimeter) and connecting them with lines; the points at the end of each line describe a vector. Alternatively, piecewise curve fitting along pixel threshold boundaries can be used; techniques for performing such piecewise curve fitting are known in the art. For example, see the following literature describing curve fitting: Michael Plass and Maureen Stone. 1983. Curve fitting with piecewise parametric cubics. SIGGRAPH Comput. 17, 3 (July 1983), 229 239. DOI: https doi.org / 10.1145 / 964967.801153. Wenping Wang, Helmut Pottmann, and Yang Liu. 2006. Fitting B spline curves to point clouds by curvature based squared distance minimization. ACM Trans. Graph. 25, 2 (April 2006), 214-238. DOI: https doi.org / 10.1145 / 1138450.1138453. Gerald Farin. 1993. Curves and surfaces for computer aided geometric design (3rd ed.): a practical guide. Academic Press, USA. In an alternative embodiment, at least some of the segments or regions may be manually selected. Once the contour or perimeter is defined, Figure 3AAn embodiment of the method may attempt to reduce the size of the spatial metadata in operation 207; in one embodiment, this may be accomplished by reducing the number of nodes along the perimeter (e.g., combining two lines into a line that approximates the two lines). Once the spatial metadata and image statistics are generated, they may be encoded and stored in association with the corresponding image used to create the metadata in operation 209; the association may be a tag that identifies the image. The metadata may be stored separately from the image and include the tag so that the synthesis process can find the appropriate metadata for the image. In one embodiment, the metadata may be stored in a markup language or scalable vector graphics format (which web browsers and GPU renderers already support).
[0047] Figure 3B Another example of how an image as one of the content elements to be synthesized can be processed to generate metadata (such as spatial metadata and image statistics) is shown. A source image is received in operation 221 and converted to a black and white image 223. The image can then be quantized in operation 225 to generate a set of segments based on the quantization (e.g., brightness quantization as described above). The metadata processing system can then track each segment in operation 227 to produce a set of vectors describing the boundaries or perimeters of each segment. The metadata processing system can then reduce the spatial metadata in operation 229 and store the spatial metadata in operation 231.
[0048] Each image in the set of content elements may be described by an abstraction that may be processed when synthesizing a displayable unit. The abstraction may provide a statistical representation of the image for use in synthesis and rendering (perceptually meaningful content modifications based on the synthesized displayable unit) while requiring significantly less storage space than a stored version of the image. The abstraction may be a mathematical representation that describes the image without requiring data for each individual pixel. Figure 4A and Figure 4B Two examples of such image abstractions are provided that may be used in one or more embodiments.
[0049] Abstractions of images can use textures or noise textures, and such abstractions can successfully capture enough data for synthesis and still have a small storage (and transmission) "footprint". Figure 4AThe example shown uses a noise texture abstraction to generate an abstraction of an image, which can be saved as metadata (referred to as abstract metadata) used during the synthesis and rendering process. This abstract metadata can then be processed during synthesis without using the actual image to decide how to modify content (either in the image or other content around the image) during synthesis and rendering. In one embodiment, each segment or region of an image can be associated with a mathematically defined texture property (e.g., defined by 1 / f noise, where f is the frequency of the pixel data). The example of a noise texture can be created based on the techniques and methods described in the following published paper: Kunkel, T. and Daly, S. (2020), 57-1: Spatiotemporal Noise Targets Inspired by Natural Imagery Statistics. SID Workshop Technical Paper Abstracts, 51:842-845. (See https: / / doi.org / 10.1002 / sdtp.14001 ). In one embodiment using these methods, a random seed image is selected (or a varying random seed is selected), and then the process generates frequency components based on the selected alpha (α) value, which produce different two-dimensional noise texture images. In one embodiment, the metadata processing system can select a larger alpha value (e.g., α=4 or 5) for HDR images than for non-HDR images; natural images (e.g., images of forests or river valleys) can benefit from alpha values in the range of 2 to 3. The alpha value affects the frequency distribution of the noise.
[0050] like Figure 4A As shown in FIG. 2 , an image 251 (as one of the content elements to be synthesized into a displayable unit (such as a web page)) can be processed into a final texture abstraction 269 using a noise texture abstraction process 257. The final texture abstraction 269 can provide a more refined image representation than the simple vector representation 255 created by the simple vector abstraction process 253 using the threshold map, as described above in conjunction with Figure 3A and Figure 3BAs described. In one embodiment, the noise texture abstraction process 257 may start with a vector-based mask (from the vector abstraction process 253) so that different portions of the image may be processed separately using the noise texture abstraction process 257 according to the vector-based mask. The noise texture abstraction process 257 may start with an analysis 259, where a set of data used in the noise texture abstraction process 257 is derived; the data may include brightness data (e.g., minimum and maximum L), texture distribution data, object location data (e.g., locations of important objects, where the locations are expressed as vectors), data about dominant colors, and image boundary data. The noise texture abstraction process 257 may then generate a noise attribute list and a vector attribute list 267 in operation 261. The noise attribute list may identify noise textures, such as noise textures 263 and 265, that have been created to represent one or more portions of the image; the vector attribute list may identify the location of each noise texture used to represent one or more portions of the image. These noise textures may be based on the image fragments they are to replace, for example, the minimum and maximum brightness of the noise textures are approximately the same as the minimum and maximum pixel values in the image fragments to be represented by the noise textures in the generated metadata. Thus, a texture abstraction can include multiple different noise textures that are composited together to represent the entire image. The storage (and transmission) size of an image texture abstraction can be significantly smaller than the actual image, and can be processed faster at compositing and rendering time; furthermore, the texture abstraction can be tailored to provide enough data (but not too much) for the compositing and rendering process.
[0051] Another method of creating an image abstraction is to divide the image into multiple segments and quantize each segment based on the image metadata within each segment. For example, the average brightness value within each segment can be used with a set of thresholds (e.g., three thresholds: less than 0.1 nits, greater than 0.1 nits and less than 600 nits, and greater than 600 nits) to classify each segment into one of the quantization levels. This method is as follows Figure 4B As shown. Source image 301 is one of the content elements to be synthesized and rendered in a displayable unit (such as a web page). Source image 301 is segmented into a plurality of segments, such as segments 305, 307, and 309, and each segment is then quantized to one of different brightness levels. Then, a noise abstract texture may be applied to each segment, such as segments 305A, 307A, and 309A; these noise abstract textures may be based on a combination of Figure 4A Although Figure 4B The segments in are rectangular, but segments of other shapes may be used in one or more embodiments. For example, the shape of each segment may be defined by any perimeter based on image statistics as described above (e.g., see the description of operation 203 above).
[0052] Figure 3A, Figure 3B , Figure 4A and Figure 4B The example shown in can create spatially local metadata within an image. The spatially local metadata can specify the location of each fragment (and its boundaries) within the image, and can also specify one or more image statistics describing each fragment (e.g., average brightness, maximum brightness, minimum brightness, color-related attributes (such as dominant color), color statistics, etc.). The spatially local metadata within the image can help when the designer adjusts the image, for example, by cropping the image. Consider a case where the image contains some dark content (e.g., a forest) and a sun that is generally very bright (e.g., 10,000 nits); for example, the image may be a photo of a forest with the sun over the forest on a bright, cloudless day. If the image is cropped to exclude the sun (e.g., during the process of designing a displayable unit (e.g., a web page), or during compositing due to screen size adjustment or dynamic web layout or animation resulting in cropping), the spatially local metadata can allow the system (e.g., compositing and rendering system) to use only the spatially local metadata within the cropped image (excluding the sun). Selecting only the spatially local metadata within the cropped image will change the maximum brightness value, and may result in changes in how the cropped image (or other images near the cropped image) are modified during compositing and rendering. In one embodiment, metadata nodes (having such spatially local metadata) may be spatially distributed across the image (e.g., in a grid) to ensure adequate coverage of the image. Such metadata may include a description of the spatial tonal distribution in the image. In one embodiment, the methods described in U.S. Patent No. 10,977,764 (assigned to applicant Dolby Laboratories Licensing Corporation) may be used for such distributed metadata nodes.
[0053] Once metadata is created and saved for compositing and rendering, for example, Figure 2A The content elements and metadata may then be used by a data processing system for synthesizing and rendering displayable units, as in operation 109 in FIG. Figure 5AA simplified version of a method that can be performed by such a data processing system is provided. In operation 351, the data processing system can receive content elements of a displayable unit and can also receive generated metadata so that the content elements can be synthesized in a perceptually meaningful manner based on the metadata. In operation 353, the data processing system analyzes the metadata to determine how to modify one or more content elements; the analysis can include calculating distances between portions of the content elements and analyzing the metadata based on these distances. The calculated distance can be compared to a distance threshold based on a user's field of view. If the distance exceeds such a field of view, then modification may not be required. In operation 355, the data processing system modifies one or more content elements. As further explained below, the modification can be performed before synthesizing the content elements, after synthesizing the content elements, or both before and after synthesis. The modification can be based in part on the spatial arrangement of the content elements on the displayable unit and in part on the image metadata and the temporal metadata. Once the displayable unit is synthesized and rendered into the frame buffer, it can be displayed in operation 357.
[0054] Figure 5B and Figure 5C Provides metadata based on the generated Figure 2A A more detailed example of a method and pipeline for synthesizing and rendering a displayable unit using the generated metadata stored in operation 109 of FIG. 100 is now described. Figure 5B For a content element having previously generated associated metadata, the method reads the stored metadata in operation 402 . Figure 5B Examples include the case where at least one content element lacks previously generated metadata. In operation 401 during the synthesis process, the content element may be processed to generate metadata; operation 401 may include any one or more of the above methods for generating metadata (such as Figure 2A , Figure 2C , Figure 3A , Figure 3B , Figure 4A or Figure 4B). All metadata may then be provided to operation 403, which analyzes a set of inputs including metadata (from 401 and 402), content in content elements, and layout description 407 (e.g., HTML or CSS or other format describing the arrangement of various content elements on a displayable unit) to determine how to modify one or more content elements based on the inputs. The inputs may also include current data (e.g., real-time data) regarding the state of viewing conditions 405 (e.g., the type of display, ambient light conditions around one or more displays, etc.). In general, the appearance analysis in operation 403 attempts to determine how the physical arrangement and layout of the content will affect the perceived appearance of the content elements. Operation 403 provides one or more outputs to appearance optimizers 409A and 409B, which modify one or more images based on the outputs of operation 403; multiple appearance optimizers allow for parallel processing. In alternative embodiments, there may be only a single appearance optimizer. The adjusted pixel values are then provided to a renderer 411 to create a composite page or other displayable unit 413, which is then displayed 415 on one or more display devices (e.g., an OLED display or a VR or AR headset, etc.).
[0055] Figure 5C More details about operation 403 are shown. Figure 5C As shown in the example of , the data processing system can synthesize the spatial metadata (rather than the content elements themselves) onto the displayable unit during operation 403, and then calculate the distance and brightness difference values between all elements (represented by the spatial metadata) of a given physical layout on the displayable unit. In fact, operation 403 can be regarded as a preliminary synthesis operation that places the spatial metadata (rather than the actual content elements) on the displayable unit using the spatial metadata; this synthesis occurs before the synthesis using the actual content elements in operation 411. Operation 403 can also apply one or more glare models known in the art. Operation 403 can then calculate modifications such as adjustments or offsets; for example, the content elements can include mapping values to ensure that all pixel values are between preselected maximum and minimum brightness values, and these mapping values can be adjusted by operation 403 based on the appearance analysis in operation 403. The modification can be a modification made to a single image content element before synthesis, or a modification made to a group of image content elements after synthesis, or both. In addition, if the contrast between the text and its background is not within the desired range, the appearance of the text can be adjusted; generally speaking, in one embodiment, the minimum contrast of the text should be about 4:1 and the maximum contrast should not exceed about 10:1. Text that has too much contrast relative to the background can cause visual discomfort, while text that has too little contrast can be difficult to read. Low-contrast text near images with glare can also be difficult to read. Fig. 6A , Figure 6B , Figure 6C and Fig.6D Various examples of possible modifications are provided.
[0056] Fig. 6A Operation 403 is shown, along with an example of how an appearance optimizer (e.g., appearance optimizer 409A) can modify 427 one or more images based on analysis 425 of spatial metadata of one or more content elements. In this example, a dark image ("pixel / region 1") to the left of an image where glare is detected ("pixel / region 2") would be modified as follows. Fig. 6A Operation 403 determines the distance between the two images with the largest difference in maximum brightness; Operation 403 may also receive input regarding viewing conditions (which may include the viewer's field of view), such as Fig. 6A Based on the glare detected in the right image, the maximum brightness of the dark image on the left is increased, and the chroma or color saturation is also increased. These adjustments can be made by Figure 5B The appearance optimizer shown is performed. There are known techniques in the art for detecting glare, such as glare models known in the art. For example, see https: / / www.lrc.rpi.edu / programs / Transportation / pdf / glareCalculation.pdf. These glare models can use the deBoer scale to classify glare (from obvious to unbearable). These glare models allow designers of displayable pages to control the perceived impact of glare. For example, if the content creator desires, the perceived impact of glare can be creatively modified. For example, it may be desirable to avoid any glare in the rendered image, have a certain amount of glare, or deliberately expect glare (for example, to create a sense of discomfort when telling a story).
[0057] Figure 6B A flow chart is shown, which shows Fig. 6A A method similar to the method shown, except that Figure 6B The method in may be an optional trim pass method for adjusting an image. Metadata for the optional trim pass may be associated with a content element and used to further adjust the content element beyond what the initial analysis may suggest. In this case, the dark image is adjusted to take into account glare near the dark image. A dark image is detected 451 near the image where glare is detected, and a color volume map offset preference 453 is defined to modify the appearance of the dark image.
[0058] Figure 6C Shown in the synthesis Figure 6C2. An example of modification of the content elements before the five content elements 503, 504, 506, 508 and 502 shown in FIG. The first three content elements are modified using tone mapping (or color volume mapping); specifically, tone mapping 505 modifies the appearance of content element 503, tone mapping 507 modifies the appearance of content element 504, and tone mapping 509 modifies the appearance of content element 506. In each of these modifications, the pixel values are modified according to the analysis of the metadata described above to improve the appearance of the final composite unit (HDR canvas 511 with custom dynamic range). The SDR content elements 508 and 502 are mapped to perceptual quantization (PQ as described in ITU-R Rec. BT.2100) containers for display on the HDR canvas using processes known in the art. After the modified content elements are composited into the HDR canvas 511 (in the frame buffer), the displayable units are displayed on display devices 513 and 515. In another embodiment, the modification may occur after the content element is composited onto the canvas of the displayable unit; for example, all content elements may be mapped after compositing. In yet another embodiment, the modification may occur before and after compositing.
[0059] Composite content typically includes video and animation composited with static images and text. Web pages typically include video, text, and static images (often from different sources) composited together on the same web page. Videos and animations can have images that change dramatically over time (e.g., a night scene in a movie followed by a daylight scene of a bright beach in the movie) or contain image elements that appear or disappear over time or based on triggers. How content appears around a movie on the page depends on spatial metadata and temporal metadata; in addition, content in the movie can be modified. Temporal metadata typically changes over time as videos and animations change, and may result in the modification of one or more content elements. Fig.6D An example 551 of how spatial and temporal metadata may be used to modify one or more content elements while playing three scenes 553, 555, and 557 of two videos on a displayable unit is shown. The methods described for the static page may be used within each scene to modify the content based on the metadata within each scene. If the video content is Dolby Vision content using existing dynamic metadata based on Dolby Vision, the dynamic metadata may be compared to the spatial metadata and adjusted throughout playback.
[0060] Additionally, the transition of the temporally local metadata from scene 553 to scene 555 and scene 557 need not be abrupt. Rather, in one embodiment, the transition may be gradual or temporally smooth, such as by following an adaptive temporal process (based on adaptive models of light, dark, and color adaptation known in the art). For example, 403 may change the mapping metadata identified in scene 553 to gradually transition to the mapping metadata identified in scene 555 over time. This may also help optimize and better utilize the available color volume of the output display and avoid signal clipping, compression, or loss of contrast.
[0061] Figure 7 An example of a data processing system 800 is shown, which can be used for one or more embodiments described herein. For example, the system 800 can be used to perform any of the methods described herein, such as Figure 2A , Figure 2C , Figure 3A , Figure 3B , Figure 4A , Figure 4B , Figure 5A , Figure 5B , Figure 5C , Fig. 6A , Figure 6B , Figure 6C and Fig.6D Such a data processing system can create metadata and is Figure 2B Further, such a data processing system may be Figure 1B Any of the systems shown. Such a system may include a web browser that synthesizes and renders a web page using the metadata described herein. Note that although Figure 7 Various components of the device are illustrated, but are not intended to represent any particular architecture or manner of interconnecting these components, as such details are not relevant to the present disclosure. It will also be understood that network computers and other data processing systems or other consumer electronic devices with fewer or perhaps more components may also be used with embodiments of the present disclosure. In one embodiment, the data processing system may be a smartphone or other mobile device.
[0062] As in Figure 7As shown in FIG. 8 , a device 800 as a form of a data processing system includes a bus 803, which is coupled to a microprocessor 805 and a ROM (read only memory) 807 and a volatile RAM 809 and a non-volatile memory 811. The microprocessor 805 can obtain instructions from the memory 807, 809, 811 and execute the instructions to perform the operations described above. The microprocessor 805 may include one or more processing cores. The bus 803 interconnects these different components together, and also interconnects these components 805, 807, 809 and 811 to a display controller and a display device 813 and peripheral devices, such as an input / output (I / O) device 815 (which may be a touch screen, a mouse, a keyboard, a modem, a network interface, a printer, one or more cameras, and other devices known in the art). Typically, the input / output device 815 is coupled to the system through an input / output controller 810. The volatile RAM (random access memory) 809 is typically implemented as a dynamic RAM (DRAM), which requires continuous power supply in order to refresh or maintain the data in the memory.
[0063] The non-volatile memory 811 is typically a magnetic hard drive or a magnetic optical drive or an optical drive or a DVD RAM or flash memory or other type of memory system that can maintain data (e.g., large amounts of data) even after the system is powered off. Typically, the non-volatile memory 811 will also be random access memory, but this is not required. Although Figure 7 The non-volatile memory 811 is shown as a local device directly coupled to the rest of the components in the data processing system, but it should be understood that embodiments of the present disclosure may utilize non-volatile memory that is remote from the system (e.g., a network storage device coupled to the data processing system via a network interface such as a modem, an Ethernet interface, or a wireless network). As is known in the art, the bus 803 may include one or more buses connected to each other through various bridges, controllers, and / or adapters.
[0064] The part described above can be implemented with logic circuits such as special logic circuits or with processing cores of other forms of microcontrollers or execution program code instructions. Therefore, the process taught by the above discussion can be executed with program codes such as machine executable instructions, and these machine executable instructions make the machine executing these instructions perform certain functions. In this context, "machine" can be a machine (for example, an abstract execution environment, such as a "virtual machine" (for example, a Java virtual machine), an interpreter, a common language runtime, a high-level language virtual machine, etc.) that converts intermediate form (or "abstract") instructions into processor-specific instructions and / or an electronic circuit (for example, a "logic circuit" implemented with a transistor) arranged on a semiconductor chip (designed to execute instructions, such as a general-purpose processor and / or a special-purpose processor). The process taught by the above discussion can also be executed by an electronic circuit (as a substitute for a machine or in combination with a machine) that is designed to execute a process (or a part thereof) without executing a program code.
[0065] The present disclosure also relates to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the desired purpose, or the apparatus may include a general-purpose device that is selectively activated or reconfigured by a computer program stored in the device. Such a computer program may be stored in a non-transitory computer-readable storage medium, such as but not limited to any type of disk, including a floppy disk, an optical disk, a CD-ROM, and a magneto-optical disk, a DRAM (volatile), a flash memory, a read-only memory (ROM), a RAM, an EPROM, an EEPROM, a magnetic card or an optical card, or any type of medium suitable for storing electronic instructions, and each is coupled to a device bus.
[0066] A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, non-transitory machine-readable media include read-only memory ("ROM"); random access memory ("RAM"); magnetic disk storage media; optical storage media; flash memory devices; and the like.
[0067] Articles of manufacture may be used to store program code. Articles of manufacture storing program code may be embodied in, but not limited to, one or more non-transitory memories (e.g., one or more flash memories, random access memories (static, dynamic, or other)), optical disks, CD-ROMs, DVD ROMs, EPROMs, EEPROMs, magnetic or optical cards, or other types of machine-readable media suitable for storing electronic instructions. Program code may also be downloaded from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a data signal embodied in a propagation medium (e.g., via a communication link (e.g., a network connection)) and then stored in a non-transitory memory (e.g., DRAM or flash memory or both) of the client computer.
[0068] The foregoing detailed description is presented in terms of algorithms and symbolic representations of operations on data bits within a device memory. These algorithmic descriptions and representations are tools used by those skilled in the art of data processing to most effectively convey the substance of their work to others skilled in the art. An algorithm here is generally considered to be a consistent sequence of operations that produces a desired result. These operations are operations that require physical manipulation of physical quantities. Typically, but not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times (mainly for common reasons) to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.
[0069] It should be borne in mind, however, that all of these terms and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise clearly stated from the above discussion, it should be understood that throughout the specification, discussions utilizing terms such as "receiving," "determining," "sending," "terminating," "waiting," "changing," etc., refer to actions and processes of manipulating data represented as physical (electronic) quantities within registers and memories of a device and transforming that data into other data similarly represented as physical quantities within a device memory or register or other such information storage device, transmission device, or display device, or similar electronic computing device.
[0070] The process and display presented herein are not inherently associated with any particular device or other apparatus. According to the teachings herein, various general purpose systems can be used together with the program, or it can be demonstrated that it is convenient to build a more specialized device to perform the described operation. The required structure of various these systems will be apparent from the following description. In addition, the present disclosure is not described with reference to any particular programming language. It will be understood that various programming languages can be used to implement the teachings of the present disclosure described herein.
[0071] Exemplary Embodiments
[0072] The following text presents numbered embodiments in a claim-like format, and it should be understood that these embodiments may be presented as claims in one or more future applications, such as one or more continuation or divisional applications.
[0073] Although separate embodiments are described in detail below, it should be understood that these embodiments may be combined or modified in part or in whole. In addition, each of these embodiments may also be expressed as a machine-readable medium or a data processing system, rather than a method.
[0074] Embodiment 1. A method for processing data, the method comprising:
[0075] Receiving a set of elements, the set of elements comprising content and data representing a position or positioning rule of each element in the set of elements on a displayable unit;
[0076] Identify one or more types of content within each element;
[0077] generating a set of metadata from the set of elements, the set of metadata used to create the composite content from the set of elements on the displayable unit when displaying the composite content, the set of metadata comprising: (1) spatial data about the elements and (2) image metadata about at least some of the elements; and
[0078] The generated set of metadata is stored in association with each element.
[0079] Embodiment 2. A method as described in Embodiment 1, wherein the displayable unit is one of: (a) a page, paper, folio or other content unit; or (b) a web page; or (c) part or all of a screen of a display device; or (d) content generated at least in part by a computer program.
[0080] Embodiment 3. The method according to embodiment 2, wherein the method further comprises:
[0081] The set of elements and the generated set of metadata are transmitted in response to a request for the web page, and wherein the data representing the position or positioning rule is included in a description of the displayable element in hypertext markup language.
[0082] Embodiment 4. A method as described in any of the preceding embodiments, wherein the one or more types of content include at least one of the following: (a) high dynamic range (HDR) image content; (b) standard dynamic range (SDR) image content; (c) text content; or (d) user interface content for receiving input from a user.
[0083] Embodiment 5. The method of any of the preceding embodiments, wherein the spatial data comprises vector-based spatial data defining approximate boundaries of each element in the group of elements on the displayable unit.
[0084] Embodiment 6. A method as described in any of the preceding embodiments, wherein the image metadata includes color volume attributes or image statistics of at least some elements in the group of elements.
[0085] Embodiment 7. A method as described in any of the preceding embodiments, wherein the image statistics include one or more of the following: the maximum brightness of the image; the diffuse white level and white point of the image, the minimum brightness of the image; the average brightness of the image; or the median brightness of the image.
[0086] Embodiment 8. A method as described in any of the preceding embodiments, wherein the set of metadata further describes temporal changes of the content over time, and wherein the set of metadata describes a rate of change associated with the temporal changes of the content over time, the rate of change indicating the rate at which metadata in the set of metadata changes from one scene to the next over time.
[0087] Embodiment 9. A method as described in any of the preceding embodiments, wherein the image metadata includes data about detected glare or data used to detect glare in at least one element in the group of elements.
[0088] Embodiment 10. A method as described in any of the preceding embodiments, wherein the image metadata includes a texture abstraction of at least one element in the group of elements.
[0089] Embodiment 11. The method of embodiment 10, wherein the texture abstraction is derived from a Fourier analysis-based representation of at least one element in the set of elements.
[0090] Embodiment 12. A method as described in any of the preceding embodiments, wherein the image metadata includes a quantized representation of at least one element in the group of elements.
[0091] Embodiment 13. A method as described in any of the preceding embodiments, wherein an application programming interface (API) is used to cause the generation of the set of metadata, wherein the API links a metadata generation component in a data processing system with a web page creation software.
[0092] Embodiment 14. The method of any one of Embodiments 9 to 13, wherein the detected glare is classified as one of disabling glare or discomfort glare.
[0093] Embodiment 15. A method as described in any of the preceding embodiments, wherein the set of metadata is stored in a scalable vector graphics format.
[0094] Embodiment 16. A method for processing data, the method comprising:
[0095] receiving a set of metadata for creating composite content from a set of elements to be composited together for display on a displayable unit, the set of metadata comprising: (1) spatial data about the elements and (2) image metadata about at least some of the elements;
[0096] processing the set of metadata to determine how to modify one or more elements based on the set of metadata;
[0097] modifying one or more elements based on the set of metadata; and
[0098] The composite content having the modified one or more elements is rendered to display the displayable unit on a display device.
[0099] Embodiment 17. A method as described in Embodiment 16, wherein the displayable unit is one of the following: (a) a page, paper, folio or other content unit; or (b) a web page; or (c) part or all of the screen of a display device; or (d) content generated at least in part by a computer program.
[0100] Embodiment 18. A method as described in Embodiment 17, wherein the receiving is in response to a request for the web page, and the set of metadata is received by a web browser, and wherein at least a portion of the modification occurs after the content is synthesized.
[0101] Embodiment 19. A method as described in any one of Embodiments 16 to 18, wherein the synthesized content includes at least one of the following: (a) high dynamic range (HDR) image content; (b) standard dynamic range (SDR) image content; (c) text content; or (d) user interface content for receiving input from a user.
[0102] Embodiment 20. A method as described in any one of Embodiments 16 to 19, wherein the modification is based at least in part on the spatial data and the image metadata.
[0103] Embodiment 21. A method as described in any one of Embodiments 16 to 20, wherein the spatial data includes vector-based spatial data that defines an approximate boundary of each element in the group of elements on the displayable unit, and the approximate boundary is used to calculate the distance between at least two elements in the group of elements.
[0104] Embodiment 22. A method as described in any one of Embodiments 16 to 21, wherein the image metadata includes color volume attributes or image statistics of at least some of the elements in the group of elements.
[0105] Embodiment 23. A method as described in any one of Embodiments 16 to 22, wherein the image statistics include one or more of the following: the maximum brightness of the image; the minimum brightness of the image; the average brightness of the image; or the median brightness of the image.
[0106] Embodiment 24. A method as described in any one of Embodiments 16 to 23, wherein the set of metadata also describes temporal changes of the content over time, and wherein the set of metadata describes a rate of change associated with the temporal changes of the content over time, the rate of change indicating the rate at which metadata in the set of metadata changes from one scene to the next over time.
[0107] Embodiment 25. A method as described in any of Embodiments 16 to 24, wherein the image metadata includes data about detected glare or data used to detect glare in at least one element in the group of elements, and wherein the modification reduces the detected glare.
[0108] Embodiment 26. The method of any one of embodiments 16 to 25, wherein the method further comprises:
[0109] determining a distance between a first element and a second element in the group of elements on the displayable unit, and determining a difference in image data statistics between the first element and the second element;
[0110] One or both of the first element or the second element are modified based on the determined distance and the determined difference.
[0111] Embodiment 27. A method as described in any one of Embodiments 16 to 26, wherein the modification of the first element or the second element occurs before or after synthesizing the content.
[0112] Embodiment 28. A method as described in any one of Embodiments 16 to 27, wherein the set of metadata is received in a scalable vector graphics format.
[0113] Embodiment 29. A method as described in any one of Embodiments 16 to 28, wherein the image metadata includes a texture abstraction of at least one element in the group of elements.
[0114] Embodiment 30. A method as described in any one of Embodiments 16 to 29, wherein the modification takes into account one or more of the following: (a) the on-screen and off-screen status of the content in the displayable unit; (b) the display device used to display the content in the displayable unit; (c) the surrounding viewing environment around the display device displaying the displayable unit; or (d) the viewing distance of a viewer of the displayable unit.
[0115] Embodiment 31. A non-transitory machine-readable medium storing executable program instructions, which, when executed by one or more data processing systems, cause the one or more data processing systems to perform a method as described in any one of Embodiments 1 to 30.
[0116] Embodiment 32. A data processing system configured to execute the method as described in any one of embodiments 1 to 30.
Claims
1. A method for processing data, the method comprising: receiving a set of elements, the set of elements comprising content and data representing a position or positioning rule of each element of the set of elements on a displayable unit; Identify one or more types of content within each element; Generating a set of metadata from the set of elements, the set of metadata being used to create the composite content from the set of elements on the displayable unit when displaying the composite content, the set of metadata comprising: (1) spatial data about the elements and (2) image metadata about at least some of the elements, wherein the spatial data comprises vector-based spatial data that defines approximate boundaries of each element in the set of elements on the displayable unit, and wherein the image metadata comprises color volume attributes or image statistics of at least some of the elements in the set of elements; and wherein the image statistics comprise one or more of: a maximum brightness of the image; a diffuse white level and white point of the image, a minimum brightness of the image; an average brightness of the image; or a median brightness of the image; and A generated set of metadata is stored in association with each element.
2. The method of claim 1, wherein: The displayable unit is one of: (a) a page, paper, folio or other unit of content; or (b) a web page; or (c) part or all of a screen of a display device; or (d) content generated at least in part by a computer program.
3. The method of claim 2, wherein: The method further comprises: The set of elements and the generated set of metadata are transmitted in response to a request for the web page, and wherein the data representing the position or positioning rules is included in a description of the displayable element in hypertext markup language.
4. A method as claimed in any one of the preceding claims, wherein: The one or more types of content include at least one of the following: (a) high dynamic range (HDR) image content; (b) standard dynamic range (SDR) image content; (c) text content; or (d) user interface content for receiving input from a user.
5. A method as claimed in any one of the preceding claims, wherein: The set of metadata also describes temporal changes in the content over time, and wherein the set of metadata describes a rate of change associated with the temporal changes in the content over time, the rate of change indicating a rate at which metadata in the set of metadata changes from one scene to the next over time.
6. A method as claimed in any one of the preceding claims, wherein: The image metadata includes data regarding detected glare or data used to detect glare in at least one element of the set of elements, and wherein the detected glare is classified as one of disabling glare or discomfort glare.
7. A method as claimed in any one of the preceding claims, wherein: The image metadata includes a texture abstraction of at least one element of the set of elements.
8. The method of claim 7, wherein: The texture abstraction is derived from a Fourier analysis based representation of at least one element of the set of elements.
9. A method as claimed in any one of the preceding claims, wherein: The image metadata includes a quantized representation of at least one element of the set of elements.
10. A method as claimed in any one of the preceding claims, wherein: An application programming interface (API) is used to cause generation of the set of metadata, the API linking metadata generation components in the data processing system with the web page creation software.
11. A method as claimed in any one of the preceding claims, wherein: The set of metadata is stored in a scalable vector graphics format.
12. A non-transitory machine-readable medium storing executable program instructions which, when executed by one or more data processing systems, cause the one or more data processing systems to perform the method of any one of claims 1 to 11.
13. A data processing system configured to execute the method according to any one of claims 1 to 11.
Citation Information
Patent Citations
Polymeric ultraviolet light stabilisers containing hindered alkyl amines and polymer compositions containing them
EP0002022A1
Viewport independent image coding and rendering
US10977764B2
User interface and graphics composition with high dynamic range video
US20150245004A1
Heterophasic propylene copolymer
US20200062942A1
User-guided image segmentation methods and products
WO2021113400A1
