Data processing system

Abstract

Data processing system. A data processing system includes: encoding circuitry operable to encode an array of data elements; decoding circuitry operable to decode an encoded version of the array of data elements; and consumer circuitry operable to use the array of data elements. The encoding circuitry encodes the array of data elements to produce an encoded version of the array of data elements by dividing the array of data elements into a plurality of separate blocks, generating respective representations for representing the different blocks into which the array of data elements is divided, and generating data representing the respective representations to represent the array of data elements as the encoded version of the array of data elements. The decoding circuitry decodes at least a portion of the encoded version of the array of data elements to produce a decoded version of the array of data elements, and the consumer circuitry uses at least a portion of the decoded version of the array of data elements. The encoding circuitry uses data indicating a resolution to control generation of a representation for representing at least one of the blocks into which the array of data elements is divided.

Description

Technical Field

[0001] The present invention relates to data processing systems, and more particularly to techniques for encoding data in data processing systems. Background Art

[0002] In data processing systems, it is common to encode arrays of data elements, such as arrays of image data values ​​(e.g., frames of video data or graphics data for display) to compress the data and thereby reduce bandwidth and storage consumption. This is particularly desirable in data processing devices, such as portable devices, where processing resources and capabilities may be limited.

[0003] To encode an array of data elements, the array of data elements is often divided into smaller blocks (sub-regions) of data elements and encoded block by block.

[0004] U.S. Patent Application No. 2013 / 0034309 describes a lossless encoding technique for encoding an array of data elements, in which a quadtree representation is generated for each block of the data array, wherein each leaf node of the quadtree represents a corresponding data element of the data array, and wherein data values ​​for the tree nodes are set so that the data value for the data element represented by the leaf node is given by the sum of the data value of the leaf node in the tree and the data values ​​of each previous parent node in the tree branch to which the leaf node belongs. Summary of the invention

[0005] Applicants believe that there is scope for improvement in techniques for encoding arrays of data elements.

[0006] According to an aspect of the present invention, there is provided a method of operating a data processing system, the data processing system comprising:

[0007] encoding circuitry operable to encode the array of data elements;

[0008] decoding circuitry operable to decode an encoded version of the array of data elements; and

[0009] a consumer circuit operable to consume the array of data elements;

[0010] The method comprises the following steps:

[0011] The encoding circuit encodes the data element array to produce an encoded version of the data element array by dividing the data element array into a plurality of separate blocks, generating respective representations for representing the different blocks into which the data element array is divided, and generating data representing the respective representations to represent the data element array as the encoded version of the data element array;

[0012] Decoding circuitry decodes at least a portion of the encoded version of the array of data elements to produce a decoded version of the array of data elements; and

[0013] The consumer circuit uses at least a portion of the decoded version of the array of data elements;

[0014] The method further comprises the following steps:

[0015] providing data to the encoding circuitry indicating a resolution to be used by the consumer circuitry for at least a region of the array of data elements; and

[0016] The encoding circuitry uses data indicative of a resolution to be used by the consumer circuitry to control generation of a representation representing at least one block into which the array of data elements is divided.

[0017] According to another aspect of the present invention, there is provided a data processing system, the data processing system comprising:

[0018] encoding circuitry operable to encode the array of data elements;

[0019] decoding circuitry operable to decode an encoded version of the array of data elements; and

[0020] a consumer circuit operable to consume the array of data elements;

[0021] in,

[0022] The encoding circuit is configured to encode the data element array to produce an encoded version of the data element array by: dividing the data element array into a plurality of separate blocks, generating respective representations for representing the different blocks into which the data element array is divided, and generating data representing the respective representations to represent the data element array as the encoded version of the data element array;

[0023] The decoding circuit is configured to decode at least a portion of the encoded version of the array of data elements to produce a decoded version of the array of data elements; and

[0024] The consumer circuit is configured to use at least a portion of the decoded version of the array of data elements;

[0025] And among them,

[0026] The data processing system is configured to provide data to the encoding circuitry indicating a resolution to be used by the consumer circuitry for at least a region of the array of data elements; and

[0027] The encoding circuit is configured to control generation of a representation representing at least one block into which the array of data elements is divided using data indicative of a resolution to be used by the consumer circuit.

[0028] Various embodiments of the present invention are directed to a data processing system in which an array of data elements, such as an array of image data values ​​(e.g., a frame of video data or graphics data for display), is encoded, decoded, and then used (e.g., displayed). Encoding the data array (e.g., as it is generated) and then subsequently decoding the data array when it is to be used (e.g., when it is to be displayed) can advantageously reduce bandwidth and storage consumption of the data processing system.

[0029] In the present invention, a data array is encoded by generating a corresponding representation, such as a tree representation, for each of a plurality of separate distinct blocks into which the array is divided, and then generating data representing each (tree) representation, for example in a manner similar to that described in U.S. Patent Application No. 2013 / 0034309 (the contents of which are incorporated herein by reference in their entirety). Such encoding of a data array, as described in U.S. Patent Application No. 2013 / 0034309, is a particularly efficient and convenient technique for encoding and compressing a data array.

[0030] Furthermore, in various embodiments of the invention, data indicating the resolution of at least one region of the decoded data array that the consumer circuit is to use is provided to the encoding circuit, and the encoding circuit is configured to use this data to control its encoding operation, in particular to control the generation of a (tree) representation for representing at least one block of the data array. Thus, for example, instead of simply generating each (tree) representation for representing each block of the data array in the same way (e.g. using the same algorithm), one or more or each (tree) representation for representing one or more or each block of the data array may be generated in a different way (e.g. using a different algorithm) (i.e. depending on the resolution of the corresponding region of the decoded data array that the consumer circuit is (ultimately) to use).

[0031] As will be described in more detail below, this facilitates greater flexibility and control over encoding operations, and in particular can facilitate using even less data to represent a data array (thereby facilitating compression of the data array) relative to the original form of the data array and relative to the compressed data array described in U.S. Patent Application No. 2013 / 0034309.

[0032] For example (and in various embodiments), where a consumer circuit is to use (e.g., display) a particular region of a decoded data array at a lower (e.g., reduced) resolution, the encoding circuitry may use data provided to the encoding circuitry to control generation of a (tree) representation during the encoding process such that a higher (e.g., improved) level of compression is used, such as by encoding one or more blocks (e.g., blocks corresponding to the particular region) of the data array using lossy compression (such as a lossy version of the encoding technique described in U.S. Patent Application No. 2013 / 0034309) (described in more detail below).

[0033] On the other hand, where the consumer circuit is to use (e.g., display) another particular region of the decoded data array at a higher (e.g., increased or normal) resolution, the encoding circuitry may use the data provided to the encoding circuitry to control the generation of a (tree) representation during the encoding process such that a lower (e.g., normal or reduced) level of compression is used, such as by encoding one or more blocks (e.g., blocks corresponding to another particular region) of the data array using lossless compression (e.g., the lossless compression techniques described in U.S. patent application Ser. No. 2013 / 0034309).

[0034] This means that where appropriate, at least some areas of the array can be compressed using higher compression levels (such as by using lossy compression), which in turn means that the overall memory bandwidth and power requirements of the system can be reduced. Thus, the memory bandwidth and power used to process a given data array can be reduced, and / or a larger data array can be processed using the same amount of memory bandwidth and power.

[0035] Moreover, this can be done without significantly affecting the quality of the data array when used (e.g. displayed) by the consumer circuitry. This is because when one or more regions of (a decoded version of) the array of data elements are to be used (e.g. displayed) at a lower resolution (e.g. at a reduced resolution relative to the (highest) resolution at which the data array was generated), some data will actually need to be discarded, and it will not matter at which stage in the processing this discarding occurs. Discarding this data during the encoding process allows this data to be discarded at an earlier stage in the processing and in a particularly convenient and efficient manner.

[0036] It will thus be appreciated that the present invention provides an improved method of operating a data processing system.

[0037] The invention also extends to the operation of the encoding circuit alone to encode an array of data elements in the manner of the invention.

[0038] Thus, according to another aspect of the present invention, there is provided a method of encoding an array of data elements using encoding circuitry of a data processing system, the method comprising the following steps:

[0039] The encoding circuit encodes the data element array to produce an encoded version of the data element array by dividing the data element array into a plurality of separate blocks, generating respective representations for representing the different blocks into which the data element array is divided, and generating data representing the respective representations to represent the data element array as the encoded version of the data element array;

[0040] The method further comprises the following steps:

[0041] providing data to encoding circuitry indicating a resolution to be used for at least one region of the array of data elements; and

[0042] The encoding circuitry uses data indicating a resolution at which at least one region of the array of data elements is to be used to control generation of a representation for representing at least one block into which the array of data elements is divided.

[0043] According to another aspect of the present invention, there is provided a data processing system, the data processing system comprising:

[0044] encoding circuitry configured to encode the data element array to produce an encoded version of the data element array by dividing the data element array into a plurality of separate blocks, generating respective representations for representing the different blocks into which the data element array is divided, and generating data representing the respective representations to represent the data element array as the encoded version of the data element array;

[0045] wherein the data processing system is configured to provide the encoding circuit with data indicating a resolution at which at least one region of the array of data elements is to be used; and

[0046] The encoding circuitry is configured to control generation of a representation representing at least one block into which the array of data elements is divided, using data indicating a resolution at which at least one region of the array of data elements is to be used.

[0047] The array of data elements encoded by the encoding circuit may be any suitable data array. It should include a plurality of data elements (entries), each occupying a different position in the array. The data array may be an image (may represent an image). In various embodiments, the data array is, for example, a frame of (image) (e.g., color) data for display, or a graphics texture. It will also be possible to make the array of data elements include an array of other graphics data, such as an array of depth data.

[0048] In a preferred embodiment, the arrays of data elements are frames generated for display for an application (such as a game) and preferably for an augmented reality or virtual reality application. Thus, each array of data elements (frame) preferably comprises an image to be displayed.

[0049] The data array element data values ​​may take any suitable and desired form, and will depend on the nature of the data array being encoded, e.g., whether it is a texture, an image, a frame, etc. For example, in the case of a texture, the data array element data values ​​should be texture data (texel values). Such texture data may include, for example, a set of color values ​​(red, green, blue (RGB)), a set of color and transparency values ​​(red, green, blue, alpha (RGBa)), a set of brightness and chrominance values, a set of shadow (light) map values, a set of normal map (bump map) values, z-values ​​(depth values), stencil values, brightness values ​​(brightness texture), brightness alpha texture and / or gloss map, etc.

[0050] In the case of an image or frame for display, the data array element data value should be pixel and / or sample position data (data value). Such pixel data may include, for example, appropriate color (RGB) values ​​or brightness and chrominance values, etc.

[0051] The array of data elements may be generated in any suitable manner. The array of data elements may be generated by a generator circuit (and the data processing system may include a generator circuit), which may include, for example, a graphics processing unit (GPU), a video processor / codec or a video engine (video processing unit (VPU)), a digital camera image signal processor (ISP), an image processor and / or a central processing unit (CPU), etc. If desired, there may be more than one generator circuit. In a preferred embodiment, the generator circuit is a graphics processor and / or a video processor.

[0052] The generator circuit should generate its array in an appropriate manner, for example, by rendering the array in the case of a graphics processor, by appropriately decoding the input encoded video data in the case of a video processor, from the captured image in the case of a digital camera image signal processor (ISP), etc.

[0053] Once generated, the array of data elements should (and preferably is) provided (sent) to encoding circuitry for encoding.

[0054] The encoding circuitry may comprise any suitable circuitry operable to encode the array of data elements to produce an encoded version of the array of data elements, and the encoding circuitry may encode the array of data elements in any suitable manner.

[0055] The encoding circuit is preferably operable to encode the array of data elements generated by the generator circuit. Thus, the method preferably comprises the step of: the encoding circuit encoding the array of data elements generated by the generator circuit. The encoding circuit may encode only a portion of the entire data array (e.g. a frame, image or texture) generated by the generator circuit, but preferably encodes the entire array (e.g. a frame, image or texture) generated by the generator circuit.

[0056] To facilitate this, the encoding circuitry may (and in various embodiments does) form part of a generator circuitry, which may be, for example, the (integrated) encoding circuitry of a graphics processing unit (GPU), a video processor / codec or a video engine (video processing unit (VPU)), a digital camera image signal processor (ISP), an image processor and / or a central processing unit (CPU), etc. Thus, in these embodiments, the generator circuitry both generates an array of data elements and encodes the array (using its encoding circuitry).

[0057] In various other embodiments, the encoding circuit may be independent of the generator circuit, for example, may be a circuit (e.g., an integrated circuit) independent of the generator circuit, for example, may include an "independent" encoding unit, for example, which is configured to receive (and encode) an array of data elements from the generating circuit.

[0058] Once generated (encoded), the encoded array of data elements may be stored, for example, in a buffer such as a frame buffer, a memory, from which it may then be read (e.g., by a decoding circuit). The memory in which the encoded array may be stored may include any suitable memory, and may be configured in any suitable and desired manner. For example, the memory may be an on-chip memory with the encoding circuit, or may be an external memory. In an embodiment, the memory is an external memory, such as the main memory of the entire data processing system. The memory may be a dedicated memory for this purpose, or it may be a part of a memory also used for other data.

[0059] The encoding circuit is operable to encode the data element array to produce an encoded version of the data element array by dividing the data element array into a plurality of separate blocks, generating respective representations representing the different blocks into which the data element array is divided, and generating data representing the respective representations to represent the data element array as the encoded version of the data element array.

[0060] Thus, in various embodiments, in an encoding operation, the data array is divided into a plurality of separate regions or blocks, and a corresponding representation, such as a tree representation, is generated for each different block (region) into which the data array is divided. That is, a plurality of (e.g., tree) representations are generated, one (e.g., tree) representation for each block into which the data element array is divided. Data representing the plurality of representations is then generated to produce an encoded version of the data element array.

[0061] The blocks (regions) into which the array of data to be encoded is divided can take any suitable and desired form. Each block should include a subset of the data elements (positions) in the array, that is, corresponding to a specific area of ​​the array. In an embodiment, the array is divided into non-overlapping and regularly sized and shaped blocks. In an embodiment, the blocks are square, but other structures can be used (if desired). The blocks can correspond to the block sizes that will be used in other ways in the data processing system under discussion. Thus, for example, in the case of a tile-based graphics processing system, the blocks can correspond to the tiles (having the same size and configuration) on which the rendering process of the graphics processing system operates.

[0062] In an embodiment, the data array is divided into 16x16 blocks (i.e., blocks of 16x16 array positions (entries)). In one such structure, a single tree representation is generated for each 16x16 block. Thus, in the case of a texture map, for example, a separate tree representation would be generated for each (non-overlapping) 16x16 texel region of the texture map, while in the case of an image or frame, a tree representation would be generated for each 16x16 pixel or sample position region of the image or frame.

[0063] Other structures are of course possible. For example, instead of generating a single tree representation for a 16x16 block, four trees could be generated, each representing an 8x8 or 16x4 block within the 16x16 block (so the data array would actually be divided into 8x8 or 16x4 blocks).

[0064] Each representation may include any suitable representation for representing a block of the data array. In various preferred embodiments, one or more or each representation includes a tree representation, such as described in U.S. Patent Application No. 2013 / 0034309. Thus, according to various embodiments, the encoding circuit is operable to encode the data element array to produce an encoded version of the data element array by dividing the data element array into a plurality of separate blocks, generating a corresponding tree representation for representing each of the different blocks into which the data element array is divided, and generating data representing each tree representation to represent the data element array as an encoded version of the data element array. Other representations may (and in various embodiments are) be used.

[0065] The encoding operation will be described in more detail below.

[0066] The decoding circuitry may comprise any suitable such circuitry operable to decode an encoded version of the array of data elements, and may decode the encoded version of the array of data elements in any suitable manner.

[0067] The decoding circuitry is preferably operable to decode the array of data elements encoded by the encoding circuitry.To do so, the decoding circuitry preferably receives at least part of an encoded version of the array of data elements, for example by reading the encoded array from a memory storing the encoded array.

[0068] The decoding circuitry shall (read and) decode at least a portion of the encoded version of the array of data elements to produce a decoded version of the array of data elements. The decoding circuitry shall (and preferably does) (read and) decode (encode) at least those portions of the data array to be used by the consumer circuitry. This may include (read and) decode some (but not all) of the encoded data array (e.g., frame, image or texture), but preferably includes (read and) decode all (the entirety) of the encoded data array (e.g., frame, image or texture).

[0069] The decoded version of the array of data elements produced by the decode circuit should (and preferably does) include at least a portion, and preferably all, of the array of data elements produced by the generator circuit.

[0070] Once generated (decoded), the decoded data array should (and preferably is) provided by (eg, sent from) the decoding circuit to the consumer circuit for use.

[0071] To facilitate this, the decoding circuitry may (and in various embodiments does) form part of the consumer circuitry, which may be, for example, the (integrated) decoding circuitry of a display controller or a graphics processing unit (GPU) etc. Thus, in these embodiments, the consumer circuitry both decodes (using its decoding circuitry) the encoded version of the array of data elements and uses the decoded version of the array of data elements.

[0072] In various other embodiments, the encoding circuit may be independent of the consumer circuit, for example, be a circuit (e.g., an integrated circuit) independent of the consumer circuit, for example, may include an "independent" decoding unit that may be configured to decode the array of data elements and provide (send) it to the consumer circuit.

[0073] The decoding circuitry is preferably operable to decode the data arrays in the manner described in U.S. Patent Application No. 2013 / 0034309. As will be described in more detail below, the encoding operations of various embodiments are preferably configured to be transparent to the decoding circuitry, i.e., such that the decoding operations required to decode each array of encoded data elements (and each block of each array of encoded data elements) are the same (and do not need to be changed) regardless of differences in the manner in which (e.g., tree) representations are generated for different data arrays and / or different blocks of data arrays in the encoding operations.

[0074] Thus, the method may comprise the steps of: decoding at least a portion of an encoded version of the array of data elements by: determining a value to be used for each of some or all of the data elements of the data array using data representing a representation, the representation representing some or all of the data elements of the data array.

[0075] Similarly, in various embodiments, the decoding circuitry may be configured to determine values ​​to be used for data elements of some or all of the data elements of the data array using data representing a representation representing some or all of the data elements of the data array.

[0076] In various embodiments, the method may include the steps of: decoding at least a portion of a decoded version of the array of data elements by processing;

[0077] determining values ​​of nodes of branches of the tree representing some or all of the data elements of the data array using data representing the tree, the tree representing some or all of the data elements of the data array; and

[0078] The value to be used for the data element of the data array is determined by summing the determined values ​​for the leaf node of the branch of the tree and each previous parent node in the branch of the tree to which the leaf node belongs.

[0079] Similarly, in various embodiments, the decoding circuit may be configured to:

[0080] determining values ​​of nodes of branches of the tree representing some or all of the data elements of the data array using data representing a tree representing some or all of the data elements of the data array; and

[0081] The value to be used for the data element of the data array is determined by summing the determined values ​​for the leaf node of the branch of the tree and each previous parent node in the branch of the tree to which the leaf node belongs.

[0082] The decoding operation is described in more detail in U.S. Patent Application No. 2013 / 0034309, which is incorporated herein by reference.

[0083] The consumer circuit may comprise any suitable such circuitry operable to use an array of data elements, and may use such an array of data elements in any suitable manner.

[0084] The consumer circuit is preferably operable to use a decoded array of data elements generated (decoded) by the decoding circuit. The consumer circuit may use (e.g. display) only a portion (some but not all) of a decoded version of an array of data elements (e.g. a frame, image or texture), but preferably uses (e.g. displays) the entire (entire) decoded array (e.g. a frame, image or texture). The array of data elements used (e.g. displayed) by the consumer circuit should (and preferably does) include at least a portion of the array of data elements generated by the producer circuit and preferably includes the entirety of the decoded version.

[0085] The nature of the consumer circuit will depend on the nature of the data array used.

[0086] Thus, for example, where the array of data elements comprises a frame or image for display (e.g., generated by a graphics processing unit (GPU), a video processor, a digital camera image signal processor (ISP), an image processor and / or a central processing unit (CPU), etc.), the consumer circuitry may then comprise, for example, a display controller or a display processor operable to provide the array of data elements (a decoded version thereof) to a display for display. Thus, using the decoded version of the array of data elements preferably comprises providing the (optionally processed) decoded version of the array of data elements (e.g., a frame or image) to a display for display (i.e., displaying the (optionally processed) decoded version of the array of data elements (e.g., a frame or image)).

[0087] In the case where the array of data elements comprises a texture (e.g. generated by a graphics processing unit (GPU)), the consumer circuitry may then comprise, for example, a graphics processing unit (GPU). In this case, the consumer graphics processing unit (GPU) may be the same as the producer graphics processing unit (GPU) (i.e., the graphics processing unit (GPU) may generate and then subsequently use the texture), or the consumer graphics processing unit (GPU) may be a different graphics processing unit (GPU) than the producer graphics processing unit (GPU). Thus, using the decoded version of the array of data elements preferably comprises using the decoded version of the array of data elements (e.g., texture) during a (subsequent) rendering operation. In this case, the array (e.g., texture) may be used in the (subsequent) rendering operation in any suitable manner according to any suitable graphics processing technique (which requires the use of the texture).

[0088] In a preferred embodiment, the consumer circuit comprises a display processor operable to provide the array (eg, a frame or image) to a display for display.

[0089] The display processor may comprise any suitable and desired display processor operable to provide an array of data (e.g., a frame or image) (e.g., generated by a generator circuit of a data processing system) to a display for display. The display processor is preferably operable to read the data of the array (encoded or decoded) from a memory, optionally decode the encoded array (as described above), optionally perform appropriate "display processing" on those arrays, and then provide the optionally processed arrays to the display for display. In a preferred embodiment, the display processor provides the array of data to the display via a "direct" connection to the display, i.e., is operable to stream the data for the array (frame or image) to the display, rather than transferring the array to the display via, for example, a memory. The display processor may have a wired or wireless connection to the display.

[0090] Thus, the display processor may comprise an input stage operable to read data of an input (encoded or decoded) data array from a memory, the data being stored in the memory. In a preferred embodiment, the input stage comprises a memory read subsystem, the memory read subsystem preferably comprising a read controller, such as for example a direct memory access (DMA) read controller, the read controller being configured (operable) to read data of the input data array from a memory storing the input data array.

[0091] The display processor may include an output stage operable to provide an output (decoded) data array (frame or image) for display to a display, e.g., such that the output data array for display is displayed on the display. The output stage preferably includes appropriate timing control functionality for the display (e.g., the output stage is configured to send pixel data to the display with appropriate horizontal and vertical blanking periods). The output stage preferably sends the output data array (frame or image) directly to the display for display, e.g., and preferably, as a stream of data elements (pixels) for display. The data may be provided via a wired or wireless connection to the display (as desired).

[0092] In these embodiments, the data processing system preferably includes a display operable to display the array of data (eg, images or frames for display).

[0093] The display will operate to display the received data array (frame or image) (for example and preferably in the normal manner for the operation of the display in question). Thereby, the data elements (pixels) for the data array will be appropriately scanned onto the panel of the display to display the data array.

[0094] It will be appreciated that, as described above, in various embodiments, an array of data elements (e.g., a frame, image, or texture) is generated, encoded, decoded, and then used (e.g., displayed). While it will be possible to encode and decode the array of data elements only after generation and before use (display), in various embodiments, the array of data elements is also processed after generation and before use (display), e.g., subjected to image, graphics, or other processing.

[0095] Where present, such processing may include any suitable such processing and may be performed by any suitable element of the data processing system in any suitable manner. In various embodiments, the array of data elements may be processed prior to encoding and / or after decoding. For example (and as described above), a display processor may perform suitable display processing (e.g., on the decoded array of data elements), such as compositing, scaling, etc., on the decoded array of data elements before providing the processed decoded array of data to a display for display.

[0096] Other configurations would be possible.

[0097] In the present invention, data indicating the resolution to be used by the consumer circuit for at least one region of (a decoded version of) the array of data elements (i.e., resolution indicating data) is provided to the encoding circuit, and the encoding circuit uses this data to control the generation of a representation for representing at least one block (tree) into which the array of data elements is divided. Thus, preferably, data indicating the resolution to be used by the consumer circuit for one or more regions of (a decoded version of) the array of data elements is generated and provided to the encoding circuit for use by the encoding circuit when encoding the array of data elements.

[0098] In various embodiments, the data processing system is configured to cause the consumer circuit (e.g., a display processor or a GPU) to use (decoded) data element arrays according to (having) one or more resolutions. Preferably, the consumer circuit (e.g., a display processor or a GPU) uses (decoded) data element arrays according to (having) a plurality of different resolutions. Correspondingly, the resolution indication data preferably indicates a plurality of different resolutions to be used by the consumer circuit.

[0099] The plurality of different resolutions may be any suitable set of resolutions. There may be a highest resolution to be used (e.g., displayed) for one or more regions of the data array, and then there may be one or more lower resolutions to be used (e.g., displayed) for other regions of the data array. There may be only two different resolutions: a highest resolution and a lower resolution. Alternatively, three or more different resolutions for the array may be used, including a highest resolution for the data array and two or more lower resolutions.

[0100] A plurality of different resolutions (used by the consumer circuitry and indicated by the resolution indicating data) may relate to each of the plurality of (e.g., consecutive) data arrays, i.e. the data processing system may be configured such that each of the plurality of different (e.g., consecutive) data arrays is used according to a respective different resolution. That is, a plurality of different resolutions may be used (and indicated by the resolution indicating data), each resolution being used for each of the plurality of different (e.g., consecutive) data arrays. In this case, where using the data arrays comprises providing the data arrays to a display (i.e., displaying the data arrays), the method may comprise the following steps: providing a sequence of consecutive data arrays to the display (i.e., displaying a sequence of consecutive (decoded) data arrays), wherein each of the plurality of different data arrays of the sequence is provided to the display (i.e., displayed at that resolution) at a respective different resolution (and the resolution indicating data may indicate each of the respective different resolutions).

[0101] Additionally or alternatively, a plurality of different resolutions (used by the consumer circuitry and indicated by the resolution indicating data) may relate to each of a plurality of different regions of the data array, i.e. the data processing system may be configured such that each of a plurality of different data array regions of the data array is used according to a respective different resolution. That is, a plurality of resolutions may be used (and indicated by the resolution indicating data), each resolution being used for each of a plurality of different data array regions of the data array. In this case, where using the data array comprises providing the data array to a display (i.e. displaying the data array), the method may comprise the steps of providing the data array to a display (i.e. displaying (decoding) the data array), wherein each of the plurality of different regions of the data array is provided to the display (i.e. displayed at that resolution) at a respective different resolution (and the resolution indicating data may indicate each of the respective different resolutions).

[0102] Thus, the resolution-indicative data may indicate a resolution to be used by the consumer circuitry for only one region of (a decoded version of) the array of data elements, or may indicate resolutions to be used for a plurality of different regions of (a decoded version of) the array of data elements. The resolution-indicative data may indicate some but not all resolutions to be used for (a decoded version of) the array of data elements (in which case it may be assumed (by the encoding circuitry) that any non-indicated regions will be used by the consumer circuitry according to, for example, a default resolution), but preferably indicates a resolution to be used for (each and every region of) (a decoded version of) the entire array of data elements.

[0103] In a preferred embodiment, the resolution-indicating data indicates a resolution to be used by the consumer circuit for each of a plurality of different regions into which (a decoded version of) the array of data elements is divided.

[0104] The different regions into which the (decoded) data array is divided (and the indication provided for a region which different resolution is to be used for that region) may comprise any suitable and desired subdivision of the (decoded) array into a plurality of regions. The regions preferably all have the same size and configuration, and are preferably rectangular, preferably square. Each region may correspond to a single data element (pixel) in the decoded array, but preferably each region corresponds to a preferably rectangular and preferably square block of a plurality of data elements (pixels) in the decoded array.

[0105] In a preferred embodiment, the regions into which the (decoded) data array is divided (the indication of which different resolution is to be used for the region being provided for the region) correspond to the plurality of blocks into which the data array is divided for encoding purposes. Thus, preferably the method comprises the step of providing (and the data processing system is preferably configured to provide) to the encoding circuitry data indicating the resolution to be used by the consumer circuitry for each different block into which the array of data elements is divided.

[0106] The resolution-indicative data may take any suitable and desired form, and may indicate different resolutions to be used in any suitable and desired manner.

[0107] In a preferred embodiment, this data is in the form of a two-dimensional array representing (a region of) the (decoded) data array, with each data element position within the array then indicating which resolution is to be used for that region within the data array. Such an array may for example be in the form of a 2D texture or "density map" which describes which resolution is to be used for a corresponding region of the data array.

[0108] In this case, a "density map" may indicate the resolution (subdivision into regions) to be used for the corresponding regions at any desired level of granularity (resolution) over the regions of the data array. Thus, each "density map" data element preferably corresponds to a region that includes a given block of multiple data elements (pixels) in the data array.

[0109] In a particularly preferred embodiment, the two-dimensional array (density texture / map) has a specific size, such as 64x64, 100x100, 128x128 or 256x256 data element positions, and each data element position in the array is then mapped to a corresponding area (block) of the data array.

[0110] In these embodiments, the resolution to be used for a given region of the data array may be indicated in any suitable and desired manner. For example, the data may indicate a single resolution (e.g., using an integer value) of a plurality of different resolutions to be used for the array region (and each array region). The data indicating the different resolutions to be used may also or instead be provided in other forms, for example, by means of a lookup table, or expressed by some form of functional, or in the form of a bounding box (e.g., a 2D axis-aligned bounding box) defining the range (region) in which the highest resolution is to be used (all other regions are then assumed to be used at lower resolutions) (and in one embodiment, this is the case). The latter structure may be particularly useful where only two different resolutions are to be used.

[0111] In various embodiments, a determination is made as to which resolution the consumer circuit will use and then resolution indicative data is generated based on this (which is then provided to the encoding circuit). The determination of which resolution the consumer circuit will use and the generation of data indicating this may be performed by any suitable and desired component or processor of the data processing system in any suitable manner. For example, the determination of which resolution is to be used where for the array and the generation of data indicating this determination (such as a density map / texture) may and preferably is performed by the application that needs to display the frame.

[0112] The resolution-indicative data should (and in various embodiments is) be determined and generated before the data element array is encoded by the encoding circuit (and then provided to the encoding circuit). Thus, in various embodiments, the resolution at which at least one region of (the decoded version of) the data element array is to be used by the consumer circuit is known before the data element array is encoded and before the decoded version of the data element array is used by the consumer circuit.

[0113] In the present invention, the encoding circuit uses data indicative of a resolution to be used by the consumer circuit to control the generation of a representation representing at least one block into which the array of data elements is divided.

[0114] The encoding circuitry may use the data to control the generation of a representation representing only one block of the data array, but preferably uses the data to control the generation of a representation representing each of a plurality of different blocks of the array of data elements. The encoding circuitry may use the data to control the generation of representations representing some but not all of the blocks of the array of data elements (in which case it may be assumed (by the encoding circuitry) that any unindicated blocks are to be encoded in a default manner, for example), but preferably uses the data to control the generation of representations representing each and every block (all blocks) into which the array of data elements is divided.

[0115] The encoding circuitry should (and preferably does) use the data to control the generation of a representation representing at least those blocks of the data array corresponding to (within) at least one region indicated by the resolution indication data, for example, to control the generation of a representation representing at least those blocks of the data array indicated by the resolution indication data.

[0116] Thus, for example, where the resolution-indicating data indicates the resolution of each of the plurality of different regions (as described above), the encoding circuitry preferably uses the data to control the generation of at least those blocks of the data array corresponding to (within) the plurality of different regions. In a preferred embodiment, where the resolution-indicating data indicates the resolution of each of the plurality of different blocks into which the data array is divided for encoding purposes (as described above), the encoding circuitry uses the data to control the generation of a representation for representing each of the plurality of blocks. Thus, the method preferably comprises the step of: the encoding circuitry uses (and the encoding circuitry is preferably configured to use) the data to control the generation of a representation for representing each of the plurality of different blocks into which the array of data elements is divided.

[0117] The generation of a representation for representing a block may be controlled in any suitable manner.In a preferred embodiment, the encoding circuitry uses data to determine how a representation should be generated for a block (and generates the representation of the block in the manner so determined).

[0118] Preferably, the encoding circuitry uses the data to select one (type of) representation for representing the block from a plurality of possible (types of) representations and then generates the selected (type of) representation for representing the block. Thus, the encoding circuitry will encode the data array in accordance with the resolution indication data which effectively indicates to the encoding circuitry which of the different (types of) representations should be used for respective regions (blocks) of the data array.

[0119] Thus, the method preferably comprises the following steps: the encoding circuit uses (and the encoding circuit is preferably configured to use) the data to select, for each block of at least one block, a representation (of type) from a plurality of representations (of type) for representing the block; and generates the selected representation (of type) for representing the block.

[0120] The multiple (types of) representations may be any suitable set of representations. In a preferred embodiment, the multiple representations differ with respect to fidelity (loss). That is, each possible (type of) representation preferably represents a block with a respective different fidelity (loss).

[0121] In a preferred embodiment, there is a highest fidelity (type of) representation and one or more lower fidelity (type of) representations. There may be only two different representations: the highest fidelity representation and the lower fidelity representation. Alternatively, three or more different fidelity representations may be used, including the highest fidelity representation and two or more lower fidelity representations.

[0122] Thus, in a preferred embodiment, the encoding circuit uses the resolution indication data to select, for each of at least one block, a higher fidelity (type of) representation for representing the block or a lower fidelity (type of) representation for representing the block, and generates the selected (type of) representation for representing the block.

[0123] In a preferred embodiment, the highest fidelity (type of) representation comprises a lossless (type of) representation (i.e., wherein the representation used to represent a given block is generated in a lossless manner), and one or more lower fidelity (type of) representations comprise a lossy (type of) representation (i.e., wherein the representation used to represent a given block is generated in a lossless manner).

[0124] This is preferably such that when the highest fidelity lossless representation is used as part of the encoding operation to generate a representation for representing a particular block, the (overall) encoding operation for encoding the particular block is lossless. On the other hand, when a lower fidelity lossy representation is used as part of the encoding operation to generate a representation for representing another particular block, the (overall) encoding operation for encoding the other particular block is lossy.

[0125] Thus, in a preferred embodiment, the encoding circuit uses the resolution indication data to select, for each block of at least one block, a lossless (type of) representation to represent the block or a lossy (type of) representation to represent the block, and generates the selected (type of) representation to represent the block.

[0126] In the case where there are multiple lossy (types of) representations, each of the multiple lossy (types of) representations may differ with respect to the degree (level) of loss. Thus, there may be a least lossy (type) representation and one or more lossy (types of) representations. In these embodiments, the encoding circuitry may use the data to select, for each of the at least one block, a degree (level) of loss for representing the block (and use the selected degree (level) of loss to generate a representation for representing the block).

[0127] In a preferred embodiment, where the data indicates that the consumer circuit will use a higher resolution for a region of (a decoded version of) the data array (for example, where the data indicates that the region is to be displayed at a higher resolution), a higher fidelity representation of blocks of the data array corresponding to (within the region) is generated, and where the data indicates that the consumer circuit will use a lower resolution for a region of (a decoded version of) the data array (for example, where the data indicates that the region is to be displayed at a lower resolution), a lower fidelity representation of blocks of the data array corresponding to (within the region) is generated.

[0128] Most preferably, where the data indicates that the consumer circuit is to use the highest resolution for a region of the (decoded version) of the data array (e.g. where the data indicates that the region is to be displayed at the highest resolution), then a highest fidelity (preferably lossless) representation representing the blocks of the data array corresponding to (within) the indicated region is generated. Correspondingly, where the data indicates that the consumer circuit is to use a lower resolution for a region of the (decoded version) of the data array (e.g. where the data indicates that the region is to be displayed at a lower resolution), then a lower fidelity (preferably lossy) representation representing the blocks of the data array corresponding to (within) the indicated region is generated.

[0129] The method of representing each representative block can be selected as desired.

[0130] In various specific preferred embodiments, at least one of the plurality of (types of) representations comprises a tree representation. Preferably, the plurality of (types of) representations comprises a plurality of different (types of) tree representations that differ with respect to their fidelity (loss), e.g., wherein each (type of) tree representation preferably represents a block having a respective different fidelity (loss). In a preferred embodiment, there is a highest fidelity (preferably lossless) (type of) tree representation and one or more lower fidelity (preferably lossy) (type of) tree representation.

[0131] In a preferred embodiment, the highest fidelity representation includes the tree representation described in U.S. Patent Application No. 2013 / 0034309 (which application is incorporated herein by reference), and one or more of the one or more lower-fidelity tree representations may include a lossy version of the tree representation described in U.S. Patent Application No. 2013 / 0034309.

[0132] In these embodiments, one or more or each of the tree representations is preferably configured such that the data values ​​of the nodes of the tree are set so that for a data element of the data array represented by a leaf node of the tree, the data value indicated by the tree is given by the sum of the data value of the leaf node in the tree and the data values ​​of each previous parent node in the tree branch to which the leaf node belongs.

[0133] Preferably, one or more or each tree is configured such that each leaf node of the tree represents a corresponding data element of the data array, and the data values ​​for the nodes of the tree are preferably arranged such that for a data element of the data array represented by a leaf node of the tree, the data value indicated by the tree is given by the sum of the data value of the leaf node in the tree and the data values ​​of each previous parent node in the branch of the tree to which the leaf node belongs. Thus, in order to reproduce the data value for the data element represented by a leaf node of the tree, the value of the leaf node in the tree and the values ​​of all parent nodes in the tree along the branch in the tree on which the leaf node resides are preferably added together.

[0134] In these embodiments, each tree can be generated in any suitable manner. In an embodiment, each node of the tree will have multiple child nodes, each child node representing a corresponding non-overlapping and equal-sized region of the data array region represented by the parent node, except for the terminal leaf nodes representing the independent data elements themselves.

[0135] In an embodiment, the tree is represented in the form of a quadtree, i.e., a quadtree is generated to represent the data array block. However, if desired, other tree structures may be used, i.e., each node has more or less than four child nodes. A hybrid tree structure may also be used, for example, the tree structure is a quadtree, but for the next level to the bottom level, each node has only two child nodes (i.e., the parent node of a leaf node each has only two child leaf nodes).

[0136] In the case where the tree representation comprises a quadtree, then each quadtree will accordingly have a root node representing the entire block of the data array that the quadtree is encoding. The root node will then have four child nodes (because it is a quadtree), each child node representing a corresponding non-overlapping and equally sized region (and in embodiments, a quadrant) of the block of the data array represented by the root node (quadtree). Each child node of the root node will then have four child nodes, each child node representing a corresponding non-overlapping and equally sized region (and in embodiments, a quadrant) of the region of the block of data represented by the child node (quadtree) of the root node, and so on (i.e., each node has four child nodes, each child node representing a corresponding non-overlapping and equally sized region of the block of the data array represented by the parent node), until a leaf node representing an independent data element (e.g., texel or pixel) of the data array.

[0137] Thus, for example, in the case of a quadtree for a 16x16 block of data elements, there will be a root node representing the entire 16x16 block. The root node will have four child nodes, each representing an 8x8 block of data elements within the 16x16 block. Each such child node will have four child nodes, each representing a 4x4 block of data elements within the 8x8 block in question, and so on, until a leaf node representing an independent data element.

[0138] A tree (or trees), such as a quadtree, may be generated such that each leaf node of the tree corresponds to a given data element (e.g., texel or pixel) of a data array block. Thus, there will be one leaf node in the tree for each data element in the data array block that the tree is to represent.

[0139] Each node of a tree (e.g., a quadtree) may have a corresponding data value associated with it. The data values ​​of the nodes of the tree may be arranged so that the data value of a data element of a data array represented by a leaf node of the tree is given by the sum of the data value of the leaf node in the tree and the data values ​​of each previous parent node in the branch of the tree to which the leaf node belongs. Thus, the data value associated with and stored for each node of the tree may be (and in embodiments is) the value required to reproduce the desired leaf node value when all node values ​​along the branches of the tree are summed together in an appropriate manner.

[0140] The data values ​​to be associated with (set for) each node of the tree may be determined in any suitable manner. In an embodiment, they are determined by performing two processing (data) passes.

[0141] In a first processing pass, each leaf node in the tree is initialized with (set to) the value that the tree would indicate for the data element represented by (corresponding to) the leaf node in the data array block to be encoded, and each non-leaf node is initialized with (set to) a selected value, e.g., based on the values ​​of the leaf nodes in the tree. In an embodiment, this calculation is performed from the bottom of the tree upwards.

[0142] The value to which each non-leaf node is set in this first processing (data) round is based on and / or related to the value of its child nodes in an embodiment. In an embodiment, it is determined in a predetermined manner according to the value of its child nodes. In an embodiment, each non-leaf node is initialized (set to this value) with the value of one of its child nodes. In various such embodiments, in this processing round, each non-leaf node is initialized (set to this minimum value) with the minimum value of its child nodes (the value of its lowest value child node). In other embodiments, in this processing round, each non-leaf node is initialized (set to this maximum value) with the maximum value of its child nodes (the value of its highest value child node).

[0143] The second (subsequent) processing round in the embodiment then subtracts the value of its parent node from each node. This is done for all nodes except the root node (which has no parent), and this is again done in a bottom-up manner in the embodiment. The node values ​​produced after the second round are then the values ​​associated with (set for) each node in the tree.

[0144] In these embodiments, the (loss) fidelity of each of the multiple (types of) tree representations may be controlled in any suitable manner.

[0145] In various preferred embodiments, the lossy tree representation may be generated by truncating the tree. Thus, in various embodiments, one or more of the one or more lower-fidelity tree representations may include a truncated tree representation, such as a truncated version of the tree representation described in U.S. Patent Application No. 2013 / 0034309. This may have the effect of reducing the size of the tree, and thus may reduce the number of bits required to encode a block of the data array (although it may be that the number of bits required to encode the truncated tree is the same as the number of bits required to encode the "original" non-truncated tree).

[0146] In these embodiments, rather than the tree including multiple terminal leaf nodes representing each (and every) of the independent data elements of the data array block, the tree may be truncated by omitting (e.g., by removing from the tree) the tree nodes representing at least some or all of the independent data elements of the data array block (i.e., so that the tree does not include (non-include) these nodes). The tree may also be truncated by omitting (e.g., by removing from the tree) one or more other higher-level nodes from the tree (i.e., so that the tree does not include (non-include) these nodes).

[0147] Truncations may be performed at any level within the tree representation, and will truncate all of the children of a particular non-leaf node, or only some but not all of the children of a particular non-leaf node.

[0148] Thus, in an embodiment, one or more entire levels of the tree are truncated (e.g., one or more of the lowest levels of the tree may be truncated), and the tree representation preferably omits nodes of one or more entire levels of the tree. For example, the entire lowest level of the tree may be truncated by omitting (e.g., by removing from the tree) nodes representing individual data elements of a data array block.

[0149] Additionally or alternatively, only a portion of (some but not all nodes of) one or more levels of the tree may be truncated, i.e., the tree is made to preferably omit some but not all nodes of one or more levels of the tree. Thus, for example, a portion of the lowest level of the tree may be truncated by omitting (e.g., by removing from the tree) nodes representing only some (but not all) independent data elements of a data array block. It is also possible to truncate (omit) only a portion of (some but not all nodes of) any other higher level of the tree.

[0150] In these embodiments, any number of nodes may be omitted from the tree. Preferably, the tree is truncated by omitting (causing the tree to omit) all child nodes of an integer number of non-leaf nodes (at any suitable one or more levels of the tree). In this case, any suitable number of child nodes of the non-leaf nodes may be truncated (may be omitted from the tree). For example, in the case of a quadtree representation, one, two, three (or all four) child nodes of any one or more non-leaf nodes may be omitted (may be removed from the tree).

[0151] Thus, in these embodiments, the truncated tree representation may include a root node that represents the entire block of the data array that the tree is encoding. The root node will then have at least one child node (e.g., one, two, three, or four child nodes in the case of a quadtree representation), each child node representing a respective non-overlapping and equally sized region (and in embodiments, a quadrant) of the block of the data array represented by the root node (quadtree).

[0152] Each child node of the root node may then have its own number of child nodes, e.g., zero child nodes or at least one child node (e.g., one, two, three, or four child nodes in the case of a quadtree representation), each child node representing a corresponding non-overlapping and equally sized region (and in an embodiment, a quadrant) of the region of the data block represented by the child nodes (quadtree) of the root node, and so on (i.e., each node has zero or at least one child node (e.g., one, two, three, or four child nodes in the case of a quadtree representation), each child node representing a corresponding non-overlapping and equally sized region of the data array block represented by the parent node).

[0153] Each truncated tree representation can be generated in any suitable manner (e.g., in a manner corresponding to the above manner). In various embodiments, a "minimum" tree representation can be generated in the above manner, and then one or more or all nodes of the tree can be appropriately discarded to obtain a truncated tree representation.

[0154] Other configurations would be possible.

[0155] In various preferred embodiments, a lossy tree representation may be generated by including in the tree an indication that one or more sub-regions of a block of a data array represented by the tree representation are copies of one or more other sub-regions of the data array.

[0156] Thus, for example, for each particular non-leaf node among one or more particular non-leaf nodes in the tree representation, in addition to or in lieu of including the particular node's corresponding (e.g., minimum) data value in the tree (as described above), the tree representation may also include an indication associated with the particular node, wherein the indication indicates (to the decoder) that the particular non-leaf node (and preferably its child nodes) should be treated as a copy of another non-leaf node (and preferably its child nodes).

[0157] In this case, the children of a particular non-leaf node may be truncated (omitted) from the tree, for example in the manner described above. This will therefore have the effect of reducing the size of the tree and therefore the number of bits required to encode a block of data arrays.

[0158] Thus, with respect to one or more non-leaf nodes of the tree representation, the lossy tree representation may include an indication associated with the non-leaf node that indicates (to the decoder) that the non-leaf node (and preferably its child nodes) is a copy of another non-leaf node (and preferably its child nodes).

[0159] In various preferred embodiments, instead of (or in addition to) truncating the tree representation (omitting nodes from the tree representation) in the manner described above, the values ​​of at least some of the "minimum" data values ​​associated with each node (as described above) can be modified in a manner that reduces the number of bits required to encode the values. Thus, in various embodiments, one or more of the one or more lower-fidelity tree representations can include a modified tree representation. For example, this can be done by reducing at least some of the minimum data values. Reducing the values ​​of at least some of the minimum data values ​​associated with each node allows for a greater degree of compression of the "minimum tree", i.e., such that the number of bits required to represent the tree representation can be reduced.

[0160] In these embodiments, the data values ​​associated with each node may be modified or reduced in any suitable manner.

[0161] For example, one or more or each minimum data value may be reduced by a set (eg, predefined) amount, for example, by subtracting the set amount from the one or more or each minimum data value.

[0162] Additionally or alternatively, one or more or each minimum data value may be reduced by dividing the value by a set (e.g., predefined) amount, for example, by dividing one or more or each minimum data value by a set amount. One or more or each minimum value may, for example, be halved, quartered, etc.

[0163] In various further embodiments, one or more or each minimum data value may be set to zero.

[0164] In these embodiments, modifications may be made at any level within the tree representation, and may modify all of the child nodes of a particular non-leaf node, or only some but not all of the child nodes of a particular non-leaf node.

[0165] In an embodiment, the value of one or more entire levels of the tree's nodes is modified, for example, the value of one or more lowest level nodes of the tree may be modified (reduced). For example, the value of the entire lowest level node of the tree may be modified (reduced).

[0166] Additionally or alternatively, the values ​​of only a portion of the nodes of one or more levels of the tree may be modified (reduced). Thus, for example, the values ​​of only a portion of the nodes (some but not all nodes) of the lowest level of the tree may be modified (reduced). The values ​​of only a portion of (some but not all nodes) of any other higher level of the tree may also be modified (reduced).

[0167] In these embodiments, the values ​​associated with any number of nodes may be modified (reduced). Preferably, the modified tree representation is generated by modifying (reducing) the values ​​associated with all child nodes of an integer number of non-leaf nodes (at any suitable one or more levels of the tree). In this case, the values ​​of the child nodes of any suitable number of non-leaf nodes may be modified (reduced). For example, in the case of a quadtree representation, the values ​​of the child nodes of one, two, three (or all four) of any one or more non-leaf nodes may be modified (reduced).

[0168] In these embodiments, each modified tree representation may be generated in any suitable manner (e.g., in a manner corresponding to that described above). In various embodiments, a "minimum value" tree representation may be generated in the manner described above, and then one or more or all values ​​of the nodes of the tree may be appropriately modified or reduced to obtain a modified tree representation. Thus, for example, one or more or all minimum data values ​​determined in the second processing pass may be modified or reduced.

[0169] Thus, in various embodiments, in a second (subsequent) processing round, after subtracting the value of each node from its parent node, one or more or each of the resulting values ​​may be modified or reduced, and the node values ​​so modified may then be the values ​​associated with (set for) each node in the tree.

[0170] In these embodiments, where the children of a particular non-leaf node are modified (e.g., reduced or zeroed), the data value included in the tree for the particular non-leaf node may be the minimum data value determined in the second processing round. An improvement to this would be to instead set the data value of the particular non-leaf node to the average of the originally determined minimum data values ​​of the children. This would improve the fidelity of the block without (significantly) increasing the number of bits required to represent the block.

[0171] Thus, in various embodiments, when generating a tree representation, one or more or each non-leaf node may be set to the average value of its child nodes (and preferably, one or more or each child node may be set to zero). Correspondingly, the encoding circuitry may be configured to generate a lossy tree representation by setting one or more non-leaf nodes in the tree to the average value of its child nodes.

[0172] In various further embodiments, where the tree representation is a lossy tree representation, the value indicated by the tree for a data element represented (corresponding to) by a leaf node in the data array to be encoded may include a modified or approximate version of the original (actual) (as generated) value of the data element represented (corresponding to) by the leaf node in the data array to be encoded.

[0173] Thus, although the value that the tree will indicate for a data element in a data array block (i.e., the value to which the corresponding leaf node will be set in the first data round) may be the actual value of that data element in the data array of a lossless representation of the original data array block, in the case where the tree will be a lossy representation of the data array block, the value that the tree will indicate for the data element in the data array block (i.e., the value to which the corresponding leaf node will be set in the first data round) may be an approximation of the actual value.

[0174] In these embodiments, one or more or each data element value may be modified (approximated) in any suitable manner.

[0175] In various embodiments, the block may be subjected to filtering prior to the first processing pass. For example, the block may be low pass filtered, for example, to remove higher frequency components. This may have the effect of reducing the difference, thereby improving compression in the manner described above, and also reducing "block" artifacts, thereby providing an improved "smoother" effect.

[0176] Any suitable filtering technique may be used. In various embodiments, each block or sub-block of the data array may be subjected to a frequency domain transform, such as a discrete cosine transform (DCT), high frequency components may be attenuated (removed), and the data may then be transformed back to the spatial domain (before encoding as described above). Alternatively, direct spatial filtering may be used.

[0177] Although as described above, in various specific preferred embodiments, one or more of the multiple (types of) representations include a tree representation, it will also be possible that one or more of the multiple (lower fidelity) (types of) representations include other (types of) tree representations.

[0178] For example, in various embodiments, one or more or each block may be represented by a constant (single) value (eg, a constant (single) color). That is, the representation used to represent the block may include a constant (single) value.

[0179] Thus, one or more of the multiple (lower fidelity) representations may include a representation for representing a block in which a single data value represents multiple data elements of the block. Preferably, the single data value represents a data element for the entire block.

[0180] The single value used in these embodiments may be selected as desired. For example, the single value may include a default (e.g., predetermined) value. Additionally or alternatively, the single value may include an average value, such as the average of all data elements in the block or sub-block in question. Thus, the encoding circuitry may generate a representation for representing a block by generating an average value of the data elements of the block or one or more sub-regions of the block.

[0181] Other configurations would be possible.

[0182] Once a (tree) representation (such as a quadtree) representing the data array is generated, it is then necessary to generate data representing the (tree) representation, which can then be stored as an encoded version of the original data array. The generated and stored data representing the (tree) representation can take any suitable and desired form. However, in an embodiment, it represents the (tree) representation in an encoded form. In an embodiment, the data representing the (tree) representation (generated and stored as a representation of the (tree) representation) is a compressed representation of the (tree) representation representing the data array (or a portion of the data array).

[0183] In embodiments where the representation comprises a tree representation, the data generated and stored to represent the tree comprises a set of data indicating tree node values ​​as described above, and may also comprise a set of data used (and for such identification) when identifying data for respective tree nodes in the set of data indicating tree node values ​​(e.g., a bit count tree). The set of node value identification data may then be used by a decoder to identify data for a given tree node within the stored set of data representing tree node values.

[0184] Compression of (tree) representations and bit count trees are described in more detail in U.S. Patent Application No. 2013 / 003430, which is incorporated herein by reference.

[0185] The techniques of various embodiments may be used in conjunction with various variable resolution techniques. One exemplary technique that may be improved using the techniques of the present invention is so-called "foveated" or "variable resolution" rendering (also known as "shot matching" rendering).

[0186] Variable-resolution ("foveated") rendering is a rendering technique in which one or more portions of a frame (image) to be displayed are rendered at a higher resolution, but one or more other portions of the frame are rendered at a lower resolution.

[0187] For example, this can be used for so-called "XR" displays, such as augmented reality (AR) and / or virtual reality (VR), head-mounted display (HMD) systems. In these embodiments, the data processing system can be operated to track the movement of the user's head / gaze (so-called head pose tracking). The head orientation (pose) data can then be used to determine how images should actually be displayed to the user for the user's current head position (viewing direction), and images (frames) can therefore be rendered (e.g., by setting the camera (viewpoint) orientation based on the head orientation data) so that appropriate images can be displayed based on the user's current viewing direction.

[0188] Variable-resolution ("foveated") rendering recognizes that areas of the frame that the user is looking directly at may need to be displayed at a higher resolution to be visually acceptable, while peripheral areas of the frame that the user is not looking directly at may be displayed at a lower resolution while still appearing visually acceptable. This can then be used to reduce the processing burden on the data processing system by displaying the peripheral areas at a lower resolution rather than displaying the entire frame at the highest required "foveated" resolution.

[0189] Variable resolution rendering can be performed by identifying one or more "fixed points" where higher resolution areas of the frame will be displayed, with areas farther away from the fixed point being displayed at lower resolutions. Thus, each fixed point can indicate the highest resolution area of ​​the frame, and in some cases, can correspond to the center of the eye's retina (fovea).

[0190] When performing variable resolution rendering, the location of the highest resolution region of the frame may be determined in any suitable and desired manner (e.g., a fixed point). For example, some form of head tracking or eye tracking (head pose tracking) system may be used to attempt to identify where the user is looking at the image to identify the region of the frame that should be displayed at the highest resolution. The location of the higher resolution region of the frame may also or instead be based on other factors, such as lens distortion.

[0191] In accordance with various embodiments, variable resolution rendering may be performed by a generator processing unit generating a single (e.g., full or high) resolution version of a frame to be displayed, and the encoding process may actually be used to change the resolution of one or more regions of a frame to generate an output frame for which different regions of the frame have different resolutions.

[0192] Thus, according to various embodiments, the generator circuit is configured to generate (and the method comprises the step of generating) for a frame to be displayed a single (eg full resolution or high resolution) version of the frame to be displayed.

[0193] The frame may then be encoded by encoding circuitry. The encoding circuitry preferably uses the resolution indication information to control generation of a tree representation for at least one block into which the frame is divided. According to various embodiments, this is done such that the encoded version of the frame is encoded at varying resolutions, i.e. such that different regions of the frame have different resolutions.

[0194] This allows foveated (variable resolution) frames to be generated for display in a particularly efficient and effective manner, for example for use in augmented or virtual reality display systems, and this can, for example, save storage bandwidth when compared to other foveated (variable resolution) rendering techniques.

[0195] In these embodiments, the highest resolution region of the frame comprises a "foveal" view to be displayed at a fixed location where the highest resolution frame is to be displayed. One or more lower resolution versions of the frame may be, and preferably are, used and displayed away from the fixed point, e.g., toward the periphery of the output frame being displayed.

[0196] In these embodiments, the determination of which representation to use for any given position in the overall output array (frame) may be performed as desired. In the case of foveated rendering, appropriate head tracking or eye tracking information may be used to identify which resolution should be displayed where in the output array (frame), and / or information related to the characteristics of the display system may also or instead be used. For example, in the case of a head mounted display, the center of the output array (frame) may be identified and indicated as the area where the highest resolution version should be displayed, with areas toward the periphery of the output array (frame) then displayed at lower resolutions.

[0197] Other configurations would be possible.

[0198] Although the present invention is described above with specific reference to the generation and processing of a single array (frame), as will be appreciated by those skilled in the art, the operations performed in the manner of the present invention are preferably performed for a plurality of arrays (frames) to be displayed, for example, and preferably for each array (frame) of a sequence of a plurality of arrays (frames) to be displayed, for example. Thus, in an embodiment, the operations performed in the manner of the present invention are used to generate a sequence of a plurality of output arrays (frames for display to a user), and correspondingly, the operations performed in the manner of the present invention are preferably repeated for a plurality of output arrays (frames to be displayed).

[0199] Thus, for example, the generator processing unit will generate a sequence of multiple output arrays (frames to be displayed), and the display processor then processes these arrays (frames) appropriately to provide these arrays to the display for display.

[0200] The display in the present invention can be any suitable and desired form of display, and can include any suitable and desired parts and elements that the display can include, such as and preferably, a display panel, a display driver circuit for scanning frame data to the display panel, and a display receiver for receiving data to be displayed on the display panel. The display can also include a suitable local (on-chip) frame data storage, such as a frame buffer. The display can include a local display (screen) and / or an external display of the entire data processing system (device). There can be more than one display output (if desired).

[0201] In a particularly preferred embodiment, the display comprises a head mounted display, for example and preferably for virtual reality and / or augmented reality display. Thus, in this case, the display should and preferably does comprise a display panel for displaying frames to a user, and a lens through which the user will view the displayed frames.

[0202] Correspondingly, in a preferred embodiment, the display has associated viewing orientation determination (e.g., head tracking) sensors that preferably periodically generate viewing tracking information based on the current and / or relative position of the display and are operable to periodically provide the viewing orientation to a processing unit of the data processing system.

[0203] Thus, another aspect of the present invention includes a head-mounted display device, which includes a data processing system of any one or more of the aspects and embodiments of the present invention described herein. Correspondingly, another aspect of the present invention includes a method for operating a head-mounted display device, the method comprising the following steps: operating the head-mounted display device in any one or more of the aspects and embodiments of the present invention.

[0204] In addition to the specific units and components required for operation in the manner of the present invention, the data processing system may also and preferably does include one or more and preferably all of any other suitable and desired components, units, processors, etc. that the data processing system may include. Thus, the data processing system may include one or more and preferably all of a central processing unit, a main processor, a graphics processing unit, a video processor, an image signal processor, a camera, a system bus, and a memory controller. In a preferred embodiment, the data processing system includes and / or communicates with one or more memories and / or storage devices that store the data described herein and / or store software for performing the processes described herein.

[0205] The methods and apparatus of the techniques described herein may be implemented in any suitable way, for example, in hardware or software and in (and included in) any suitable device or component.

[0206] For example, the actual means or components used to store data in the manner of the techniques described herein will depend, for example, on the nature of the data array being stored. Thus, for example, in the case of graphics textures, a suitable processor such as a personal computer may be used to generate and store textures in the manner of the techniques described herein, such as by an application developer, and then such stored textures provided, for example, as part of game content. In the case where the stored data arrays are frames for display, then the processor may accordingly be a graphics processor that generates and stores data in the desired manner.

[0207] Similarly, on the data reading (decoding) side of the operation, for example in the case of texture data, the processor may be a graphics processor that reads (decodes) a stored array of data, while in the case of frames for display, the processor may be a display controller for a display that reads (decodes) a stored array of data.

[0208] In an embodiment, the technology described herein is implemented in a graphics processor, a display controller, an image signal processor, a video decoder, or a video encoder, whereby the technology described herein also extends to a graphics processor, a display controller, an image signal processor, a video decoder, or a video encoder, which are configured to use the methods of the technology described herein, or a device including the technology described herein, or to operate according to the methods of any one or more embodiments of the technology described herein. Subject to any hardware necessary to perform the specific functions discussed above, such a graphics processor, display controller, image signal processor, video decoder, or video encoder may otherwise include any one or more or all of the usual functional units included in the graphics processor, display controller, image signal processor, video decoder, or video encoder. In an embodiment, the methods and devices of the technology described herein are implemented in hardware, and in an embodiment are implemented on a single semiconductor platform.

[0209] The techniques described herein are particularly, but not exclusively, suitable for use in low power portable devices. Thus, in an embodiment, the techniques described herein are implemented in a portable device such as a mobile phone or a PDA.

[0210] Similarly, the memory storing the data representing the tree representing the data array may comprise any suitable such memory, and may be configured in any suitable and desired manner. For example, it may be an on-chip buffer, or it may be external memory (and in practice it is more likely to be external memory). Similarly, the memory may be dedicated memory for this purpose, or it may be part of a memory also used for other data. In an embodiment, this data is stored in the main memory of a system incorporated into the graphics processor.

[0211] In the case of an array of texture data, in an embodiment the memory is a texture buffer of the graphics processing system (which may be, for example, on-chip or in external memory, as desired). Similarly, in the case of a frame for display, in an embodiment the memory is a frame buffer for the graphics processing system and / or for a display to which the output of the graphics processing system is to be provided.

[0212] In an embodiment, all data representing the tree representing the data array is stored in the same physical memory, but this is not required.

[0213] Other storage structures would of course be possible.

[0214] The techniques described herein may be implemented in any suitable system, such as a suitably configured microprocessor-based system. In an embodiment, the techniques described herein are implemented in a computer and / or microprocessor-based system.

[0215] The various functions of the techniques described herein can be performed in any desired and suitable manner. For example, the functions of the techniques described herein can be implemented in hardware or software (as desired). Thus, for example, the various functional elements of the techniques described herein can include suitable processors, controllers, functional units, circuits, processing logic, microprocessor structures, etc., which are operable to perform various functions, etc., such as appropriate dedicated hardware elements and / or programmable hardware elements that can be programmed to operate in a desired manner.

[0216] It should also be noted here that, as will be appreciated by those skilled in the art, various functions of the techniques described herein, etc. may be repeated and / or performed in parallel on a given processor. Equally, various processing stages may share processing circuits, etc. (if desired).

[0217] Furthermore, any one or more or all of the processing stages and engines of the present invention may be embodied as processing stage circuitry, for example in the form of one or more fixed function units (hardware) (processing circuitry), and / or in the form of programmable processing circuitry that can be programmed to perform desired operations. Equally, any one or more of the processing stages and processing stage circuitry of the present invention may be provided as separate circuit elements to any one or more of other processing stages or processing stage circuitry, and / or any one or more or all of the processing stages and processing stage circuitry may be formed at least in part by shared processing circuitry.

[0218] Those skilled in the art will also appreciate that all described implementations of the technology described herein may include any one or more or all of the optional features of the technology described herein, as appropriate.

[0219] Methods according to the techniques described herein may be implemented at least in part using software (e.g., a computer program). It will be seen that embodiments of the techniques described herein include: computer software that, when installed on a data processing device, is specifically adapted to perform the methods described herein; a computer program element that includes computer software code portions for performing the methods described herein when the program element is run on the data processing device; and a computer program that includes code means that are adapted to perform all the steps of the methods described herein when the program is run on a data processing system. The data processing system may be a microprocessor, a programmable FPGA (field programmable gate array), or the like.

[0220] The technology described herein also extends to a computer software carrier, which includes such software, which when used to operate a graphics processor, a renderer, or other system including a data processing device, combines with the data processing device to cause the processor, renderer, or system to perform the steps of the method of the technology described herein. Such a computer software carrier can be a physical storage medium, such as a ROM chip, RAM, flash memory, CD ROM, or disk.

[0221] It will be further understood that not all steps of the methods of the technology described herein need to be performed by computer software, and thus, a broader embodiment of the technology described herein includes computer software, and such software is installed on a computer software carrier for performing at least one of the steps of the methods set forth herein.

[0222] The techniques described herein may therefore be suitably embodied as a computer program product for use with a computer system. Such an implementation may include a series of computer-readable instructions fixed on a tangible, permanent medium, such as a computer-readable medium, for example, a disk, CD ROM, ROM, RAM, flash memory, or hard disk. The series of computer-readable instructions embodies all or part of the functions previously described herein.

[0223] Those skilled in the art will appreciate that such computer readable instructions may be written in a variety of programming languages ​​for use with many computer architectures or operating systems. Further, such instructions may be stored using any storage technology (storage technology now or in the future includes but is not limited to semiconductor, magnetic or optical), or sent using any communication technology (communication technology now or in the future includes but is not limited to optical, infrared or microwave). It is contemplated that such computer program products may be distributed as removable media, with printed or electronic documentation, e.g., off-the-shelf software; preloaded with a computer system, e.g., on a system ROM or fixed disk; or distributed over a network (e.g., the Internet or the World Wide Web) from a server or bulletin board. BRIEF DESCRIPTION OF THE DRAWINGS

[0224] Various embodiments of the present invention will now be described, by way of example only, and with reference to the accompanying drawings, in which:

[0225] Figure 1 schematically illustrates a data array that may be encoded according to an embodiment of the present invention;

[0226] Figure 2 schematically illustrates the generation of a quadtree representing an array of data elements according to an embodiment of the present invention;

[0227] Figure 3 Schematically illustrates a data processing system in which a data array encoded according to the present invention may be used;

[0228] Figure 4 shows an exemplary density texture that may be used in embodiments of the present invention;

[0229] Figure 5 schematically illustrates a process of truncating values ​​of a data array according to an embodiment of the present invention;

[0230] Figure 6 schematically illustrates a decision process for truncating values ​​of a data array by averaging according to an embodiment of the present invention; and

[0231] Figure 7 The decision process for truncating values ​​of a data array by filtering according to an embodiment of the present invention is schematically illustrated. DETAILED DESCRIPTION

[0232] Figure 1 An exemplary data array 30 that can be encoded in the manner of the present invention is schematically shown. Data array 30 is a two-dimensional data array containing a plurality of data elements (i.e., containing data array entries at a plurality of specific positions within the array). Data array 30 can be an array of any suitable and desired data, such as data representing an image.

[0233] In the context of graphics processing, the data array may be, for example, a texture map (i.e., an array of texture elements (texels)) or a data array representing a frame to be displayed (in which case the data array may be an array of pixels to be displayed). In the case of a texture map, each data entry (position) in the data array will represent an appropriate texel value (e.g., a set of color values, such as RGBA, or luminance and chrominance values ​​for a texel). In the case of a frame for display, each data entry (position) in the array will indicate a set of color values ​​(e.g., RGB values) to be used to display the frame on a display.

[0234] In the present invention, the data array 30 is encoded and compressed to provide a set of data representing the data array 30 which can then be stored in a memory, and from which data values ​​of individual data elements in the data array 30 can be obtained by decoding the data representing the data array 30.

[0235] An embodiment of a process for encoding and compressing data array 30 will now be described.

[0236] In this embodiment, if Figure 1 As shown, in order to encode and compress the data array 30, the data array 30 is first divided into a plurality of non-overlapping, equally sized and uniform blocks 31, each block corresponding to a particular region of the data array 30. In the present embodiment, each block 31 of the data array corresponds to a block of 16x16 elements (positions) within the data array 30 (i.e., a block of 16x16 texels in the case of a texture map). (Other structures would of course be possible).

[0237] Each such block 31 of the data array 30 is then encoded to provide a compressed representation of the block 32 of the data array 30 .

[0238] To do this, a particular form of quadtree representation is first generated which represents the blocks 31 of the data array 30. This is done as follows.

[0239] The quadtree is constructed with a root node representing the entire data block 31 (and thus the entire 16x16 block in this embodiment). This root node then has four child nodes, each representing a corresponding non-overlapping, uniform, and equally sized 8x8 sub-block 32 of the 16x16 block 31 of the data array 30. Each such 8x8 sub-block representing a child node then itself has four child nodes, each representing a corresponding non-overlapping, uniform, and equally sized 4x4 sub-block 33 of the 8x8 sub-block 32, and so on, until the 16 leaf nodes of the quadtree are reached, each representing a respective independent data element of the 16x16 block 31 of the data array 30.

[0240] The data value of each node of the quadtree is determined by performing two processing (data) passes.

[0241] In the first processing round of this embodiment, each leaf node of the quadtree is initialized with the value of the data element of the block 31 of the data array 30 to which the leaf node corresponds (each leaf node is set to this value). Then, each non-leaf node is initialized with the minimum value of its child nodes (the value of its lowest valued child node) (is set to this minimum value). This calculation is performed from bottom to top.

[0242] A second processing round is then performed, where each node except the root node (which has no parent) has its parent node's value subtracted. This is again done from the bottom up.

[0243] The node values ​​after this second processing pass are then the node values ​​to be used for the nodes of the quadtree representing the block 31 of the data array 30 .

[0244] The effect of this process is that the value of the data element in the data array represented by a leaf node will be given by the sum of the value of the leaf node and the values ​​of all previous nodes along the quadtree branch on which the leaf node resides (in other words, in order to determine the value of the data element represented by a leaf node of the quadtree from the quadtree representation of the data array, the value of the leaf node and the values ​​of all previous nodes along the quadtree branch on which the leaf node resides must be summed (added together)).

[0245] Figure 2 Construction of such a quadtree is illustrated for a representative 4x4 block 40 of data elements.

[0246] like Figure 2 As shown, a quadtree 45 representing a 4x4 array of data elements 40 has a root node 41 having four child nodes 42, each corresponding to a respective 2x2 block 48 of the 4x4 block 40 of data elements. Each such child node 42 of the quadtree 45 then has 4 child nodes forming leaf nodes 43 of the quadtree 45. The leaf nodes 43 of the quadtree 45 each correspond to a respective independent data element 49 of the 4x4 block 40 of data elements.

[0247] like Figure 2 As shown and described above, in the first processing round 44, each leaf node 43 in the quadtree 45 is set to the value of its corresponding data element 49 in the data element block 40 (i.e., the value of the data element 49 to which the leaf node 43 corresponds), and each non-leaf node 41, 42 in the tree 45 representing the data array 40 is set to the minimum value of its child nodes. This calculation is performed from the bottom up.

[0248] Then, in a second processing round 46, each node except the root node 41 (which has no parent) has the value of its parent node subtracted from it. This again proceeds from the bottom up.

[0249] The node values ​​47 obtained after the second processing round are the node values ​​of the tree representing the data array 40 , and these node values ​​are stored as representing the data array 40 .

[0250] In the present embodiment, a separate such quadtree representation is constructed for each different component (data channel) of the data elements of the data array being encoded. Thus, for example, where each data element has four components (such as RGBA color components), a separate quadtree representation of the above form will be constructed for each color component, i.e., such that there will be a "red" component quadtree, a "green" component quadtree, a "blue" component quadtree, and an "alpha" component quadtree.

[0251] Once the difference values ​​to be stored for each node of the quadtree representing the block 30 of the data array have been determined in the manner described above, an entropy coding scheme is then used to generate a set of data representing the "difference" quadtree.

[0252] The entropy encoding process used in the present embodiment essentially determines how many bits are used to signal each difference value of a quadtree representing a data array. Once this is done, a set of data representing a quadtree is generated and stored to represent the node values ​​of a quadtree representing a block of the data array, the quadtree representing the block of the data array using the determined number of bits to signal each difference value.

[0253] Furthermore, to facilitate decoding of the stored data representing the node values ​​of the quadtree, a set of data, also in the form of a quadtree, is generated and stored indicating the number of bits that have been used to signal the respective difference values.

[0254] Thus, the encoding and storage of data representing the difference values ​​of quadtree nodes representing blocks of a data array is based on a quadtree maintained in parallel with the tree representing the values ​​of the data array, the quadtree representing (and allowing to be derived) the number of bits that have been used to signal the difference of the children of the quadtree node representing the values ​​of the data elements of the associated block of the data array (thus, the parallel trees can accordingly be viewed as "bit count" trees).

[0255] This process is described in more detail in US 2013 / 0034309, which is incorporated herein by reference.

[0256] In the operation of encoding the data array 30 in the manner of the present embodiment, the data for the data array may be processed in any suitable and desired manner. For example, a suitable processor or processing circuit may read the original data array to be compressed from a memory, and / or receive a data stream corresponding to the original data array to be compressed, and then process the data stream accordingly, e.g., divide it into blocks, generate the required quadtree, and then generate data representing the quadtree and store the data representing the tree in a memory, e.g., in a memory and / or on a removable storage medium, etc.

[0257] As described above, in the present embodiment, the process will accordingly first include generating a "minimum value" quadtree of the above-described form for the data array or for each block into which the data array has been divided, wherein the leaf nodes of the tree correspond to the respective data elements of the data array (or block of the data array). Then, based on the number of bits required (to be used for the indication) for indicating the values ​​of each node of the minimum value quadtree in a set of data representing the node values ​​of the minimum value quadtree, a bit count quadtree is obtained. Then, a set of data representing the bit count quadtree, together with a set of data representing the node values ​​of the minimum value quadtree (in the form of differences), will be generated, the set of node value indication data being configured according to and having the configuration indicated by the corresponding bit count quadtree.

[0258] The set of data representing the tree node values ​​and the set of data representing the corresponding bit count tree thus generated will then be appropriately stored to serve as a compressed set of data representing the data array.

[0259] The above mainly describes the way in which an encoded version of a data array is generated and stored in a memory for use in the present embodiment. When the data array stored in this way is to be used (for example, applied to a fragment to be rendered) (in the case where the stored data array is a texture map for graphics processing), then the decoding process will basically include the reverse process of the above encoding process.

[0260] Thus, a decoding device such as a graphics processor (e.g., where the stored data array is a texture map) or a display controller (e.g., where the stored data array is a frame to be displayed) will read the relevant sub-block data from the appropriate location in memory and decode the data to determine the value of the data element of interest.

[0261] The decoder will first determine the required bit count tree node values ​​from the stored data representing the bit count tree, and then use these determined bit count tree node values ​​to identify the data of the relevant nodes of the quadtree representing the data element values ​​of the data array block of interest, and use these node values ​​to determine the value of the data element of interest.

[0262] As part of this process, the decoder will look at the bit count node value accordingly and interpret the stored data representing the quadtree of data element values ​​representing the data block and determine the values ​​of the nodes of the quadtree accordingly.

[0263] This process may then be repeated for each data element of interest (for which a value is desired).

[0264] Figure 3 The structure of a data processing system 14 is schematically shown, which can store and use data arrays that have been stored in the manner of the present invention.

[0265] Figure 3 The data processing system 14 includes: a main processor, which includes a central processing unit (CPU) 7; a graphics processing unit (GPU) 2; a video codec 1; an image signal processor (ISP) 12 (i.e., for example, communicating with a camera 12); a display processor (display processing unit (DPU)) 5; and a memory controller 8. Figure 3 As shown, these units communicate via interconnect 9 and access off-chip memory 3.

[0266] In this system, the GPU 2, video codec 1, ISP 13 and / or CPU 7 will generate frames (images) to be displayed, and the display processor 5 will then provide these frames for display to the display 4. For the rest of the system, the display 4 can be local or remote and can have a wired or wireless connection to the display processor 5.

[0267] In use of the system, an application such as a game executing on the main processor (CPU), for example, will need to display frames on the display 4. To do this, the application 10 will submit appropriate commands and data to the driver 11 of the graphics processing unit 2 executing on the CPU 7. The driver 11 will then generate appropriate commands and data to cause the graphics processing unit 2 to render appropriate frames for display and store these frames in appropriate frame buffers (e.g., in the main memory 3). The display processor 5 will then read these frames into a buffer for the display, from which the frames are then read out and displayed on the display panel of the display 4.

[0268] Frames for display may also or instead be streamed from the processing unit that generates the frames (eg, GPU 2 or ISP 13) to the display processor (if desired), for example, via a buffer.

[0269] Figure 3 The illustrated data processing system 14 may be used to provide frames for display to a so-called "XR" display, such as an augmented reality (AR) and / or virtual reality (VR), head mounted display (HMD) system. In this case, the display 4 of the system may include a suitable head mounted display, the head mounted display including, among other things: a display screen (panel) for displaying the frames to be viewed to a user wearing the head mounted display; one or more lenses, the one or more lenses being in the viewing path between the user's eyes and the display screen; and one or more sensors for tracking the pose of the user's head (their head position and / or their viewing (gaze) direction) in use (when displaying images to the user on the display).

[0270] In head mounted AR / VR display operation, appropriate images to be displayed to each eye will be rendered by, for example, the graphics processing unit 2 in response to appropriate commands and data from an application requiring an augmented reality or virtual reality display, such as a game (e.g., executing on CPU 7).

[0271] In such a configuration, the system will also operate to track the movement of the user's head / gaze (so-called head pose tracking). This head orientation (pose) data is then used to determine how images should actually be displayed to the user for the user's current head position (viewing direction), and the images (frames) are rendered accordingly (e.g. by setting the camera (viewpoint) orientation based on the head orientation data) so that an appropriate image can be displayed based on the user's current viewing direction.

[0272] A technique used to help facilitate the generation of frames for display at an appropriate rate in an “XR” display system is so-called “foveated” rendering or “variable resolution” rendering.

[0273] Variable-resolution ("foveated") rendering is a rendering technique in which one or more portions of a frame (image) to be displayed are rendered at a higher resolution, but one or more other portions of the frame are rendered at a lower resolution.

[0274] This is based on the fact that areas of the frame that the user is looking at directly may need to be rendered at a higher resolution in order to be visually acceptable, while peripheral areas of the frame that the user is not looking at directly may be rendered at a lower resolution while still appearing visually acceptable. This can then be used to reduce the rendering burden on, for example, a graphics processing unit (GPU) that generates the frame for display by rendering the peripheral areas at a lower resolution rather than rendering the entire displayed frame at the highest required ("foveated") resolution.

[0275] Variable resolution rendering can be performed by identifying one or more "fixed points" where higher resolution areas of the frame will be rendered, with areas farther from the fixed points being rendered at lower resolutions. Thus, each fixed point can indicate the highest resolution area of ​​the frame, and in some cases, is intended to correspond to the center of the eye's retina (fovea).

[0276] Variable resolution rendering can also be used to account for (and exploit) lens distortion introduced by a head mounted display (HMD). For example, high resolution head mounted displays used for virtual reality applications often use lenses with severe pincushion distortion characteristics. This pincushion distortion can be corrected by passing the rendered image through a barrel distortion. The effect of this is to magnify (for each eye) the rendered image towards the center of the display, while shrinking the peripheral areas. The effect of this then is that the peripheral areas can be rendered at a lower quality than the central magnified area without any significant loss in the overall visual effect for the user.

[0277] When performing variable resolution rendering, the location of the highest resolution region of the frame may be determined in any suitable and desired manner (e.g., a fixed point). For example, some form of head tracking or eye tracking (head pose tracking) system may be used to attempt to identify where the user is looking at the image to identify the region of the frame that should be rendered at the highest resolution. The location of the higher resolution region of the frame may also or instead be based on other factors, such as lens distortion.

[0278] Various embodiments involve performing the encoding technique described above (and in U.S. Patent Application No. 2013 / 0034309) to encode a data array 30, such as a frame for display, in a lossy manner but in a manner that is transparent to the decoder, e.g., such that only modifications with respect to U.S. Patent Application No. 2013 / 0034309 are on the encoding side and such that the decoder can operate in a normal manner.

[0279] This can be particularly useful and applicable in situations such as foveated rendering, where lower resolution (i.e., lossy) regions of the overall frame 30 are acceptable, and the locations of these regions are known in advance (so this information can be provided to the encoder). The encoder can then use the information about the foveated regions to modify how it encodes the individual blocks 31. This works because in the encoding techniques described above, each block or "super-block" 31 is encoded independently and can therefore be treated differently from any other super-blocks that make up the overall frame 30.

[0280] This allows foveated rendering to be achieved by generating a frame 30 at a higher resolution, then encoding higher resolution regions of the frame 30 (such as the "fixed point") using lossless coding, and encoding lower resolution regions of the frame (further away from the fixed point) using lossy coding. Thus, the bandwidth associated with variable resolution images can be reduced through local truncation compression.

[0281] This allows the encoder to produce coded images with less bandwidth, where the data is intentionally discarded during the encoding step. This can provide lossy bandwidth savings for foveated rendering, where the periphery of the image can be stored as a reduced effective resolution. These coded images are compressed in a way that allows them to be decoded by existing decoders.

[0282] Thus, in various embodiments, a lossy version of the above encoding techniques is performed for foveated rendering.Lossy encoding techniques can be used to reduce bandwidth in foveated rendering.

[0283] Lossy encoding can also be performed for other variable resolution techniques. For example, in video decoding, it may be known that the final result of the decoding process will be displayed as a thumbnail. In this case, data may be discarded during encoding (as described above), knowing that the result will still be acceptable.

[0284] In the context of shot-matched foveated rendering, there will be a priori knowledge of what pixel density is expected on the image. This density should be provided to the encoder, for example, during rendering at super-block granularity, although other structures are possible.

[0285] Thus, in various embodiments, when encoding a frame using, for example, foveated rendering or other variable resolution techniques, information (e.g., a "density map") is provided to the encoder that indicates the desired resolution across the frame (in different areas of the frame), and the encoder will then use the provided "density map" to adjust its encoding of blocks (or groups of blocks) across the frame to determine whether to encode areas of the frame in a lossy or lossless manner.

[0286] Thus, the encoder will receive the image data (frame) to be encoded together with a description of how to lossily compress different areas of the image (frame). The lossy compression description may be in the form of, for example, a 2D map that tells the encoder what compression to use where, and may be predetermined based on knowledge of the head mounted display being used, for example.

[0287] In this embodiment, the 2D map may be in the form of a two-dimensional array representing (a region of) the output frame to be displayed, with each data element position within the array then indicating which resolution version of the frame to use for a given region (block) within the output frame.

[0288] Figure 4 An exemplary density texture 400 is shown. In this figure, the darkest texture positions 401 show where a higher resolution version of the frame will be displayed, the lightest texture positions 402 show where the lowest resolution version of the frame will be displayed, and the "mid-grey" texture positions 403 show where a combination of the highest resolution and lowest resolution regions of the frame should be displayed. In practice, each of these density texture positions will be represented by a suitable (greyscale) value (which will be stored for them) which will indicate which resolution version or versions of the frame are to be used for the output frame region corresponding to the density texture position in question.

[0289] Data indicating different resolution versions of a frame to be used for various output frame regions may also be provided in other forms, for example by means of a lookup table, or expressed by some form of function, or in the form of a bounding box (e.g., a 2D axis-aligned bounding box) defining an area of ​​extent where the highest resolution version (level) should be used (all other regions are then generated from lower resolution (level) versions of the frame).

[0290] Once the graphics processor 2 has generated and encoded the appropriate frame for display, the display processor 5 will read the frame for display from the memory 3, decode the frame, and provide the appropriate output frame (eg, one frame for each eye) for display.

[0291] The encoding selection may be per "superblock" or for groups of superblocks (some combination of superblocks), such as a 32x32 pixel block. The encoding selection will also be made within the resolution of a given superblock.

[0292] When lossy encoding is to be used (eg, in lower resolution regions of a frame), the encoding can be made lossy in a number of different ways (and, eg, the level of lossiness can be varied if desired).

[0293] As mentioned above, the preferred encoding technique is a minimum tree delta encoding scheme. The encoded image is effectively a collection of "superblocks" (16x16 or 32x8 pixels), each described by a fixed-size header and a variable-size payload. Typically, each superblock consists of a set of 4x4 pixel subblocks, each stored as a 2-level 4-way tree, although the header can also encode a constant color block without a payload.

[0294] The most extreme loss would be to encode the superblock as a constant color. This would give a peak compression rate of 16 bytes / 256 pixels = 1 / 2 bit per pixel. It would also be possible to encode subblocks within a superblock as a constant color.

[0295] A less lossy technique would be to truncate the tree early and / or mark the subblocks in the superblock as copies of each other.

[0296] Starting from this, the encoding can be configured to reduce the number of bits required to store the differences (Δ) in the difference tree. This can be done by reducing the size of the differences that need to be stored at each level in the tree.

[0297] In the most extreme case, the parent node can be set to the average of the child nodes, and the child nodes can all be set to zero. Thus, for example, while calculating the minimum tree, the average of the data elements can be calculated instead, and all child nodes can be set to zero, thus reducing their bit count to zero.

[0298] Figure 5 An example of a 4x4 block 40 of truncated data elements 50 is shown. Figure 5 (A) shows the original non-truncated values ​​of individual data elements 59 of the 4x4 array of data elements 50, i.e., a 4x4 tile, without truncation. Figure 5 As shown in (B), the data elements of each 2x2 sub-block 58 of the 4x4 array 50 can be averaged, that is, 4x4 tiles, level 1 truncation. Additionally or alternatively, as Figure 5 As shown in (C), the data elements of the entire 4x4 array 50 can be averaged, that is, 4x4 tiles, level 2 truncation.

[0299] These techniques can be performed at any level within the difference tree, and will modify only some but not all child nodes (e.g., set them to zero), thereby changing the degree of compression. In the case of averaging groups of four data elements, the averaging can be performed with the equivalent of a right shift rather than integer division.

[0300] Additionally or alternatively, the parent node may be set to a value that allows its difference with the child node to be reduced so that a smaller difference needs to be encoded, thereby reducing the amount of encoding required to encode the child node's difference. For example, the child node difference may also be halved (rather than zeroed) when encoding. For example, the value of Δ may also be simply reduced, rather than zeroing them, to reduce the amount of encoding required to encode the difference.

[0301] Performing any of these steps will still produce a valid encoded image, but with a small (or zero) payload size. The fact that the above encoding technique encodes super-blocks individually means that the decision of how much to encode / discard can be made per super-block, so the compression rate can vary across the image. Thus, the above scheme provides a simple means of truncating data in a limited set of steps.

[0302] Figure 6 The decision process for truncation by averaging according to an embodiment is shown. Figure 6 As shown, the process 60 may be performed for each 4x4 block 40 of data elements in the data array. For each level 61 in the tree representation, if truncation is to be performed at that level 62, the average of the sub-pixels is calculated 63 and each sub-pixel value is replaced with the average 64. Processing then continues as described above to find the minimum sub-pixel value 65 and replace each sub-pixel value with a difference relative to the minimum sub-pixel value 66. The process is performed from the bottom of the tree upwards, thus continuing with the parent level 67.

[0303] Also like Figure 6 As shown, if truncation is to be performed 68 for a superblock, the average of the values ​​of the 4x4 tiles is calculated 69 and the superblock is encoded as a constant color 70, i.e., the average. In this case, all data related to the sub-blocks within the superblock are removed 71 and not used to encode the superblock.

[0304] The above technique preserves the average image intensity (depending on the surrounding rounding decision), but will reduce the frequency content of the image. In terms of appearance, this will be similar to truncating the Haar wavelet expansion of the block, and this will introduce blocky artifacts.

[0305] Therefore, a further improvement would be to apply a low pass filter before generating the minimum tree, to remove higher frequency components. This has the effect of producing smoother results when decoding lossy coded data, for example, by reducing high frequencies and making delta coding increasingly efficient, but producing smoother results.

[0306] A preferred filtering structure would be to perform a frequency domain transform (e.g., a discrete cosine transform (DCT)) on the block or superblock values, truncate the high frequency components in the frequency domain, and then transform back to the spatial domain before encoding. The DCT option may be useful in settings where the image data is subjected to a DCT for other reasons. Alternatively, direct spatial filtering may be used.

[0307] Figure 7 The corresponding decision process for truncation by filtering according to an embodiment is shown. Figure 7 As shown, the process is Figure 6 The depicted process is substantially similar, except that if truncation 62 is to be performed at the level, the sub-pixels 72 are replaced with low pass filtered sub-pixels (as described above).

[0308] Other structures would be possible. For example, if the decoder knows about the lossy compression scheme, it can use a more complex storage format in its line buffer, rather than just one data value per pixel. For example, the display processor could store the decompressed data in a format similar to run-length encoding (RLE), since a row of a decompressed image would have portions of (1,2,4,8,16) pixels with the same value. If the decoder knows the exact function used by the encoder to choose how many levels to discard, the display controller can use that function to use a larger surface than would otherwise fit in its line buffer RAM. The advantage over a new lossy format is that the buffer is still compatible with decoders that are not aware that the buffer is different in any way.

[0309] These techniques can be used to enable a decoder that may be limited in the size of conventionally (i.e., non-lossy) encoded frames (images) it can handle (e.g., in terms of the payload and header or payload to header ratio it can handle, etc.) to actually process larger images (frames) by lossily encoding these larger frames to fit the constraints of the decoder (where it is acceptable to not display the final image at full resolution over the entire image).

[0310] Although the present embodiment has been described above as generating a tree representation for 16x16 blocks of data elements, other structures may be used. For example, a separate tree representation may be generated for each 8x8 block of data elements of the data array or each 16x4 block of data elements of the data array (i.e., such that the data array would be divided into 8x8 or 16x4 blocks, respectively).

[0311] Furthermore, although the present embodiments have been described above with particular reference to their use with a graphics processor and a display controller, the techniques described herein may be used for other data array processing, and in particular for other data processing architectures.

[0312] For example, the techniques in the solutions described herein can be used in image signal processors and video decoders and encoders (MPEG / h.264, etc.). In these cases, for example, the techniques in the solutions described herein can be used to encode images generated by an image signal processor that processes data received from an image sensor to form a viewable image with the data. The video encoder / decoder can, for example, decode images (video frames) encoded in the form of the techniques described herein to then compress the images using some other standard (like h.264), and use the techniques in the solutions described herein to encode frames of video data correspondingly, for example, to provide to a graphics processor or display controller.

[0313] As can be seen from the above, at least in some embodiments, the techniques described herein provide methods and apparatus for encoding data arrays that can allow the encoded data to take up less storage space (to be stored more efficiently), reduce the amount of storage traffic used to read the encoded data, and / or make the storage traffic used to read the encoded data more efficient. Thus, power consumption can be reduced.

[0314] In at least some embodiments, this is achieved by providing to the encoder data indicating a resolution to be used by the consumer circuitry for at least one region of the array of data elements, and the encoder using the data indicating the resolution to be used by the consumer circuitry to control generation of a representation for representing at least one block into which the array of data elements is divided.

Claims

1. A method of operating a data processing system, the data processing system comprising: encoding circuitry operable to encode the array of data elements; decoding circuitry operable to decode an encoded version of the array of data elements; as well as a consumer circuit operable to consume the array of data elements; The method comprises the following steps: The encoding circuit encodes the array of data elements to produce an encoded version of the array of data elements by dividing the array of data elements into a plurality of separate blocks, generating respective representations for representing the different blocks into which the array of data elements is divided, and generating data representing each representation in an encoded form to represent the array of data elements as the encoded version of the array of data elements; The decoding circuitry decodes at least a portion of the encoded version of the array of data elements to produce a decoded version of the array of data elements; and the consumer circuit using at least a portion of the decoded version of the array of data elements; The method further comprises the following steps: providing to the encoding circuit data indicating a resolution to be used by the consumer circuit for at least one region of the array of data elements; and The encoding circuit uses the data indicating the resolution to be used by the consumer circuit to control generation of the representation representing at least one block into which the array of data elements is divided and generation of data representing the representation representing the at least one block by: the encoding circuitry determines whether the data indicating a resolution indicates that the consumer circuitry is to use a relatively higher resolution for a region of the array of data elements corresponding to the at least one block, or whether the data indicating a resolution indicates that the consumer circuitry is to use a relatively lower resolution for a region of the array of data elements corresponding to the at least one block; and When the encoding circuitry determines that the data indicating a resolution indicates that the consumer circuitry is to use a relatively higher resolution for a region of the array of data elements corresponding to the at least one block: The encoding circuit selects a higher-fidelity representation to represent the at least one block, generates a higher-fidelity representation to represent the at least one block, and generates data representing the higher-fidelity representation in encoded form; and When the encoding circuitry determines that the data indicating a resolution indicates that the consumer circuitry is to use a relatively lower resolution for a region of the array of data elements corresponding to the at least one block: The encoding circuit selects a lower-fidelity representation to represent the at least one block, generates a lower-fidelity representation to represent the at least one block, and generates data representing the lower-fidelity representation in encoded form.

2. The method according to claim 1, wherein: The consumer circuit comprises a display controller operable to provide the decoded version of the array of data elements to a display for display; and The step of using the decoded version of the array of data elements includes the display controller providing the decoded version of the array of data elements to the display for display.

3. The method according to claim 1, wherein: The consumer circuit includes a graphics processing unit; and The step of using the decoded version of the array of data elements includes the graphics processing unit using the decoded version of the array of data elements when rendering an image for display.

4. The method of claim 1 or 2, comprising the encoding circuit selecting one of a lossless representation representing the at least one block and a lossy representation representing the at least one block.

5. The method of claim 1 or 2, comprising the encoding circuit selecting a tree representation for representing the at least one block.

6. The method of claim 1 or 2, comprising the encoding circuit selecting a truncation tree representation for representing the at least one block.

7. The method of claim 1 or 2, comprising the encoding circuit selecting a tree representation for representing the at least one block, wherein: The tree representation includes at least one indication associated with a node of the tree, the indication indicating that the node is a copy of another node.

8. The method according to claim 1 or 2, wherein: The lower-fidelity representation comprises a tree representation for representing the at least one block, and wherein the step of generating the tree representation comprises: The data value to be associated with each node of the tree representation is determined by the following process: In a first processing pass, each leaf node in the tree is set to the value that the tree would indicate for the data element in the array of data to be encoded represented by the leaf node, and each non-leaf node in the tree is set to the value of one of its child nodes; In the second processing round, subtract the value of each node from its parent node; and then One or more of the result values ​​are reduced to determine the data value to be associated with each node of the tree representation.

9. The method according to claim 1 or 2, wherein: The lower-fidelity representation comprises a tree representation for representing the at least one block, and wherein generating the tree representation comprises setting one or more non-leaf nodes in the tree to an average of its child nodes.

10. The method according to claim 1 or 2, wherein: The lower-fidelity representation comprises a tree representation for representing the at least one block, and wherein generating the tree representation comprises filtering the array of data elements.

11. The method according to claim 1 or 2, wherein: The lower-fidelity representation includes a representation for representing the at least one block, wherein a single data value represents a plurality of data elements of the at least one block.

12. A method of encoding an array of data elements using encoding circuitry of a data processing system, the method comprising the steps of: The encoding circuit encodes the array of data elements to produce an encoded version of the array of data elements by dividing the array of data elements into a plurality of separate blocks, generating respective representations for representing the different blocks into which the array of data elements is divided, and generating data representing each representation in an encoded form to represent the array of data elements as the encoded version of the array of data elements; The method further comprises the following steps: providing data to the encoding circuitry indicating a resolution to be used for at least one region of the array of data elements; and The encoding circuitry controls generation of the representation representing at least one block into which the array of data elements is divided and generation of data representing the representation representing the at least one block, using the data indicating the resolution to be used for the at least one region of the array of data elements, by: the encoding circuitry determining whether the data indicating a resolution indicates that a relatively higher resolution is to be used for a region of the array of data elements corresponding to the at least one block, or whether the data indicating a resolution indicates that a relatively lower resolution is to be used for a region of the array of data elements corresponding to the at least one block; and When the encoding circuitry determines that the data indicating a resolution indicates that a relatively higher resolution is to be used for a region of the array of data elements corresponding to the at least one block: The encoding circuit selects a higher-fidelity representation to represent the at least one block, generates a higher-fidelity representation to represent the at least one block, and generates data representing the higher-fidelity representation in encoded form; and When the encoding circuitry determines that the data indicating a resolution indicates that a relatively lower resolution is to be used for a region of the array of data elements corresponding to the at least one block: The encoding circuit selects a lower-fidelity representation to represent the at least one block, generates a lower-fidelity representation to represent the at least one block, and generates data representing the lower-fidelity representation in encoded form.

13. A data processing system, the data processing system comprising: encoding circuitry operable to encode the array of data elements; decoding circuitry operable to decode an encoded version of the array of data elements; as well as a consumer circuit operable to consume the array of data elements; in, The encoding circuit is configured to encode the array of data elements to produce an encoded version of the array of data elements by: dividing the array of data elements into a plurality of separate blocks, generating respective representations for representing the different blocks into which the array of data elements is divided, and generating data representing the respective representations in encoded form to represent the array of data elements as the encoded version of the array of data elements; the decoding circuitry being configured to decode at least a portion of the encoded version of the array of data elements to produce a decoded version of the array of data elements; and the consumer circuit being configured to use at least a portion of the decoded version of the array of data elements; And among them, The data processing system is configured to provide data to the encoding circuitry indicating a resolution to be used by the consumer circuitry for at least a region of the array of data elements; and The encoding circuit is configured to encode the array of data elements by using the data indicating the resolution to be used by the consumer circuit, control generation of the representation representing at least one block into which the array of data elements is divided and generation of data representing the representation representing the at least one block by: the encoding circuitry determines whether the data indicating a resolution indicates that the consumer circuitry is to use a relatively higher resolution for a region of the array of data elements corresponding to the at least one block, or whether the data indicating a resolution indicates that the consumer circuitry is to use a relatively lower resolution for a region of the array of data elements corresponding to the at least one block; and When the encoding circuitry determines that the data indicating a resolution indicates that the consumer circuitry is to use a relatively higher resolution for a region of the array of data elements corresponding to the at least one block: The encoding circuit selects a higher-fidelity representation to represent the at least one block, generates a higher-fidelity representation to represent the at least one block, and generates data representing the higher-fidelity representation in encoded form; and When the encoding circuitry determines that the data indicating a resolution indicates that the consumer circuitry is to use a relatively lower resolution for a region of the array of data elements corresponding to the at least one block: The encoding circuit selects a lower-fidelity representation to represent the at least one block, generates a lower-fidelity representation to represent the at least one block, and generates data representing the lower-fidelity representation in encoded form.

14. The data processing system according to claim 13, wherein: The consumer circuit comprises a display controller operable to provide the decoded version of the array of data elements to a display for display.

15. The data processing system according to claim 13, wherein: The consumer circuit includes a graphics processing unit.

16. The data processing system according to claim 13 or 14, wherein: The encoding circuit is configured to select one of a lossless representation for representing the at least one block and a lossy representation for representing the at least one block.

17. The data processing system according to claim 13 or 14, wherein: The encoding circuit is configured to select a tree representation for representing the at least one block.

18. The data processing system according to claim 13 or 14, wherein: The encoding circuit is configured to select a truncation tree representation for representing the at least one block.

19. The data processing system according to claim 13 or 14, wherein: The encoding circuit is configured to select a tree representation for representing the at least one block, wherein the tree representation includes at least one indication associated with a node of the tree, the indication indicating that the node is a copy of another node.

20. The data processing system according to claim 13 or 14, wherein: The lower-fidelity representation comprises a tree representation for representing the at least one block, and wherein the encoding circuit is configured to generate the tree representation by: The data value to be associated with each node of the tree representation is determined by the following process: In a first processing pass, each leaf node in the tree is set to the value that the tree would indicate for the data element in the array of data to be encoded represented by the leaf node, and each non-leaf node in the tree is set to the value of one of its child nodes; In the second processing round, subtract the value of each node from its parent node; and then One or more of the result values ​​are reduced to determine the data value to be associated with each node of the tree representation.

21. The data processing system according to claim 13 or 14, wherein: The lower-fidelity representation comprises a tree representation for representing the at least one block, and wherein the encoding circuitry is configured to generate the tree representation by setting one or more non-leaf nodes in the tree to an average value of its child nodes.

22. The data processing system according to claim 13 or 14, wherein: The lower-fidelity representation includes a representation for representing the at least one block, and wherein the encoding circuitry is configured to generate a tree representation by filtering the array of data elements.

23. The data processing system according to claim 13 or 14, wherein: The lower-fidelity representation includes a representation for representing the at least one block, wherein a single data value represents a plurality of data elements of the at least one block.

24. A data processing system, the data processing system comprising: encoding circuitry configured to encode an array of data elements to produce an encoded version of the array of data elements by dividing the array of data elements into a plurality of separate blocks, generating respective representations representing the different blocks into which the array of data elements is divided, and generating data representing the respective representations in encoded form to represent the array of data elements as the encoded version of the array of data elements; wherein the data processing system is configured to provide the encoding circuit with data indicating a resolution to be used for at least one region of the array of data elements; and The encoding circuitry is configured to encode the array of data elements by using the data indicating the resolution to be used for at least one region of the array of data elements, controlling generation of the representation representing at least one block into which the array of data elements is divided and generation of data representing the representation representing the at least one block by: the encoding circuitry determining whether the data indicating a resolution indicates that a relatively higher resolution is to be used for a region of the array of data elements corresponding to the at least one block, or whether the data indicating a resolution indicates that a relatively lower resolution is to be used for a region of the array of data elements corresponding to the at least one block; and When the encoding circuitry determines that the data indicating a resolution indicates that a relatively higher resolution is to be used for a region of the array of data elements corresponding to the at least one block: The encoding circuit selects a higher-fidelity representation to represent the at least one block, generates a higher-fidelity representation to represent the at least one block, and generates data representing the higher-fidelity representation in encoded form; and When the encoding circuitry determines that the data indicating a resolution indicates that a relatively lower resolution is to be used for a region of the array of data elements corresponding to the at least one block: The encoding circuit selects a lower-fidelity representation to represent the at least one block, generates a lower-fidelity representation to represent the at least one block, and generates data representing the lower-fidelity representation in encoded form.

25. A computer readable storage medium storing a computer program which, when executed on a processor, performs the method according to any one of claims 1 to 12.

Images