Systems and methods for concave compression of image frames in a system on a chip

Abstract

An exemplary method for intelligent compression uses a foveated compression method. First, the location of a gaze point within an image frame is determined. Next, the image frame is segmented into two or more sectors such that one of the two or more sectors is designated as a gaze sector and the remaining sectors are designated as foveated sectors. A sector can be defined by one or more tiles within the image frame. The gaze sector includes the particular tile that contains the gaze point and is compressed according to a lossless compression algorithm. The foveated sectors are compressed according to a lossy compression algorithm. As the angular distance of a foveated sector's location from the gaze sector's location increases, the compression factor can increase.

Description

[0001] This application is a divisional application of the patent application for the invention entitled "System and method for concave compression of image frames in a system on a chip", filed on September 7, 2018, with application number 201880063413.0. BACKGROUND

[0002] Portable computing devices ("PCDs") have become necessities for personal and professional use. These devices can include cellular telephones, portable digital assistants ("PDAs"), portable game consoles, palmtop computers, and other portable electronic devices. PCDs typically contain integrated circuits or system on a chip ("SoC") that include multiple components designed to work together to provide functionality to the user. For example, a SoC can contain any number of processing engines, such as modems, central processing units ("CPUs") composed of cores, graphics processing units ("GPUs"), etc., that read and write data and instructions to / from memory components on the SoC. Data and instructions are transferred between devices via a collection of lines known as a bus.

[0003] The effective size of the bus bandwidth and memory components in a PCD is critical to optimizing the functional capabilities of the processing components on the SoC and ensuring minimum required quality of service ("QoS") levels. Generally, the utilization of memory capacity and bus bandwidth is further optimized by compressing data, such that less bus bandwidth is required for transmission and less memory space is required for storage of the data. Not all data / image frames compress with the same efficiency, or in that regard, not all data / image frames require the same compression efficiency in order to maintain an adequate QoS, thus PCD designers are faced with a trade-off decision - compressing using a lossy compression method, which produces a lower quality output upon decompression, in return for the benefit of smaller memory components and bus bandwidth requirements, or alternatively, compressing using a lossless compression method, which produces a high quality output upon decompression, but requires relatively larger memory components and bus bandwidth to maintain a satisfactory QoS. Either way, designers must size the bus and memory components in view of the "worst case" compression, otherwise they risk a reduction in QoS, as measured by a variety of key performance indicators.

[0004] In short, current systems and methods for data / image frame compression known in the art dictate that PCD designers must use memory components and bus bandwidth that are oversized for most use cases in order to ensure that an acceptable level of QoS is provided. Notably, however, user visual acuity is highest at the fovea and perceptible visual detail drops off rapidly outside the focal area containing the point of gaze. Thus, high quality compression of data within image frames associated with areas outside the user's focal area can not have a significant positive impact on QoS. Therefore, there is a need in the art for an intelligent compression system and method that utilizes knowledge of the user's focal point of gaze to compress image frames in a foveated manner such that a no / low compression, high quality output compression algorithm is used on frame areas associated with the user's focal point while successively higher compression, lower quality output compression algorithms are used on frame areas associated with the user's peripheral vision. SUMMARY

[0005] Various embodiments of methods and systems for intelligent data compression in a portable computing device ("PCD") are disclosed. An exemplary method begins with determining a point of gaze within an image frame. With the point of gaze determined, the image frame is next segmented into two or more sectors such that one of the two or more sectors is designated as a gaze sector and the remaining sectors are designated as foveated sectors. The sectors can be defined by one or more tiles within the image frame. The gaze sector contains the particular tile containing the point of gaze and is compressed according to a lossless compression algorithm (e.g., JPEG) having a low compression factor. The foveated sectors are compressed according to a compression algorithm having a high compression factor relative to the algorithm used to compress the gaze sector. Embodiments of the solution can increase the compression factor of the algorithm used to compress tiles within a given foveated sector as the angular distance of the location of the foveated sector from the location of the gaze sector increases. In this manner, an image frame can be compressed according to a foveated compression method. The compressed frame can be stored in memory until later retrieved, decompressed, and rendered.

[0006] Depending on the embodiment, the image frame can be segmented into a square grid foveated compression pattern, a cross grid foveated compression pattern, a honeycomb compression pattern, etc. Further, depending on the embodiment, all tiles in a given sector can be compressed according to a single compression algorithm and compression factor or, alternatively, the compression factor can vary for tiles within a given sector (hierarchical compression). BRIEF DESCRIPTION OF DRAWINGS

[0007] In the drawings, like reference numerals refer to like parts throughout the various views unless otherwise indicated. For reference numerals with diacritical markings (e.g., 102A and 102B), the diacritical markings can be used to distinguish between two like parts existing in the same view. The diacritical markings can be omitted when it is intended that a reference numeral apply equally to all like parts within all views.

[0008] Figure 1 The effect of compressing an image frame composed of multiple data sub-units or tiles is shown;

[0009] Figure 2 A compressed data transaction for a DRAM memory component is shown, with the minimum access length ("MAL") per transaction required;

[0010] Figure 3 A series of compressed data transactions associated with an exemplary image frame is shown;

[0011] Figure 4 is a functional block diagram showing one embodiment of a system-on-a-chip for intelligent compression using a foveated compression method;

[0012] Figure 5 is a functional block diagram showing one embodiment of an image CODEC module configured to implement an intelligent compression method that leverages a foveated compression method, including lossless and lossy algorithms; Figure 4

[0013] Figure 6 Exemplary chart outputs of mathematical equations and exemplary lookup tables are shown, each specifying a compression ratio according to a given sector's distance from a foveal point;

[0014] Figure 7 Exemplary foveated sectors defined within an image frame are shown, each foveated sector containing one or more tiles of the image frame;

[0015] Figure 8 An exemplary image frame is shown segmented and compressed according to one embodiment of a solution that leverages an exemplary square grid foveated compression pattern;

[0016] Figure 9 An exemplary image frame is shown segmented and compressed according to one embodiment of a solution that leverages an exemplary hierarchical square grid foveated compression pattern;

[0017] Figure 10 An exemplary image frame is shown segmented and compressed according to one embodiment of a solution that leverages an exemplary cross grid foveated compression pattern;

[0018] ​Figure 11 An exemplary image frame is shown segmented and compressed according to one embodiment of the solution utilizing an exemplary hierarchical cross-hatched concave compression pattern;

[0019] Figure 12 An exemplary image frame is shown segmented and compressed according to one embodiment of the solution utilizing an exemplary honeycomb concave compression pattern;

[0020] Figure 13 An exemplary image frame is shown segmented and compressed according to one embodiment of the solution utilizing an exemplary hierarchical honeycomb concave compression pattern;

[0021] Figure 14 An exemplary image frame is shown segmented and compressed according to one embodiment of the solution utilizing an exemplary hierarchical honeycomb concave compression pattern;

[0022] Figure 15 is a logic flow diagram illustrating the intelligent compression method according to the solution which utilizes a concave compression method (including lossless and lossy algorithms);

[0023] Figure 16 is a functional block diagram illustrating an exemplary, non-limiting aspect of a portable computing device ("PCD") in the form of a wireless telephone for implementing the intelligent compression method and system according to the solution;

[0024] Figure 17 is a schematic diagram illustrating an exemplary software architecture of the PCD of Figure 16 ; and

[0025] Figure 18 is a diagram relating to a small concave. DETAILED DESCRIPTION

[0026] The word "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0027] In this description, the term "application" also can include files having executable content, such as: object code, scripts, byte code, markup language files, and patches. In addition, an "application" referred to herein can also include files that are not executable in nature, such as documents that can need to be opened or installed software that needs to be accessed.

[0028] In this specification, references to "DRAM" or "DDR" memory components will be understood to contemplate any of a broader class of volatile random access memory ("RAM") and will not limit the scope of the solutions disclosed herein to a specific RAM type or generation. That is, it will be understood that various embodiments of the systems and methods provide a solution for managing transactions of data that has been compressed according to lossless and / or lossy compression algorithms and are not necessarily limited to application to compressed data transactions associated with double data rate memory. Moreover, it is contemplated that certain embodiments of the solutions disclosed herein can be applicable to DDR, DDR-2, DDR-3, low power DDR ("LPDDR"), or any subsequent generation of DRAM.

[0029] As used in this specification, the terms "component," "database," "module," "block," "system," and the like are intended to refer to a computer-related entity, either hardware, firmware, a combination of hardware and software, software, or software in execution, unless the context clearly indicates otherwise. For example, a component can be, but is not limited to being, a process running on a processor, a processor, a target, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a computing device and the computing device can be a component. One or more components can reside within a process and / or thread of execution and a component can be localized, co-resident, and / or distributed amongst one computer and / or across two or more computers. In addition, these components can execute from various computer readable media having various data structures stored thereon. The components can communicate by way of local and / or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and / or across a network such as the Internet with other systems by way of the signal).

[0030] In this specification, the terms "central processing unit ("CPU")," "digital signal processor ("DSP")," "graphics processing unit ("GPU")," and "chip" are used interchangeably. Moreover, a CPU, DSP, GPU, or chip can contain one or more distinct processing components (which are generally referred to herein as "core(s)").

[0031] In this specification, the terms "engine," "processing engine," "processing component," "producer," and the like are used to refer to any component in a system on a chip ("SoC") that generates data and / or image frames and passes them to or from a memory component over a bus. Thus, an engine can refer to, but is not limited to, a CPU, a DSP, a GPU, a modem, a controller, a camera, a camcorder, and the like.

[0032] In this specification, the term "bus" refers to a collection of wires over which data can be transmitted from a processing engine to a memory component or other device located on or off the SoC. It will be understood that a bus is composed of two parts - an address bus and a data bus, where the data bus transfers the actual data and the address bus transfers information specifying the location of the data in the memory component (i.e., the address and associated metadata). The term "width" or "bus width" or "bandwidth" refers to the amount of data that can be transferred over a given bus per cycle, i.e., the "data block size." For example, a 16-byte bus can transfer 16 bytes of data at a time, while a 32-byte bus can transfer 32 bytes of data per cycle. Further, "bus speed" refers to the number of data blocks that can be transferred over a given bus per second. Similarly, "bus cycle" or "cycle" refers to the transfer of one data block over a given bus.

[0033] In this specification, the term "portable computing device" ("PCD") is used to describe any device operating on a finite capacity power supply, such as a battery. Although battery-powered PCDs have been in use for decades, technological advances in rechargeable batteries, coupled with the advent of third generation ("3G") and fourth generation ("4G") and fifth generation ("5G") wireless technology, have resulted in a variety of PCDs having a multitude of functions. Therefore, a PCD can be a cellular telephone, a satellite telephone, a pager, a PDA, a smartphone, a navigation device, a smartbook or reader, a media player, a combination of the aforementioned devices, a laptop with wireless connectivity, and the like.

[0034] In this specification, the terms "lossless" and "lossy" refer to different categories of compression algorithms or methods, and are not intended to refer to any specific algorithm. Whether a given specific compression algorithm is "lossless" or "lossy" will be recognized by one of ordinary skill in the art. Generally, and as will be appreciated by one of ordinary skill in the art, "lossless" and "lossy" are terms that describe whether all of the original data can be recovered when a file is decompressed in the compression of a data set or image frame. With "lossless" compression, every single bit of data that was originally in the uncompressed frame will be preserved after the frame is decompressed, i.e., all information is fully restored. Graphics Interchange Format ("GIF") is an exemplary image format that provides lossless compression. In contrast, lossy compression algorithms reduce a frame or data set by permanently eliminating certain information, particularly redundant information. Thus, when a file compressed with a lossy algorithm is decompressed, only a portion of the original information is still there (but the user experience can not be affected as a result). Lossy compression algorithms can be suitable for video and sound-based use cases, e.g., when a user can not be able to perceive a certain amount of information loss. JPEG image files are an exemplary image format that provides lossy compression. With lossy compression algorithms, a designer can decide how much loss to introduce (according to a compression factor associated with a given lossy compression algorithm), and trade off between file size and output image quality. The higher the compression factor of a lossy compression algorithm, the relatively smaller the size of the resulting compressed file when the file is later decompressed, and the lower the output image quality. Similarly, the lower the compression factor of a lossy compression algorithm, the relatively larger the size of the resulting compressed file when the file is later decompressed, and the higher the output image quality (approaching or meeting the output quality of lossless compression).

[0035] In this specification, the terms "image," "data set," "data," "frame," "image frame," "buffer," "file," and the like are used interchangeably. Although embodiments of the solution are described herein in the context of a producer component generating a data set in the form of an image frame (e.g., that can be generated by a camera or video subsystem), it should be understood that the solution described herein is not limited to application to image frames. Rather, it is contemplated that embodiments of the solution can be applicable to any use case that can generally benefit from compression of data.

[0036] In this specification, the terms "tile" and "cell" are used interchangeably to refer to a block of pixels that forms a subset of a larger block of data (e.g., an image frame). Depending on the embodiment of the solution, a "tile" or "cell" can exhibit any aspect ratio that is suitable for the embodiment, and thus one of ordinary skill in the art will appreciate that a "tile" or "cell" in an image frame need not be limited to having an "square" aspect ratio - that is, depending on the embodiment, a "tile" or "cell" can be rectangular.

[0037] In the present specification, the terms "sector" and "concave sector" are used interchangeably to refer to a portion of an image frame containing one or more tiles. In the context of the solution described herein, a "sector" can be of any shape or size, as long as it can be divided into one or more complete tiles.

[0038] In the present specification, the term "point of regard" refers to a location within an image frame determined to require the relatively highest output quality after decompression. The "point of regard" can correspond to a portion of the image frame perceived by the user's fovea or within a range defined at a certain standard deviation from the fovea (see Figure 18 ), rather than a portion perceived within the user's peripheral vision. The "point of regard" can correspond to a given sector or possibly a given tile of the image frame.

[0039] In the present specification, the term "saccades" refers to relatively fast movements of the user's eyes between points of regard. Depending on the particular embodiment of the solution, it can be determined whether the user is perceiving the image subject to saccades, i.e. whether the user is "quickly" switching back and forth from one point of regard to another.

[0040] In the present specification, the term "uncompressed" refers to a frame in its original, pre-compressed state, while the term "decompressed" refers to a frame that was first compressed from its uncompressed state using a compression algorithm and subsequently decompressed. Depending on the category of compression used, the dataset of a decompressed frame can or can not be identical to the dataset of the frame's original uncompressed state (lossless compression) or not (lossy compression).

[0041] As will be appreciated by those skilled in the art of frame compression, the resulting compression ratio generated by any given compression algorithm inevitably differs from frame to frame. The level of texture, lighting conditions, ISO settings, etc. in a given frame of a digitally captured video sequence can be significantly different from a different frame in the same sequence, and therefore, the relative compression level of these frames is also significantly different. For example, a frame in a video sequence capturing several people standing and talking is more likely to experience a high level of compression than a subsequent frame capturing these same people sprinting away from an explosion in the background. In short, a frame with a running subject and an explosion simply has a lot of data, and the loss of that data in compression cannot be made up if the frame is to provide a high quality rendering when decompressed at a later time.

[0042] With sufficiently large processing components, memory components, bus bandwidth, and power supply, PCD designers wouldn't need to consider trade-offs between compression algorithms; they could simply produce frames at the highest possible quality and compress them using lossless compression algorithms (if available). This would ensure that the QoS of the user experience is always at its maximum possible level. However, the limited form factor of the PCD forces designers to weigh various compression algorithms when setting the size of the components within the PCD to provide a minimum acceptable QoS for all anticipated use cases.

[0043] Therefore, PCD designers typically use a "real-worst-case" compression ratio when evaluating the bandwidth requirements and component sizes for all known use cases. Thus, the real-worst-case compression ratio is the minimum acceptable QoS required to maintain the most challenging frames that need compression (e.g., capturing a frame of people sprinting down a street away from an explosion in the background). In the real-worst-case scenario, designers make conservative size settings to ensure that memory and bus bandwidth are always sufficient regardless of the complexity of a given frame. It is worth noting that if designers set the bandwidth too low, the resulting latency in processing during problematic use cases can lead to frame drops, reduced frames per second ("FPS"), etc. Conversely, for all use cases better than the real-worst-case scenario, the system may be significantly oversized to provide the minimum acceptable QoS, resulting in higher system cost and / or higher system power consumption.

[0044] Advantageously, embodiments of the solution provide designers with the ability to "correctly set" the size of producers, memory components, and bus bandwidth to optimize power consumption and QoS levels across a range of use cases. Notably, embodiments of the solution leverage knowledge of the intra-frame gaze point (i.e., the frame region focused on by the user) to apply a hybrid-mode compression method that uses lossless compression (or possibly no compression) for the sectors of the frame containing the gaze point, while using lossy compression for the remaining sectors of the frame. In doing so, and as will be explained more fully below with reference to the accompanying drawings, the embodiments cleverly provide the use of both lossless and lossy compression algorithms in the compression of a given image frame, thereby reducing the processing and bus bandwidth required in the worst-case scenario.

[0045] For example, returning to the use case of image frames within a video sequence capturing several people sprinting along a street away from an explosion in the background, embodiments of the solution leverage knowledge about the user's focus point within the frame (e.g., the faces of the people, the background explosion, cars on the road, etc.) and instead of applying lossless compression across the entire frame, apply it only to a portion of the frame that is the object of the user's focus. In this way, intraframe data perceived outside the user's concave area (i.e., within the user's extended peripheral field of view) and therefore having little or no positive impact on QoS when compressed according to a lossless algorithm, can be compressed using a lossy compression algorithm (and later decompressed). Furthermore, it is conceivable that some embodiments could even identify the location of the user's blind spot and use a very high compression algorithm to compress the data associated with the blind spot (or alternatively, discard the data associated with the blind spot entirely instead of compressing it).

[0046] Go to Figure 1 This illustrates the effect of compressing an image frame composed of multiple data sub-units or tiles. Various embodiments may be described in the context of an image frame or a portion of an image frame composed of 256-byte tiles. However, it is important to understand that the 256-byte tile size and the individual compressed data transaction sizes are exemplary in nature and do not imply that embodiments of the solution are limited to applications with a 256-byte tile size. Furthermore, it will be understood that any specific minimum access length (“MAL”) or access block size (“ABS”) for DRAM mentioned in this specification is used for the convenience of describing the solution and does not imply that embodiments of the solution are limited to applications with specific MAL requirements. Therefore, those skilled in the art will recognize that the specific data transfer sizes, data block sizes, bus widths, MALs, etc., mentioned in this specification are provided for illustrative purposes only and do not limit the scope of the contemplated solutions to applications with the same data transfer sizes, data block sizes, bus widths, MALs, etc.

[0047] return Figure 1 The illustration shows a portion of an uncompressed image frame (aka a "buffer") depicted as containing thirty uncompressed tiles or units, each of size "K", as shown without shading. The exemplary size K could be 256 bytes; however, as explained above, the tile size is not limited to any particular size and can vary depending on the application. For ease of illustration and description, the thirty-tile portion of a larger uncompressed image frame is depicted as representing the entire image frame. As those skilled in the art will understand, the size of the uncompressed image frame can be reduced by applying a compression algorithm to each tile in a compressor block (in...). Figure 1The image CODEC module (113A) is described in the diagram to optimize its transmission on the bus, reduce overall system power consumption, and minimize its impact on storage capacity. The compression results in a compressed image frame plus a metadata file, as shown in the diagram relative to the frame. Figure 1 As can be seen in the illustration, the compressed image frame, after undergoing the compression algorithm of compression block 113A, contains the tiles from the original uncompressed image frame.

[0048] In an uncompressed image frame, each tile can be K in size, while in a compressed image frame, each tile can be K or smaller (K ​​represents possible uncompressed, K-1 bytes, K-2 bytes, K-3 bytes, ..., K=1 byte). In the illustration, the individual tiles forming the compressed image frame are represented by different shade levels depending on the degree of compression resulting from the compression algorithm applied by compression block 113A to the data held by a given tile. It is noteworthy that, as those skilled in the art will understand, compression block 113A creates an accompanying buffer for compressed image frame metadata. The compressed image frame metadata contains a record of the size, type, and attributes of each compressed tile in the compressed image frame. Since DRAM access may be limited to minimum access lengths (MALs), the size of a given compressed tile can be represented in the metadata as the number of ABS required to represent the compressed tile size (e.g., 1MAL, 2MAL, ..., nMALs). This size description in the metadata allows future buffer readers to query the memory only the minimum amount of data required to decompress each tile back to its original size K.

[0049] Figure 2 This illustrates compressed data transactions of a DRAM memory component, which satisfy the requirement that each transaction is an integer multiple of the Minimum Access Length per Transaction (“MAL”). (See example from...) Figure 2As illustrated, the length of a compressed tile can be less than an integer multiple of the minimum access length requirement of the DRAM storing it. Therefore, a request for compressed data requires a transaction containing a certain amount of useless data or "padding" that satisfies an integer multiple of the MAL requirement. Padding, indicating the absence of useful information, is added to the compressed tile data so that the transaction size is an integer multiple of the system MAL (i*MAL). Depending on the specific chip technology (e.g., LPDDR2, LPDDR3, LPDDR4, etc.) and the memory bus width (x16, x32, x64), an exemplary MAL can be 32 bytes or 64 bytes. As an example, a compressed tile of size 63 bytes can be padded with 1 byte of padding data to form a complete 64-byte transaction size (2x32B MAL or 1x64BMAL). Similarly, if the MAL is 32 bytes, a compressed tile of size 65 bytes can be padded with 31 bytes of padding data (3x32B MAL), or if the MAL is 64 bytes, it can be padded with 63 bytes of padding data to form a complete 128-byte transaction size (2x64B MAL). Note that in the examples above, the difference in compressed tile size is only 2 bytes; however, since a 65-byte compressed tile exceeds 64 bytes, its transaction must contain significantly more padding.

[0050] Figure 3 A series of compressed data transactions associated with the exemplary image frame are shown. It is noteworthy that... Figure 3 The transactions or units in the illustration (and Figure 1 and Figure 2 The units depicted in the diagram can be considered to have been compressed according to lossless or lossy compression algorithms.

[0051] The image frame is shown as having "N" columns and "M" rows of tiles. According to methods known in the art, the first four sequential tiles in the first row are shown with their uncompressed length, compressed length, and transaction length (compressed length plus padding). For ease of explanation, the first four sequential tiles are explained in the context of the concept depicted, relating to the group of tiles other than the first four sequential tiles in the first row of the image frame, as will be understood by those skilled in the art.

[0052] Observe the four exemplary sequential tiles in their uncompressed state, each tile (#1,0; #2,0; #3,0; #4,0) having a length of 256 bytes (other lengths can be anticipated). When compressed, the lengths of the four exemplary sequential tiles are 112 bytes, 56 bytes, 33 bytes, and 177 bytes, respectively. Assuming the MAL is 64 bytes, the transaction lengths for each of the four exemplary sequential tiles can be 128 bytes (112 bytes of compressed data plus 16 bytes of padding), 64 bytes (56 bytes of compressed data plus 8 bytes of padding), 64 bytes (33 bytes of compressed data plus 31 bytes of padding), and 192 bytes (177 bytes of compressed data plus 15 bytes of padding), respectively. It is worth noting that, in order to trade four exemplary sequential tiles, methods known in the art perform four transactions—one transaction per compressed tile.

[0053] Turning now to the remaining figures, embodiments of the solution are described. As will be apparent from the figures and related description below, an image frame can be subdivided into sectors, each defined by a group of one or more adjacent tiles. The gaze point location can be determined to be within a given sector. From there, advantageously, a lossless compression algorithm (or a near-lossless lossy compression algorithm with a relatively low compression factor) can be used to compress the sector containing the gaze point, while the remaining sectors in the frame are compressed according to increasingly lossy compression algorithms. Furthermore, in some embodiments, the blind spot location can be determined to be within a second given sector, which is therefore compressed using a very high-compression lossy compression algorithm, either discarded entirely or replaced with padding data, since the decompressed data associated with the blind spot has no impact on QoS. In this way, instead of applying a single compression algorithm to all tiles in the image frame, embodiments of the solution provide a hybrid compression method within the image frame.

[0054] Figure 4 This is a functional block diagram illustrating one embodiment of a system-on-chip 102 for intelligent compression (“IC”) using a concave compression method. (See from...) Figure 4 As can be understood from the illustration, monitor module 114 communicates with gaze sensor 159. Gaze sensor 159 can be configured to determine the user focus region within a given image frame generated by data / frame engine 201. The gaze location can be provided by monitor module 114 to intelligent compression (“IC”) module 101. Intelligent compression module 101 communicates with compression function database 109 and can generate a compressed map that divides the image frame into a series of sectors. Intelligent compression module 101, also communicating with compression function database 109, can also generate a compressed map that adjusts the size of concave regions in the data frame based on knowledge of a wait time factor associated with gaze sensor 159 and / or monitor module 114. It is conceivable that, depending on the wait time factor, the concave region can be defined by a single sector or multiple adjacent sectors.

[0055] The intelligent compression module 101 can also generate instructions for compressing tiles within one or more sectors as part of a compression map. The compression map can be provided from the intelligent compression module 101 to the image codec module 113B, which in turn compresses image frames received from the data / frame engine 201 based on the compression map. In this way, the image codec module 113B can compress tiles within each defined sector according to an optimal compression algorithm, such as using a lossless compression algorithm for tiles within a given sector corresponding to a gaze point determined from the gaze point sensor 159, while using a lossy compression algorithm for sectors not corresponding to the gaze point location.

[0056] Image codec module 113B can transmit compressed image frames, compressed according to the concave compression method described above and below, to memory 112 (which may contain DDR memory components) for storage via bus 211. Subsequently, before being rendered to the user via display 132, the compressed image frames can be returned to image codec module 113B for decompression based on the compression map initially generated by intelligent compression module 101. During decompression, image codec module 113B can communicate with intelligent compression module 101 or directly with compression function database 109 to identify the compression map used for compressing the image frames.

[0057] Advantageously, by using a concave compression method, embodiments of the solution can optimally compress image frames, such that a low-compression, lossless compression algorithm is used to compress only those sectors in the image frame that have the greatest impact on QoS, while a lossy compression algorithm is used to aggressively compress those sectors that have little or no impact on QoS. Moreover, in some embodiments, sectors identified as containing blind spots (whose locations can be determined by IC module 101 based on the known location of the gaze point) can be compressed according to the most aggressive lossy compression algorithm available for a particular embodiment of the solution. As will become more apparent from the following description, it is contemplated that as the distance between a sector and a given sector associated with the gaze point increases, embodiments of the solution can apply lossless compression algorithms with increasingly higher compression factors (and therefore, increasingly lower quality decompression outputs). The image codec module 113B can be divided into an encoder module for writing frames to memory and a decoder module for reading compressed frames from memory. These two modules can be co-located, or they can be physically separate and located in other blocks within the chip.

[0058] Figure 5 This illustrates a configuration for implementing a smart compression method. Figure 4A functional block diagram of an embodiment of the encoder portion of the image CODEC module 113B, wherein the intelligent compression method utilizes concave compression methods (including lossless and lossy algorithms). As relative to... Figure 4 As shown and described, producer engine 201 can provide its uncompressed frame input to image CODEC 113B, which in turn compresses the frames sector by sector according to the compression map generated by intelligent compression module 101. Depending on the sector, image codec module 113B can utilize lossless or lossy compression blocks. The determination of which compressor block can be used to compress tiles within a given sector and the compression factor of the algorithm applied to the compressor block depends on the instructions received from intelligent compression module 101. Intelligent compression module 101 can generate instructions based on compression maps or compression functions queried from compression function database 109.

[0059] Figure 6 Exemplary graph output 601 and exemplary lookup table 602 of the mathematical equations are shown, each specifying a compression ratio based on the distance between a given sector and the gaze point. The exemplary mathematical equations that generate graph output 601 and exemplary lookup table 602 can be stored in compression function database 109. (See also: From...) Figure 6 As illustrated, the intelligent compression method specifies that, depending on the position / distance of a given sector relative to the sector associated with the gaze point, a compression algorithm with an increased relative compression factor can be used on sectors within an image frame. In short, the farther a given sector is from the sector associated with the gaze point, the higher the compression factor of the compression algorithm used by the embodiment of the solution to compress that given sector.

[0060] Figure 7 Exemplary concave sectors defined within an image frame are shown, each concave sector containing one or more tiles of the image frame. (See from...) Figure 7 As the diagram illustrates, a concave sector "Lx" can be defined as encompassing any number of tiles arranged within the sector according to "n" columns and "m" rows. (See diagram from...) Figure 7 The illustration also suggests that a given concave sector “Lx” can contain tiles with any suitable aspect ratio, and therefore it is conceivable that the tiles can be square or rectangular in nature.

[0061] Figure 8 An exemplary image frame 800 is shown, illustrating segmentation and compression according to one embodiment of a solution utilizing an exemplary square grid concave compression pattern. (As shown from...) Figure 8As can be understood from the illustrations, the square grid concave compression pattern can also be described and classified as a side-corner adjacent concave compression pattern. It will be understood that although the exemplary image frame 800 is shown as being divided into a certain number of sectors, embodiments of the solution are not limited to any specific number or shape of sectors. Moreover, although in Figure 8 The sectors depicted in the diagram are square, but it is conceivable that sectors (e.g., tiles) can be defined by any aspect ratio without departing from the scope of the solution.

[0062] The compression map generated by the intelligent compression module 101 or queried from the compression function database 109 can define the size, number, pattern, and relative position of sectors. In this illustration, the sector pattern is a square grid concave compression pattern. Furthermore, the compression map can specify the specific compression algorithm (and extend to the compression factor) applied to each particular sector and how it is applied.

[0063] As in Figure 8 As can be seen in the illustration, the gaze point, indicated by a "star," is located within a given sector (hereinafter referred to as the gaze sector). The gaze point location may have been determined by the monitor module 114, which works in conjunction with the gaze point sensor 159. In turn, the given sector determined as the gaze sector is the sector in the image frame containing the gaze point.

[0064] As mentioned above, each sector "Lx" can encompass one or more entire tiles. Tiles within the gaze sector can be compressed using either a lossless compression algorithm or a very high-quality lossy compression algorithm with a relatively low compression factor. Figure 8 (marked as "L1"). Based on the depicted exemplary square grid concave compression pattern, each sector adjacent to the side or corner of the gaze sector is designated as the "L2" sector, and therefore each tile within the "L2" sector is compressed according to a compression algorithm whose compression factor is equal to or higher than the compression factor associated with the algorithm used to compress the gaze sector.

[0065] Similarly, each sector adjacent to the side or corner of sector "L2" is designated as sector "L3", and therefore, each tile within sector "L3" is compressed according to a compression algorithm with a compression factor equal to or higher than the compression factor associated with the algorithm used to compress sector "L2". Furthermore, each sector adjacent to the side or corner of sector "L3" is designated as sector "L4", and therefore, each tile within sector "L4" is compressed according to a compression algorithm with a compression factor equal to or higher than the compression factor associated with the algorithm used to compress sector "L3". The pattern continues accordingly, and as from... Figure 8As can be understood from the diagram, this enables the implementation of a concave compression method, which optimizes compression resources and minimizes the impact on QoS.

[0066] Figure 9 An exemplary image frame 900 is shown, segmented and compressed according to an embodiment of a solution utilizing an exemplary hierarchical square grid concave compression pattern. The exemplary image frame 900 has been segmented relative to the above... Figure 8 The illustration is consistent with the description. However, the compressed image of exemplary image frame 900 and... Figure 8 The difference between the diagram and the associated compression diagram is that it further includes instructions for classifying the compression patterns within a sector.

[0067] By hierarchically classifying the compression pattern within a sector, it is conceivable that embodiments of a solution utilizing a compression pattern with hierarchical instructions can generate a gradually finer-grained decrease in output quality based on the distance from the gaze sector. To this end, tiles within a given sector can be compressed according to different compression algorithms and / or algorithms associated with different compression factors. Typically, as the distance from the gaze sector increases, the compression factor of the tile will increase, as indicated by the arrows seen in the sectors of image frame 900. Although each tile in a given sector of image frame 800 can be compressed using a single compression algorithm with a single compression factor, different tiles within a given sector of image frame 900 can be compressed with different compression factors. It is noteworthy that although tiles within a given sector of image frame 900 can be compressed with different compression factors when the image frame undergoes a hierarchical square grid concave compression pattern, it is conceivable that the average compression factor of all tiles within a given sector can fall within the range of the overall compression category Lx identifier for the given sector.

[0068] For example, return to reference Figure 7 The diagram is shown in the figure. Figure 9 The sectors depicted by arrows from left to right in the diagram—the tiles in column #1 can all undergo a lossy compression algorithm with the least aggressive compression factor relative to the tiles in column #n, while the tiles in column #n can all undergo a lossy compression algorithm with the most aggressive compression factor relative to the algorithms applied to other tiles in the sector.

[0069] In view of Figure 7 Another example of the illustration, consider Figure 9The diagram illustrates a sector depicted by an arrow pointing upwards from the lower left corner to the upper right corner. The tile at position "1,m" can withstand a lossy compression algorithm with the least aggressive compression factor compared to the tile at position "n,0". The tile at position "n,0" can withstand a lossy compression algorithm with the most aggressive compression factor compared to the algorithms applied to other tiles in the sector. Furthermore, the tiles arranged diagonally from tile "1,0" to tile "n,m" can withstand a lossy compression algorithm with an average compression factor compared to the algorithms applied to other tiles in the sector.

[0070] Figure 10 An exemplary image frame 1000 is shown, segmented and compressed according to one embodiment of a solution utilizing an exemplary cross-grid concave compression pattern. It will be understood that although the exemplary image frame 1000 is shown as being segmented into a certain number of sectors, the embodiment of the solution is not limited to any specific number or shape of sectors. Moreover, although in Figure 10 The sectors depicted in the diagram are square, but it is conceivable that sectors (e.g., tiles) can be defined by any aspect ratio without departing from the scope of the solution.

[0071] The compression map generated by the intelligent compression module 101 or queried from the compression function database 109 can define the size, number, pattern, and relative position of sectors. In this illustration, the sector pattern is a cross-grid concave compression pattern. Furthermore, the compression map can specify the specific compression algorithm (and extend to the compression factor) applied to each particular sector and how it is applied.

[0072] As in Figure 10 As can be seen in the illustration, the gaze point, indicated by a "star," is located within a given sector (hereinafter referred to as the gaze sector). The gaze point location may have been determined by the monitor module 114, which works in conjunction with the gaze point sensor 159. In turn, the given sector determined as the gaze sector is the sector in the image frame containing the gaze point.

[0073] As mentioned above, each sector "Lx" can encompass one or more entire tiles. Tiles within the gaze sector can be compressed using either a lossless compression algorithm or a very high-quality lossy compression algorithm with a relatively low compression factor. Figure 10 (Labeled as "L1"). Based on the depicted exemplary cross-grid concave compression pattern, each sector adjacent to the side (but not the corner) of the gaze sector is designated as the "L2" sector, and therefore each tile within the "L2" sector is compressed according to a compression algorithm whose compression factor is equal to or higher than the compression factor associated with the algorithm used to compress the gaze sector.

[0074] Similarly, each sector adjacent to the side (but not the corner) of sector "L2" is designated as sector "L3", and therefore, each tile within sector "L3" is compressed according to a compression algorithm with a compression factor equal to or higher than the compression factor associated with the algorithm used to compress sector "L2". Furthermore, each sector adjacent to the side (but not the corner) of sector "L3" is designated as sector "L4", and therefore, each tile within sector "L4" is compressed according to a compression algorithm with a compression factor equal to or higher than the compression factor associated with the algorithm used to compress sector "L3". The pattern continues accordingly, and as from... Figure 10 As can be understood from the diagram, this enables the implementation of a concave compression method, which optimizes compression resources and minimizes the impact on QoS.

[0075] Figure 11 An exemplary image frame 1100 is shown, segmented and compressed according to an embodiment of a solution utilizing an exemplary hierarchical cross-grid concave compression pattern. The exemplary image frame 1100 has been segmented relative to the above... Figure 10 The illustration is consistent with the description. However, the compressed image of exemplary image frame 1100 and... Figure 10 The difference between the diagram and the associated compression diagram is that it further includes instructions for classifying the compression patterns within a sector.

[0076] By hierarchically classifying the compression pattern within a sector, it is conceivable that embodiments of a solution utilizing a compression pattern with hierarchical instructions can generate a gradually finer-grained decrease in output quality based on the distance from the gaze sector. To this end, tiles within a given sector can be compressed according to different compression algorithms and / or algorithms associated with different compression factors. Typically, as the distance from the gaze sector increases, the compression factor of the tile will increase, as indicated by the arrows seen in the sectors of image frame 1100. Although each tile in a given sector of image frame 1000 can be compressed using a single compression algorithm with a single compression factor, different tiles within a given sector of image frame 1100 can be compressed with different compression factors. It is noteworthy that although tiles within a given sector of image frame 1100 can be compressed with different compression factors when the image frame undergoes a hierarchical cross-grid concave compression pattern, it is conceivable that the average compression factor of all tiles within a given sector can fall within the range of the overall compression category Lx identifier for the given sector.

[0077] For example, return to reference Figure 7 The diagram is shown in the figure. Figure 11The sectors depicted by arrows from left to right in the diagram—the tiles in column #1 can all undergo a lossy compression algorithm with the least aggressive compression factor relative to the tiles in column #n, while the tiles in column #n can all undergo a lossy compression algorithm with the most aggressive compression factor relative to the algorithms applied to other tiles in the sector.

[0078] In view of Figure 7 Another example of the illustration, consider Figure 11 The diagram illustrates a sector depicted by an arrow pointing upwards from the lower left corner to the upper right corner. The tile at position "1,m" can withstand a lossy compression algorithm with the least aggressive compression factor compared to the tile at position "n,0". The tile at position "n,0" can withstand a lossy compression algorithm with the most aggressive compression factor compared to the algorithms applied to other tiles in the sector. Furthermore, the tiles arranged diagonally from tile "1,0" to tile "n,m" can withstand a lossy compression algorithm with an average compression factor compared to the algorithms applied to other tiles in the sector.

[0079] Figure 12 An exemplary image frame 1200 is shown, showing segmentation and compression according to one embodiment of a solution utilizing an exemplary cellular concave compression pattern. It will be understood that although the exemplary image frame 1200 is shown as being segmented into a certain number of sectors, embodiments of the solution are not limited to any specific number or shape of sectors. Furthermore, although in Figure 12 The sectors depicted in the diagram are square, but it is conceivable that sectors (e.g., tiles) can be defined by any aspect ratio without departing from the scope of the solution.

[0080] The compression map generated by the intelligent compression module 101 or queried from the compression function database 109 can define the size, number, pattern, and relative position of sectors. In this illustration, the sector pattern is a honeycomb concave compression pattern. Furthermore, the compression map can specify the specific compression algorithm (and extend to the compression factor) applied to each particular sector and how it is applied.

[0081] As in Figure 12 As can be seen in the illustration, the gaze point, indicated by a "star," is located within a given sector (hereinafter referred to as the gaze sector). The gaze point location may have been determined by the monitor module 114, which works in conjunction with the gaze point sensor 159. In turn, the given sector determined as the gaze sector is the sector in the image frame containing the gaze point.

[0082] As mentioned above, each sector "Lx" can encompass one or more entire tiles. Tiles within the gaze sector can be compressed using either a lossless compression algorithm or a very high-quality lossy compression algorithm with a relatively low compression factor.Figure 12 (Labeled as "L1"). According to the depicted exemplary honeycomb concave compression pattern, each horizontal sector line is offset relative to one or more of its adjacent sector lines, such that the corners of the sector do not correspond to the corners of adjacent sectors. Any sector adjacent to the gaze sector is designated as the "L2" sector, and therefore, each tile within the "L2" sector is compressed according to a compression algorithm whose compression factor is equal to or higher than the compression factor associated with the algorithm used to compress the gaze sector.

[0083] Similarly, each sector adjacent to the "L2" sector is designated as the "L3" sector, and therefore, each tile within the "L3" sector is compressed according to a compression algorithm with a compression factor equal to or higher than the compression factor associated with the algorithm used to compress the L2 sector. Furthermore, each sector adjacent to the "L3" sector is designated as the "L4" sector, and therefore, each tile within the "L4" sector is compressed according to a compression algorithm with a compression factor equal to or higher than the compression factor associated with the algorithm used to compress the L3 sector. The pattern continues accordingly, and as from... Figure 12 As can be understood from the diagram, this enables the implementation of a concave compression method, which optimizes compression resources and minimizes the impact on QoS.

[0084] Figure 13 An exemplary image frame 1300 is shown, segmented and compressed according to an embodiment of a solution utilizing an exemplary hierarchical cellular concave compression pattern. The exemplary image frame 1300 has been segmented relative to the above... Figure 12 The illustration is consistent with the description. However, the compressed image of exemplary image frame 1300 and... Figure 12 The difference between the diagram and the associated compression diagram is that it further includes instructions for classifying the compression patterns within a sector.

[0085] By hierarchically classifying the compression pattern within a sector, it is conceivable that embodiments of a solution utilizing a compression pattern with hierarchical instructions can generate a gradually finer-grained decrease in output quality based on the distance from the gaze sector. To this end, tiles within a given sector can be compressed according to different compression algorithms and / or algorithms associated with different compression factors. Typically, as the distance from the gaze sector increases, the compression factor of the tile will increase, as indicated by the arrows seen in the sectors of image frame 1300. Although each tile in a given sector of image frame 1200 can be compressed using a single compression algorithm with a single compression factor, different tiles within a given sector of image frame 1300 can be compressed with different compression factors. It is noteworthy that although tiles within a given sector of image frame 1300 can be compressed with different compression factors when the image frame undergoes a hierarchical cellular concave compression pattern, it is conceivable that the average compression factor of all tiles within a given sector can fall within the range of the overall compression category Lx identifier for the given sector.

[0086] For example, return to reference Figure 7 The diagram is shown in the figure. Figure 13 The sectors depicted by arrows from left to right in the diagram—the tiles in column #1 can all undergo a lossy compression algorithm with the least aggressive compression factor relative to the tiles in column #n, while the tiles in column #n can all undergo a lossy compression algorithm with the most aggressive compression factor relative to the algorithms applied to other tiles in the sector.

[0087] Figure 14 An exemplary image frame 1400 is shown, segmented and compressed according to an embodiment of a solution utilizing an exemplary hierarchical cellular concave compression pattern. The exemplary image frame 1400 has been segmented relative to the above... Figure 12 The illustration is consistent with the description. However, the compressed image of exemplary image frame 1400 and... Figure 12 The difference between the associated compressed diagram and the one illustrated is that it further includes instructions for grading the compressed pattern within a sector. Additionally, the compressed diagram of exemplary image frame 1400 and... Figure 13 The difference between the associated compression diagram and the diagram is that it contains different instructions for classifying the compression pattern within a sector. Provided Figure 14 The illustrations are intended to show different grading patterns, and therefore, embodiments of the solution are not limited to any particular pattern or method of compression grading within a sector.

[0088] By hierarchically classifying the compression pattern within a sector, it is conceivable that embodiments of a solution utilizing a compression pattern with hierarchical instructions can generate a gradually finer-grained decrease in output quality based on the distance from the gaze sector. To this end, tiles within a given sector can be compressed according to different compression algorithms and / or algorithms associated with different compression factors. Typically, as the distance from the gaze sector increases, the compression factor of the tile will increase, as indicated by the arrows seen in the sectors of image frame 1400. Although each tile in a given sector of image frame 1200 can be compressed using a single compression algorithm with a single compression factor, different tiles within a given sector of image frame 1400 can be compressed with different compression factors. It is noteworthy that although tiles within a given sector of image frame 1400 can be compressed with different compression factors when the image frame undergoes a hierarchical cellular concave compression pattern, it is conceivable that the average compression factor of all tiles within a given sector can fall within the range of the overall compression category Lx identifier for the given sector.

[0089] For example, return to reference Figure 7 The diagram is shown in the figure. Figure 14 The sectors depicted by arrows from left to right in the diagram—the tiles in column #1 can all undergo a lossy compression algorithm with the least aggressive compression factor relative to the tiles in column #n, while the tiles in column #n can all undergo a lossy compression algorithm with the most aggressive compression factor relative to the algorithms applied to other tiles in the sector.

[0090] In view of Figure 7 Another example of the illustration, consider Figure 14 The diagram illustrates a sector depicted by an arrow pointing upwards from the lower left corner to the upper right corner. The tile at position "1,m" can withstand a lossy compression algorithm with the least aggressive compression factor compared to the tile at position "n,0". The tile at position "n,0" can withstand a lossy compression algorithm with the most aggressive compression factor compared to the algorithms applied to other tiles in the sector. Furthermore, the tiles arranged diagonally from tile "1,0" to tile "n,m" can withstand a lossy compression algorithm with an average compression factor compared to the algorithms applied to other tiles in the sector.

[0091] Figure 15This is a logic flowchart illustrating a smart compression method 1500 according to a solution that utilizes concave compression methods (including lossless and lossy algorithms). Starting from box 1505, the user's focus is monitored. As understood by those skilled in the art of visual focus, the user's focus or gaze point on the image is perceived at the highest level of acuity. Moreover, and as can be understood from Figure 1 presented above and the definition of gaze point, acuity decreases with distance (or angle) from the user's concave area (i.e., the user's acuity decreases in the peripheral field of vision). Advantageously, by utilizing knowledge of the user's gaze point on the image, a lossy compression algorithm can be used to compress data associated with image sectors at a certain distance from the gaze point, because the relatively low output quality during decompression does not significantly affect QoS.

[0092] Returning to method 1500, in decision box 1510, it can be determined from monitoring the gaze point whether saccades have occurred, i.e., whether the user is rapidly translating from one gaze point to another. If saccades occur, a "yes" branch follows to box 1515, and the image can be compressed according to a predetermined compression ratio. However, if monitoring the gaze point determines that the user is focusing on a specific area of ​​the image, rather than switching between two gaze points within the image, a "no" branch follows to box 1520, and a concave compression pattern is generated based on the gaze point. It is worth noting that, as described above relative to... Figures 8-14 The concave compression pattern generated in frame 1520 may or may not be differentiated within the sector. Furthermore, it is conceivable that the concave compression pattern generated in frame 1520 may be, but is not limited to, a pattern relative to... Figures 8-14 Any of the exemplary concave compression patterns shown and described are consistent.

[0093] Method 1500 continues from box 1520 to box 1525. In box 1525, the image is compressed based on the generated concave compression pattern. As described in more detail above, the image frame can be segmented, and the gaze sector (i.e., the sector containing the gaze point) can be determined. The gaze sector is compressed using a lossless compression algorithm or, in some cases, a lossy compression algorithm associated with a relatively low compression factor. In this way, subsequent decompression of the image will produce the highest quality output at the image sector most relevant to user experience (and extended to QoS). As explained above, sectors outside the gaze sector can be compressed (and subsequently decompressed) according to a lossy compression algorithm associated with a relatively increasingly higher compression factor (based on distance from the gaze sector), i.e., the sector farthest from the gaze sector may undergo a lossy compression algorithm with a relatively highest compression factor. Method 1500 returns.

[0094] Figure 16This is a functional block diagram illustrating an exemplary, non-limiting aspect of a portable computing device (“PCD”) 100 in the form of a wireless telephone for implementing the intelligent compression method and system according to the solution described herein. As shown, the PCD 100 includes a system-on-a-chip 102 comprising a multi-core central processing unit (“CPU”) 110 and an analog signal processor 126 coupled together. As will be understood by those skilled in the art, the CPU 110 may include a zeroth core 222, a first core 224, and an Nth core 230. Furthermore, as will be understood by those skilled in the art, a digital signal processor (“DSP”) may also be used instead of the CPU 110.

[0095] Typically, the intelligent compression (“IC”) module 101 can be formed by hardware and / or firmware, and can be responsible for generating concave compressed images and enabling the image CODEC module 113 to compress the image in a concave manner using lossy and lossless compression algorithms. Figure 16 As shown, display controller 128 and touchscreen controller 130 are coupled to digital signal processor 110. A touchscreen display 132 external to system-on-chip 102 is coupled to display controller 128 and touchscreen controller 130. PCD 100 may further include video encoder 134, such as a phase alternating line (“PAL”) encoder, sequential and storage color television system (“SECAM”) encoder, National Television System Committee (“NTSC”) encoder, or any other type of video encoder 134. Video encoder 134 is coupled to multi-core CPU 110. Video amplifier 136 is coupled to video encoder 134 and touchscreen display 132. Video port 138 is coupled to video amplifier 136. Figure 16 As depicted, a Universal Serial Bus (“USB”) controller 140 is coupled to a CPU 110. Additionally, a USB port 142 is coupled to the USB controller 140.

[0096] Memory 112 (which may include PoP memory, cache, mask ROM / boot ROM, boot OTP memory, DDR type DRAM memory, etc.) can also be coupled to CPU 110. Subscriber identification module (“SIM”) card 146 can also be coupled to CPU 110. Furthermore, such as Figure 16 As shown, digital camera 148 can be coupled to CPU 110. In one exemplary aspect, digital camera 148 is a charge-coupled device (“CCD”) camera or a complementary metal-oxide-semiconductor (“CMOS”) camera.

[0097] like Figure 16The diagram further illustrates that the stereo audio CODEC 150 can be coupled to the analog signal processor 126. Additionally, the audio amplifier 152 can be coupled to the stereo audio CODEC 150. In one exemplary aspect, a first stereo speaker 154 and a second stereo speaker 156 are coupled to the audio amplifier 152. Figure 16 Microphone amplifier 158 is shown to also be coupled to stereo audio CODEC 150. Additionally, microphone 160 can be coupled to microphone amplifier 158. In one particular aspect, FM radio tuner 162 can be coupled to stereo audio CODEC 150. Furthermore, FM antenna 164 is coupled to FM radio tuner 162. Additionally, stereo headphones 166 can be coupled to stereo audio CODEC 150.

[0098] Figure 16 Further indications are provided that the radio frequency (“RF”) transceiver 168 can be coupled to the analog signal processor 126. An RF switch 170 can be coupled to the RF transceiver 168 and the RF antenna 172. (As indicated...) Figure 16 As shown, keyboard 174 can be coupled to analog signal processor 126. Additionally, mono headset 176 with a microphone can be coupled to analog signal processor 126. Furthermore, vibrator device 178 can be coupled to analog signal processor 126. Figure 16 A power source 188 (e.g., a battery) is also shown coupled to the on-chip system 102 via a power management integrated circuit (“PMIC”) 180. In one particular aspect, the power source 188 includes a rechargeable DC battery or a DC power source derived from an AC-to-DC transformer connected to an AC power source.

[0099] CPU 110 may also be coupled to one or more internal on-chip thermal sensors 157A and one or more external off-chip thermal sensors 157B. The on-chip thermal sensor 157A may include one or more absolute temperature proportional (“PTAT”) temperature sensors, which are based on a vertical PNP structure and are typically dedicated to complementary metal-oxide-semiconductor (“CMOS”) very large-scale integration (“VLSI”) circuits. The off-chip thermal sensor 157B may include one or more thermistors. The thermal sensor 157 may generate a voltage drop, which is converted into a digital signal by an analog-to-digital converter (“ADC”) controller (not shown). However, other types of thermal sensors 157 may be employed.

[0100] The touchscreen display 132, video port 138, USB port 142, camera 148, first stereo speaker 154, second stereo speaker 156, microphone 160, FM antenna 164, stereo headset 166, RF switch 170, RF antenna 172, keyboard 174, mono headset 176, vibrator 178, thermal sensor 157B, PMIC 180, and power supply 188 are external to the system-on-a-chip 102. However, it should be understood that... Figure 16 In exemplary embodiments of PCD 100, one or more of these devices depicted as being outside the system-on-chip 102 may reside on the chip 102 in other exemplary embodiments.

[0101] In one particular aspect, one or more method steps described herein may be implemented by executable instructions and parameters stored in memory 112 or in the form of IC module 101 and / or image CODEC module 113. Furthermore, IC module 101, image CODEC module 113, memory 112, the instructions stored therein, or combinations thereof, may be used as means for performing one or more method steps described herein.

[0102] Figure 17 This illustrates the method for performing intelligent compression. Figure 16 A schematic diagram of the exemplary software architecture of PCD is shown in Figure 1700. Figure 17 As shown, the CPU or digital signal processor 110 is coupled to the memory 112 via a main bus 211. As described above, the CPU 110 is a multi-core processor with N cores. That is, the CPU 110 includes a first core 222, a second core 224, and an Nth core 230. As is known to those skilled in the art, each of the first core 222, the second core 224, and the Nth core 230 can be used to support a dedicated application or program. Alternatively, one or more applications or programs can be distributed to process on two or more available cores.

[0103] CPU 110 may receive commands from one or more IC modules 101, which may include software and / or hardware. If implemented as software, the one or more modules 101 include instructions executed by CPU 110 that issue commands to other applications executed by CPU 110 and other processors.

[0104] The first core 222, the second core 224, up to the Nth core 230 of CPU 110 can be integrated on a single integrated circuit die, or they can be integrated or coupled on separate dies in a multi-circuit package. Designers can couple the first core 222, the second core 224, up to the Nth core 230 via one or more shared caches, and they can implement message or instruction passing via network topology (e.g., bus, ring, mesh, and cross topology).

[0105] Bus 211 may include multiple communication paths via one or more wired or wireless connections, as known in the art and described above in the definitions. Bus 211 may have additional elements (omitted for simplicity) for implementing communication, such as controllers, buffers (caches), drivers, repeaters, and receivers. Furthermore, bus 211 may include address, control, and / or data connections for implementing appropriate communication between the aforementioned components.

[0106] When the logic used by PCD 100 is implemented in software, such as Figure 17 As shown, it should be noted that one or more of the components of the startup logic 250, management logic 260, IC interface logic 270, and application and file system 290 in the application storage device 280 may be stored on any computer-readable medium for use by or in connection with any computer-related system or method.

[0107] In the context of this document, a computer-readable medium is an electronic, magnetic, optical, or other physical device / means that may contain or store computer programs and data for use in or in connection with a computer-related system or method. Various logical elements and data storage devices may be implemented in any computer-readable medium for use in or in connection with an instruction execution system, device, or apparatus (e.g., a computer-based system, a processor-containing system, or other system that can fetch and execute instructions from and from an instruction execution system, device, or apparatus). In the context of this document, "computer-readable medium" can be any means that can store, communicate, propagate, or transmit programs for use in or in connection with an instruction execution system, device, or apparatus.

[0108] Computer-readable media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, apparatuses, or propagation media. More specific examples of computer-readable media (a non-exhaustive list) will include the following: electrical connections having one or more wires (electronic), portable computer disks (magnetic), random access memory (RAM) (electronic), read-only memory (ROM) (electronic), erasable programmable read-only memory (EPROM, EEPROM, or flash memory) (electronic), optical fiber (optical), and portable compressed optical disc read-only memory (CDROM) (optical). Note that computer-readable media can even be paper or other suitable media on which programs are printed, because programs can be captured electronically (e.g., via optical scanning of paper or other media), then compiled, interpreted, or otherwise processed (if necessary), and then stored in computer memory.

[0109] In an alternative embodiment where one or more of the startup logic 250, management logic 260, and possible IC interface logic 270 are implemented in hardware, the respective logics may be implemented by any one or a combination of the following techniques, each of which is well known in the art: one or more discrete logic circuits having logic gates for implementing logic functions on data signals, application-specific integrated circuits (ASICs) having appropriate combinational logic gates, one or more programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0110] Memory 112 is a non-volatile data storage device, such as flash memory or solid-state memory. Although depicted as a single device, memory 112 may be a distributed storage device with separate data storage devices coupled to digital signal processor 110 (or another processor core).

[0111] The startup logic 250 includes one or more executable instructions for selectively identifying, loading, and executing a smart compression selection program. The startup logic 250 can identify, load, and execute the selected smart compression program. An exemplary selection program can be found in the program storage device 296 of the embedded file system 290. When executed by one or more core processors in the CPU 110, the exemplary selection program can operate according to one or more signals provided by the IC module 101 to implement the smart compression method.

[0112] Management logic 260 includes one or more executable instructions for terminating IC programs on one or more corresponding processor cores, and selectively identifying, loading, and executing more suitable replacement programs. Management logic 260 is configured to perform these functions during operation or when PCD 100 is powered on and used by an operator of the device. Replacement programs can be found in program storage device 296 of embedded file system 290.

[0113] Interface logic 270 includes one or more executable instructions for presenting, managing, and interacting with external input to observe, configure, or otherwise update information stored in embedded file system 290. In one embodiment, interface logic 270 may operate in conjunction with manufacturer input received via USB port 142. This input may contain one or more programs to be deleted from or added to program storage device 296. Alternatively, the input may contain edits or changes to one or more programs in program storage device 296. Furthermore, the input may identify one or more changes or replacements to one or both of startup logic 250 and management logic 260. As an example, the input may contain changes to the compression factor associated with a specific type of compression algorithm used for sector "Lx" and / or a preferred segmentation compression pattern.

[0114] Interface logic 270 enables the manufacturer to controllably configure and adjust the end-user experience under defined operating conditions on the PCD 100. When memory 112 is flash memory, one or more of the information in boot logic 250, management logic 260, interface logic 270, application storage device 280, or embedded file system 290 can be edited, replaced, or otherwise modified. In some embodiments, interface logic 270 may allow the end user or operator of the PCD 100 to search, locate, modify, or replace information in boot logic 250, management logic 260, application storage device 280, or embedded file system 290. The operator can use the resulting interface to make changes that will be implemented at the next boot of the PCD 100. Alternatively, the operator can use the resulting interface to make changes that will be implemented during runtime.

[0115] The embedded file system 290 includes a hierarchically arranged memory management storage device 292. In this respect, the file system 290 may include a reserved portion of its total file system capacity for storing information used to configure and manage various IC algorithms used by the PCD 100.

[0116] In the process or process flow described in this specification, certain steps naturally precede other steps so that the invention functions as described. However, the invention is not limited to the order of the described steps if such order does not alter the function of the invention. That is, it will be appreciated that some steps may be performed before, after, or in parallel (substantially) with other steps without departing from the scope and spirit of the invention. In some cases, certain steps may be omitted or not performed without departing from the invention. Furthermore, words such as "then," "following," and "next" are not intended to limit the order of steps. These words are used only to guide the reader in describing the exemplary method.

[0117] Furthermore, for example, based on the flowcharts and associated descriptions in this specification, those skilled in the art of programming can write computer code or identify suitable hardware and / or circuitry to implement the disclosed invention without difficulty. Therefore, the disclosure of specific program code instruction sets or detailed hardware arrangements is not considered necessary for a full understanding of how to implement and use the invention. The inventive function of the claimed computer implementation process has been explained in more detail in the above description and in conjunction with the accompanying drawings, which illustrate the various process flows.

[0118] In one or more exemplary aspects, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on or transmitted on a computer-readable medium. Computer-readable media includes computer storage media and communication media, wherein the communication media includes any media that facilitates the transfer of a computer program from one place to another. Storage media may be any available media that is accessible to a computer. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, or any other media that can be used to carry or store the required program code in the form of instructions or data structures and that is accessible to a computer.

[0119] Therefore, although the selected aspects have been shown and described in detail, it should be understood that various substitutions and modifications may be made therein without departing from the spirit and scope of the invention as defined in the following claims.

Claims

1. A method for intelligent data compression in a portable computing device (PCD), the method comprising: The gaze point within an image frame is determined by a gaze point sensor, wherein the gaze point sensor is configured to determine the user focus area within a given image frame; The image frame is divided into two or more sectors, wherein the two or more sectors include a gaze sector and one or more concave sectors, the gaze sector includes the gaze point, each concave sector does not include the gaze point, and each sector includes multiple tiles; The image frame is compressed such that the gaze sector is compressed according to a compression algorithm with a low compression factor, and the one or more concave sectors are compressed according to a compression algorithm with a high compression factor, wherein each corresponding patch within a particular concave sector undergoes a different compression factor based on the distance between the gaze sector and the corresponding patch, and each compression factor of the corresponding patch within the concave sector has an amount allocated corresponding to the distance between the gaze sector and the corresponding patch; and The compressed image frame is stored.

2. The method according to claim 1, wherein the compression algorithm with a low compression factor is a lossless compression algorithm.

3. The method according to claim 1, wherein the compression algorithm with a high compression factor is a lossy compression algorithm.

4. The method according to claim 1, further comprising: A compressed image is generated, wherein the size of the concave sector is adjusted.

5. The method of claim 4, wherein the size of the concave sector is based on knowledge of a waiting time factor associated with the gaze point sensor.

6. The method of claim 1, wherein dividing the image frame into two or more sectors comprises arranging the sectors in a square grid concave compression pattern.

7. The method of claim 1, wherein dividing the image frame into two or more sectors comprises arranging the sectors in a cross-grid concave compression pattern.

8. The method of claim 1, wherein dividing the image frame into two or more sectors comprises arranging the sectors in a honeycomb concave compression pattern.

9. A system for intelligent data compression in a portable computing device (PCD), the system comprising: The intelligent compression module, monitor module, gaze point sensor, and image CODEC module are collectively configured as follows: The gaze point sensor is used to determine the gaze point within an image frame, wherein the gaze point sensor determines the user focus area within a given image frame; The image frame is divided into two or more sectors, wherein the two or more sectors include a gaze sector and one or more concave sectors, the gaze sector includes the gaze point, each concave sector does not include the gaze point, and each sector includes multiple tiles; The image frame is compressed such that the gaze sector is compressed according to a compression algorithm with a low compression factor, and the one or more concave sectors are compressed according to a compression algorithm with a high compression factor, wherein each corresponding patch within a particular concave sector undergoes a different compression factor based on the distance between the gaze sector and the corresponding patch, and each compression factor of the corresponding patch within the concave sector has an amount allocated corresponding to the distance between the gaze sector and the corresponding patch; and The compressed image frame is stored.

10. The system of claim 9, wherein the compression algorithm having a low compression factor is a lossless compression algorithm.

11. The system of claim 9, wherein the compression algorithm having a high compression factor is a lossy compression algorithm.

12. The system according to claim 9, wherein the intelligent compression module generates a compression map, and the compression map adjusts the size of the concave sector.

13. The system of claim 12, wherein the size of the concave sector is based on knowledge of a waiting time factor associated with the gaze point sensor.

14. The system of claim 9, wherein dividing the image frame into two or more sectors comprises arranging the sectors in a square grid concave compression pattern.

15. The system of claim 9, wherein dividing the image frame into two or more sectors comprises arranging the sectors in a cross-grid concave compression pattern.

16. The system of claim 9, wherein dividing the image frame into two or more sectors comprises arranging the sectors in a honeycomb concave compression pattern.

17. A system for intelligent data compression in a portable computing device (PCD), the system comprising: A means for determining a gaze point within an image frame includes a gaze point sensor, wherein the gaze point sensor is configured to determine a user focus region within a given image frame. A means for dividing the image frame into two or more sectors, wherein the two or more sectors include a gaze sector and one or more concave sectors, the gaze sector includes the gaze point, each concave sector does not include the gaze point, and each sector includes multiple tiles. A means for compressing the image frame such that the gaze sector is compressed according to a compression algorithm with a low compression factor and the one or more concave sectors are compressed according to a compression algorithm with a high compression factor, wherein each corresponding patch within a particular concave sector undergoes a different compression factor based on the distance between the gaze sector and the corresponding patch, each compression factor of the corresponding patch within the concave sector having an amount allocated corresponding to the distance between the gaze sector and the corresponding patch; and A means for storing the compressed image frame.

18. The system of claim 17, wherein the compression algorithm having a low compression factor is a lossless compression algorithm.

19. The system of claim 17, wherein the compression algorithm having a high compression factor is a lossy compression algorithm.

20. The system of claim 17, further comprising: A device for generating a compressed map, wherein the compressed map adjusts the size of the concave sector.

21. The system of claim 20, wherein the size of the concave sector is based on knowledge of a waiting time factor associated with the gaze point sensor.

22. The system of claim 17, wherein the means for dividing the image frame into two or more sectors includes means for arranging the sectors in a square grid concave compression pattern.

23. The system of claim 17, wherein the means for dividing the image frame into two or more sectors includes means for arranging the sectors in a cross-grid concave compression pattern.

24. The system of claim 17, wherein the means for dividing the image frame into two or more sectors includes means for arranging the sectors in a honeycomb concave compression pattern.

25. A computer program product comprising a computer-readable program adapted to be executed to implement a method for intelligent data compression in a portable computing device (PCD), the method comprising: The gaze point within an image frame is determined by a gaze point sensor, wherein the gaze point sensor is configured to determine the user focus area within a given image frame; The image frame is divided into two or more sectors, wherein the two or more sectors include a gaze sector and one or more concave sectors, the gaze sector includes the gaze point, each concave sector does not include the gaze point, and each sector includes multiple tiles; The image frame is compressed such that the gaze sector is compressed according to a compression algorithm with a low compression factor, and the one or more concave sectors are compressed according to a compression algorithm with a high compression factor, wherein each corresponding patch within a particular concave sector undergoes a different compression factor based on the distance between the gaze sector and the corresponding patch, and each compression factor of the corresponding patch within the concave sector has an amount allocated corresponding to the distance between the gaze sector and the corresponding patch; and The compressed image frame is stored.

26. The computer program product of claim 25, wherein the compression algorithm having a low compression factor is a lossless compression algorithm.

27. The computer program product of claim 25, wherein the compression algorithm having a high compression factor is a lossy compression algorithm.

28. The computer program product of claim 25, the method further comprising generating a compressed image, the compressed image adjusting the size of the concave sector.

29. The computer program product of claim 28, wherein the size of the concave sector is based on knowledge of a waiting time factor associated with the gaze point sensor.

30. The computer program product of claim 25, wherein dividing the image frame into two or more sectors comprises arranging the sectors in a square grid concave compression pattern.

31. The computer program product of claim 25, wherein dividing the image frame into two or more sectors comprises arranging the sectors in a cross-grid concave compression pattern.

32. The computer program product of claim 25, wherein dividing the image frame into two or more sectors comprises arranging the sectors in a honeycomb concave compression pattern.

Images