Image processing apparatus, method and related device

Abstract

The present disclosure provides an image processing device, method and related equipment, the device comprises: a layer processing submodule corresponding to layer data in a to-be-processed image, and a lookup table storage block corresponding to the layer processing submodule; the lookup table storage block is configured to: in a case where the layer data processed by the layer processing submodule is in a first dynamic range format, store mapping data of a first type; in a case where the layer data processed by the layer processing submodule is switched to a second dynamic range format, store mapping data of a second type; wherein the mapping data of the first type and the mapping data of the second type multiplex the same physical storage space in the lookup table storage block; the layer processing submodule is used for dynamic range conversion processing of the layer data. Through the cross-format multiplexing mechanism, the problem of excessive storage overhead caused by reserving storage resources for each format can be avoided.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an image processing apparatus, method and related equipment. Background Technology

[0002] In the field of image processing, with the continuous development of display technology, image data exhibits a trend of coexistence of multiple dynamic range formats. For example, High Dynamic Range (HDR) images and Standard Dynamic Range (SDR) images, or different types of HDR formats such as HDR10 and HLG (Hybrid Log-Gamma), are widely used in various display devices. In order to correctly present image content on display devices with different dynamic ranges, it is usually necessary to perform dynamic range conversion processing on the image data.

[0003] In related technologies, especially in multi-layer image compositing scenarios, it is typically necessary to configure a separate Look-Up Table (LUT) storage space for each dynamic range format involved in the multi-layer process. This LUT stores conversion parameters such as tone mapping curves and color gain coefficients for the corresponding format. When the dynamic range format of the image source changes, it is necessary to switch to the corresponding Look-Up Table storage space to read the data. Due to the storage issues involved in multiple Look-Up Tables corresponding to various dynamic range formats, this results in significant storage resource consumption. Summary of the Invention

[0004] This disclosure provides an image processing apparatus, method, and related equipment.

[0005] In a first aspect, this disclosure provides an image processing apparatus, comprising: a layer processing submodule corresponding to layer data in an image to be processed, and a lookup table storage block corresponding to the layer processing submodule;

[0006] The lookup table storage block is configured to: store first type of mapping data when the layer data processed by the layer processing submodule is in a first dynamic range format; and store second type of mapping data when the layer data processed by the layer processing submodule is switched to a second dynamic range format; wherein the first type of mapping data and the second type of mapping data reuse the same physical storage space in the lookup table storage block.

[0007] The layer processing submodule is configured to perform dynamic range conversion processing on the layer data based on the mapping data stored in the lookup table storage block.

[0008] Secondly, this disclosure provides an image processing method based on the image processing apparatus described above. The method includes: performing dynamic range conversion processing on layer data according to the mapping data stored in the lookup table storage block through the layer processing submodule.

[0009] Thirdly, this disclosure provides an electronic device, including the image processing apparatus described above.

[0010] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method.

[0011] Fifthly, this disclosure provides a computer program product comprising computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the method described above.

[0012] In the embodiments provided in this disclosure, the lookup table storage block is configured to store either a first type of mapping data or a second type of mapping data, thereby enabling the two different types of mapping data to reuse the same physical storage space. This facilitates the dynamic loading of the corresponding mapping data based on the dynamic range format currently being processed by the layer. Through this cross-format reuse mechanism, the problem of excessive storage overhead caused by reserving storage resources for each format can be avoided, significantly improving the utilization rate of storage units and greatly saving storage space.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0015] Figure 1 This is a schematic diagram of an optional structure of the image processing apparatus provided in an embodiment of the present disclosure;

[0016] Figure 2 A schematic diagram of the hardware circuitry for storing a lookup table in the image dynamic range conversion processing of one implementation is shown.

[0017] Figure 3 A schematic diagram of the electro-optic conversion function module is shown.

[0018] Figure 4 A schematic diagram of the storage pool module in one implementation is shown;

[0019] Figure 5 A schematic diagram of an improved storage pool module provided in one example of this disclosure is shown. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] Unless otherwise specified, the various embodiments and features of this disclosure may be combined with each other. As used herein, the term "and / or" includes any and all combinations of one or more of the associated enumerated entries.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0023] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0024] In image processing, when the dynamic range format of layer data changes, conventional implementations require configuring independent lookup table storage space for each format, resulting in significant storage resource consumption and increased chip area and hardware cost. To effectively reduce storage resource consumption while supporting multiple dynamic range format switching, this disclosure provides an image processing apparatus. The image processing apparatus includes: a layer processing submodule corresponding to the layer data in the image to be processed, and a lookup table storage block corresponding to the layer processing submodule. The lookup table storage block is configured to: store first-type mapping data when the layer data processed by the layer processing submodule is in a first dynamic range format; and store second-type mapping data when the layer data processed by the layer processing submodule is switched to a second dynamic range format. The first-type mapping data and the second-type mapping data reuse the same physical storage space in the lookup table storage block. The layer processing submodule is configured to: perform dynamic range conversion processing on the layer data based on the mapping data stored in the lookup table storage block.

[0025] Figure 1 This is a schematic diagram of an optional structure of the image processing apparatus provided in an embodiment of the present disclosure, such as... Figure 1 As shown, the device includes a processing module 11 and a storage module 12. The processing module 11 includes a layer processing submodule 111 corresponding to the layer data in the image to be processed; the storage module 12 includes a lookup table storage block 121 corresponding to the layer processing submodule 111. The number of layer processing submodules 111 can be one or more, and similarly, the number of lookup table storage blocks 121 can also be one or more. This application does not limit the specific details.

[0026] The lookup table storage block 121 is configured to: store first-type mapping data when the layer data processed by the layer processing submodule 111 is in a first dynamic range format; and store second-type mapping data when the layer data processed by the layer processing submodule 111 is switched to a second dynamic range format. The first-type and second-type mapping data reuse the same physical storage space within the lookup table storage block. The layer processing submodule 111 is configured to: perform dynamic range conversion processing on the layer data based on the mapping data stored in the lookup table storage block. When the layer processing submodule needs to read or write to the lookup table storage block, it can send read / write instructions to the lookup table storage block, including format information in the instructions to indicate the specific format of the layer data processed by the layer processing submodule 111.

[0027] The processing module can be a core unit in the image processing device used to perform image data processing. The storage module can refer to a memory unit in the image processing device used to store data, such as static random access memory, dynamic random access memory, register files, or other forms of volatile or non-volatile storage media. The storage module can be integrated inside the processing module or externally as a separate chip; this invention does not limit this.

[0028] In image processing architectures, particularly in scenarios such as graphical user interface rendering, video compositing, and game screen generation, the image to be processed can be composed of multiple independent layers, each corresponding to a set of layer data. For example, in a video playback scenario, this could include video content layers, subtitle layers, menu layers, cursor layers, etc.; in a game scenario, it could include background layers, character layers, special effects layers, user interface layers, etc. Each layer of data can be processed independently, and finally, the layers are combined by a compositing module to form a complete output image.

[0029] A layer processing submodule refers to a logical unit within a processing module used to process data from a single layer. When a processing module needs to process multiple layers, multiple layer processing submodules can be configured to work in parallel, with each submodule processing one layer of data. Layer processing submodules can be hardware acceleration units within the processing module or software-implemented processing threads; this invention does not limit the specific implementation.

[0030] A lookup table storage block refers to a storage area allocated within a storage module for the layer processing submodule, used to store the mapping data required by the corresponding layer during dynamic range conversion. Dynamic range format is used to characterize the brightness range representation format used for image data. For example, dynamic range formats can include SDR, HDR, etc. The HDR format can be further divided into HDR10, HLG (Hybrid Log-Gamma), and other formats.

[0031] Accordingly, the first dynamic range format and the second dynamic range format are used to represent two different dynamic range formats. For example, the first dynamic range format can be HDR10, and the second dynamic range format can be HLG; or, the first dynamic range format can be HDR format, and the second dynamic range format can be SDR format. Those skilled in the art can flexibly define the specific types of the first and second dynamic range formats according to the actual application scenario, and the present invention does not limit this.

[0032] Mapping data refers to the parameter data required to convert image data from one dynamic range format to another. Mapping data is typically stored in the form of lookup tables for fast pixel-level conversion. Depending on the dynamic range format, mapping data can include various types such as tone mapping curves, color gain coefficients, luminance mapping tables, and chrominance mapping tables.

[0033] The first and second types of mapping data refer to the mapping data corresponding to the first dynamic range format and the second dynamic range format, respectively. These two types of mapping data typically differ in data structure, numerical range, and update frequency. For example, the first type of mapping data can be a static tone mapping table for the first dynamic range format, while the second type of mapping data can be a dynamically metadata-driven tone mapping table for the second dynamic range format. Although the two types of mapping data are different, they can reuse the same physical storage space in the lookup table storage block.

[0034] Physical storage space refers to the actual hardware storage units within the storage module. By reusing the same physical storage space, different types of mapping data can occupy the same storage area in a time-sharing manner, without each needing to occupy a separate storage area. When the dynamic range format of the layer data changes, the new mapping data will overwrite the old mapping data, thus achieving dynamic switching under limited storage resources.

[0035] In specific implementation, the layer processing submodule can be configured to: perform dynamic range conversion processing on the layer data according to the mapping data stored in the lookup table storage block. Among them, dynamic range conversion processing refers to: the image processing device processes the input layer data according to the stored mapping data so that the processed layer data is adapted to the target dynamic range format. The specific processing methods include, but are not limited to: conversion between different dynamic range format standards, and real-time adjustment and optimization of mapping parameters frame by frame or scene by scene according to metadata under the same format standard. It can be seen that in this embodiment, dynamic range conversion processing includes: conversion processing between different dynamic range formats, and mapping data update processing between different image frames under the same dynamic range format. Specifically, it can include the following two situations: (1) Cross-format conversion: For example, when switching from the first dynamic range format to the second dynamic range format, it is necessary to update the entire set of mapping data, so as to perform dynamic range conversion processing based on the updated entire set of mapping data. (2) Dynamic processing within the same format: For example, when processing the first dynamic range format, since the metadata changes frame by frame, the mapping data required for each frame may be different, and it is necessary to continuously update the mapping data, so as to perform dynamic range conversion processing based on the updated mapping data.

[0036] In this disclosure, the layer processing submodule serves as the core of dynamic range conversion, capable of initiating data access requests and performing pixel-level calculations. The lookup table storage block, on the other hand, acts as a dedicated storage medium, handling the physical storage, updating, and output of the mapped data. Optionally, the two can work collaboratively through a one-to-one binding relationship: the layer processing submodule does not need to directly participate in the storage management of the mapped data, but instead drives the lookup table storage block to schedule its internal storage space by sending format switching commands or read requests. This approach allows the computational logic and storage management to be independent of each other, facilitating the reuse of mapped data in different dynamic range formats within the same physical storage space and providing a hardware foundation for multi-layer parallel processing. For example, when the lookup table storage block receives an indication signal from the layer processing submodule that the current layer data is in the first dynamic range format, it responds to the signal by writing the first type of mapping data into the internal multiplexed physical storage space for the layer processing submodule to read and execute the conversion processing of the current frame. When the layer processing submodule detects a change in the bitstream and determines that the processed layer data has switched to the second dynamic range format, it sends a format switching trigger instruction to the lookup table storage block. In response to the instruction, the lookup table storage block writes the second type of mapping data into the same physical storage space without changing the physical storage space address mapping, and completes the seamless switching of the stored content by updating the status flag bit, thereby supporting the layer processing submodule to directly call the new format mapping data for dynamic range conversion in the next frame.

[0037] Therefore, in this disclosure, the lookup table storage block is configured to store either type 1 or type 2 mapping data, allowing the two different types of mapping data to reuse the same physical storage space. This facilitates the dynamic loading of the corresponding mapping data based on the current dynamic range format being processed by the layer. Through this cross-format reuse mechanism, the problem of excessive storage overhead caused by reserving storage resources for each format can be avoided, significantly improving the utilization rate of storage units and greatly saving storage space.

[0038] exist Figure 1 In the illustrated embodiment, the structure of the image processing device is exemplarily described by taking the integration of the layer processing submodule 111 within the processing module 11 and the integration of the lookup table storage block 121 within the storage module 12 as an example. This architecture enables appropriate isolation between layer processing functions and storage management functions at the physical or logical level, facilitating modular design and centralized management.

[0039] However, those skilled in the art will understand that this disclosure does not limit the specific hierarchical relationship or physical affiliation between the layer processing submodule and the lookup table storage block in the overall layout of the image processing apparatus. In other alternative embodiments, the image processing apparatus can adopt various alternative architectural layouts. In one approach, the lookup table storage block does not need to be configured inside the storage module; it can be attached externally to the storage module as an independent functional unit, or even set up independently of the processing module and the storage module. The lookup table storage block can still maintain a correspondence with the layer processing submodule through a data bus or dedicated interface, and provide mapping data in response to the read requests of the layer processing submodule. Simultaneously, when the layer data processed by the layer processing submodule undergoes dynamic range format switching, it reuses and stores different types of mapping data within its own physical storage space. This approach can reduce the occupation of internal storage space in the storage module and is suitable for design scenarios where storage module resources are scarce. In yet another approach, the lookup table storage block can be directly integrated into the processing module, for example, as a dedicated cache or local storage resource within the processing module. Each layer processing submodule can be configured with its own dedicated lookup table storage block within the processing module, or multiple layer processing submodules can share the same lookup table storage block within the processing module in a time-sharing manner. This approach can shorten the reading path of mapped data, reduce access latency, and is suitable for scenarios with high real-time requirements for dynamic range conversion.

[0040] Alternatively, the image processing device may not include the two top-level functional units of an integrated processing module and a storage module, but directly include at least one layer processing submodule and at least one lookup table storage block corresponding to that layer processing submodule. Each layer processing submodule and each lookup table storage block establishes a corresponding connection through direct mapping or routing configuration. In this architectural layout, the lookup table storage blocks can be deployed in the image processing device in various ways, such as distributed, centralized, or hybrid. It should be noted that in the image processing device, an integrated processing module and a storage module are set up, and the layer processing submodule is configured within the processing module, while the lookup table storage blocks are configured within the storage module (e.g., ...). Figure 1 As shown in the figure, this facilitates the implementation of logical function division and resource sharing management at the system level, and is especially suitable for application scenarios where there is a regular one-to-one or one-to-many mapping relationship between the layer processing submodule and the lookup table storage block. Of course, those skilled in the art can flexibly choose any one of the above implementation methods, or combine them, according to actual needs such as chip back-end design, hardware resource distribution, power consumption and area constraints. In addition, those skilled in the art can also make various modifications and variations to the above embodiments:

[0041] In one optional implementation, to better adapt to multi-layer scenarios, the number of layer processing sub-modules can be configured to be multiple, each used to process multiple layers of data in the image to be processed. Furthermore, the number of lookup table storage blocks can be configured to be multiple, with each lookup table storage block corresponding to one of the multiple layer processing sub-modules. Each lookup table storage block stores the mapping data corresponding to its respective layer processing sub-module. By allocating an independent layer processing sub-module and a corresponding lookup table storage block to each layer, different types of mapping data can reuse the same physical storage space within a single layer processing channel. This ensures both the ability to process multiple layers in parallel and maximizes storage efficiency within each layer. Therefore, by configuring an independent layer processing sub-module for each layer, multiple layers can perform dynamic range conversion processing in parallel, significantly improving image processing throughput and reducing display latency.

[0042] In one alternative implementation, considering that the dynamic range conversion process requires the collaborative operation of multiple types of mapping data, such as luminance mapping and color mapping (which require different lookup tables), a consideration is given. Furthermore, different types of mapping data may differ in data characteristics, update frequency, and access patterns. For example, luminance mapping tables may be updated frequently with scene changes, while color mapping tables are relatively stable. Mixing all types of mapping data in the same storage area would lead to management complexity and prevent optimization for the characteristics of different types of mapping data. To further optimize the organization of mapping data while achieving storage reuse, in the case where the first type of mapping data includes a first lookup table and a second lookup table of the first type, and the second type of mapping data includes a first lookup table and a second lookup table of the second type, each lookup table storage block may include: a first lookup table unit and a second lookup table unit. Specifically, when the layer data processed by the layer processing submodule is in a first dynamic range format, the first lookup table unit stores a first lookup table of the first type, and the second lookup table unit stores a second lookup table of the first type. When the layer data processed by the layer processing submodule is in a second dynamic range format, the first lookup table unit stores a first lookup table of the second type, and the second lookup table unit stores a second lookup table of the second type. The first lookup table of the first type and the first lookup table of the second type share the same physical storage space in the first lookup table unit, and the second lookup table of the first type and the second lookup table of the second type share the same physical storage space in the second lookup table unit.

[0043] By dividing the lookup table storage block into a first lookup table unit and a second lookup table unit, and establishing a correspondence between the mapping data type and the lookup table unit, different types of lookup tables with corresponding functions (such as brightness-related tables and color-related tables) can be stored in different lookup table units, enabling the layer processing submodule to access multiple types of mapping data in parallel. At the same time, within each lookup table unit, a storage reuse mechanism between different dynamic range formats is still retained, thus supporting the collaborative processing of multiple types of mapping data while maintaining storage efficiency.

[0044] The first type of lookup table refers to a first type of mapping data corresponding to a first dynamic range format, typically used for performing the first type of operation in dynamic range conversion. For example, it could be a tone mapping table for luminance mapping. The second type of lookup table refers to a second type of mapping data corresponding to a first dynamic range format, typically used for performing the second type of operation in dynamic range conversion. For example, it could be a color gain table for color mapping. The second type of lookup table refers to a first type of mapping data corresponding to a second dynamic range format, functionally corresponding to the first type of lookup table, but for a different dynamic range format. For example, it could be a luminance mapping table for a second dynamic range format. The second type of lookup table refers to a second type of mapping data corresponding to a second dynamic range format, functionally corresponding to the second type of lookup table, but for a different dynamic range format.

[0045] Accordingly, the first lookup table unit refers to the first storage area allocated within the lookup table storage block, used to store lookup tables related to the first type of operation (such as luminance mapping). The first lookup table unit has its own independent address space and access interface. The second lookup table unit refers to the second storage area allocated within the lookup table storage block, used to store lookup tables related to the second type of operation (such as color mapping). The second lookup table unit is independent of the first lookup table unit and can be accessed in parallel.

[0046] By storing different types of mapping data in different lookup table units, the layer processing submodule can read data from multiple units simultaneously, enabling parallel processing of luminance and color mapping and improving the efficiency of dynamic range conversion. Furthermore, each lookup table unit independently reuses storage between formats, making the storage management logic clearer. The first lookup table unit only needs to focus on updating the luminance-related table, and the second lookup table unit only needs to focus on updating the color-related table, reducing management complexity. Moreover, it facilitates the adoption of targeted optimization strategies based on the characteristics of the data stored in different lookup table units. For example, for frequently updated luminance mapping tables, the write path can be optimized; for relatively stable color mapping tables, the read path can be optimized.

[0047] Furthermore, in practical applications, different dynamic range formats typically possess different technical characteristics. For example, for dynamic metadata-driven formats, their mapping data needs to be dynamically updated as the scene or frame changes, including scene-by-scene tone mapping parameters and color enhancement parameters; while for static parametric formats, their mapping data is relatively fixed, mainly consisting of static luminance gain and chrominance gain coefficients. Accordingly, when the first dynamic range format is a dynamic metadata-driven format, the first lookup table of the first type can be a dynamic tone mapping table, and the second lookup table of the first type can be a dynamic color gain table; when the second dynamic range format is a static parametric format, the first lookup table of the second type can be a static luminance gain table, and the second lookup table of the second type can be a static chrominance gain table. Dynamic metadata-driven formats refer to a type of high dynamic range format characterized by containing dynamically changing metadata in the video stream. This metadata can provide optimization parameters scene-by-scene, frame-by-frame, and even pixel-by-pixel, enabling the display device to make precise adjustments based on content characteristics. Static parametric formats refer to a type of high dynamic range format characterized by using fixed parameter curves for photoelectric conversion and not containing dynamically changing metadata. A dynamic tone map table refers to tone mapping data used in a dynamically metadata-driven format, whose content is dynamically updated as the scene or frame changes. Dynamic tone maps are typically generated by parsing dynamic metadata, accurately mapping the brightness range of the source content to the brightness range of the target display device while preserving the scene's visual intent. A dynamic color gain table refers to color mapping data used in a dynamically metadata-driven format, whose content is dynamically updated as the scene or frame changes. Dynamic color gain tables typically include gamut mapping coefficients, saturation adjustment parameters, etc., used for accurate conversion between different color spaces and can enhance colors based on content characteristics. A static luminance gain table refers to luminance mapping data used in a statically parameterized format, whose content is relatively fixed and does not change with the scene. Static luminance gain tables are typically generated based on standard electro-optical transfer functions (EOTF) or opto-electrical transfer functions (OETF), such as lookup tables corresponding to perceptual quantizer (PQ) curves or HLG curves. A static chrominance gain table refers to color mapping data used in a statically parameterized format, whose content is relatively fixed.

[0048] By configuring corresponding mapping data according to the specific type of the dynamic range format, dynamic range conversion processing can fully utilize the characteristics of each format. For dynamic metadata-driven formats, fine-grained adjustments can be achieved scene-by-scene or frame-by-frame; for static parametric formats, standardized and efficient conversion can be achieved. The update frequency of the dynamic tone map table and dynamic color gain table can be matched with the format characteristics, thereby avoiding unnecessary update overhead. For static formats, there is no need to calculate mapping data in real time; for dynamic formats, updates can be performed only when necessary, improving overall efficiency.

[0049] In certain situations, multiple display modes or enhanced versions may exist within the same dynamic range format. For example, the HDR_VIVID_HLG mode based on the HLG format requires an additional tone mapping table on top of the basic HLG conversion to achieve vivid image quality; the HDR_VIVID_PQ mode based on the PQ format also requires additional mapping data to achieve specific image quality enhancements. If only one set of mapping data is configured for each dynamic range format, it cannot meet the needs of different enhanced modes within the same format. If independent storage space is configured for each enhanced mode, it will break the original storage reuse mechanism and increase storage overhead. Therefore, in order to further support the mapping needs of multiple enhanced modes within the same dynamic range format while maintaining the advantages of storage reuse, the above lookup table storage block can also be configured to: when the layer data processed by the layer processing submodule is an enhanced first dynamic range format, further store a third type of mapping data; wherein, the third type of mapping data includes: a third lookup table; and each lookup table storage block also includes: a third lookup table unit for storing the third lookup table. By adding a third lookup table unit, multiple enhanced modes within the dynamic range format can be extended to support it without changing the core storage reuse mechanism, demonstrating excellent functional scalability. Furthermore, the targeted adjustment parameters provided by the third lookup table unit allow for further fine-tuning of the image after dynamic range conversion, meeting the image quality requirements of different content types (movies, sports, animation) and different viewing environments (bright, dark). Optionally, the third lookup table unit can store the corresponding mapping data only when needed (i.e., when processing enhanced formats). For scenarios that do not support enhanced modes (i.e., the basic format), this unit can be disabled, thus maintaining efficient storage resource utilization. Compared to the traditional approach of configuring complete storage space independently for each enhanced mode, this embodiment significantly saves storage resources. The enhanced first dynamic range format refers to a format that overlays a specific image quality enhancement algorithm or display mode on top of the basic dynamic range format. Enhanced formats are extensions or optimizations of the basic format. For example, it may include the following formats: (1) HDR_VIVID_HLG: an enhanced mode based on the hybrid log-gamma format, which achieves vivid image quality by overlaying a dynamic tone map table and a dynamic color gain table, and additionally using a third lookup table; (2) HDR_VIVID_PQ: an enhanced mode based on the perceptual quantization format, which achieves image quality enhancement by overlaying a dynamic tone map table and a dynamic color gain table. Correspondingly, the third type of mapping data refers to the additional mapping data corresponding to the enhanced format, which is used to further adjust the image quality or enhance the effect on top of the basic dynamic range conversion. The third type of mapping data may be a brightness adjustment table, a color enhancement table, a contrast adjustment table, a tone map table for a specific scene, etc.The third lookup table corresponds to the third type of mapping data and can be a one-dimensional or multi-dimensional lookup table. The third lookup table unit refers to a newly added third storage area in the lookup table storage block, used to store the third type of mapping data. The third lookup table unit is independent of the first and second lookup table units and can be accessed in parallel.

[0050] Furthermore, when processing consecutive image frames, the mapping data itself may be updated frame by frame. For example, for dynamic metadata-driven formats, the tone mapping table for each frame may be different; even for static formats, the mapping data may need to be updated due to user adjustments or environmental changes. If, during the updating of mapping data, the layer processing submodule is still reading old mapping data to process the current frame, read-write conflicts will occur, leading to processing errors or abnormal display. If the mapping data is updated only after the current frame is processed, the processing of the next frame will be delayed, affecting the smoothness of the display. Therefore, in order to update the mapping data safely and efficiently while ensuring the continuity of image processing and avoiding read-write conflicts, each lookup table unit in each lookup table storage block of this disclosure may further include: a first unit partition and a second unit partition operating in a ping-pong manner; wherein, the first unit partition and the second unit partition are used to alternately store the mapping data corresponding to the current frame and subsequent frames during frame switching. The ping-pong method is a double-buffer working mechanism, which achieves conflict-free data reading and writing through the alternating use of two buffers (partitions). While one buffer is used for reading (current frame processing), another buffer is used for writing (next frame data update). At frame switching, the roles of the two buffers are reversed: the original read buffer becomes the write buffer, and the original write buffer becomes the read buffer. This cycle repeats, forming a ping-pong alternating working pattern. The first unit partition can be the first storage partition within each lookup table unit, used to store a portion of the mapping data. The second unit partition can be the second storage partition within each lookup table unit, used to store the other portion of the mapping data. The first and second unit partitions are usually the same size, have independent address spaces, and can be accessed in parallel. Frame switching refers to the moment when image processing transitions from the current frame to the next frame. During frame switching, the layer processing submodule begins processing the new image frame, and the source of the mapping data also needs to switch to the data corresponding to the new frame. The mapping data corresponding to the current frame refers to the mapping data used to process the current image frame, stored in the partition currently in a readable state. The mapping data corresponding to subsequent frames refers to the mapping data used to process subsequent image frames (usually the next frame), stored in the partition currently in a writable state, waiting to switch to a readable state when the subsequent frame is processed. By employing a ping-pong partitioning mechanism, read and write operations always target different physical partitions, avoiding read-write conflicts and ensuring the correctness and stability of image processing. Furthermore, updates to the mapped data can be performed simultaneously with the processing of the current frame, without waiting for the current frame to complete, achieving non-blocking pipelined operations, ensuring the continuity of image processing, and preventing screen stuttering or delays. The ping-pong working mode is not only applicable to alternating between two partitions but can also be extended to the cyclical use of three or more partitions. Those skilled in the art can choose an appropriate number of partitions and working mode according to actual needs; this invention does not limit this selection.

[0051] To facilitate accurate control of the read / write status of the first and second unit partitions operating in a ping-pong manner, each lookup table storage block can be further configured with status flag bits. These status flag bits indicate the read / write status of the first and second unit partitions of each lookup table cell within the lookup table storage block. This allows the processing module to read the mapping data corresponding to the current frame from the unit partition in the readable state and write the mapping data corresponding to subsequent frames to the unit partition in the writable state. The status flag bits can be one or more binary bits representing the current operating state of the storage partition. The status flag bits can be stored in registers or are part of the internal control logic of the lookup table storage block. Each status flag bit typically corresponds to a specific storage partition, indicating whether the storage partition is currently in a readable or writable state. A readable state means that the storage partition currently stores valid data, which can be read by the layer processing submodule for dynamic range conversion processing. A writable state means that the storage partition can currently be written with new mapping data, and the write operation will not affect the currently ongoing read operation. The read / write state is the specific value represented by the status flag bits; "0" represents the readable state, and "1" represents the writable state. By clearly indicating the current status of each partition through status flags, the processing module can accurately perform read and write operations, avoid data errors caused by accessing the wrong partition, and improve the reliability of the system.

[0052] In one optional implementation, the status flag bit corresponding to each lookup table storage block is toggled upon detecting a mapping data update event corresponding to the layer processing submodule, thereby changing the read / write status of the first and second unit partitions. The mapping data update event can include: a frame switching event for switching image frames within the same dynamic range format, and a dynamic range format switching event for changing the dynamic range format. This method allows for dynamic adjustment of the ping-pong partition's role based on actual frame switching or format switching events, synchronizing mapping data updates with the image processing process. This ensures data consistency and avoids unnecessary switching overhead. A mapping data update event is an event that triggers the need to update mapping data. When such an event occurs, the currently used mapping data may no longer be applicable to subsequent image frames, requiring the loading of new mapping data. Therefore, the mapping data update event is the trigger condition for ping-pong partition switching. A frame switching event is an event that occurs when image processing transitions from the current frame to the next. For dynamic metadata-driven formats, the mapping data for each image frame may differ, thus requiring updates to the mapping data during each frame switch. Frame switching events are typically triggered by a vertical synchronization signal or a frame completion interrupt. The dynamic range format switching event refers to an event that occurs when the dynamic range format of layer data changes. For example, when the display mode switches from HDR mode to SDR mode, the format switching event is triggered. The value of the status flag can be changed by flipping the operation, thereby changing the read / write status of the first and second unit partitions.

[0053] When the layer processing submodule detects a need for bitstream switching (i.e., a mapping data update event), such as when the current frame has finished processing and the next frame requires switching to a dynamic range format, the layer processing submodule can send a switching request to the lookup table storage block. Upon receiving this switching request, the lookup table storage block triggers an internal status flag to flip (e.g., from 0 to 1), thereby changing the read / write status of the first and second unit partitions. At this point, a partition that was previously in a readable state becomes writable, used to write mapping data for the next frame; a partition that was previously writable becomes readable, used to provide mapping data for the current frame. This method enables seamless switching between ping-pong operations.

[0054] In one specific implementation, the detection of the mapping data update event and the toggling of the status flag bit can be achieved as follows: The layer processing submodule monitors the vertical synchronization signal or frame completion interruption in the image processing pipeline to detect in real time whether a bitstream switch (i.e., a mapping data update event) has occurred. When a mapping data update event is detected, the layer processing submodule sends a control signal to the lookup table storage block to instruct it to switch its status. The lookup table storage block can be configured with a status flag bit, which can be stored in a dedicated control register within the lookup table storage block. When the lookup table storage block receives a toggling instruction from the layer processing submodule, it triggers the status flag bit to toggle (e.g., from 0 to 1, or from 1 to 0). The toggling of the status flag bit causes a change in the read / write status of the first and second unit partitions: the partition that was originally in a readable state becomes writable, used to write the mapping data for the next frame; the partition that was originally in a writable state becomes readable, used to provide the mapping data for the current frame. In this way, seamless switching of ping-pong operations can be achieved, ensuring data consistency and processing continuity.

[0055] Furthermore, another embodiment of this invention provides an image processing method based on the aforementioned image processing apparatus. This method includes: storing first-type mapping data in a lookup table storage block when the layer data processed by the layer processing submodule is in a first dynamic range format; storing second-type mapping data in a lookup table storage block when the layer data processed by the layer processing submodule is switched to a second dynamic range format; wherein the first-type mapping data and the second-type mapping data reuse the same physical storage space in the lookup table storage block; and reading the mapping data from the lookup table storage block through the layer processing submodule, and performing dynamic range conversion processing on the layer data based on the mapping data. By utilizing the reused storage of different types of mapping data in the lookup table storage block, format switching and conversion processing can be quickly completed without reloading the storage space when the dynamic range format of the layer data changes.

[0056] In one implementation, the layer processing submodule can be implemented as a hardware acceleration unit, and the dynamic range conversion processing can be completed through a hardware pipeline. After the processing module configures the mapping data in the lookup table storage block, the layer processing submodule reads the mapping data from the lookup table storage block in each clock cycle and performs parallel conversion on the input pixels. In another implementation, the layer processing submodule can be implemented as a software thread, and the dynamic range conversion processing is completed by executing program instructions through the central processing unit. The processing module loads the mapping data from the lookup table storage block into memory, and the layer processing submodule performs the conversion pixel by pixel using a software algorithm.

[0057] Optionally, the above method further includes: upon detecting a mapping data update event corresponding to the layer processing submodule, controlling the state flag bit of the lookup table storage block corresponding to the layer processing submodule to flip, thereby changing the read / write status of the first and second unit partitions in the lookup table storage block. The mapping data update event can be triggered by a hardware signal (such as a frame synchronization signal), and the state flag bit is automatically flipped by hardware. The processing module requires no software intervention, only needing to synchronously update the state flag bit when the event occurs. Alternatively, the mapping data update event can also be detected by software, and the state flag bit is triggered to flip by the software by writing to a control register. The software performs the flipping operation only after confirming that the new mapping data has been written.

[0058] Specifically, when performing dynamic range conversion on layer data using the layer processing submodule based on the mapping data stored in the lookup table storage block, this can be achieved as follows: when processing the layer data of the current frame, read the mapping data corresponding to the current frame from the cell partition in the readable state to perform dynamic range conversion on the current frame; and write the mapping data corresponding to the subsequent frames to the cell partition in the writable state.

[0059] To facilitate understanding, the following example illustrates the specific implementation details of the image processing apparatus provided in this application. Specifically, the image processing apparatus in this example is a dynamic HDR lookup table storage device capable of simultaneously switching multiple layer bitstreams.

[0060] This example belongs to the field of display engine technology, and specifically relates to the hardware circuit design of lookup table storage during image dynamic range conversion processing. In the field of image processing, with the continuous development of display technology, image data exhibits a trend of coexistence of multiple dynamic range formats. For example, standard dynamic range images and high dynamic range images, or different types of HDR formats such as HDR10, Hybrid Log-Gamma HLG, HDR10+, and HDR Vivid, are widely used in various display devices. To correctly present image content on display devices with different dynamic ranges, dynamic range conversion processing of image data is usually required.

[0061] In relevant image processing architectures, especially for multi-layered image compositing scenarios, a separate lookup table storage space is typically configured for each dynamic range format or each layer processing channel to store conversion parameters such as tone mapping curves and color gain coefficients. When the dynamic range format of the image source changes, the processing unit needs to switch to the corresponding lookup table storage space to read data. However, the above implementation method has at least the following problems: the lookup table storage spaces corresponding to different dynamic range formats are independent of each other, resulting in a large storage resource consumption. Especially when processing multi-layered images, each layer needs to be equipped with multiple lookup table storage spaces, further increasing chip area and hardware cost.

[0062] Table 1 lists the numbers and meanings of the dynamic range formats involved in this disclosure. Each of the dynamic range formats in Table 1 can be used as a specific implementation of the aforementioned first dynamic range format or second dynamic range format.

[0063] Table 1

[0064]

[0065] Figure 2 A schematic diagram of the hardware circuitry for storing a lookup table in the image dynamic range conversion process is shown in one implementation. For example... Figure 2 As shown, the hardware circuit mainly includes a memory pool, a front-end processing module, a layer blending module, and a back-end processing module. The storage resources are connected to the front-end processing module, and the front-end processing module is connected to the back-end processing module. Figure 2 The example shown uses four layers for illustration. Correspondingly, the front-end processing module includes four sets of photoelectric conversion functions and four sets of color correction modules. The output of the color correction module is provided to the back-end processing module for processing through the layer blending module. The back-end processing module includes photoelectric conversion functions, an error diffusion module, a debanding module, an inverse tone mapping module, and an output module. The front-end processing module can also be called the High Dynamic Range Front-End (HDR_FE) module, the layer blending module can also be called the Overlay module, the back-end processing module can also be called the High Dynamic Range Back-End (HDR_BE) module, and the memory pool can be called the Memory Pool (MEM_POOL) module.

[0066] like Figure 2 As shown, the front-end processing module performs dynamic range conversion on the input multi-layer data, the layer blending module overlays and blends the processed multi-layer data, and the back-end processing module adapts the format according to the dynamic range format of the output display device. The memory pool stores various lookup tables required during the dynamic range conversion process.

[0067] The following is about Figure 2 Each module in the document will be introduced separately:

[0068] I. Front-end processing module

[0069] The front-end processing module contains multiple image processing layers for dynamic range conversion (VLC) processing. The number of image processing layers in this module can be configured according to actual application requirements. For example, in one specific implementation, this module can contain four image processing layers, corresponding to multiple video sources, such as the main screen, floating window, and subtitle layer. Each image processing layer contains an Electro-Optical Transfer Function (EOTF) module and a Color Conversion (CC) module. The layer data processed by each image processing layer is a specific implementation of the aforementioned layer data; each image processing layer is a specific implementation of the aforementioned layer processing submodule.

[0070] (1) Electro-optical conversion function module

[0071] The electro-optic conversion function module is responsible for converting the input nonlinear electrical signal image data into linear optical signal image data for output. In one specific implementation, the input data can be 10 bits of nonlinear electrical signal image data, and the output data can be 18 bits of linear optical signal image data.

[0072] The electro-optic conversion function module determines the data processing branch based on the dynamic range format of the image processing layer. This dynamic range format can include standard dynamic range format, static high dynamic range format, and dynamic high dynamic range format, etc. The specific type of dynamic range format can be indicated by a format configuration signal (e.g., denoted as eotf_mode). Internally, the electro-optic conversion function module may contain multiple lookup table modules, each used to store different types of mapping data.

[0073] Figure 3 A schematic diagram of the electro-optic conversion function module is shown. Figure 3As shown, the electro-optic conversion function module can be further divided into a front-end conversion module, a dynamic range processing module, and a back-end conversion and output module. Specifically, the electro-optic conversion function module can include dynamic processing and static processing sections. For example, the front-end conversion module further includes multiple modules such as a gamma fitting function module, an HLG fitting function module, a DICOM fitting function module, a PQ fitting function module, and a linear fitting function module. These modules are used to process dynamic tables (also called dynamic mapping tables) or static tables (also called static mapping tables), respectively. The output results of the above fitting function modules are transmitted to the dynamic range processing module via the electro-optic conversion mode selection and routing module. The dynamic range processing module further includes a dynamic processing submodule for processing dynamic tables and a static processing submodule for processing static tables. The output results of the dynamic processing submodule or the static processing submodule are output through the back-end conversion and output module.

[0074] The backend conversion and output modules can be implemented using multiplexers. Correspondingly, the multiplexer ensures that at any given time, only data from the dynamic processing submodule or the static processing submodule is received. Therefore, the dynamic processing submodule and the static processing submodule will not output data simultaneously, thus providing a prerequisite for the time-sharing multiplexing mechanism mentioned later.

[0075] (2) Color conversion module

[0076] The color conversion module unifies the image data after dynamic conversion by each image processing layer to the same color gamut, ensuring consistency in subsequent processing. The color conversion module can also contain dynamic and static tables, and its storage method is similar to that of the electro-optical conversion function module.

[0077] (3) Storage pool module

[0078] The storage pool module stores all lookup tables required by each image processing layer in the front-end processing module. In one specific implementation, the storage pool module can store multiple lookup table groups, each corresponding to one image processing layer. The number of lookup table groups (SHADOW_GROUP) can be configured according to the number of image processing layers. For example, when the front-end processing module contains 4 image processing layers, the storage pool module can store 4 lookup table groups, each corresponding to one image processing layer. In addition, the storage pool module can also include an idle lookup table group, which can be used for bitstream switching scenarios. This idle lookup table group can also be called a shadow lookup table group.

[0079] Figure 4 A schematic diagram of the storage pool module in one implementation is shown. For example... Figure 4As shown, with the front-end processing module containing four image processing layers, the memory pool contains a total of five lookup table groups, namely... Figure 4 The lookup table group consists of lookup table group 0, lookup table group 1, lookup table group 2, lookup table group 3, and shadow lookup table group.

[0080] Each lookup table group (LUT_GROUP) can contain all the lookup tables required by the electro-optical conversion function module, specifically including dynamic tables and static tables. For example, each lookup table group can contain an electro-optical conversion function mapping table, a color gain mapping table, an HLG hue mapping table, a luminance gain mapping table (also called a static luminance gain table), a chrominance gain mapping table (also called a static chrominance gain table), etc. Each lookup table group is a specific implementation of the aforementioned lookup table storage block; the various lookup tables contained within each lookup table group are specific implementations of the lookup tables stored in the first lookup table unit, the second lookup table unit, and the third lookup table unit.

[0081] exist Figure 4 The implementation of the storage pool module, as shown, involves the following stream switching process: First, when the next frame of a certain image processing layer in the current processing module requires stream switching, the lookup table entries needed for the next frame's stream are written into the shadow lookup table group of the storage pool module within the processing cycle of the current frame. Second, at the start of the next frame, the image processing layer directly calls the pre-configured lookup table entries in the shadow lookup table group to complete the stream switching. Finally, after the switching is complete, the lookup table group used by the previous frame of the image processing layer becomes idle, becoming a new shadow lookup table group for use by subsequent stream switching of other image processing layers. Stream switching primarily refers to scenarios where the encoding format or dynamic range format of the video signal corresponding to the image processing layer changes. Through this mechanism, each image processing layer can use the shadow lookup table group to pre-store the lookup table entries needed for the next frame when performing stream switching, thereby achieving parallel switching capability across multiple image processing layers.

[0082] In cases where the bitrate remains constant, image processing for dynamic high dynamic range formats may require switching the dynamic table frame-by-frame. For example, in dynamic metadata-driven formats such as HDR10+ or ​​HDR_VIVID, the mapping data for each frame may differ. To address this requirement, in Figure 4 A ping-pong table access design can be used to achieve frame-by-frame switching. For example, in... Figure 4 In this model, the electro-optic conversion function mapping table, color gain mapping table, and HLG tone mapping table in each lookup table group are all dynamic mapping tables, which may need to be switched frame by frame. Therefore, the mapping tables, color gain mapping tables, and HLG tone mapping tables in each lookup table group are all designed to use a ping-pong switching method, that is, two storage areas are allocated for each dynamic mapping table, so as to use the two storage areas to perform ping-pong switching between different image frames. Figure 4 Both the luminance gain mapping table and the chrominance gain mapping table in the lookup table are static mapping tables. Since static mapping tables do not need to be updated frequently as long as the bitstream format remains unchanged, each static table in each lookup table group can be configured with only one storage area. Accordingly, Figure 4 Each dynamic mapping table in the system requires 18 bits × 1024 depth × 4 × 2 of storage space. Here, 18 bits represent the data width of each entry in a single lookup table. 1024 depth represents the number of entry groups in a single lookup table, i.e., the depth. 4 indicates that the number of parallel replicas of the lookup table storage resources is 4 sets; for example, each dynamic mapping table can be configured with four storage areas to support a more flexible preloading strategy. 2 indicates that each set of lookup table storage resources uses a double-buffered or ping-pong buffering mechanism. The storage space for each static mapping table is 18 bits × 1024 depth × 4. Therefore, since static mapping tables do not require double buffering, the storage space required for each dynamic mapping table is typically twice that required for each static mapping table.

[0083] (3) Layer blending module

[0084] The layer blending module is used to overlay and fuse image data processed by multiple image processing layers in the front-end processing module. This module synthesizes the linear optical signal image data output by each image processing layer according to a preset stacking order to generate complete image frame data.

[0085] (4) Backend processing module

[0086] The backend processing module is used to perform format adaptation processing on the image data output by the layer blending module according to the dynamic range format of the input and output.

[0087] In one specific implementation, input data can go through four processing branches: High Dynamic Range to Standard Dynamic Range (HDR to SDR), Standard Dynamic Range to High Dynamic Range (SDR to HDR), Standard Dynamic Range to Standard Dynamic Range (SDR to SDR), and High Dynamic Range to High Dynamic Range (HDR to HDR). All branches must pass through a photoelectric conversion function module to convert linear signals into nonlinear electrical signals for output to the display device.

[0088] The specific processing logic for each branch is as follows:

[0089] High dynamic range to standard dynamic range branch: Before entering the photoelectric conversion function module, it needs to pass through the Local Tone Mapping (LTM) module and the color conversion module to respectively achieve contrast enhancement, preservation of shadow and highlight details, and color space conversion.

[0090] Standard Dynamic Range to High Dynamic Range Branch: After the output of the photoelectric conversion function module, it needs to go through the Inverse Tone Mapping (ITM) module to simulate the high dynamic range highlights from the standard dynamic range data.

[0091] Therefore, it is evident that the above implementation method has at least the following drawbacks:

[0092] (1) Waste of storage resources

[0093] In the above approach, each lookup table group contains all dynamic and static mapping tables, without considering the need for storage resource reuse in different scenarios. For example, when the image processing layer processes static formats, the storage area corresponding to the dynamic tables is idle; when processing dynamic formats, the storage area corresponding to the static tables is also idle. Therefore, Figure 4 The independent configuration of data in the lookup table leads to redundant storage resources, increasing chip area and hardware costs. To address this issue, this example effectively reduces storage resource consumption by reusing the same physical storage space in the lookup table storage block for different types of mapping data.

[0094] (2) Design redundancy

[0095] The above method requires an additional shadow lookup table group to implement stream switching, increasing hardware design complexity. Furthermore, this shadow lookup table group is only used for stream switching and remains idle in other scenarios, further exacerbating resource waste. To address this issue, this example abandons the shadow lookup table group and directly utilizes the double-buffering mechanism of the dynamic lookup table within each lookup table group to implement stream switching, thereby avoiding the storage overhead of the shadow lookup table group.

[0096] (3) Stream switching is restricted

[0097] In the above method, since only one shadow lookup table group is configured, and this shadow lookup table group is shared by multiple lookup table groups, only one image processing layer can complete the bitstream switching per frame at most. When multiple image processing layers need to switch bitstreams simultaneously or continuously, they cannot respond in time, which may lead to screen stuttering or switching delays. To address this issue, this example abandons the shadow lookup table group and directly utilizes the double-buffering mechanism of the dynamic lookup table within each lookup table group to achieve bitstream switching. Each lookup table group can complete the bitstream switching independently without affecting each other. Therefore, the four image processing layers in each frame can achieve bitstream switching simultaneously, improving processing flexibility and efficiency.

[0098] To comprehensively address the aforementioned issues, this example designs a lookup table hardware storage device to support the reuse of the same storage resource for lookup tables occupied during different dynamic range conversions. Through a ping-pong table access design, the dynamic HDR bitstream is switched frame by frame. Figure 5 A schematic diagram of the improved storage pool module provided in this example is shown. This storage pool module is a specific implementation of the storage module mentioned above. Figure 5 As shown, with four image processing layers, four lookup table groups can be designed. Each lookup table group can contain multiple lookup table units. In one specific implementation, each lookup table group can contain a first lookup table unit (which can be denoted as lookup table 0, or LUT0), a second lookup table unit (which can be denoted as lookup table 1, or LUT1), and a third lookup table unit (which can be denoted as lookup table 2, or LUT2), for a total of three lookup table units. Each lookup table unit consists of multiple memory units (MEM).

[0099] Specifically, each lookup table unit can consist of 8 storage units, each with an 18-bit width and a depth of 1024 (i.e., each storage unit can store 1024 18-bit data items). These 8 storage units can be divided into two ping-pong tables: the first ping-pong table (which can be denoted as TABLE0) and the second ping-pong table (which can be denoted as TABLE1). In one specific partitioning method, storage units numbered MEM0 to MEM3 constitute the first ping-pong table, and storage units numbered MEM4 to MEM7 constitute the second ping-pong table.

[0100] The first ping-pong table described above is a specific implementation of the first unit partitioning, and the second ping-pong table described above is a specific implementation of the second unit partitioning. Within each lookup table unit, the alternating use of the two ping-pong tables achieves non-blocking updates of the mapped data.

[0101] Additionally, each lookup table group can be configured with a status flag (which can be denoted as metadata_id) to indicate the usage status of the first and second ping-pong tables. In one specific implementation, the status flag can be automatically toggled by hardware logic. For example, when the status flag is 0, it indicates that the first ping-pong table is currently occupied (in a readable state), and the second ping-pong table is idle (in a writable state); when the status flag is 1, it indicates that the second ping-pong table is currently occupied (in a readable state), and the first ping-pong table is idle (in a writable state). The toggling of the status flag can be triggered by a frame synchronization signal or a format switching event, achieving automatic switching at the hardware level.

[0102] By using the status flags, the processing module can accurately determine which ping-pong table to read the mapping data from and which ping-pong table to write the mapping data for the subsequent frame to.

[0103] Different lookup table cells within each lookup table group can store different types of mapping data, and the same lookup table cell can be reused for different types of mapping data. Specifically:

[0104] The first lookup table unit can be reused from a dynamic tone map table (eetf_map, i.e., an electro-optical conversion map table based on dynamic metadata) and a static luminance gain table (GAIN_Y). The dynamic tone map table is used for non-linear luminance mapping of dynamic high dynamic range formats (such as HDR10+, HDR_VIVID_PQ, HDR_VIVID_HLG, etc., i.e., the aforementioned dynamic metadata-driven formats); the static luminance gain table is used for luminance gain adjustment of specific static formats (such as HDR_VIVID_STATIC_HLG format, i.e., a specific form of the aforementioned static parameterized format). These two maps are not used simultaneously, therefore, they can reuse the same physical storage space.

[0105] The aforementioned dynamic tone mapping table is a specific implementation of the first lookup table of the first type (when the first dynamic range format is a dynamic metadata-driven format); the aforementioned static brightness gain table is a specific implementation of the first lookup table of the second type (when the second dynamic range format is a static parameterized format).

[0106] The second lookup table unit can be reused from a dynamic color gain table (color_gain_map) and a static chroma gain table (GAIN_C). The dynamic color gain table is used to dynamically adjust color saturation; the static chroma gain table is used to apply a fixed gain adjustment to the chroma (C) channel. These two mapping tables will not be used simultaneously, so they can reuse the same physical storage space.

[0107] The aforementioned dynamic color gain table is a specific implementation of the first type of second lookup table (when the first dynamic range format is a dynamic metadata-driven format); the aforementioned static chroma gain table is a specific implementation of the second type of second lookup table (when the second dynamic range format is a static parameterized format).

[0108] The third lookup table unit can be used to store a hybrid log-gamma tone map (hlg_tone_map). This map is a tone mapping curve designed for the hybrid log-gamma format to achieve specific image enhancement effects. In one specific implementation, this map is used when the dynamic range format is HDR_VIVID_HLG (i.e., number 6 in Table 1). The aforementioned hybrid log-gamma tone map is a specific implementation of the third lookup table.

[0109] Table 2 shows the usage of each lookup table group in the storage pool module under each dynamic range format.

[0110] Table 2

[0111]

[0112] As shown in Table 2:

[0113] For static formats numbered 0 to 3 (standard dynamic range and static high dynamic range), none of the lookup table cells are used, indicating that these formats do not require additional lookup tables for dynamic range conversion, or that their conversion parameters can be provided in other ways.

[0114] For the dynamic high dynamic range formats numbered 4 to 6, the first lookup table cell stores the dynamic tone map (eetf_map), and the second lookup table cell stores the dynamic color gain map (color_gain_map). Among them, the HDR_VIVID_HLG format numbered 6 additionally uses a third lookup table cell to store the mixed log-gamma tone map (hlg_tone_map).

[0115] For the HDR_VIVID_STATIC_HLG format number 7, the first lookup table cell stores the static luminance gain table (GAIN_Y_MAP), the second lookup table cell stores the static chrominance gain table (GAIN_C_MAP), and the third lookup table cell is unused.

[0116] As shown in Table 2, the first lookup table unit can be reused by the dynamic tone map table (eetf_map) and the static luminance gain table (GAIN_Y_MAP); the second lookup table unit can be reused by the dynamic color gain table (color_gain_map) and the static chrominance gain table (GAIN_C_MAP). This reuse mechanism allows the same physical storage space to store different types of mapping data in different scenarios, thereby achieving efficient utilization of storage resources.

[0117] In addition, due to Figure 5Each lookup table cell in each lookup table group is configured with two cell partitions to support ping-pong operations. Therefore, based on the ping-pong access design, each image processing layer can support frame-by-frame bitstream switching. A specific example is given below to illustrate this.

[0118] Example scenario: An image processing layer needs to switch from HDR_VIVID_PQ format (number 5) to HDR_VIVID_HLG format (number 6).

[0119] Initial state: In the current frame, the dynamic range format of this image processing layer is HDR_VIVID_PQ (number 5). At this time, the status flag (metadata_id) is 0, indicating that the first ping-pong table (MEM0 to MEM3) is currently occupied. The dynamic tone map table (eetf_map) used by this layer occupies the first ping-pong table of the first lookup table unit, and the dynamic color gain table (color_gain_map) occupies the first ping-pong table of the second lookup table unit. The third lookup table unit is not used.

[0120] Switching preparation: The system detects that the next frame requires switching to HDR_VIVID_HLG format (number 6). Within the processing cycle of the current frame, the mapping data required for the next frame is written into the second ping-pong table (MEM4 to MEM7) of each lookup table unit:

[0121] Write the dynamic tone map (eetf_map) required for the next frame into the second ping-pong table of the first lookup table unit; where eetf_map can be understood as a specific form of electro-optic conversion function mapping table.

[0122] Write the dynamic color gain table (color_gain_map) required for the next frame into the second ping-pong table of the second lookup table unit; where color_gain_map can be understood as a specific form of color gain mapping table.

[0123] The required mixed log-gamma tone map (hlg_tone_map) for the next frame is written into the second ping-pong table of the third lookup table unit. hlg_tone_map can be understood as a specific form of HLG tone map.

[0124] Since the processing module is reading mapping data from the first ping-pong table and writing subsequent frame data to the second ping-pong table during the current frame processing, the read and write operations are for different physical partitions, so no read and write conflicts will occur.

[0125] Switching Execution: At the start of the next frame, the status flag (metadata_id) flips to 1, indicating that the second ping-pong table has become readable. This image processing layer directly calls the mapping data pre-written into the second ping-pong table of the previous frame, completing a seamless bitstream switch.

[0126] After the switch is complete, the first ping-pong table becomes idle and becomes a new writable partition for subsequent frames to write mapping data.

[0127] The status flag bit flipping in the above example can be triggered by the frame synchronization signal or a format switching event, and is completed automatically by hardware logic. Through this ping-pong operation mechanism, the image processing layer can achieve frame-by-frame switching of dynamic high dynamic range formats without the need for additional lookup table resources.

[0128] In summary, the key points in this example include the following aspects:

[0129] First, a single image processing layer can achieve dynamic range frame-by-frame switching without additional lookup table resources.

[0130] By setting up a ping-pong table structure within each lookup table unit and configuring status flags to indicate read / write status, the image processing layer can write the mapping data required for the next frame into an idle ping-pong table while processing the current frame. During frame switching, the read / write roles are reversed by flipping the status flags. This design allows a single image processing layer to achieve frame-by-frame switching of dynamic range formats without additional lookup table resources. The above design is a specific implementation of the first and second unit partitioning and status flag control mechanism that operates in a ping-pong manner.

[0131] Second, each image processing layer is configured with a corresponding lookup table group, which supports multiple image processing layers to complete dynamic range frame-by-frame switching simultaneously.

[0132] In this example, each image processing layer can be configured with its own dedicated lookup table group. Each lookup table group contains a complete ping-pong table structure and can independently perform frame-by-frame switching operations. Therefore, when multiple image processing layers exist, each layer can simultaneously perform frame-by-frame switching of dynamic range format without interference. This design enables the system to support the concurrent processing requirements of multiple video sources (such as the main screen, floating windows, subtitle layers, etc.). The above design is a specific implementation of the one-to-one correspondence between the multiple layer processing submodules and multiple lookup table storage blocks.

[0133] Third, based on the lookup table group reuse rules, efficient reuse of lookup table resources is achieved during the conversion process of different dynamic and high dynamic range formats.

[0134] In this example, through reasonable lookup table unit partitioning and reuse rules, different types of mapping data can share the same physical storage space: the first lookup table unit reuses a dynamic tone mapping table (eetf_map) and a static luminance gain table (GAIN_Y); the second lookup table unit reuses a dynamic color gain table (color_gain_map) and a static chrominance gain table (GAIN_C). These two reuse rules are based on the following conditions: the two reused mapping tables typically operate on the same image channel (luminance channel or chrominance channel), and the input and output bit widths and data formats are consistent. Therefore, it is convenient to occupy the same physical storage space in different scenarios in a time-sharing manner, achieving efficient utilization of storage resources. The above reuse rules are a specific implementation of the first type of mapping data and the second type of mapping data reusing the same physical storage space in the lookup table storage block.

[0135] In summary, the embodiments of the present invention have at least the following beneficial effects compared to the prior art:

[0136] First, storage resource utilization efficiency is significantly improved. Through lookup table resource reuse design and precise allocation strategy, different types of mapped data can be reused in the same physical storage space in a time-sharing manner, avoiding resource idleness caused by full storage. Compared with existing technologies, it can save about 30% of storage resources, effectively reducing chip area and hardware costs.

[0137] Second, the bitstream switching capability is significantly enhanced. By eliminating redundant shadow group design, each image processing layer relies on the ping-pong table design within its dedicated lookup table group to achieve the ability to switch bitstreams frame by frame simultaneously across multiple image processing layers. This breaks through the limitation of single-frame, single-layer switching in existing technologies, improving the system's concurrent processing capability and response speed.

[0138] Third, the complexity of hardware design is effectively reduced. By simplifying the composition structure and data read / write logic of the lookup table group, there is no need to maintain the idle state and switching scheduling of the shadow group, which improves the simplicity and reliability of the hardware design and reduces the difficulty of chip design and verification complexity.

[0139] This disclosure also provides an electronic device, including the image processing apparatus described above.

[0140] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0141] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method. The computer-readable storage medium may be volatile or non-volatile.

[0142] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0143] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0144] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0145] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0146] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0147] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0148] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0149] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0150] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0152] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. An image processing apparatus, characterized in that, include: A layer processing submodule corresponding to the layer data in the image to be processed, and a lookup table storage block corresponding to the layer processing submodule; The lookup table storage block is configured to: store first type of mapping data when the layer data processed by the layer processing submodule is in a first dynamic range format; and store second type of mapping data when the layer data processed by the layer processing submodule is switched to a second dynamic range format; wherein the first type of mapping data and the second type of mapping data share the same physical storage space in the lookup table storage block in a time-sharing manner. The layer processing submodule is configured to perform dynamic range conversion processing on the layer data based on the mapping data stored in the lookup table storage block.

2. The apparatus according to claim 1, characterized in that, The image to be processed contains multiple layers of data, and the number of layer processing sub-modules is also multiple, wherein each layer processing sub-module corresponds one-to-one with a layer of data. Furthermore, there are multiple lookup table storage blocks, and each lookup table storage block corresponds one-to-one with a multiple layer processing sub-module; wherein, each lookup table storage block is used to store the mapping data corresponding to the corresponding layer processing sub-module.

3. The apparatus according to claim 1, characterized in that, The first type of mapping data includes: a first lookup table of the first type and a second lookup table of the first type; the second type of mapping data includes: a first lookup table of the second type and a second lookup table of the second type. Furthermore, each lookup table storage block includes: a first lookup table unit and a second lookup table unit; wherein, when the layer data processed by the layer processing submodule is in a first dynamic range format, the first lookup table unit is used to store a first lookup table of the first type, and the second lookup table unit is used to store a second lookup table of the first type; When the layer data processed by the layer processing submodule is in the second dynamic range format, the first lookup table unit is used to store the first lookup table of the second type, and the second lookup table unit is used to store the second lookup table of the second type. The first lookup table of the first type and the first lookup table of the second type share the same physical storage space in the first lookup table unit, and the second lookup table of the first type and the second lookup table of the second type share the same physical storage space in the second lookup table unit.

4. The apparatus according to claim 3, characterized in that, When the first dynamic range format is a dynamic metadata-driven format, the first lookup table of the first type is a dynamic tone mapping table, and the second lookup table of the first type is a dynamic color gain table. When the second dynamic range format is a static parameterized format, the first lookup table of the second type is a static luminance gain table, and the second lookup table of the second type is a static chrominance gain table.

5. The apparatus according to claim 3 or 4, characterized in that, The lookup table storage block is also configured to further store third-type mapping data when the layer data processed by the layer processing submodule is an enhanced first dynamic range format; The third type of mapping data includes a third lookup table; and each lookup table storage block further includes a third lookup table unit for storing the third lookup table.

6. The apparatus according to claim 3 or 4, characterized in that, Each lookup table cell in each lookup table storage block further includes: a first cell partition and a second cell partition that operate in a ping-pong manner; The first unit partition and the second unit partition are used to alternately store the mapping data corresponding to the current frame and the subsequent frames during inter-frame switching.

7. The apparatus according to claim 6, characterized in that, Each lookup table storage block is further configured with status flag bits; The status flag is used to indicate the read / write status of the first and second unit partitions of each lookup table unit in the lookup table storage block, so that the layer processing submodule can read the mapping data corresponding to the current frame from the unit partition in the readable state and write the mapping data corresponding to the subsequent frame to the unit partition in the writable state.

8. The apparatus according to claim 7, characterized in that, The status flag bit corresponding to each lookup table storage block is flipped when a mapping data update event corresponding to the layer processing submodule is detected, so as to change the read and write status of the first unit partition and the second unit partition in the lookup table storage block. The mapping data update events include: a frame switching event for switching image frames under the same dynamic range format, and a dynamic range format switching event for changing the dynamic range format.

9. An electronic device, characterized in that, include: The image processing apparatus according to any one of claims 1-8.

10. An image processing method, implemented based on the image processing apparatus according to any one of claims 1-8, characterized in that, The method includes: When the layer data processed by the layer processing submodule is in the first dynamic range format, the first type of mapping data is stored in the lookup table storage block; When the layer data processed by the layer processing submodule is switched to the second dynamic range format, the second type of mapping data is stored in the lookup table storage block; wherein, the first type of mapping data and the second type of mapping data share the same physical storage space in the lookup table storage block in a time-sharing manner. The layer processing submodule performs dynamic range conversion on the layer data based on the mapping data stored in the lookup table storage block.

11. The method according to claim 10, characterized in that, The method further includes: when a mapping data update event corresponding to the layer processing submodule is detected, controlling the state flag bit of the lookup table storage block corresponding to the layer processing submodule to flip, so as to change the read and write status of the first unit partition and the second unit partition in the lookup table storage block.

12. The method according to claim 11, characterized in that, The step of performing dynamic range conversion on the layer data through the layer processing submodule, based on the mapping data stored in the lookup table storage block, includes: When processing the layer data of the current frame, read the mapping data corresponding to the current frame from the cell partition that is in a readable state in order to perform the dynamic range transformation of the current frame; Furthermore, the mapping data corresponding to the subsequent frames is written to the writable cell partition.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 10-12.

14. A computer program product comprising computer-readable code, characterized in that, When the computer-readable code is run in an electronic device, the processor in the electronic device performs the method of any one of claims 10-12.

Images