Image data rearrangement device, system and method

Abstract

The invention discloses an image data rearrangement device, system and method, and the device comprises a writing module which is disposed on a common writing path between a plurality of function modules and a memory, and is used for responding to a writing request of the function modules for a first data stream, carrying out the rearrangement of the first data stream, and obtaining a second data stream, writing the second data stream into the memory; the first data stream comprises a first number of first data units, and pixels in the first data units are in an initial arrangement format; the second data stream comprises a second number of second data units; pixels in the second data unit are in a target arrangement format; the target arrangement format is continuous and tight arrangement of effective data; and the reading module is deployed on a common reading path between the plurality of functional modules and the memory and is used for responding to a reading request of the functional modules, reading the second data stream from the memory, converting the second data stream into an initial arrangement format and sending the second data stream to the functional modules.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image data rearrangement apparatus, system and method. Background Technology

[0002] In multimedia SoCs (System-on-a-Chip), high-resolution images such as 4K and 8K often use non-byte aligned color depths such as 10-bit, 12-bit, and 14-bit. To adapt to the processing logic of processors and hardware IPs (independent hardware modules with specific functions, such as ISP image signal processors and DPU data processing units), this type of pixel data needs to be aligned to 16-bit (half-word) transmission, resulting in a large number of invalid gaps in the data and causing bandwidth waste. Summary of the Invention

[0003] The technical solution provided in this application is as follows:

[0004] The first aspect of this application provides an image data rearrangement apparatus, comprising:

[0005] A write module, deployed on a common write path between multiple functional modules and a memory, is used to respond to a write request from a functional module for a first data stream, rearrange the first data stream to obtain a second data stream, and write the second data stream into the memory; the first data stream includes a first number of first data units, and the pixels in the first data units are arranged in an initial arrangement format; the second data stream includes a second number of second data units, and the pixels in the second data units are arranged in a target arrangement format; the target arrangement format is that valid data is arranged continuously and closely.

[0006] A read module deployed on a common read path between the multiple functional modules and the memory is used to read the second data stream from the memory in response to a read request from the functional module, convert the second data stream into the initial arrangement format, and send it to the functional module.

[0007] In one possible implementation, the write module writes the second data stream into the memory, including:

[0008] The write module caches the second data stream, and when the length of the cached data meets the preset address alignment condition, it writes the cached second data stream into the memory.

[0009] In one possible implementation, the functional module is used to offset the initial write address in the write request based on the base address offset corresponding to the functional module, to obtain the write base address corresponding to the functional module, and to send a first control signal containing the write base address corresponding to the functional module.

[0010] The write module includes:

[0011] The write cache includes: multiple first cache units, each of which independently corresponds to a data stream;

[0012] A write reordering module is configured to respond to a write request from the functional module for a first data stream, and if the first data stream meets preset conditions, reorder the first data stream to obtain a second data stream with valid data arranged continuously and closely, and cache the second data stream in the first target cache unit corresponding to the first data stream; and

[0013] When the first target cache unit is full, the write base address corresponding to the functional module is obtained from the first control signal. Based on the cumulative number of writes of the first target cache unit, the write base address corresponding to the functional module is offset to obtain the memory start write address. The second data stream in the first target cache unit is written into the memory using the memory start write address.

[0014] In one possible implementation, the first control signal further includes a first control parameter; the first control parameter is used to indicate whether to perform rearrangement processing on the first data unit;

[0015] The write module further includes:

[0016] The first data path selection module is configured to respond to the second control signal by sending the first data stream to the write reordering module if the first control parameter indicates that the first data stream needs to be rearranged; and by directly writing the first data stream into the memory if the first control parameter indicates that the first data stream does not need to be processed.

[0017] In one possible implementation, the first control signal further includes: a first identifier for identifying the first data stream;

[0018] The write reordering module caches the second data stream into the first target cache unit corresponding to one of the first data streams, including:

[0019] The write reordering module selects the cache unit corresponding to the first identifier from the plurality of cache units and determines it as the target cache unit, and caches the second data stream into the first target cache unit.

[0020] In one possible implementation, the functional module is used to offset the initial read address in the read request based on the base address offset corresponding to the functional module, to obtain the read base address corresponding to the functional module, and to send a second control signal containing the read base address corresponding to the functional module.

[0021] The read module includes:

[0022] The read cache includes: multiple second cache units, each of which independently corresponds to a data stream;

[0023] The read request processing module is used to respond to the second control signal, offset the read base address corresponding to the functional module based on the capacity of the second cache unit and the number of times the functional module reads from the memory, to obtain the memory start read address, and read the second data unit from the memory at the memory start read address and store it into the second target cache unit corresponding to the first data stream until the first number is reached;

[0024] The read reordering module is used to read the second data units from the second target cache unit in the order of the cache until the second quantity is reached, convert the second quantity of second data units into the initial reordering format, and send it to the functional module.

[0025] In one possible implementation, the functional module is used to offset the initial read address in the read request based on the base address offset corresponding to the functional module, to obtain the read base address corresponding to the functional module, and to send a second control signal containing the read base address corresponding to the functional module.

[0026] The read module includes:

[0027] A read request processing module, configured to, in response to the second control signal, convert the read base address into a memory address range corresponding to the second data stream according to the target arrangement format of the second data stream; and,

[0028] Based on the memory address range, the second data stream is read from the memory;

[0029] The read reordering module is used to convert the second data stream into the initial reordering format and send it to the functional module.

[0030] In one possible implementation, the second control signal further includes: a first identifier for identifying the first data stream;

[0031] The read request processing module reads the second data unit from the memory at the memory start read address and stores it in the second target cache unit corresponding to the first data stream, including:

[0032] The read request processing module selects the cache unit corresponding to the first identifier from the plurality of cache units and determines it as the second target cache unit corresponding to the first data stream, and reads the second data unit from the memory with the memory start read address and stores it in the second target cache unit.

[0033] In another aspect, this application provides an image data rearrangement system, comprising:

[0034] The functional module is used to initiate a write request or a read request; the write request corresponds to a first data stream; the first data stream includes a first number of first data units and the first data units are arranged in an initial format;

[0035] A control module, connected to the functional module, a write module deployed on a common write path between the multiple functional modules and the memory, and a read module deployed on a common read path between the multiple functional modules and the memory, is used to output a first control signal or a second control signal according to the write request or the read request;

[0036] The first arbitration module, connected to the control module, the write module, and the read module, is used to schedule the first control signal or the second control signal corresponding to the write request or read request initiated by the multiple functional modules to the write module or the read module.

[0037] The second arbitration module is connected to the write module, the read module, and the memory, and is used to coordinate the write module or the read module to write data to or read data from the memory;

[0038] The write module is configured to, in response to the first control signal, rearrange the first data stream to obtain a second data stream, and write the second data stream into the memory; the second data stream includes a second number of second data units; the second data units are arranged in a target format; the target format is that valid data is arranged continuously and closely.

[0039] The read module is configured to, in response to the second control signal, read the second data stream from the memory, convert the second data stream into the initial arrangement format, and send it to the functional module.

[0040] A third aspect of this application provides an image data rearrangement method, comprising:

[0041] A write module deployed on a common write path between multiple functional modules and a memory responds to a write request from a functional module for a first data stream, rearranges the first data stream to obtain a second data stream, and writes the second data stream into the memory; the first data stream includes a first number of first data units and the first data units are arranged in an initial arrangement format; the second data stream includes a second number of second data units and the second data units are arranged in a target arrangement format; the target arrangement format is that valid data is arranged continuously and closely.

[0042] A read module deployed on a common read path between the multiple functional modules and the memory responds to a read request from a functional module by reading the second data stream from the memory, converting the second data stream into the initial arrangement format, and sending it to the functional module. Attached Figure Description

[0043] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 This is a schematic diagram of the structure of an image data rearrangement device provided in Embodiment 1 of this application;

[0045] Figure 2 A schematic diagram of the initial arrangement format between pixels in a first data unit provided in this application;

[0046] Figure 3 A schematic diagram of a rearrangement provided in this application;

[0047] Figure 4 Another schematic diagram of a rearrangement provided for this application;

[0048] Figure 5 Another schematic diagram of a rearrangement provided in this application;

[0049] Figure 6 A schematic diagram of the structure of a cache unit provided in this application;

[0050] Figure 7 A schematic diagram illustrating a processing scenario for a write module provided in this application;

[0051] Figure 8 A schematic diagram of the structure of a write module provided in this application;

[0052] Figure 9 A schematic diagram illustrating a processing scenario for a read module provided in this application;

[0053] Figure 10 A schematic diagram of the structure of a read module provided in this application;

[0054] Figure 11 This is another schematic diagram of an image data rearrangement device provided in this application. Detailed Implementation

[0055] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0056] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0057] The terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0058] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Reference Figure 1 This is a schematic diagram of the structure of an image data rearrangement device provided in Embodiment 1 of this application, as shown below. Figure 1 As shown, the image data rearrangement device may include, but is not limited to, a write module and a read module.

[0060] In this embodiment, from a hardware design perspective, the write module can be connected to the common bus (e.g., AXI bus) transmission link of the multimedia subsystem. The input and output interfaces of the write module both conform to the protocol standard of the common bus, and are hard-connected to the bus hardware (e.g., arbitration module), functional modules, and memory through shared physical signal line groups.

[0061] From a functional implementation perspective, the write module can be deployed on a common write path between multiple functional modules (which can be represented as functional module 0, functional module 1, ..., functional module N) and the memory. This common write path can be a dedicated functional path built based on three logical sub-channels in the common bus: write address (AW), write data (W), and write response (B). All write requests and corresponding data initiated by functional modules must be transmitted through this path. The write module, as a necessary hardware processing node on this path, receives and processes the first data stream in all write directions.

[0062] Functional modules may include independent units with specific image processing capabilities, such as image signal processors (ISPs), graphics processing units (GPUs), display controllers (Display), and video encoders. Functional modules can perform their own image processing tasks by reading and writing data to memory.

[0063] When a functional module needs to write processed data to memory, it can send a write request through a data interaction interface (such as the AXIMaster interface, AHB-Lite interface, or APB interface). The write request can be transmitted to the write module via a common write path.

[0064] Within a functional module, pixels used in image acquisition, rendering, and other processes can include either non-byte-aligned pixels or byte-aligned pixels.

[0065] Non-byte aligned pixels can be understood as pixels whose bit width cannot be divided by 8 bits (1 byte) or 16 bits (2 bytes). For example, the effective data bit width of 10-bit, 12-bit, and 14-bit pixels is neither equal to 1 byte (8 bits) nor occupies 2 bytes (16 bits), and therefore cannot directly meet the basic processing requirements of processors and hardware IPs.

[0066] Byte-aligned pixels can be understood as pixels whose bit width is divisible by 8 bits (1 byte) or 16 bits (2 bytes). For example, the bit width of 8-bit and 16-bit pixels can be directly matched to the processing specifications of processors and hardware IPs without additional adaptation.

[0067] For processors (such as CPUs, DSPs, GPUs, etc.), instruction execution or data reading and writing is generally required in units of a set bit width (e.g., 16 bits (half word)) (to simplify hardware logic and instruction design). Therefore, functional modules will actively align the original pixels before outputting data.

[0068] The specific alignment process may include: the functional module retains the high-order bits and pads the low-order bits with zeros for each non-byte-aligned pixel, thus expanding the bit width of the non-byte-aligned pixel to a set bit width, resulting in a byte-aligned pixel. The low-order zero-padded portion can be considered as invalid data without actual image information.

[0069] Since the bit width of the byte-aligned pixels perfectly matches the bit width required by the processor and hardware IP, no alignment processing is needed after the functional module has completed its internal processing.

[0070] In this embodiment, data transmission can follow the Burst transmission length of the data transmission bus (e.g., AXI bus, AHB bus) (i.e., continuously transmitting N data units of the same bit width, such as N being 4, 8, or 16). The specific data organization and processing logic is as follows:

[0071] The functional module can integrate several byte-aligned pixels (extended non-byte-aligned pixels or native byte-aligned pixels) into a first data unit according to the single transmission bit width of the data transmission bus (e.g., 64-bit, 128-bit, or 256-bit). The bit width of a single first data unit is consistent with the single transmission bit width of the bus.

[0072] In the first data unit, the pixels are arranged in the initial format, meaning that there are invalid data gaps between the pixels.

[0073] For example, when the bit width is set to 16 bits, such as Figure 2 As shown, a 10-bit pixel (i.e., a non-byte-aligned pixel) occupies the high 10 bits of the 16-bit array after expansion, with the remaining 6 bits being invalid data. This means there are 6-bit invalid data gaps between pixels in the first data unit. Similarly, a 12-bit pixel (i.e., a non-byte-aligned pixel) occupies the high 12 bits after expansion, with the remaining 4 bits being invalid data. This also means there are 4-bit invalid data gaps between pixels in the first data unit. A 14-bit pixel (i.e., a non-byte-aligned pixel) occupies the high 14 bits after expansion, with the remaining 2 bits being invalid data. This means there are 2-bit invalid data gaps between pixels in the first data unit. A 16-bit pixel (i.e., a byte-aligned pixel) occupies the entire 16-bit width and has no invalid gaps.

[0074] In this embodiment, the number of consecutively transmitted data units supported by the bus can be a first quantity, and the first quantity of consecutive first data units (each 128 bits) can be integrated into a first data stream. For example, if the burst transmission length is 8, eight consecutive first data units of the same bit width can be combined into a first data stream.

[0075] If the data transmission bus has a single transmission width of 128 bits, the functional module can integrate 8 extended 16-bit pixels (8×16bit=128bit) into a first data unit (total width 128 bits).

[0076] The write module can be used to respond to the write request of the functional module for the first data stream, rearrange the first data stream to obtain a second data stream, and write the second data stream into the memory.

[0077] The second data stream may include a second number of second data units; the second data units are arranged in a target format; the target format is that the effective data is arranged continuously and closely.

[0078] In this embodiment, the write module can traverse all first data units in the first data stream and extract the valid pixel data in each unit one by one according to the original arrangement order of the first data units (ignoring the low-order invalid padding). Using a set transmission bandwidth (128 bits for a single transmission of the data transmission bus in this embodiment) as the unit, the extracted valid data is arranged continuously and tightly to reassemble into a complete second data stream, ensuring that the data order is completely consistent with the original first data stream, without any misalignment or loss.

[0079] For example, such as Figure 3 As shown, the first data stream of 10-bit pixels contains eight 128-bit first data units (i.e., the Burst transmission length is 8, which can be represented as Brust8). Each first data unit contains eight pixels (e.g., P07~00). In each pixel, only the high 10 bits are valid data and the low 6 bits are invalid data. The total amount of valid data in each first data unit is 8 × 10 bits = 80 bits, and the total amount of valid data in the first data stream is 8 × 80 bits = 640 bits.

[0080] After the write module extracts 640 bits of valid data, it can arrange the 640 bits of valid data continuously and tightly in 128-bit units according to the original order of the first data unit, and reassemble them into a second data stream. The second data stream contains 5 consecutive 128-bit second data units (that is, the Burst transmission length is 5, which can be represented as Brust5).

[0081] It should be noted that the rearrangement process allows for the concatenation of valid data across first data units. Therefore, a single second data unit may contain the remaining valid data from the previous first data unit and a portion of the valid data from the next first data unit. For example, the 128-bit pixels (Packer (P00~12.8L)) contained in second data unit 0 include both 80 bits of valid data from the 8 pixels in first data unit 0 and 48 bits of valid data from the first 5 pixels in first data unit 1 (80 bits + 48 bits = 128 bits). The remaining second data units follow this logic to tightly concatenate valid data across units, ultimately forming a second data stream without any invalid gaps.

[0082] like Figure 4 As shown, the first data stream of 12-bit pixels contains 8 first data units of 128 bits each (i.e., the Burst transmission length is 8, which can be represented as Brust8). Each first data unit contains 8 pixels (e.g., P07~00). Only the high 12 bits of each pixel are valid data, and the low 4 bits are invalid data. The total amount of valid data in each first data unit is 8 × 12 bits = 96 bits, and the total amount of valid data in the first data stream is 8 × 96 bits = 768 bits.

[0083] After the write module extracts 768 bits of valid data, it can arrange the 768 bits of valid data continuously and tightly in 128-bit units according to the original order of the first data unit, and reassemble them into a second data stream. The second data stream contains 6 consecutive 128-bit second data units (that is, the Burst transmission length is 6, which can be represented as Brust6).

[0084] It should be noted that the rearrangement process allows for the concatenation of valid data across first data units. Therefore, a single second data unit may contain the remaining valid data from the previous first data unit and a portion of the valid data from the next first data unit. For example, the 128-bit pixels (Packer (P00~10.8L)) contained in second data unit 0 include both 96 bits of valid data from the 8 pixels in first data unit 0 and 32 bits of valid data from the first 3 pixels in first data unit 1 (96 bits + 32 bits = 128 bits). The remaining second data units follow this logic to tightly concatenate valid data across units, ultimately forming a second data stream without any invalid gaps.

[0085] like Figure 5As shown, the first data stream of 14-bit pixels contains 8 first data units of 128 bits each. Each first data unit contains 8 pixels (e.g., P07~00). Only the high 14 bits of each pixel are valid data, and the low 2 bits are invalid data. The total amount of valid data in each first data unit is 8 × 14 bits = 112 bits. The total amount of valid data in the first data stream is 8 × 112 bits = 896 bits.

[0086] After the write module extracts 896 bits of valid data, it can arrange the 896 bits of valid data continuously and tightly in 128-bit units according to the original order of the first data unit, and reassemble them into a second data stream (which can be represented as Burst7). The second data stream contains 7 consecutive 128-bit second data units.

[0087] It should be noted that the rearrangement process allows for the concatenation of valid data across first data units. Therefore, a single second data unit may contain the remaining valid data from the previous first data unit and a portion of the valid data from the next first data unit. For example, the 128-bit pixels (Packer (P00~9.2L)) contained in second data unit 0 include both 112 bits of valid data from the 8 pixels in first data unit 0 and 16 bits of valid data from the first 2 pixels in first data unit 1 (112 bits + 16 bits = 128 bits). The remaining second data units follow this logic to tightly concatenate valid data across units, ultimately forming a second data stream without any invalid gaps.

[0088] The write module can follow the protocol standard of the common bus to write the second data stream into the memory.

[0089] From a hardware design perspective, the read module can be connected to the common bus transmission link of the multimedia subsystem. The input and output interfaces of the read module both conform to the protocol standard of the common bus, and are hard-connected to bus hardware (such as the arbitration module), functional modules, and memory through shared physical signal lines.

[0090] From a functional implementation perspective, the read module can be deployed on a common read path between the multiple functional modules and the memory. This common read path can be a dedicated functional path built based on two logical sub-channels, read address (AR) and read data (R), in the common bus. All read requests initiated by the functional modules and the corresponding data returned from the memory must be transmitted through this path. The read module, as a necessary hardware processing node on this path, receives and processes the second data stream in all read directions.

[0091] When a functional module needs to access image data, it can send a read request through the data interaction interface. The read request can be transmitted to the read module through a public read path.

[0092] The read module can be used to read the second data stream from the memory in response to a read request from the functional module, convert the second data stream into the initial arrangement format, and send it to the functional module.

[0093] The read module can read a second data stream from memory in accordance with the common bus protocol standard.

[0094] The second data stream consists of tightly packed valid data (without any invalid gaps), for example, in the format of Burst5 (10-bit pixels), Burst6 (12-bit pixels), or Burst7 (14-bit pixels), which is exactly the same format as that written by the write module.

[0095] Since the target arrangement format of the second data stream does not match the initial arrangement format required by the functional module, the read module can convert the second data stream into the initial arrangement format.

[0096] Converting the second data stream into the initial arrangement format may include, but is not limited to:

[0097] Based on the effective bit width of a single pixel, the closely packed pixels in the second data stream are split and restored to independent effective pixels in the same order as the first data stream;

[0098] Perform high-bit retention and low-bit zero padding on each independent valid pixel, and extend its bit width to a set bit width (e.g., 16 bits) to obtain byte-aligned pixels;

[0099] According to the data transmission bus Burst transmission specification (consistent with the Burst length requested by the functional module, such as Burst8), the extended byte-aligned pixels are integrated into several first data units.

[0100] For example, in a 10-bit pixel scene, the second data stream is in Burst5 format, containing five 128-bit second data units, with a total effective data volume of 640 bits (corresponding to 64 independent effective pixels of 10 bits).

[0101] The 640-bit tightly packed valid data is split into 10-bit granularity (i.e., the effective bit width of a single pixel) and restored to 64 independent valid pixels of 10 bits each (in the same order as when extracted by the write module, with no misalignment).

[0102] For each 10-bit independent valid pixel, perform the operation of retaining the high bits and padding the low bits with zeros to expand it into a 16-bit byte aligned pixel. That is, the high 10 bits are valid data and the low 6 bits are filled with invalid zero values.

[0103] According to the AXI bus Burst8 specification, 64 16-bit aligned pixels are integrated into 8 128-bit first data units (8×8 pixels = 64 pixels, 8×128 bits = 1024 bits).

[0104] In this embodiment, the write module's close arrangement of valid data completely eliminates invalid gaps in the transmission of non-byte aligned pixels, significantly improving the utilization of bus and memory bandwidth and reducing system power consumption. The read module accurately restores the data format, ensuring that the functional modules can be used directly without adjusting their internal processing logic, thus meeting the real-time processing requirements of high-resolution images such as 4K / 8K.

[0105] Meanwhile, the write and read modules are deployed on a common write path and a common read path, eliminating the need to design and embed write and read modules separately for each functional module, greatly reducing redundant hardware circuit design and significantly reducing the overall chip area.

[0106] Furthermore, the write and read modules are deployed on common write and read paths, eliminating the need to modify the internal structure of existing functional modules. This avoids the additional costs and risks of customizing externally purchased hardware IPs, the risks of modifying hardware IPs, and the complexity of interface compatibility design for newly developed hardware IPs. Moreover, through a standardized rearrangement logic, all functional modules are uniformly adapted, greatly reducing software complexity.

[0107] As another optional embodiment of this application, an image data rearrangement device is provided in Embodiment 2 of this application. This embodiment is mainly an implementation of the writing module in Embodiment 1. The writing module writes the second data stream into the memory, including:

[0108] The write module caches the second data stream, and when the length of the cached data meets the preset address alignment condition, it writes the cached second data stream into the memory.

[0109] When the write module rearranges the first data stream into the second data stream (Burst5, Burst6, Burst7), it mainly focuses on the tight arrangement of valid data and ignores the 4K boundary issue. That is, the start address or end address of the second data stream may cross the 4K-byte physical page boundary of the DDR memory (DDR memory cells are divided into 4K-byte pages, and cross-page transmission will cause a sharp increase in transmission latency and may even violate the data transmission bus protocol).

[0110] For example, a second data stream of 10-bit pixels (Burst5, total length 80 bytes), if its starting address is 4000 bytes (only 96 bytes away from the 4K boundary (4096 bytes)), then the 80 bytes of data in this Burst5 will occupy 4000~4079 bytes, without spanning pages; but the next Burst5 (80 bytes) will occupy 4080~4159 bytes, of which 4080~4095 bytes belong to the previous 4K page, and 4096~4159 bytes belong to the next 4K page, forming cross-page transmission, which affects transmission efficiency.

[0111] Therefore, in this embodiment, the second data stream can be cached. When the length of the cached data meets the preset address alignment conditions (e.g., meets the 4K boundary requirements), the cached second data stream is written into the memory, thus completely avoiding the page crossover problem.

[0112] In this embodiment, the write module can detect the cumulative data length of the cached second data stream in real time, provided that the write address of the first second data stream is aligned with the 4K boundary, and determine whether the cumulative data length is 128 bytes (the total length in Burst8 format).

[0113] For example, 128 bytes is the standard length of the Burst8 bus, and 4096 words (4K) are divisible by 128 bytes (4096÷128=32). Therefore, after the second data stream in the cache is rounded up to 128 bytes, it can fall within a single 4K page without the risk of page crossover.

[0114] In this embodiment, a caching mechanism is added to the write module to temporarily store the rearranged second data stream and write it to the memory only when the length of the cached data meets the preset address alignment condition. This ensures that all written data blocks fall within a single 4K physical page, completely avoiding the additional timing delays and data transmission bus protocol violations caused by cross-page transmission, and significantly improving the stability and efficiency of data transmission.

[0115] On the other hand, the caching process only changes the timing of data writing, without modifying the content and order of valid data. While ensuring the bandwidth utilization improvement and power consumption reduction brought about by the close arrangement of valid data, it does not require additional modification to the hardware structure of functional modules, common buses or memory, thus maintaining good compatibility.

[0116] As another optional embodiment of this application, an image data rearrangement device is provided in embodiment 3 of this application. This embodiment is mainly an implementation of the write module in embodiment 1. In this embodiment, the functional module can be used to offset the initial write address in the write request based on the base address offset corresponding to the functional module, to obtain the write base address corresponding to the functional module, and to issue a first control signal containing the write base address corresponding to the functional module.

[0117] In the multimedia subsystem, multiple functional modules such as the ISP, GPU, and display controller can share DDR storage resources. If all modules directly use their own initial write addresses to write data, conflicts such as overlapping storage areas and data overwriting between different modules are very likely to occur.

[0118] Therefore, each functional module is designed with a pre-defined base address offset, which clearly defines the offset range of its own storage area relative to the DDR global address.

[0119] Each functional module corresponds to a unique dedicated area in DDR for its write base address, which can avoid storage conflicts when multifunctional modules share resources.

[0120] Of course, in this embodiment, a first control signal containing the base address offset corresponding to the functional module and the initial write address in the write request can also be sent to the write module, and the write module can offset the initial write address in the write request based on the base address offset corresponding to the functional module to obtain the base address corresponding to the functional module.

[0121] The format for sending the base address offset corresponding to the functional module may include:

[0122] AXI_MASTER_AWADDR[31:27]=AXI_SLAVE_AWADDR[31:27];

[0123] AXI_MASTER_AWADDR[26:6]=AXI_SLAVE_AWUSER[26:6];

[0124] AXI_MASTER_AWADDR [5:0]=0.

[0125] AXI_SLAVE_AWADDR[31:27] can represent the high 5 bits of the base address. The high 5 bits ([31:27]) can be used to identify the high-order storage area of ​​DDR (such as different bank groups, storage channels). This part is independent of the data arrangement format (byte alignment / tight arrangement). Whether the data has been rearranged or not, the macroscopic storage area of ​​DDR to which it belongs will not change. Therefore, when the write module determines the memory start write address, it can directly inherit this high-order segment (i.e., AXI_MASTER_AWADDR[31:27] (which can represent the high 5 bits of the memory start write address) = AXI_SLAVE_AWADDR[31:27]), which simplifies the address generation logic and ensures the consistency of the data storage area.

[0126] AXI_SLAVE_AWUSER[26:6] can represent the base address offset corresponding to the functional module. AXI_MASTER_AWADDR[26:6] can represent the base address offset corresponding to the functional module extracted by the write module.

[0127] In this embodiment, the data size of a single frame of 8K resolution (8192×4320), 16-bit pixels (2 bytes per pixel) (i.e., the largest image) is 8192×4320×2 bytes = 73,728,000 bytes, which is 0x4380000 bytes in hexadecimal. The binary representation of 0x4380000 bytes requires 27 bits, meaning the total bit width for storing this image is 27 bits.

[0128] To meet the total bit width of 27 bits, the core offset segment of the write base address can be set to [26:6] (21 bits), while the high bit segment is [31:27] (5 bits), which can completely cover the bit 0~bit 26 range required for 0x4380000 bytes, and can meet the storage requirements of the largest image. Therefore, there is no need to extend the bit width, which simplifies the hardware design and avoids resource waste.

[0129] Writing modules can include, but are not limited to:

[0130] The write cache includes: multiple first cache units, each of which independently corresponds to a data stream.

[0131] In multimedia image processing scenarios, an image plane (such as the Y plane, U plane, and V plane of YUV) is itself an independent data stream. This is because the pixel data (luminance data Y, chrominance data U / V) of different planes need to be transmitted and stored independently, and subsequently read and processed independently. Therefore, each image plane needs to be allocated a first buffer unit as an independent data stream.

[0132] To adapt to the common bus transmission specification, the capacity of the first buffer unit can be equal to the burst transmission length. For example, if the burst transmission length is burst8, the capacity of the first buffer unit can be 128 bytes (i.e., 128 bytes) (which can be represented as B = 8 × 16 bytes = 128 bytes, where B represents the capacity of the first buffer unit). A further refined design could be: the first buffer unit could include 8 sub-buffer units (which can be represented as buffer lines), each sub-buffer unit being 128 bits (i.e., 16 bytes) in size (consistent with the single transmission width of the data transmission bus).

[0133] In this embodiment, the number of multiple first buffer units (which can be represented as S) can be based on the minimum requirements of the multimedia subsystem (e.g., ISP subsystem, video encoding subsystem, audio subsystem, etc.) and must cover the maximum number of parallel data streams in the target scenario (e.g., when supporting dual-channel YUV image processing, at least 6 sets of buffer lines are required, corresponding to 2 channels × 3 planes), to ensure that all parallel data streams have dedicated buffer units.

[0134] In this embodiment, as Figure 6 As shown, the first cache unit may include, but is not limited to, multiple sub-cache units.

[0135] AxID (data stream identifier): can be used to accurately route write requests from different data streams to the corresponding BufferLine, avoiding mismatches between multiple data streams.

[0136] Addr: Used to mark the starting address of memory when the current Buffer Line writes the second data stream to the AXI Master (i.e., the DDR controller).

[0137] Offset: For the write module, it is used to mark the current write offset address of the Buffer Unit, that is, the starting position of the next data to be written in the Buffer Unit. The offset is automatically accumulated after data is written to ensure continuous data storage.

[0138] The write reordering module is used to respond to the write request of the functional module for the first data stream. If the first data stream meets the preset conditions, the module reorders the first data stream to obtain a second data stream with valid data arranged continuously and closely, and caches the second data stream in the first target cache unit corresponding to the first data stream.

[0139] The first data stream may satisfy preset conditions, including but not limited to: there are invalid data gaps between pixels in the first data unit of the first data stream.

[0140] In this embodiment, the first data stream can be rearranged based on the size of the invalid data gaps between pixels in the first data unit. The specific rearrangement process can be found in the relevant description in Embodiment 1, and will not be repeated here.

[0141] The write reordering module can also be used to obtain the write base address corresponding to the functional module from the first control signal when the first target cache unit is full, offset the write base address corresponding to the functional module based on the cumulative number of writes of the first target cache unit to obtain the memory start write address, and write the second data stream in the first target cache unit into the memory using the memory start write address.

[0142] In this embodiment, each first cache unit can independently maintain a counter. Each time a full cache write operation to DDR is completed, the counter is incremented by 1 (initially 0), and the value of the counter is used as the cumulative number of writes.

[0143] In this embodiment, the address offset can be determined based on the cumulative number of writes to the first target cache unit and the capacity of the first target cache unit. The write base address corresponding to the functional module is offset based on the address offset to obtain the memory start write address.

[0144] In this embodiment, corresponding to the implementation where the functional module sends a first control signal containing the base address offset corresponding to the functional module and the initial write address in the write request to the write module, obtaining the write base address corresponding to the functional module from the first control signal may include, but is not limited to:

[0145] The write module can extract the high 5 bits from the initial write address according to XI_MASTER_AWADDR[31:27]=AXI_SLAVE_AWADDR[31:27], and concatenate them with AXI_MASTER_AWADDR[26:6] and AXI_MASTER_AWADDR[5:0] to obtain the write base address.

[0146] In this embodiment, there is no need to design separate caching logic for each functional module. Data temporary storage and reading are achieved solely through the first cache unit of the write module, further simplifying hardware design complexity and reducing chip area footprint. Furthermore, the standardized Burst8 write format is better suited to the page access characteristics of DDR memory, allowing multiple data blocks to be transferred continuously with a single page activation, further optimizing overall read and write performance.

[0147] Furthermore, by determining the number of multiple first buffer units according to the maximum parallel data flow requirements of the multimedia subsystem, physical isolation and parallel caching of multiple data streams can be achieved, avoiding interference between data streams from different image planes, and adapting to multi-channel and multi-plane image processing scenarios.

[0148] In this embodiment, the following section will describe the entire process of the write module (core is Write Burst Packer) writing the second data stream when processing different TransactionIDs, taking into account the core characteristics of 10-bit pixels (the first data stream is Burst8, and the rearranged second data stream is Burst5).

[0149] For example, such as Figure 7 As shown, the Transaction ID can be used to identify the processing task of the write module for each Burst8 input data. Different Transaction IDs correspond to different processes: "data reception → rearrangement → buffer filling → whether to read → output control → response feedback", as detailed below:

[0150] Transaction ID 0:

[0151] The Slave Input (AXI Slave Input Interface) receives 10-bit pixel Burst8 data (i.e., one implementation of the first data stream) sent by the receiving function module, which is then rearranged into Burst5 data (i.e., one implementation of the second data stream) by the Burst Packer (write reordering module).

[0152] Fill the Buffer Unit (first buffer unit) 0~4 (a total of 5 entries) with the complete Burst5 data.

[0153] After filling is complete, the total number of Burst5 data cached in the Buffer Line is checked. If there are only 5 data entries, which is less than "8 data entries" (i.e., the length of the Burst5 data cached in the Buffer Line is less than 128 bytes), an AXI BRESP (write response) is returned to the functional module to inform the functional module that the data cached in this batch has been completed.

[0154] Buffer Line Read (buffered read control module) does not need to read data and remains in a dummy (idle) state.

[0155] There is no need to output data to DDR (i.e., one implementation of memory) through Master Output (the only outlet for the write module to write data to DDR, following the AXI bus protocol), and Master Output also remains dummy.

[0156] The buffer line awaits subsequent data packets to be filled.

[0157] Transaction ID 1:

[0158] The Slave Input receives new Burst8 data, which is then converted into Burst5 data by the Burst Packer.

[0159] First, fill the first 3 entries of the Burst5 data into Buffer Units 5-7. At this point, the total number of entries in the Buffer Line reaches 8 (0-7). Read out the 8 entries of data in the Buffer Line (i.e., Buffer Line Read) and write them to DDR through MasterOutput.

[0160] During the Outstanding period of data writing to DDR (the concurrent processing window allowed by the bus), the remaining two entries of this Burst5 data are filled into Buffer Unit 0~1.

[0161] After the DDR returns an AXI BRESP indicating that the data writing is complete, the response is forwarded to the functional module to complete the entire process for this batch.

[0162] Transaction ID 2:

[0163] The Slave Input receives new Burst8 data and converts it into Burst5 packets.

[0164] Fill the Buffer Units 2-6 (5 records in total) with the complete Burst5 data.

[0165] The total number of cached entries is 5 (combined with the 2 remaining entries from before, totaling 7 entries). Since this is less than 8 entries, AXI BRESP is returned directly to the functional module.

[0166] Since there is no data output, Buffer Line Read and Master Output remain dummy.

[0167] Transaction ID 3:

[0168] Slave Input receives new Burst8 data and converts it to Burst5 data.

[0169] First, fill the first one of the Burst5 data into Buffer Unit 7. At this point, the total buffer size is 8 data entries. Then, read out the 8 data entries from the Buffer line and write them to DDR through Master Output.

[0170] During the Outstanding period, the remaining 4 entries from this Burst5 data are filled into Buffer Units 0-3.

[0171] After DDR returns AXI BRESP, it is forwarded by the write module to the function module.

[0172] Transaction ID 4:

[0173] Slave Input receives new Burst8 data and converts it to Burst5 data.

[0174] First, fill the first 4 entries of the Burst5 data into Buffer Units 4-7, and then fill the buffer with a total of 8 entries. Read the 8 entries of data from the Bufferline and write them into DDR.

[0175] During the Outstanding period, the remaining 1 record of this Burst5 data is filled into Buffer Unit 0.

[0176] The write module can forward the DDR's AXI BRESP to the function module.

[0177] Transaction ID 5:

[0178] Slave Input receives new Burst8 data and converts it to Burst5 data.

[0179] Fill the Buffer Units 1-5 (5 records in total) with the complete Burst5 data.

[0180] The total number of cached entries is 5 (6 in total, including the 1 remaining entry from before). Since this is less than 8 entries, AXI BRESP is returned directly to the functional module.

[0181] Since there is no data output, Buffer Line Read and Master Output remain dummy.

[0182] Transaction ID 6:

[0183] The Slave Input receives new Burst8 data and converts it to Burst5 data.

[0184] First, fill the first two entries of the Burst5 data into Buffer Units 6 and 7, and then fill the buffer with a total of 8 entries. Read the 8 entries of data from the Bufferline and write them to DDR.

[0185] During the Outstanding period, the remaining 3 entries of the current Burst5 data are filled into Buffer Units 0-2.

[0186] Forward the AXI BRESP from DDR to the function module.

[0187] Transaction ID 7:

[0188] The Slave Input receives new Burst8 data and converts it to Burst5 data.

[0189] The Burst5 data is completely filled into Buffer Units 3-7 (a total of 5 entries). Combined with the 3 remaining entries from before, the total number of entries in the buffer is 8. The 8 entries of data in the Buffer line are read out and written to DDR.

[0190] After DDR returns AXI BRESP, the write module forwards AXI BRESP to the function module to complete the batch processing.

[0191] It should be noted that while sending Burst8 data, the functional module can also send the write base address (Pack AddrTrans). When the buffer line is full of 8 data entries, the write module can determine the starting write address in memory based on the write base address, and write the data in the buffer line to DDR based on the starting write address in memory.

[0192] Of course, if no Burst8 data is received, Slave Input, Burst Packer, BufferLine Read, and Master Output all remain in an IDLE state.

[0193] As another optional embodiment of this application, an image data rearrangement device is provided in Embodiment 4 of this application. This embodiment is mainly an implementation of the write module in Embodiment 3. In this embodiment, the first control signal further includes a first control parameter; the first control parameter is used to indicate whether to perform rearrangement processing on the first data unit.

[0194] In this embodiment, the first control parameter may include, but is not limited to:

[0195] AWUSER[1:0] (pixel bit width identifier): Used to accurately represent the bit width type of pixels in the first data stream. 0 can represent byte-aligned pixels (no additional adaptation processing required); 1 can represent 10-bit pixels; 2 can represent 12-bit pixels; 3 can represent 14-bit pixels.

[0196] AWUSER[2] (Working mode control bit, i.e. AWUSER.PACK): Used to switch the core working mode of the write module. 0 can represent Bypass (bypass mode, i.e. no need to rearrange the first data stream); 1 can represent EnablePacker (enable packing mode, i.e. rearrange the first data stream).

[0197] Reference Figure 8 The write module may further include:

[0198] The first data path selection module is configured to respond to the second control signal by sending the first data stream to the write reordering module if the first control parameter indicates that the first data stream needs to be rearranged; and by directly writing the first data stream into the memory if the first control parameter indicates that the first data stream does not need to be processed.

[0199] In this embodiment, the functional module can send the write request and the first control signal (AWUSER signal) for the first data stream to the common write path; the first AXI Arbiter (first arbitration module) can arbitrate the write requests of multiple functional modules, determine the current transmission priority, and then forward the write request and the AWUSER signal to the AXI Slave interface (first input / output interface) of the Write Packer (write module).

[0200] The AXI DEMUX (first data path selection module) can parse the AWUSER signal: if AWUSER.PACK=0 (bypass mode, i.e., no need to process the first data stream), the write request is directly distributed to the AXI MUX (first response demultiplexing module); if AWUSER.PACK=1 (packing is required, i.e., reordering processing is required), the write request is distributed to the Write BurstPacker (i.e., write reordering module).

[0201] The Write Burst Packer receives the first data stream (e.g., Burst8 data), extracts the valid data and arranges it closely according to the pixel bit width identifier (10, 12, 14 bits) in the AWUSER signal, and converts it into the second data stream (e.g., Burst5 / 6 / 7 data), while recording the write base address carried by the AWUSER signal.

[0202] The Write Burst Buffer can write the second data stream (e.g., Burst5 / 6 / 7 data) into the corresponding first target cache unit in the Write Burst Buffer (i.e., multiple first cache units); when the first target cache unit is full, the write base address corresponding to the functional module is obtained from the first control signal, and the write base address corresponding to the functional module is offset based on the cumulative number of writes of the first target cache unit to obtain the memory start write address, and the second data stream in the first target cache unit is written to DDR at the memory start write address.

[0203] AXI MUX can aggregate data streams and send them to the second AXI Arbiter via the AXI Master (second input / output interface) of the Write Packer. The second AXI Arbiter (second arbitration module) can arbitrate the write data of multiple functional modules, determine the current transmission priority, and then forward the data to DDR. The AXI Master strictly follows the AXI protocol and writes data to DDR according to the memory start write address.

[0204] After the DDR completes the data writing, it can return an AXI Response via the AXI bus. This response is fed back to the AXI Slave via the WritePacker internal link, and then the AXI Slave returns it to the functional module to inform it that the write operation is complete.

[0205] In this embodiment, the working mode control bit in the first control signal can accurately distinguish between bypass mode (AWUSER.PACK=0) and packing mode (AWUSER.PACK=1), enabling the write module to adaptively process two types of pixel data streams: byte-aligned (e.g., 8-bit, 16-bit) and non-byte-aligned (10-bit, 12-bit, 14-bit). This eliminates the need for an additional independent processing module, avoiding redundant processing of byte-aligned data that does not require rearrangement, reducing unnecessary hardware overhead and bus latency. Furthermore, the pixel bit width identifier accurately identifies the pixel bit width, ensuring the targeted and accurate rearrangement processing, and further improving bandwidth utilization.

[0206] The introduction of the first data path selection module enables intelligent data flow splitting. Combined with the aggregation function of the first response demultiplexing module, the write module can seamlessly accommodate the differentiated write requirements of different functional modules while maintaining the uniformity of the common write path. This eliminates the need to modify the internal structure and bus topology of existing functional modules, significantly reducing hardware modification costs and compatibility risks.

[0207] As another optional embodiment of this application, an image data rearrangement device is provided in embodiment 5 of this application. This embodiment is mainly an implementation of the write rearrangement module in embodiment 3. In this embodiment, the first control signal may further include: a first identifier for identifying the first data stream.

[0208] The first identifiers of different first data streams are different.

[0209] The write reordering module caches the second data stream into the target cache unit corresponding to the first data stream, specifically including:

[0210] The write reordering module selects the cache unit corresponding to the first identifier from the plurality of cache units to determine the first target cache unit, and caches the second data stream into the target cache unit.

[0211] For example, the first cache unit described in Embodiment 3 contains an AxID (data stream identifier) ​​field. When the write reordering module receives the first identifier, it can search in multiple first cache units for a matching AxID (data stream identifier) ​​field.

[0212] If a match is found, the first cache unit to which the matched AxID (data stream identifier) ​​field belongs is the first target cache unit.

[0213] Of course, since each of the first cache units independently corresponds to one data stream, the functional module can also construct a mapping relationship between the first identifier and the first cache unit identifier. For the first data stream, the corresponding first target cache unit identifier is included in the first control signal and sent to the write module. Accordingly, the write module can directly determine the first target cache unit based on the first target cache unit identifier.

[0214] In this embodiment, the write module directly locks the first target cache unit by matching the first identifier, which can avoid the problem of multiple data stream confusion and overwriting caused by rough matching of timing or address, and adapt to complex scenarios such as multiple image planes.

[0215] Furthermore, by comparing the first identifier in the first control signal with the existing AxID field in the first cache unit, no new mapping table is needed, which can reduce hardware modification costs and design complexity.

[0216] As another optional embodiment of this application, an image data rearrangement device is provided in embodiment 6 of this application. This embodiment is mainly an implementation of the reading module in embodiment 1. In this embodiment, the functional module is used to offset the initial reading address in the reading request based on the base address offset corresponding to the functional module, to obtain the reading base address corresponding to the functional module, and to issue a second control signal containing the reading base address corresponding to the functional module.

[0217] The read module may include:

[0218] The read cache includes multiple second cache units, each of which independently corresponds to a data stream.

[0219] In this embodiment, the second buffer unit can be understood with reference to the first buffer unit described above, and will not be repeated here. The difference is that the Offset in the second buffer unit can be used to mark the starting position of the next data read in the Buffer Unit. The offset is automatically accumulated after data is read to ensure sequential reading.

[0220] The read request processing module is configured to respond to the second control signal, offset the read base address corresponding to the functional module based on the capacity of the second cache unit and the number of times the functional module reads from the memory, obtain the memory start read address, and read the second data unit from the memory at the memory start read address and store it in the second target cache unit corresponding to the first data stream until the first number is reached.

[0221] In this embodiment, a counter (initially 0) can be maintained independently for each second cache unit corresponding to the first data stream. The counter is incremented by 1 only when the cached data in the second cache unit is insufficient to meet the unpacking requirement once (i.e., the number of second data units in the second cache unit has not reached the second quantity), and the read request initiated by the read request processing module to the memory has been successfully received and returned. The value of this counter can be used as the number of times the functional module reads from the memory.

[0222] In this embodiment, the offset can be obtained by multiplying the capacity of the second cache unit by the number of times the functional module is read from the memory.

[0223] The starting read address of the memory is obtained by offsetting the read base address corresponding to the functional module based on the offset.

[0224] The read reordering module is used to read the second data units from the second target cache unit in the order of the cache until the second quantity is reached, convert the second quantity of second data units into the initial reordering format, and send it to the functional module.

[0225] The detailed process of converting the second number of second data units into the initial arrangement format can be found in the relevant description in Embodiment 1, and will not be repeated here.

[0226] In this embodiment, the reading and unpacking process of the reading module when processing different Transaction IDs will be described below, taking into account the core characteristics of 10-bit pixels (the first data stream is Burst8, and the rearranged second data stream is Burst5).

[0227] For example, such as Figure 9 As shown, Transaction ID 0:

[0228] The Slave Input (CMD, command input interface) detects the Burst8 read request issued by the functional module and sends the read base address (Addr Trans) in the read request to the read request processing module (Master Output (CMD)).

[0229] The Master Output (CMD) determines the starting address of the memory read based on the read base address and sends a read instruction (DDR read) containing the starting address of the memory read to the DDR.

[0230] DDR returns eight second data units through the Master Input (RSP), i.e., the data input interface, and caches the eight second data units sequentially into Buffer Line Buffer Units 0~7.

[0231] After the Buffer Line is filled, five core data entries (corresponding to one Burst5 data stream, 640 bits of valid data) are extracted from Buffer Units 0-4 and sent to the Burst Unpacker (read reordering module). The Burst Unpacker unpacks them into Burst8-byte aligned data and feeds it back to the functional module through Slave Output. Buffer Units 5-7 retain the remaining data for subsequent ID unpacking.

[0232] Transaction ID 1:

[0233] The Slave Input (CMD) receives new Burst8 read requests. The process of reading data from DDR can be found in the relevant description in Transaction ID0, and will not be repeated here.

[0234] There are 3 residual data entries in Buffer Units 5-7, which is less than 5 entries required for unpacking. The Master Output (CMD) sends a new Burst8 read command to the DDR. Of the 8 second data units returned by the DDR, the first 5 entries are cached in Buffer Units 0-4 (to complete the cache).

[0235] Five data entries are extracted from Buffer Units 5-7 (3 remaining entries) + Buffer Units 0-1 (2 newly filled entries), unpacked into Burst8 format by Burst Unpacker, and then fed back to the functional module via Slave Output.

[0236] The last three unused data entries from the DDR return data are cached in Buffer Units 5-7.

[0237] Transaction ID 2:

[0238] The Slave Input (CMD) receives a Burst8 read request (ARID=2). The Burst Unpacker can synchronously check the Buffer Line status. At this time, Buffer Units 2-6 all have valid data (a total of 5 records, meeting the unpacking requirements), so there is no need to trigger the Master Output (CMD) to initiate a new read request to DDR. Both the Master Output (CMD) and the Master Input (RSP) can remain dummy.

[0239] Five data entries are extracted directly from Buffer Units 2-6 (i.e., Buffer Read Directly), unpacked into Burst8 format by BurstUnpacker, and then fed back to the functional module via Slave Output.

[0240] Transaction ID 3:

[0241] The Slave Input (CMD) receives a Burst8 read request, reads 8 second data units from DDR, and buffers the first 7 data units (i.e., the first 7 second data units) into Buffer Line Buffer Units 0 to 6.

[0242] Five data entries are extracted from Buffer Unit 7 (1 remaining entry) + Buffer Units 0~3 (4 newly cached entries), unpacked by Burst Unpacker, and then fed back to the functional module.

[0243] The last unused data from the DDR return is cached in Buffer Unit 7.

[0244] Transaction ID 4:

[0245] The Slave Input (CMD) receives Burst8 read requests. The first four data entries returned by DDR are first filled into BufferUnit 0~3.

[0246] Extract 5 data entries from Buffer Units 4-7 (4 remaining entries) + Buffer Unit 0 (1 newly cached entry), and then unpack and feed them back to the function module.

[0247] The last four unused data entries from the DDR return data are cached in Buffer Units 4-7.

[0248] Transaction ID 5:

[0249] The Slave Input (CMD) receives the Burst8 read request. The Read Unpacker checks the Buffer Line status. At this time, Buffer Units 1 to 5 all have valid data (a total of 5 records, which meets the unpacking requirements), so there is no need to send a new read request to DDR.

[0250] Five data entries are extracted directly from Buffer Units 1-5, unpacked by Burst Unpacker, and then fed back to the functional module via SlaveOutput.

[0251] Transaction ID 6:

[0252] The Slave Input (CMD) receives Burst8 read requests.

[0253] The current Buffer Units 6-7 are empty, with less than 5 unpacking requests. The Master Output (CMD) sends a new Burst8 read command to the DDR. The first 6 data returned by the DDR are first buffered in Buffer Units 0-5.

[0254] Five entries are extracted from Buffer Units 6-7 (reserved) + Buffer Units 0-2 (3 new entries), and then unpacked by Burst Unpacker and fed back to the functional module.

[0255] The last two unused data entries from the DDR return data are cached in Buffer Units 6 and 7.

[0256] Transaction ID 7:

[0257] The Slave Input (CMD) receives a Burst8 read request (ARID=7). The Read Unpacker checks the BufferLine status. At this time, there is valid data in Buffer Units 3 to 7 (a total of 5 records, which meets the unpacking requirements), so there is no need to send a new read request to DDR.

[0258] Five data entries are extracted directly from Buffer Units 3-7, unpacked into Burst8 format by Burst Unpacker, and then fed back to the functional module via Slave Output.

[0259] In this embodiment, by configuring an independent second cache unit for each data stream, physical isolation storage of multiple data streams is achieved. This avoids data confusion and overwriting issues when multiple image planes or multi-functional modules read data in parallel, ensuring that the tightly packed data of each data stream is cached sequentially and processed independently. Furthermore, the second cache unit is precisely adapted to the difference between the DDR transfer rate and the processing rate of the read-reordering module. By temporarily storing the second data unit returned from DDR, the risk of processing interruption or data loss caused by the rate mismatch is mitigated. This allows the read-reordering module to read sufficient data from the cache (reaching the second quantity) as needed before starting unpacking, significantly improving the stability of data transmission and format conversion.

[0260] As another optional embodiment of this application, an image data rearrangement device is provided in embodiment 7 of this application. This embodiment is mainly an implementation of the reading module in embodiment 1. In this embodiment, the functional module is used to offset the initial reading address in the reading request based on the base address offset corresponding to the functional module, to obtain the reading base address corresponding to the functional module, and to issue a second control signal containing the reading base address corresponding to the functional module.

[0261] The read module may include:

[0262] The read request processing module is configured to, in response to the second control signal, convert the read base address into the memory address range corresponding to the second data stream according to the target arrangement format of the second data stream.

[0263] In this embodiment, the total length of the second data stream can be determined according to the target arrangement format of the second data stream, that is, by multiplying the bit width (128 bits) of a single second data unit by the second quantity (e.g., 5 for Burst5, 6 for Burst6, and 7 for Burst7).

[0264] In this embodiment, the value obtained by subtracting one from the total length of the second data stream can be added to the read base address to obtain the end address of the memory address range. The read base address can be used as the starting address to obtain the memory address range (i.e., read base address ~ read base address + (total length of the second data stream - 1)).

[0265] For example, if the second data stream is in Burst5 format, the functional module needs to read 64 independent valid pixels of 10 bits each (total valid data amount of 640 bits). The corresponding second data stream is 1 Burst5 (5 128-bit units, total length of 640 bits). Therefore, the memory address range = read base address ~ read base address + 639 bits (including the starting address, total of 640 bits).

[0266] The read request processing module can also be used to read the second data stream from the memory based on the memory address range.

[0267] The read request processing module can issue a read command to the memory, explicitly requesting the reading of the second data unit within the address range of read base address to read base address + (total length of the second data stream - 1).

[0268] The memory can return the second number of second data units according to the read command, without any cache or temporary storage.

[0269] The read reordering module is used to convert the second data stream into the initial reordering format and send it to the functional module.

[0270] The detailed process of converting the second number of second data units into the initial arrangement format can be found in the relevant description in Embodiment 1, and will not be repeated here.

[0271] In this embodiment, the read request processing module responds to the second control signal and converts the read base address into the memory address range corresponding to the second data stream according to the target arrangement format of the second data stream; and, based on the memory address range, reads the second data stream from the memory, which can eliminate the additional latency caused by cached storage and read / write, and significantly reduce the end-to-end data transmission latency compared to the cached solution, adapting to timing-sensitive scenarios such as real-time output of the display controller and low-latency video decoding.

[0272] Meanwhile, the cacheless design significantly reduces storage resource consumption, simplifies hardware architecture and control logic, and reduces chip area and power consumption.

[0273] As another optional embodiment of this application, an image data rearrangement device is provided in embodiment 8 of this application. This embodiment is mainly an implementation of the reading module in embodiment 6. In this embodiment, the functional module issues a second control signal containing the reading base address corresponding to the functional module, including: the functional module issues a second control signal according to the data arrangement method corresponding to the initial reading address in the reading request; the second control signal may include a second control parameter, which is used to indicate whether to perform conversion processing on the reading base address.

[0274] When a functional module initiates a read request, it first determines the arrangement of the data to be read: if it is reading byte-aligned data (such as 8-bit / 16-bit pixels with no invalid gaps), it sets the second control parameter to 0; if it is reading tightly packed data (such as 10-bit / Burst5), it sets the second control parameter to 1 and sends a second control signal containing the read base address.

[0275] The read module also includes:

[0276] The command path selection module is configured to respond to the second control signal. If the second control parameter indicates that the read base address needs to be converted, the second control signal is sent to the read request processing module. If the second control parameter indicates that the initial read address needs to be converted, the second data unit is read directly from the memory based on the initial read address.

[0277] If the second control parameter = 1 (requiring conversion), it can be determined that the data is in a tightly packed format and needs to be unpacked. The second control signal is forwarded to the read request processing module, which can calculate the address offset based on the capacity of the second cache unit and the number of reads, generate the memory start read address, read the second data stream from DDR and store it in the second target cache unit, according to the logic of Example 6.

[0278] If the second control parameter = 0 (conversion processing disabled), the data can be determined to be in byte-aligned format, and unpacking is unnecessary. Data is read directly from the initial read address, skipping the address translation and cache allocation in the read request processing module, thus achieving bypass processing.

[0279] The read module may further include:

[0280] The second data path selection module is used to store the second data unit read from the memory into the second cache unit corresponding to the functional module or to directly return the second data unit read from the memory to the functional module according to the data arrangement of the second data unit read from the memory.

[0281] If the read data is tightly packed (corresponding to the second control parameter = 1), the second data path selection module can distribute the second data unit to the corresponding second target cache unit, and then feed it back to the function module after being unpacked by the read reordering module.

[0282] If the read data is byte-aligned (corresponding to the second control parameter = 0), the second data path selection module can directly forward the second data unit to the functional module, skipping the second buffer unit and the read reordering module, and return directly to the functional module through the AXI Slave, thus achieving low-latency bypass.

[0283] In this embodiment, refer to Figure 10 The read module may further include:

[0284] AXI Slave: The command and data input interface of Read Unpacker, which receives read requests from functional modules (such as ISP and GPU) and serves as the output interface for unpacked data, returning the byte-aligned data stream to the functional modules.

[0285] AXI Master: The core interface for the interaction between Read Unpacker and DDR. It is responsible for sending the read command (including the memory start address) modified by Read Burst CMD to DDR, and at the same time receiving the second data unit returned by DDR. It is the only channel for command issuance and data reception between Read Unpacker and DDR.

[0286] AXI MUX (RSP): Response multiplexing module, responsible for aggregating the processing results of two paths: one is the "unpacking path" (byte-aligned data stream after being buffered by Read Burst Buffer and unpacked by Read Burst Unpacker), and the other is the "bypass path" (data stream read directly from DDR without unpacking). Both types of data streams are returned to the upstream IP through AXI Slave, realizing centralized output of the response.

[0287] Among them, functional modules (such as GPUs) generate Burst8 read requests (including ARUSER signals, i.e., the second control signals) due to processing needs, and send them to the public read path through the AXI Master interface.

[0288] The first AXI Arbiter can arbitrate read requests from multiple functional modules and then forward the requests to the AXI Slave interface of ReadUnpacker.

[0289] The command path selection module, namely AXI DEMUX (CMD), parses the ARUSER signal in the read request: if ARUSER.UNPACK=0 (bypass mode), the request is directly dispatched to AXI MUX (CMD); if ARUSER.UNPACK=1 (requires unpacking), the request is dispatched to Read Burst CMD.

[0290] The Read Burst CMD (i.e., the read request processing module) receives the read request that needs to be unpacked. Based on the pixel width and base address offset in the ARUSER signal, it modifies the initial read address (to adapt to the address rules of closely packed data in DDR), generates the memory start read address, and then sends the read command containing the memory start read address to AXI MUX (CMD).

[0291] AXI MUX (CMD) can aggregate read commands from the "bypass path" and the "address-translated path" and send them to the second AXI Arbiter (second arbitration module) through the AXI Master interface of ReadUnpacker.

[0292] The second AXI Arbiter can arbitrate read commands from multiple functional modules, determine the current transmission priority, and forward the read command to the DDR. The DDR returns the second data unit based on the memory start read address, which is then received by the AXI Master.

[0293] AXI DEMUX (RSP) (i.e., the second data path selection module) stores the second data units read from the memory into the corresponding second cache unit in the Read BurstBuffer (i.e., multiple second cache units) according to the data arrangement of the second data units read from the memory, or directly returns the second data units read from the memory to the functional module to parse the RID in the returned data and distribute it to the corresponding Read BurstBuffer (matching the corresponding cache unit in the case of multiple data streams); the Read Burst Buffer caches the tightly packed Burst8 data stream and extracts the Burst5 / 6 / 7 data streams as needed (matching the pixel bit width).

[0294] The Read Burst Unpacker extracts the Burst5 / 6 / 7 data stream (matching pixel bit width) from the Read Burst Buffer as needed.

[0295] Read Burst Unpacker unpacks Burst5 / 6 / 7 data streams into byte-aligned Burst8 data streams (high bits retain valid data, low bits are padded with zeros).

[0296] AXI MUX (RSP) can aggregate the "unpacked data stream" and the "bypass data stream" and return them to the functional module through the AXI Slave interface; after receiving the byte-aligned data, the functional module completes the read process.

[0297] In this embodiment, the image data rearrangement device may further include a control module (i.e., AxUSER Plugin).

[0298] The control module can be integrated into the AXI Master output of the functional module to process and modify the initial control signal of the AXI Master output to obtain the first control signal or the second control signal.

[0299] The control module sends the first control signal or the second control signal to the write module or the read module.

[0300] In this embodiment, the AxUSER Plugin is integrated into the AXI Master output rather than inside the functional module. This eliminates the need to modify the core logic and hardware structure of the functional module. Signal processing is completed solely through an external control module at the output. This avoids the R&D risks and costs associated with custom IP modifications (especially since customizing externally purchased IP is extremely difficult) and ensures the original stability of the functional module.

[0301] In this embodiment, different functional modules may have different read / write requirements for the memory. For example, such as Figure 11 As shown, IP0 in IP0_WRAP, which encapsulates control and functional modules, can complete its own processing without reading data from memory. It only needs to write the processed results to memory for use by other IPs. Therefore, IP0 can send write requests through a public write path.

[0302] In IP1_WRAP, which encapsulates control and functional modules, the core function of IP1 (e.g., the display controller) is to perform subsequent processing and output based on existing image data. It does not need to generate new data to write to memory; it only needs to read data already stored by other IPs from DDR. Therefore, IP1 can send read requests through a common read path and retrieve data from memory.

[0303] The core function of IP2 in IP2_WRAP, which encapsulates both control and functional modules, or IP3 in IP3_WRAP, which encapsulates both control and functional modules, is to read existing data, process it, and then store new data. This requires first acquiring the raw data, processing it, and then saving the result. For example, an Image Signal Processor (ISP) needs to first read the raw image data captured by the camera from DDR (read request), perform noise reduction, sharpening, and other processing, and then write the optimized image data to DDR (write request). Or,

[0304] Graphics Processing Unit (GPU): It needs to read texture data and model data from DDR (read request), perform rendering processing, and then write the rendering results back to DDR (write request).

[0305] Therefore, IP2 can send write requests through the public write path, send read requests through the public read path, and retrieve data from memory.

[0306] Of course, some functional modules (which can be represented as IP4 in IP4_WRAP, which encapsulates control and functional modules) can process multiple independent image data streams (multiple image planes) simultaneously. Each data stream corresponds to an independent read / write interaction, thus generating multiple write requests and / or read requests. For example, an ISP that supports multi-channel video processing can process raw data from two cameras simultaneously. Each data stream needs to initiate a separate read request (to read its own raw data) and a write request (to store its own processing results), thus forming multiple read / write requests.

[0307] In another embodiment of this application, an image data rearrangement system is provided, comprising:

[0308] A functional module is used to initiate a write request or a read request; the write request corresponds to a first data stream; the first data stream includes a first number of first data units and the first data units are arranged in an initial format.

[0309] The control module is connected to the functional module, a write module deployed on a common write path between the multiple functional modules and the memory, and a read module deployed on a common read path between the multiple functional modules and the memory, and is used to output a first control signal or a second control signal according to the write request or the read request.

[0310] The first arbitration module, connected to the control module, the write module, and the read module, is used to dispatch the first control signal or the second control signal corresponding to the write request or read request initiated by the multiple functional modules to the write module or the read module.

[0311] The second arbitration module, connected to the write module, the read module, and the memory, is used to coordinate the write module or the read module to write data to or read data from the memory.

[0312] The write module is configured to, in response to the first control signal, rearrange the first data stream to obtain a second data stream, and write the second data stream into the memory; the second data stream includes a second number of second data units; the second data units are arranged in a target arrangement format; the target arrangement format is that valid data is arranged continuously and closely.

[0313] The read module is configured to, in response to the second control signal, read the second data stream from the memory, convert the second data stream into the initial arrangement format, and send it to the functional module.

[0314] The functions and interactions of each module can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0315] In another embodiment of this application, an image data rearrangement method is provided, which may include, but is not limited to, the following steps:

[0316] Step S101: The write module deployed on the common write path between multiple functional modules and the memory responds to the write request of the functional module for the first data stream, rearranges the first data stream to obtain a second data stream, and writes the second data stream into the memory; the first data stream includes a first number of first data units and the first data units are arranged in an initial arrangement format; the second data stream includes a second number of second data units; the second data units are arranged in a target arrangement format; the target arrangement format is that the effective data is arranged continuously and closely.

[0317] Step S102: The read module deployed on the common read path between the multiple functional modules and the memory responds to the read request of the functional module, reads the second data stream from the memory, converts the second data stream into the initial arrangement format, and sends it to the functional module.

[0318] In this embodiment, the detailed process of steps S101-S102 can be referred to the relevant description in the image data rearrangement device, and will not be repeated here.

[0319] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0320] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0321] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0322] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An image data rearrangement apparatus, comprising: a write module disposed on a common write path between a plurality of functional modules and a memory, configured to perform rearrangement processing on a first data stream in response to a write request of the first data stream from the functional modules, to obtain a second data stream, and to write the second data stream into the memory; the first data stream comprises a first number of first data units, and the first data units are in an initial arrangement format; the second data stream comprises a second number of second data units; and the second data units are in a target arrangement format; and the target arrangement format is a valid data continuous and compact arrangement format; a read module disposed on a common read path between the plurality of functional modules and the memory, configured to read the second data stream from the memory in response to a read request of the functional modules, to convert the second data stream into the initial arrangement format, and to send the second data stream to the functional modules. 2.The image data rearrangement apparatus of claim 1, wherein the write module writes the second data stream into the memory, comprising: the write module buffers the second data stream, and writes the buffered second data stream into the memory when a length of the buffered data meets a preset address alignment condition. 3.The image data rearrangement apparatus of claim 1, wherein the functional modules are configured to offset an initial write address in the write request based on a base address offset corresponding to the functional modules, to obtain a write base address corresponding to the functional modules, and to issue a first control signal comprising the write base address corresponding to the functional modules; the write module, comprising: a write buffer comprising a plurality of first buffer units, each of the first buffer units independently corresponding to a data stream; a write rearrangement module configured to perform rearrangement processing on a first data stream in response to a write request of the first data stream from the functional modules, to obtain a second data stream in a valid data continuous and compact arrangement format, and to buffer the second data stream in a first target buffer unit corresponding to the first data stream, if the first data stream meets a preset condition; and when the first target buffer unit is full, obtaining the write base address corresponding to the functional modules from the first control signal, offsetting the write base address corresponding to the functional modules based on a cumulative write-out number of the first target buffer unit, to obtain a memory start write address, and writing the second data stream in the first target buffer unit into the memory using the memory start write address. 4.The image data rearrangement apparatus of claim 3, wherein the first control signal further comprises a first control parameter; and the first control parameter is used to indicate whether to perform rearrangement processing on the first data units; the write module, further comprising: a first data path selection module configured to send the first data stream to the write rearrangement module if the first control parameter indicates that the first data stream needs to be rearranged, in response to the second control signal. if the first control parameter represents that the first data stream does not need to be processed, the first data stream is directly written into the memory.

5. The image data rearranging apparatus of claim 3, the first control signal further comprising: a first identifier for identifying the first data stream; the write rearrangement module caches the second data stream to a first target cache unit corresponding to the first data stream, including: the write rearrangement module selects a cache unit corresponding to the first identifier from the plurality of cache units as a target cache unit, and caches the second data stream to the first target cache unit.

6. The image data rearrangement apparatus according to claim 1, wherein the function module is configured to offset an initial read address in the read request based on a base address offset corresponding to the function module to obtain a read base address corresponding to the function module, and issue a second control signal containing the read base address corresponding to the function module. the read module, comprising: a read cache, including: a plurality of second cache units, each of the second cache units independently corresponding to a data stream; a read request processing module configured to, in response to the second control signal, offset the read base address corresponding to the function module based on a capacity of the second cache unit and a number of times of reading from the memory for the function module to obtain a memory start read address, and read the second data unit from the memory to a second target cache unit corresponding to the first data stream at the memory start read address until the first number is reached. a read rearrangement module configured to read the second data unit from the second target cache unit in a cached order until the second number is reached, convert the second number of second data units into the initial arrangement format, and send to the function module.

7. The image data rearrangement apparatus according to claim 1, wherein the function module is configured to offset an initial read address in the read request based on a base address offset corresponding to the function module to obtain a read base address corresponding to the function module, and issue a second control signal containing the read base address corresponding to the function module. the read module, comprising: a read request processing module configured to, in response to the second control signal, convert the read base address into a memory address range corresponding to the second data stream according to the target arrangement format of the second data stream, and read the second data stream from the memory based on the memory address range; a read rearrangement module configured to convert the second data stream into the initial arrangement format and send to the function module.

8. The image data rearranging apparatus of claim 6, the second control signal further comprising: a first identifier for identifying the first data stream; the read request processing module reads the second data unit from the memory to a second target cache unit corresponding to the first data stream at the memory start read address, including: the read request processing module selects a cache unit corresponding to the first identifier from the plurality of cache units as the second target cache unit corresponding to the first data stream, and reads the second data unit from the memory to the second target cache unit at the memory start read address.

9. An image data rearrangement system, comprising: a function module configured to initiate a write request or a read request; the write request corresponds to a first data stream; the first data stream comprises a first number of first data units and the first data units are in an initial arrangement format; a control module connected to the function module, a write module deployed on a common write path between the function module and a memory, and a read module deployed on a common read path between the function module and the memory, configured to output a first control signal or a second control signal according to the write request or the read request; a first arbitration module connected to the control module, the write module and the read module, configured to schedule the first control signal or the second control signal corresponding to the write request or the read request initiated by the function module to the write module or the read module; a second arbitration module connected to the write module, the read module and the memory, configured to coordinate the write module or the read module to write data to or read data from the memory; the write module, configured to perform rearrangement processing on the first data stream in response to the first control signal, to obtain a second data stream, and write the second data stream into the memory; the second data stream comprises a second number of second data units; the second data units are in a target arrangement format; and the target arrangement format is a continuous and compact arrangement of valid data; the read module, configured to read the second data stream from the memory in response to the second control signal, convert the second data stream into the initial arrangement format, and send the second data stream to the function module.

10. An image data rearrangement method, comprising: a write module deployed on a common write path between a plurality of function modules and a memory, configured to perform rearrangement processing on a first data stream in response to a write request of the function module for the first data stream, to obtain a second data stream, and write the second data stream into the memory; the first data stream comprises a first number of first data units and the first data units are in an initial arrangement format; the second data stream comprises a second number of second data units; the second data units are in a target arrangement format; and the target arrangement format is a continuous and compact arrangement of valid data; a read module deployed on a common read path between the plurality of function modules and the memory, configured to read the second data stream from the memory in response to a read request of the function module, convert the second data stream into the initial arrangement format, and send the second data stream to the function module.

Images