Volume video parallel coding and decoding accelerator based on octree partitioning

Abstract

The invention discloses a volume video parallel coding and decoding accelerator based on octree partitioning, and relates to the field of volume video coding and decoding, the volume video parallel coding and decoding accelerator comprises an interpolation module and an MLP calculation module of a binary multi-resolution Hash grid, the binary multi-resolution Hash grid is obtained by adopting an octree partitioning strategy to cut volume elements in a volume video, and the MLP calculation module is used for calculating the volume elements in the volume video. The interpolation module is used for processing linear interpolation calculation of the binarized multi-resolution hash grid to obtain a multi-dimensional interpolation vector, and the MLP calculation module adopts a reconfigurable systolic array, distributed access and layer fusion technology to perform MLP calculation on the interpolation vector in a pipelined manner by taking a block as a unit to obtain a multi-dimensional interpolation vector; and predicting entropy model information output by the entropy model corresponding to the characteristic value of each individual primitive. According to the method, the problems of high calculation complexity, extremely high requirements on calculation resources and bandwidth, difficulty in realizing efficient deployment on edge calculation equipment with limited resources and the like in a volume video encoding and decoding process are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of volume video coding, in particular to a volume video parallel coding accelerator based on octree partitioning. BACKGROUND

[0002] With the rapid development of new generation immersive media technology, volume video (also known as holographic video) as a new media form capable of truly reproducing three-dimensional scenes is being applied more and more widely. Compared with traditional two-dimensional video, volume video expresses through multiple frames of static three-dimensional scenes, each frame represents a three-dimensional scene through thousands to millions of volume primitives, each volume primitive contains three-dimensional spatial coordinates and multiple attribute parameters, thereby providing a more immersive visual experience. However, this data expression method also brings a huge data volume challenge. The cost of storing and transmitting uncompressed volume video data is extremely high, which seriously limits its practical application. Therefore, the volume video codec actually encodes and decodes the attribute parameters of a large number of irregular volume primitives.

[0003] As a core component of modern compression systems, entropy coding realizes efficient compression representation of information by utilizing the statistical redundancy characteristics in data. The current mainstream volume video coding methods mainly include two categories: one is the PNG-based compression method, which maps volume primitive parameters to a two-dimensional grid and processes them with mature lossless image coding technology, which has the advantage of fast coding and decoding speed, but the compression efficiency is limited; the other is the ANS (Asymmetric Numeral Systems) compression method based on context modeling, which can realize near-optimal compression performance through three key steps of hash grid neural network prediction of parameter distribution, adaptive quantization adjustment and entropy coding, and is the most advanced compression technology at present.

[0004] However, although the ANS compression method has a significant compression efficiency, it has a high computational complexity, especially in the aspects of hash grid interpolation and multi-layer perceptron (MLP) calculation, which presents obvious hardware implementation challenges. The parameters of the hash grid usually reach MB level, and the interpolation process requires a large number of irregular memory accesses, resulting in a memory bottleneck; while the MLP calculation involves a large number of multiply-accumulate operations and intermediate data exchange, which requires a high demand for computing resources and bandwidth, making it difficult to achieve efficient deployment on resource-constrained edge computing devices.

[0005] It is worth noting that volume video data usually has high sparsity and spatial locality characteristics in three-dimensional space, which provides an important optimization space for parallel hardware acceleration. How to fully utilize these characteristics and design an efficient parallel coding acceleration architecture has become an important direction of current research. SUMMARY

[0006] The application aims to provide an octree partitioning-based volumetric video parallel encoding and decoding accelerator aiming at the above-mentioned technical problems.

[0007] In a first aspect, the application provides an octree partitioning-based volumetric video parallel encoding and decoding accelerator, comprising an interpolation module of a binary multi-resolution hash grid and an MLP calculation module, the binary multi-resolution hash grid being obtained by cutting a volumetric primitive in a volumetric video by using an octree partitioning strategy, the interpolation module being used for processing linear interpolation calculation of the binary multi-resolution hash grid to obtain a multi-dimensional interpolation vector, and the MLP calculation module being used for pipeline processing of MLP calculation on the interpolation vector in a block unit by using a reconfigurable systolic array, distributed memory access and layer fusion technology, and predicting entropy model information output by an entropy model corresponding to a feature value of each volumetric primitive.

[0008] As a preferred, the specific process of the octree partitioning strategy is as follows:

[0009] The three-dimensional coordinates of each volumetric primitive in the volumetric video are normalized and quantized into unsigned integer representation space coordinates according to the accuracy requirement;

[0010] The space coordinates are converted by Morton coding to obtain corresponding Morton codes, and all the volumetric primitives are sorted based on the Morton codes to obtain a volumetric primitive linear octree structure;

[0011] The depth of the octree is determined, the volumetric primitive linear octree structure is linearly octree partitioned based on the depth of the octree to obtain a plurality of space blocks corresponding to different depths, the space blocks containing invalid volumetric primitives are identified and removed, only the space blocks containing valid volumetric primitives are reserved, the Morton code corresponding to the three-dimensional coordinates of each volumetric primitive in each space block containing valid volumetric primitives is combined with the binary hash table to construct a corresponding local hash grid, and all the local hash grids constitute a binary multi-resolution hash grid.

[0012] As a preferred, the interpolation module comprises a control unit, a point storage unit, a hash storage unit, a calculation unit, a pipeline scheduling unit and a result cache unit; the control unit is used for receiving and decoding external instructions, initiating off-chip memory access requests according to the instructions, and determining basic parameters of the calculation unit and the pipeline scheduling unit in the working process; the pipeline scheduling unit is used for generating point coordinate reading control information and vertex reading control information respectively and sending them to the point storage unit and the hash storage unit to read the point coordinates and the feature values of the vertices, generating calculation pipeline control information and sending it to the calculation unit in combination with the point coordinates and the feature values of the vertices, the calculation unit comprising a multiplier and an accumulator, is used for completing weighted summation between the point coordinates and the feature values of the vertices according to the mathematical principle of interpolation to obtain interpolation results and directly store them in the result cache unit, and the point coordinates exist in the form of the Morton code corresponding to the three-dimensional coordinates of the volumetric primitive.

[0013] As preferred, the hash storage unit adopts a multi-bank structure including B banks, the hash storage unit is configured to store hash tables of different resolutions in different banks in a cyclic distribution manner, each vertex in the local hash grid stores a 4-bit feature value; the pipeline scheduling unit includes a prefetch buffer module working in a ping-pong operation mode; the prefetch buffer module has B register groups, each register group has a corresponding bank, and the register groups are input to the prefetch buffer module in parallel from multiple bank dimensions, the prefetch buffer module retrieves the local hash grid of the resolution of the current processing input point coordinate according to the input point coordinate, generates addresses of 8 vertices on the basic unit of the local hash grid of the resolution of the current processing input point coordinate in series through a hash mapping function, and gradually transmits 8 4-bit feature values stored on the 8 vertices to the register groups.

[0014] As preferred, the calculation unit is configured to expand and induce all expressions of the product of the distances from the point to the vertices in each dimension into 8 similar terms, aggregate the feature values of the vertices as coefficients, realize the merging of the same type of transformation, take the product of the distances from the point to the vertices as elements, and calculate the interpolation result.

[0015] As preferred, the reconfigurable systolic array includes a plurality of 4x4 systolic array tensor calculation cores, and the 4x4 systolic array tensor calculation core supports multiplication and accumulation operations.

[0016] As preferred, the 4x4 systolic array tensor calculation core includes a memory control unit, a data delay unit, a calculation control unit and a 4x4 calculation array, the memory control unit is configured to detect the arrival of the input interpolation result and trigger the weight reading operation, the data delay unit is configured to input the weight and the corresponding interpolation result to the calculation array to perform linear layer calculation by using a data arrangement mechanism, and the calculation control unit is configured to control whether the ReLU calculation needs to be performed during the calculation of the calculation array and the time of outputting the result after the calculation of the linear layer.

[0017] As preferred, the calculation array includes 4 rows and 4 columns of basic calculation units, the basic calculation units are configured to perform multiplication and accumulation operations on the input weight and the corresponding interpolation result, and a ReLU operator is configured to perform ReLU calculation.

[0018] As preferred, the 4*4 systolic array tensor calculation core further comprises a weight input selection unit, the weights are stored in a distributed on-chip SRAM, the distributed on-chip SRAM comprises a plurality of private storage blocks, each 4*4 systolic array tensor calculation core has a corresponding private storage block, each row of 5 4*4 systolic array tensor calculation cores shares 5 private storage blocks through a switch cross interconnection matrix, and the weight input selection unit is used to select the weights input to each 4*4 systolic array tensor calculation core from the corresponding private storage block or the private storage blocks corresponding to the remaining 4 4*4 systolic array tensor calculation cores in each row of 5 4*4 systolic array tensor calculation cores.

[0019] As preferred, a FIFO buffer is inserted between linear layers, and the FIFO buffer is used to store intermediate results obtained in the calculation process of the linear layers.

[0020] Compared with the prior art, the application has the following beneficial effects:

[0021] (1) The octree partitioning-based volumetric video parallel encoding and decoding accelerator proposed in the application is an octree partitioning method for the three-dimensional structure of a volumetric video, spatial locality organization of data is realized through Morton coding and spatial sorting, the encoding and decoding process can be executed in parallel, processing delay is significantly reduced, and the method can fully utilize the spatial sparsity characteristics of the volumetric video, effectively prune the hash grid, and greatly reduce the amount of code stream data transmission.

[0022] (2) The octree partitioning-based volumetric video parallel encoding and decoding accelerator proposed in the application designs an interpolation module specially for the binarized multi-resolution hash grid, the interpolation module effectively avoids storage access conflicts through a multi-Bank storage structure design and a prefetch mechanism, reduces repeated calculation overhead through a vertex pre-aggregation and similar item merging optimization method, and significantly improves the interpolation calculation energy efficiency based on a low-bit multiplier designed pipeline architecture.

[0023] (3) The octree partitioning-based volumetric video parallel encoding and decoding accelerator proposed in the application designs a reconfigurable systolic array supporting layer fusion, can efficiently process linear layer calculation of different sizes, realizes efficient reuse of weight matrices through distributed on-chip storage SRAM, realizes reuse of input activation values through a systolic array data stream, supports layer fusion calculation of multiple linear layers, and effectively reduces storage access of intermediate results. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0025] Figure 1 This is a schematic diagram of the encoding and decoding process of a volume video parallel encoding and decoding accelerator based on octree block partitioning, as an embodiment of this application.

[0026] Figure 2 This is a schematic diagram of the architecture of a volume video parallel encoding and decoding accelerator based on octree block partitioning, as an embodiment of this application.

[0027] Figure 3 This is a schematic diagram of the interpolation module of the volume video parallel encoding and decoding accelerator based on octree block partitioning, as an embodiment of this application.

[0028] Figure 4 This is a schematic diagram of the prefetch and full pipeline data flow of a volume video parallel encoding and decoding accelerator based on octree block partitioning, as an embodiment of this application.

[0029] Figure 5 The interpolation process of the volume video parallel encoding and decoding accelerator based on octree block division, as an embodiment of this application, is illustrated in the principle diagram of merging like terms.

[0030] Figure 6 This is a schematic diagram of the structure of the MLP computing module of the volume video parallel encoding and decoding accelerator based on octree block partitioning, as an embodiment of this application.

[0031] Figure 7 This is a schematic diagram of the basic structure of the 4×4 systolic array tensor computation kernel of the volume video parallel encoding and decoding accelerator based on octree block partitioning, as an embodiment of this application.

[0032] Figure 8 This is a flowchart illustrating the systolic array memory access mode (taking 3 systolic arrays as an example) of a volume video parallel encoding and decoding accelerator based on octree block partitioning according to an embodiment of this application.

[0033] Figure 9 This is a schematic diagram illustrating the implementation of a layer fusion data stream using an octree-based volume video parallel encoding / decoding accelerator, as an embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0035] This application provides an octree-based block-based volume video parallel encoding and decoding accelerator, including an interpolation module for a binarized multi-resolution hash grid and an MLP computation module. The binarized multi-resolution hash grid is obtained by cropping volume primitives in the volume video using an octree block-based strategy. The interpolation module is used to process the linear interpolation computation of the binarized multi-resolution hash grid to obtain multi-dimensional interpolation vectors. The MLP computation module uses reconfigurable systolic array, distributed memory access, and layer fusion technology to perform MLP computation on the interpolation vectors in a pipelined manner on a block-by-block basis, predicting the entropy model information output by the entropy model corresponding to the feature value of each volume primitive.

[0036] In a specific embodiment, the octree block partitioning strategy proceeds as follows:

[0037] The three-dimensional coordinates of each volume primitive in the volume video are normalized and quantized into spatial coordinates represented by unsigned integers according to the accuracy requirements;

[0038] The spatial coordinates are transformed by Morton coding to obtain the corresponding Morton code. All volume primitives are sorted based on the Morton code to obtain a linear octree structure of volume primitives.

[0039] The depth of the octree is determined, and the linear octree structure of volume primitives is divided into blocks based on the depth of the octree to obtain several spatial blocks corresponding to different depths. Spatial blocks containing invalid volume primitives are identified and removed, and only spatial blocks containing valid volume primitives are retained. The Morton code corresponding to the three-dimensional coordinates of each volume primitive in each spatial block containing valid volume primitives is combined with a binary hash table to construct a corresponding local hash grid. All local hash grids constitute a binary multi-resolution hash grid.

[0040] Specifically, the volumetric video parallel encoding and decoding accelerator based on octree block partitioning proposed in the embodiments of this application employs a parallel encoding and decoding algorithm. This parallel encoding and decoding algorithm adopts a hierarchical processing flow, such as... Figure 1As shown, the three-dimensional coordinates of the volume primitives are first normalized, mapping them to the (0,1) interval, and retaining 8-bit or 16-bit binary representations of the spatial coordinates according to precision requirements. Then, the spatial coordinates are transformed using Morton coding to obtain the corresponding Morton codes. The Morton codes corresponding to all volume primitives are sorted to obtain a linear octree structure of the volume primitives. The linear octree structure is then divided into blocks according to different depths. Depth 0 corresponds to the 8 spatial blocks obtained after the first division, depth 1 corresponds to one of the 8 spatial blocks obtained after the first division followed by a second division, and so on, to obtain spatial blocks of different depths. After determining the octree depth, empty blocks (i.e., spatial blocks containing invalid volume primitives) are automatically identified and removed, retaining only spatial blocks containing valid volume primitives. Each spatial block containing valid volume primitives independently corresponds to a local hash grid, allowing for parallel interpolation calculations. This block-based strategy ensures that each processing unit only needs to access local data, significantly reducing the number of off-chip memory accesses. Finally, MLP computation is performed in a pipelined manner on a block-by-block basis to predict the entropy model distribution of each parameter, thus completing the encoding and decoding process.

[0041] The choice of octree depth needs to take into account the size of the hash grid parameters and hardware resource constraints. Ideally, the hash grid parameters of each spatial block should be fully stored in on-chip memory (usually less than 64KB) to make full use of spatial locality, so that the interpolation calculation of each spatial block only needs to read the data from off-chip memory once.

[0042] Furthermore, the embodiments of this application employ 8-bit linear quantization technology for the weights used in the MLP calculation process, which significantly improves calculation efficiency while ensuring calculation accuracy and further reduces the amount of code stream transmission.

[0043] The architecture of a volumetric video parallel encoding / decoding accelerator based on octree block partitioning proposed in this application is shown in Figure 2. This architecture mainly includes an interpolation module for a binarized multi-resolution hash grid and an MLP computation module based on a reconfigurable systolic array. These two modules work together, and the overall hardware architecture transmits data via the AXI4 protocol and instructions via the AXI-Lite protocol, realizing a complete encoding / decoding process from hash grid interpolation to entropy model prediction.

[0044] The interpolation module handles linear interpolation calculations for multi-resolution hash grids. The input is a binarized multi-resolution hash grid, and the output is a multi-dimensional interpolation vector of the volume primitives. The interpolation module employs specially optimized data flow and computational structures to address the challenges of irregular memory access and computational redundancy. The MLP computation module uses a reconfigurable systolic array architecture, supporting linear layer computations of different scales. It reduces intermediate data storage through layer fusion technology. This MLP computation module takes a multi-dimensional interpolation vector as input and outputs entropy model information corresponding to each feature value in the volume primitive, including mean, variance, and quantization step size.

[0045] The global cache mainly stores intermediate results, input data (point coordinates of volume primitives and binary hash tables) and output data (entropy model information corresponding to each feature value in the volume primitive) during the calculation process of the interpolation module and the MLP calculation module.

[0046] The entire architecture adopts a design concept that combines distributed storage and parallel computing, which not only ensures processing efficiency but also provides good scalability, enabling it to adapt to volumetric video encoding and decoding needs of different scales and complexities.

[0047] In a specific embodiment, the interpolation module includes a control unit, a point storage unit, a hash storage unit, a computation unit, a pipeline scheduling unit, and a result cache unit. The control unit is used to receive and decode external instructions, control the point storage unit and hash storage unit to initiate off-chip memory access requests according to the instructions, and determine the basic parameters of the computation unit and pipeline scheduling unit during the working process. The pipeline scheduling unit is used to generate point coordinate reading control information and vertex reading control information respectively and send them to the point storage unit and hash storage unit to read the feature values ​​of point coordinates and vertices, generate computation pipeline control information and send it to the computation unit in combination with the feature values ​​of point coordinates and vertices. The computation unit includes a multiplier and an accumulator, which are used to perform a weighted summation between the feature values ​​of point coordinates and vertices according to the mathematical principle of interpolation, obtain the interpolation result and directly store it in the result cache unit. The point coordinates exist in the form of Morton code corresponding to the three-dimensional coordinates of the volume primitive.

[0048] Embodiments of this application propose an interpolation module for a binary multi-resolution hash grid, supporting fully pipelined 3D and 2D interpolation on a multi-resolution hash grid composed of binary vertices. The overall hardware architecture is as follows: Figure 3As shown, the interpolation module mainly consists of a control unit, a storage unit, a computation unit, and a pipeline scheduling unit. The storage unit includes point storage units and hash storage units. The control unit receives and decodes instructions from external sources, controlling each storage unit to initiate external memory access requests and determining the basic parameters for the computation unit and pipeline scheduling unit during operation. Upon receiving an instruction, the control unit learns from the instruction the number of points to be processed, the number of hash vertex feature values, and their base addresses in external storage. It then sequentially initiates memory access requests, storing the read point coordinates and vertex feature values ​​into the point storage unit and hash storage unit, respectively. After this data is prepared, the pipeline scheduling unit begins operation, acting as a bridge between the hash storage unit, point storage unit, and computation unit, transmitting the point coordinates and vertex feature values ​​to the computation unit according to a specific data flow. During this process, the pipeline scheduling unit generates a series of control information specific to each clock cycle, including the values ​​of the memory access pointers for the point storage unit and hash storage unit, the current point coordinates and resolution being processed by the computation unit, etc., thereby ensuring that the interpolation of all points in the current task across multiple dimensions and resolutions can be completed completely and correctly. The computation unit mainly consists of multipliers and accumulators, performing a weighted summation between point coordinates and vertex feature values ​​according to the mathematical principles of interpolation. The result is directly stored in the result cache unit until all interpolation processes of the current task are completed. Then, the control unit initiates another memory access request and outputs the result.

[0049] In a specific embodiment, the hash storage unit adopts a multi-bank structure, which includes B banks. The hash storage unit is used to store hash tables of different resolutions into different banks in a circular allocation manner. Each vertex in the local hash grid stores a 4-bit feature value. The pipeline scheduling unit includes a prefetch cache module that operates in the form of ping-pong operation. The prefetch cache module has B register groups, each of which has a corresponding bank. Inputs are fed into the prefetch cache module in parallel from multiple bank dimensions. The prefetch cache module retrieves the local hash grid of its current resolution based on the coordinates of the input point being processed. It then serially generates the addresses of the 8 vertices on the basic unit of the local hash grid of its current resolution through a hash mapping function, and gradually transfers the 8 4-bit feature values ​​stored on the 8 vertices to the register groups.

[0050] Specifically, this interpolation module mainly optimizes two problems existing in binarized multi-resolution linear interpolation:

[0051] (1) Irregular memory access leads to low bandwidth utilization and low computational throughput;

[0052] (2) The additional computational overhead caused by high precision requirements and redundant calculations.

[0053] To address the first problem mentioned above, the interpolation module proposed in the embodiments of this application designs the hash storage unit as a multi-bank structure and provides a prefetch logic to solve the impact of irregular memory accesses on computing performance during the interpolation process.

[0054] During interpolation, irregular memory access stems from the contradiction between the arrangement of multi-dimensional hash grid vertices in memory and the addressing method required for computation. Taking three-dimensional interpolation as an example, on the one hand, a three-dimensional hash grid, as a three-dimensional array-like data structure, typically needs to be flattened along different dimensions when arranged in storage, with the three dimensions being unfolded with different priorities, thus forming a hierarchical arrangement. On the other hand, interpolation computation requires finding the hash grid in which each specific input point coordinate belongs, obtaining the 8 vertex feature values ​​of that hash grid, and then performing a weighted sum based on the distance from the point coordinate to the vertex in each dimension. However, the 8 vertices of the same hash grid are not arranged contiguously in storage, but involve address jumps across dimensions. This means that the process of reading vertex values ​​can only be performed intermittently and serially, reducing bandwidth utilization and limiting the parallelism of computational units. This problem is further exacerbated in binary multi-resolution hash grids because each vertex feature value only occupies 1 bit, far less than the width that a memory interface can typically provide. Meanwhile, when the hash grid has multiple resolutions and multiple tables at each resolution, the dimensions of the hierarchical expansion required for storing and arranging vertex feature values ​​increase further, making the problem of irregular memory access more significant.

[0055] like Figure 4As shown, the hash storage unit in the interpolation module proposed in this application consists of four banks. Vertex feature value data from the outside is divided into different data packets based on resolution and stored in different banks. The hash storage unit polls the banks in a round-robin fashion, so that data packets of each resolution are evenly distributed to different banks. At the same time, there are multiple tables for each resolution (4 tables in a typical task), which is transformed into each vertex feature value being expanded from 1 bit to multiple bits (e.g., 4 bits), so that the multiple binarized tables are merged into a single table of multiple bits. This design allows data of different resolutions to be stored independently and accessed in parallel, and allows multiple tables of the same resolution to be stored and accessed centrally, reducing the problem of irregular memory access to the x, y, z dimensions. In one example, the interpolation dimension is 3, the number of hash table copies is 4, then the prefetch cycle per resolution vertex is 8, and the computation cycle is 4; the computation-memory access ratio is 1-2, so 2 banks are set as a group, and a total of 4 banks are instantiated to form a ping-pong operation, with a ping-pong processing cycle of 8; where bank0 and bank1 are read in (represented by solid lines): dimension 8 is serial, dimension 4 is parallel, and bank dimension is parallel; bank3 and bank4 are output (represented by dashed lines): dimension 8 is parallel, dimension 4 is serial, and bank dimension is serial.

[0056] Furthermore, a prefetch buffer module (referred to as the "prefetch buffer") is placed within the pipeline scheduling unit to prefetch the hash storage units, addressing the issues caused by irregular memory accesses in the x, y, and z dimensions, and supporting fully pipelined computation. This prefetch buffer module exists in the form of a transposed register array; in one embodiment, the transposed register array contains four register groups. Once the vertex feature value data is ready in the hash storage unit, the interpolation module enters the computation state. At this point, the prefetch buffer operates in a ping-pong manner, that is, within a certain computation cycle, prefetching is performed while providing the prefetched data to the computation unit. Specifically, the banks in the hash storage unit are paired up and switch their working states to perform a ping-pong operation, and the prefetch buffer has four register groups corresponding to these banks. In the stage of fetching data from the latter to the former, the prefetch buffer retrieves the hash grid containing the currently processed input point coordinates, serially generating addresses for accessing eight vertices at the current resolution, and progressively collecting eight multi-bit vertex feature values ​​representing multiple tables into the register set. In the stage of providing data to the computation unit, the data in the register set is accessed in parallel along the x, y, and z dimensions, and sequentially traversed along the dimensions of different tables at the current resolution. Each cycle, the computation unit obtains eight vertex feature values ​​from one grid within one table at the current resolution, enabling parallel computation of eight vertices while isolating computations of different tables within the pipeline, ensuring ordered computation. Through this prefetch strategy, the computation unit can operate at full load throughout the pipeline until all points have been processed across all resolutions and all tables. The above principle still applies to the two-dimensional interpolation process.

[0057] In a specific embodiment, the calculation unit is used to expand and summarize all expressions of the product of distances from points to vertices in each dimension into 8 similar terms, directly aggregate the feature values ​​of vertices as coefficients, realize the merging of transformations of the same type, and calculate the interpolation result using the product of distances from points to vertices as elements.

[0058] Specifically, despite having a fully pipelined data flow, the computing unit still faces challenges from redundant computations in handling the interpolation process itself. The computing unit proposed in the embodiments of this application achieves vertex pre-aggregation through mathematical transformations, which can maximize the efficiency of computation.

[0059] Redundant calculations in the interpolation process arise from the repeated calculation of identical parts of the result during weighted summation. For example... Figure 5As shown, the standard calculation method uses vertex feature values ​​as basic elements and the product of the distances from each point to the vertex in each dimension as weights (coefficients) for weighted summation. However, the weights of different vertices are actually composed of the same type of terms. Calculating the weight for each vertex separately essentially involves redundant calculation of this information, thus generating unnecessary overhead. In particular, practical tasks require high interpolation accuracy, and large-bit-width decimal calculations are not hardware-friendly, exacerbating the problems caused by redundant calculations.

[0060] The computational unit proposed in the embodiments of this application pre-aggregates vertices. That is, instead of treating vertices as basic elements and the distance product as coefficients, it expands all expressions of the distance product, summarizes them into eight forms of basic elements, and then directly aggregates the vertex feature values ​​as coefficients, achieving a transformation that combines like terms. The calculation of coefficients at the vertex feature value aggregation point is very simple because each vertex feature value is only binary data, and the basic elements generated from the high-precision distance data only need to be used directly after one calculation, thus improving computational efficiency. For the two-dimensional interpolation process, the distance product does not produce third-order basic elements, and the number of vertex feature values ​​is reduced from eight to four. By selectively disabling some corresponding parts based on the three-dimensional interpolation logic, support for two-dimensional interpolation can be achieved.

[0061] In a specific embodiment, the reconfigurable systolic array includes several 4×4 systolic array tensor computation kernels, which support multiplication and accumulation operations.

[0062] In a specific embodiment, the 4×4 systolic array tensor computation core includes a memory access control unit, a data delay unit, a computation control unit, and a 4×4 computation array. The memory access control unit is used to detect the arrival of the input interpolation results and trigger the weight reading operation. The data delay unit is used to use a data processing mechanism to input the weights and their corresponding interpolation results into the computation array to perform linear layer computation. The computation control unit is used to control whether the computation array needs to perform ReLU computation during the linear layer computation process and the time for outputting the results after completing the linear layer computation.

[0063] In a specific embodiment, the computing array includes 4 rows and 4 columns of basic computing units. The basic computing units are used to perform multiplication and accumulation operations on the input weights and their corresponding interpolation results, and are configured with ReLU operators to perform ReLU calculations.

[0064] In a specific embodiment, the 4×4 systolic array tensor computation core further includes a weight input selection unit. The weights are stored in a distributed on-chip SRAM, which contains several private storage blocks. Each 4×4 systolic array tensor computation core has its own corresponding private storage block. The five 4×4 systolic array tensor computation cores in each row share five private storage blocks through a switch cross-connection matrix. The weight input selection unit is used to select whether the weights input to each 4×4 systolic array tensor computation core come from its corresponding private storage block or the private storage blocks corresponding to the other four 4×4 systolic array tensor computation cores in each row.

[0065] In a specific embodiment, a FIFO buffer is inserted between the linear layers. The FIFO buffer is used to store intermediate results obtained during the calculation of the linear layer.

[0066] Specifically, the overall architecture of the MLP computing module proposed in the embodiments of this application is as follows: Figure 6 As shown, this MLP computing module employs a hybrid design combining distributed on-chip SRAM storage blocks and a reconfigurable systolic array. In one example, the MLP computing module contains 4×4 systolic array tensor computing cores with 48 rows and 20 columns. Each 4×4 systolic array tensor computing core supports 8-bit fixed-point multiplication and 32-bit accumulation operations, enabling efficient matrix multiplication and addition operations. The control unit adopts a hierarchical scheduling mechanism: the global control unit is responsible for overall task distribution and data flow scheduling, while the local control units independently manage the running status of each 4×4 systolic array tensor computing core.

[0067] like Figure 6 As shown, the distributed storage blocks are grouped into 12 groups of 5 blocks each (corresponding to Banks 0-4). Each row of 5 4×4 systolic array tensor computation cores is fully interconnected with the corresponding group's storage blocks via a crossbar switch matrix. Each 4×4 systolic array tensor computation core can access any storage block within its group via a 32-bit bus. A single storage block provides a total peak bandwidth of 160 bits through 5 parallel 32-bit interfaces. The crossbar switch matrix can dynamically configure the connection relationships according to computational needs, thus flexibly supporting matrix operations of different shapes.

[0068] The internal buffer consists of five FIFO buffers. The output of each 4×4 systolic array tensor computation core is sequentially fed into the corresponding FIFO buffer via a 32-bit bus for temporary storage. This design supports the reconstruction of the MLP computation module into two consecutive linear layers of 4×4 systolic array tensor computation cores: intermediate results from the first layer can be directly passed to the second layer for further computation without external storage, effectively reducing frequent access to the global cache and lowering data transfer overhead.

[0069] The data scaling calculation module is responsible for restoring the MLP output result before quantization. To improve computational efficiency, the embodiments of this application perform 8-bit quantization on both weights and activation values. Therefore, they need to be multiplied by the corresponding scaling factor in the output stage to restore the true numerical accuracy. The activation values ​​mentioned in the embodiments of this application are the interpolation results output by the interpolation module.

[0070] The nonlinear function calculation module is responsible for performing activation function operations. It uses a multi-segment linear approximation method to fit common nonlinear functions (such as Sigmoid, GELU, etc.). On the hardware side, it achieves low-overhead and highly flexible activation calculation through lookup tables.

[0071] This architecture is specifically optimized for two key issues in MLP computing: first, reducing the overhead of intermediate data handling through distributed storage and efficient data flow design; and second, improving the utilization and flexibility of computing resources through reconfigurable pulsating arrays.

[0072] refer to Figure 7 The basic structure of the 4×4 systolic array tensor computation kernel mainly includes: a memory controller unit, a data delay block, a computation controller unit, and a 4×4 computation array. The 4×4 computation array contains 4 rows and 4 columns of basic computation units. The memory controller unit intelligently detects the arrival of input activation values ​​and dynamically triggers weight read operations. The data delay block uses a simplified data processing mechanism to ensure that data correctly enters the 4×4 computation array. The computation controller unit mainly controls whether ReLU calculation is needed and when the calculation is completed and the output result is displayed. The basic computation units adopt a fixed output data stream pattern. Each basic computation unit integrates a configurable ReLU operator, supporting seamless integration of linear and activation layers, avoiding additional data storage and transmission overhead.

[0073] To address the storage and access challenges of large-scale weight matrices, the MLP computation module in this application employs a distributed on-chip SRAM storage scheme. For example... Figure 8 As shown, each row of five 4×4 systolic array tensor computation cores shares five private storage blocks. The weight input for each 4×4 systolic array tensor computation core can be selected from either the private storage block or the parent array. This design ensures both proximity access to weight data and provides flexible data path configuration.

[0074] Through a reconfigurable architecture and dynamic resource allocation, the MLP computing module achieves efficient integration between multiple linear layers. For example... Figure 9As shown, computational and storage resources are dynamically allocated according to the computational needs of each linear layer, and FIFO buffers are inserted between linear layers to balance throughput differences, achieving seamless pipelined processing. This design significantly reduces off-chip storage access to intermediate results and improves overall computational efficiency.

[0075] In summary, by employing three technologies—systolic array, distributed memory access, and layer fusion computing—the embodiments of this application achieve data reuse of input activation values, weights, and intermediate layer calculation results, thereby greatly improving the computational efficiency of MLP.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A volume video parallel encoding and decoding accelerator based on octree block partitioning, characterized in that, The system includes an interpolation module for a binarized multi-resolution hash grid and an MLP computation module. The binarized multi-resolution hash grid is obtained by cropping volume primitives in a volume video using an octree block-based strategy. The interpolation module is used to process the linear interpolation computation of the binarized multi-resolution hash grid to obtain a multi-dimensional interpolation vector. The MLP computation module uses reconfigurable systolic array, distributed memory access, and layer fusion technology to perform MLP computation on the interpolation vector in a pipelined manner on a block-by-block basis, predicting the entropy model information output by the entropy model corresponding to the feature value of each volume primitive.

2. The volume video parallel encoding and decoding accelerator based on octree block partitioning according to claim 1, characterized in that, The specific process of the octree block division strategy is as follows: The three-dimensional coordinates of each volume primitive in the volume video are normalized and quantized into spatial coordinates represented by unsigned integers according to the accuracy requirements; The spatial coordinates are transformed by Morton coding to obtain the corresponding Morton code. All volume primitives are sorted based on the Morton code to obtain a linear octree structure of volume primitives. The depth of the octree is determined, and the linear octree structure of the volume primitives is divided into blocks based on the depth of the octree to obtain several spatial blocks corresponding to different depths. Spatial blocks containing invalid volume primitives are identified and removed, and only spatial blocks containing valid volume primitives are retained. The Morton code corresponding to the three-dimensional coordinates of each volume primitive in each spatial block containing valid volume primitives is combined with a binary hash table to construct a corresponding local hash grid. All local hash grids constitute a binary multi-resolution hash grid.

3. The volume video parallel encoding and decoding accelerator based on octree block partitioning according to claim 2, characterized in that, The interpolation module includes a control unit, a point storage unit, a hash storage unit, a computation unit, a pipeline scheduling unit, and a result cache unit. The control unit is used to receive and decode external instructions, control the point storage unit and the hash storage unit to initiate off-chip memory access requests according to the instructions, and determine the basic parameters of the computation unit and the pipeline scheduling unit during the operation process. The pipeline scheduling unit is used to generate point coordinate reading control information and vertex reading control information respectively and send them to the point storage unit and hash storage unit to read the feature values ​​of point coordinates and vertices, generate computation pipeline control information and send it to the computation unit in combination with the feature values ​​of point coordinates and vertices. The computation unit includes a multiplier and an accumulator, which are used to perform a weighted summation between the feature values ​​of point coordinates and vertices according to the mathematical principle of interpolation, obtain the interpolation result and directly store it in the result cache unit. The point coordinates exist in the form of Morton code corresponding to the three-dimensional coordinates of the volume primitive.

4. The volume video parallel encoding and decoding accelerator based on octree block partitioning according to claim 3, characterized in that, The hash storage unit adopts a multi-bank structure, which includes B banks. The hash storage unit is used to store hash tables of different resolutions into different banks in a circular allocation manner. Each vertex in the local hash grid stores a 4-bit feature value. The pipeline scheduling unit includes a prefetch cache module that operates in a ping-pong manner. The prefetch cache module has B register groups, each with a corresponding bank. Inputs are fed into the prefetch cache module in parallel from multiple bank dimensions. The prefetch cache module retrieves the local hash grid of its current resolution based on the coordinates of the input point being processed. It then serially generates the addresses of the eight vertices on the basic unit of the local hash grid of its current resolution using a hash mapping function, and gradually transmits the eight 4-bit feature values ​​stored on the eight vertices to the register groups.

5. The volume video parallel encoding and decoding accelerator based on octree block partitioning according to claim 3, characterized in that, The calculation unit is used to expand and summarize all expressions of the product of distances from points to vertices in each dimension into 8 similar terms, directly aggregate the feature values ​​of the vertices as coefficients, realize the merging of transformations of the same type, and calculate the interpolation result using the product of distances from points to vertices as elements.

6. The volume video parallel encoding and decoding accelerator based on octree block partitioning according to claim 1, characterized in that, The reconfigurable systolic array includes several 4×4 systolic array tensor computation kernels, which support multiplication and accumulation operations.

7. The volume video parallel encoding and decoding accelerator based on octree block partitioning according to claim 6, characterized in that, The 4×4 systolic array tensor computation core includes a memory access control unit, a data delay unit, a computation control unit, and a 4×4 computation array. The memory access control unit is used to detect the arrival of the input interpolation results and trigger a weight read operation. The data delay unit is used to input the weights and their corresponding interpolation results into the computation array to perform linear layer computation using a data processing mechanism. The computation control unit is used to control whether the computation array needs to perform ReLU computation during the linear layer computation process and the time for outputting the results after completing the linear layer computation.

8. The volume video parallel encoding and decoding accelerator based on octree block partitioning according to claim 7, characterized in that, The computation array includes a 4-row, 4-column basic computation unit. The basic computation unit is used to perform multiplication and accumulation operations on the input weights and their corresponding interpolation results, and is configured with a ReLU operator to perform ReLU calculation.

9. The volume video parallel encoding and decoding accelerator based on octree block partitioning according to claim 7, characterized in that, The 4×4 systolic array tensor computation core also includes a weight input selection unit. The weights are stored in a distributed on-chip SRAM, which contains several private storage blocks. Each 4×4 systolic array tensor computation core has its own corresponding private storage block. A switch cross-connection matrix enables five private storage blocks to be shared by five 4×4 systolic array tensor computation cores in each row. The weight input selection unit is used to select whether the weights input to each 4×4 systolic array tensor computation core come from its corresponding private storage block or from the private storage blocks corresponding to the other four 4×4 systolic array tensor computation cores in each row.

10. The volume video parallel encoding and decoding accelerator based on octree block partitioning according to claim 7, characterized in that, A FIFO buffer is inserted between the linear layers to store intermediate results obtained during the calculation of the linear layer.

Images