A high-throughput dynamic visual frame real-time encoding processing method and system

Abstract

The application relates to the technical field of machine vision and edge computing, and discloses a high-throughput dynamic visual frame real-time encoding processing method and system, which comprises a dynamic visual frame sensor, a front-end encoding chip and a back-end processing chip. The front-end encoding chip sequentially performs sparse collection and data regularization processing, intra-frame adaptive spatial compression encoding processing, inter-frame motion perception time domain compression encoding processing and sparse feature extraction and semantic encoding processing on binary sparse event frames output by the dynamic visual frame sensor, and outputs low-dimensional sparse feature vectors to the back-end processing chip. The back-end processing chip performs multiplication and addition operation on the decoded feature vectors inside a storage array, and skips the corresponding multiplication and addition operation according to zero-value components in the feature vectors. In the inference process, the precision is dynamically selected for inference according to real-time event sparsity. The low-delay compression and high-energy-efficiency real-time inference of high-frame-rate dynamic visual frame data are realized.

Description

Technical Field

[0001] This invention relates to the fields of machine vision and edge computing technology, and in particular to a high-throughput dynamic visual frame real-time encoding processing method and system. Background Technology

[0002] Dynamic vision sensors and their derivatives, dynamic vision frame sensors, asynchronously sense pixel-level brightness changes and can output event data with high temporal resolution (up to 2000 FPS or more) and high sparsity, showing great application potential in fields such as high-speed moving target analysis, autonomous driving, and industrial defect detection. However, in high frame rate operation, the data rate of the raw sparse event stream or event frames output by these sensors can reach the Gbps level, placing severe bandwidth and storage pressure on subsequent data transmission, storage, and processing. To address these challenges, existing technologies have attempted to compress event frames using traditional video coding schemes (such as H.264) or image compression algorithms (such as JPEG) to reduce data volume. Simultaneously, processors based on the von Neumann architecture are used to perform neural network inference on the compressed data to achieve visual perception tasks such as object detection and semantic segmentation.

[0003] However, the existing processing solutions still have many shortcomings in practical applications. First, traditional video coding and image compression algorithms are mainly designed for dense frames, and their coding latency is usually greater than 1ms, which is difficult to meet the requirements of microsecond-level real-time processing in high frame rate scenarios. Second, these algorithms fail to fully utilize the natural sparsity of event frame data, limiting compression efficiency and still resulting in high data transmission and storage burdens. More importantly, when processing such data, the traditional von Neumann computing architecture requires frequent data transfer between storage and computing units, leading to significant power consumption. At the same time, existing architectures struggle to perceive and utilize the sparsity of input data during inference to skip invalid computations, failing to fully leverage the acceleration potential brought by the high sparsity of dynamic visual frame data. Furthermore, in existing solutions, coding and inference are usually implemented as independent modules or devices, with data being transferred multiple times between different modules, further exacerbating latency and power consumption issues, making it difficult to build an efficient and compact real-time intelligent vision system.

[0004] In view of the shortcomings of the existing technologies, there is an urgent need in the field for an integrated system and method that can natively adapt to the high frame rate and high sparsity characteristics of dynamic visual frame sensors, and achieve extremely low latency and high energy efficiency processing across the entire chain from data acquisition, multi-level compression coding to inference computing. Summary of the Invention

[0005] The purpose of this invention is to provide a high-throughput dynamic visual frame real-time encoding and processing method and system, which solves the problems of high latency, high power consumption and low energy efficiency caused by architecture mismatch and separate encoding and inference modules in the transmission, storage and processing of high frame rate dynamic visual frame data in the prior art.

[0006] To achieve the above objectives, the present invention provides a high-throughput dynamic visual frame real-time encoding and processing system, comprising:

[0007] A dynamic visual frame sensor, which outputs binary sparse event frames representing pixel-level brightness changes at a high frame rate.

[0008] The front-end encoding chip is integrated with the dynamic visual frame sensor in the same module. The front-end encoding chip integrates a multi-level cascaded sparse compression encoder. The sparse compression encoder receives the binary sparse event frame and sequentially performs sparse acquisition and data normalization processing, intra-frame adaptive spatial compression encoding processing, inter-frame motion perception temporal compression encoding processing, and sparse feature extraction and semantic encoding processing to output a low-dimensional sparse feature vector.

[0009] A back-end processing chip is connected to the front-end encoding chip via a high-speed interface. The back-end processing chip includes a feature decoding and mapping unit and an in-memory computing inference engine. The feature decoding and mapping unit receives and decodes the low-dimensional sparse feature vector. The in-memory computing inference engine includes an in-memory computing integrated storage array, in which the weights of the neural network are pre-stored. The in-memory computing inference engine performs multiply-add operations on the decoded feature vector within the in-memory computing integrated storage array to complete the neural network inference and outputs the target detection result, semantic segmentation result, or target tracking result.

[0010] The sparse acquisition and data normalization process specifically includes:

[0011] The binary sparse event frame is scanned in parallel for each row, and the binary sparse event frame is converted into a list of coordinates sorted by row. During the scanning process, the event density of each row is calculated to generate a row density histogram.

[0012] The intra-frame adaptive spatial compression coding process specifically includes:

[0013] For each row, run-length encoding is adaptively selected based on the event density of the corresponding row in the row density histogram, choosing between sparse row mode, dense row mode, or mixed mode.

[0014] When the event density is less than the first threshold, the sparse row mode is selected to directly encode the column coordinates of each event in the coordinate list;

[0015] When the event density is greater than the second threshold, the dense row mode is selected to encode the column coordinates of the first event in the coordinate list and the column interval between each subsequent event and the previous event.

[0016] When the event density is greater than or equal to the first threshold and less than or equal to the second threshold, the hybrid mode is selected, each row is divided into multiple column segments, and only the column segments containing events are encoded.

[0017] The inter-frame motion-aware temporal compression coding process specifically includes:

[0018] Read the row density histogram of the previous frame and calculate the absolute value of the difference in event density between each row of the row density histogram of the current frame and the row density histogram of the previous frame.

[0019] Rows whose absolute value is greater than the motion threshold are marked as moving rows, and rows whose absolute value is less than or equal to the motion threshold are marked as static rows;

[0020] For each of the motion lines, output the intra-frame adaptive spatial compression coding result of the current frame;

[0021] For each static line, calculate the bitwise XOR value of the intra-frame adaptive spatial compression coding result of the current frame and the intra-frame adaptive spatial compression coding result of the previous frame. When the bitwise XOR value is zero, output 1 bit to indicate no change. When the bitwise XOR value is non-zero, output 1 bit flag followed by the compressed representation of the non-zero part of the bitwise XOR value.

[0022] The frame header, the motion line bitmap consisting of markers for all the motion lines and the static lines, the motion line data block, and the static line difference data block are sequentially concatenated to form a time-domain compressed bitstream.

[0023] The sparse feature extraction and semantic encoding process specifically includes:

[0024] The temporal compressed bitstream is encoded using a sparse convolutional autoencoder, which includes multiple sparse convolutional layers and one fully connected layer.

[0025] The sparse convolutional layer adopts a non-zero coordinate driven architecture, extracts the position and value of all non-zero activation values ​​in the input sparse feature map, allocates multiple parallel processing units to process each non-zero position, each of the parallel processing units reads the non-zero values ​​in the neighborhood around the current non-zero position and performs a multiply-add operation with the corresponding convolutional kernel weights, and accumulates the result of the multiply-add operation to the corresponding position in the output feature map.

[0026] The fully connected layer maps the output feature map of the sparse convolutional layer into a 256-dimensional feature vector, and outputs the 256-dimensional feature vector after INT8 quantization.

[0027] The feature decoding and mapping unit decodes and maps the 256-dimensional feature vector into the input format of the in-memory computing inference engine. The in-memory computing inference engine also includes a sparse perceptual controller and an inference precision adjuster.

[0028] The sparse sensing controller identifies the zero-value components in the decoded feature vector and controls the in-memory computing array to skip the multiply-accumulate operations corresponding to the zero-value components.

[0029] The inference precision adjuster selects INT4 precision, INT2 precision, or INT1 precision for inference based on the real-time event sparsity rate obtained from the sparse acquisition and data normalization processing. When the real-time event sparsity rate is greater than a first sparsity threshold, the INT4 precision is selected; when the real-time event sparsity rate is less than or equal to the first sparsity threshold but greater than a second sparsity threshold, the INT2 precision is selected; and when the real-time event sparsity rate is less than or equal to the second sparsity threshold, the INT1 precision is selected.

[0030] The front-end encoding chip is manufactured using logic technology and is directly connected to the dynamic visual frame sensor via a chip-level parallel bus. The back-end processing chip is manufactured using in-memory computing static random access memory technology. The front-end encoding chip and the back-end processing chip are connected through a high-speed serial interface and integrated into the same imaging computing module.

[0031] A high-throughput dynamic visual frame real-time encoding processing method includes the following steps:

[0032] Acquire binary sparse event frames output by a dynamic visual frame sensor at a high frame rate;

[0033] In the front-end encoding chip, multi-level cascaded sparse compression coding is sequentially performed on the binary sparse event frame. The multi-level cascaded sparse compression coding includes sparse acquisition and data normalization processing, intra-frame adaptive spatial compression coding processing, inter-frame motion-aware temporal compression coding processing, and sparse feature extraction and semantic coding processing, outputting a low-dimensional sparse feature vector.

[0034] The low-dimensional sparse feature vector is transmitted to the back-end processing chip via a high-speed interface;

[0035] The low-dimensional sparse feature vector is decoded in the feature decoding and mapping unit of the back-end processing chip, and the decoded feature vector is mapped to the in-memory computing storage array.

[0036] The decoded feature vector is multiplied and added within the in-memory computing array to complete neural network inference, and the corresponding multiplied and added operations are skipped based on the zero-value components in the decoded feature vector.

[0037] Output object detection results, semantic segmentation results, or object tracking results.

[0038] This invention discloses a high-throughput real-time encoding processing method and system for dynamic visual frames. The front-end encoding chip sequentially performs four levels of processing on the binary sparse event frames output by the dynamic visual frame sensor: sparse acquisition and data normalization, intra-frame adaptive spatial compression encoding, inter-frame motion-aware temporal compression encoding, and sparse feature extraction and semantic encoding. This compresses massive event frames into low-dimensional sparse feature vectors and transmits them to the back-end processing chip. The back-end processing chip directly performs multiply-accumulate operations on the decoded feature vectors within a storage array to complete neural network inference. During inference, it skips invalid calculations corresponding to zero-value components based on the sparsity of the feature vectors. Simultaneously, throughout the entire encoding and inference process, it adaptively adjusts the encoding mode and computational precision according to the dynamic sparsity of the scene. Therefore, this invention achieves extremely low latency and high energy efficiency across the entire chain, from data acquisition and multi-level compression encoding to inference computation. It effectively solves the technical problems of high latency, high power consumption, and low energy efficiency caused by high latency in existing encoding schemes, high power consumption during data transport in computing architectures, inability to utilize sparsity, and the separation of encoding and inference modules. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0040] Figure 1 This is a schematic diagram of the physical structure of the high-throughput dynamic visual frame real-time encoding and processing system according to the first embodiment of the present invention.

[0041] Figure 2 This is a flowchart of the internal processing of a multi-level cascaded sparse compression encoder according to the first embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of the sparse perception computing principle of the in-memory computing inference engine in the first embodiment of the present invention.

[0043] Figure 4 This is the overall logical architecture diagram of the high-throughput dynamic visual frame real-time encoding processing method according to the second embodiment of the present invention.

[0044] Figure 5 This is a flowchart of the steps of the high-throughput dynamic visual frame real-time encoding processing method according to the second embodiment of the present invention.

[0045] In the diagram: 101 - Dynamic visual frame sensor, 102 - Front-end encoding chip, 103 - Back-end processing chip. Detailed Implementation

[0046] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0047] First embodiment:

[0048] Please refer to Figures 1 to 3 This invention provides a high-throughput dynamic visual frame real-time encoding and processing system, including a dynamic visual frame sensor 101, a front-end encoding chip 102, and a back-end processing chip 103.

[0049] This invention employs a modular design, primarily comprising three physical components: a dynamic visual frame sensor 101 chip, a front-end encoding chip 102, and a back-end processing chip 103. The dynamic visual frame sensor 101 chip is manufactured using a 65nm CIS process, has a resolution of 1280×1024, and supports global frame synchronization output up to 2000 frames per second. Each pixel independently detects changes in logarithmic illumination intensity. When the change exceeds a preset threshold, the pixel outputs "1" in the current frame; otherwise, it outputs "0". Therefore, each frame of data is a 1280×1024 binary sparse matrix, with a typical sparsity rate of 5% to 30%. At a typical sparsity rate of 10% and 2000 FPS, each binary sparse event frame contains approximately 131K non-zero pixel data, or approximately 131K non-zero pixel data / frame; for example, 1280×1024×10%=131,072. The front-end encoding chip 102 is manufactured using a 14nm Logic process and is directly connected to the dynamic visual frame sensor chip 101 via a chip-level parallel bus (such as a 64-bit data bus, operating at a frequency of 1GHz). The front-end encoding chip 102 integrates a multi-level cascaded sparse compression encoder, entirely implemented using digital logic circuits, without the involvement of an embedded processor or software. The front-end encoding chip 102 can also be referred to as a sparse encoding chip, with a chip area of ​​approximately 9mm². The back-end processing chip 103 is manufactured using a 28nm CIM-SRAM process and connects to the front-end encoding chip 102 via a high-speed serial interface. The back-end processing chip 103 includes a feature decoding and mapping unit, an in-memory computing inference engine, and a post-processing unit. The back-end processing chip 103 can also be referred to as an in-memory computing chip, with a chip area of ​​approximately 16mm². In a typical 10% sparse scenario, the total power consumption of the back-end processing chip 103 is approximately 400mW. The system adopts a pipelined parallel architecture, with an equivalent processing frame rate greater than 2000FPS.

[0050] The front-end encoding chip 102 integrates a multi-level cascaded sparse compression encoder, which is used to receive the binary sparse event frames output by the dynamic visual frame sensor 101 and perform the following four levels of processing in sequence.

[0051] (I) Level 1: Sparse Acquisition and Data Regularization

[0052] The first stage is used to convert the binary sparse event frames output by the dynamic visual frame sensor 101 into an ordered list of coordinates that is friendly to the subsequent encoder, and to statistically analyze the row density distribution.

[0053] In terms of hardware implementation, the front-end encoding chip 102 contains 128 parallel scanning engines, each responsible for 10 lines (1280 lines / 128 = 10 lines). Each engine contains 10 line counters (each 10 bits) and 10 coordinate FIFOs (each depth equal to the maximum number of events in that line, typically 100). Upon the arrival of the frame synchronization signal, all scanning engines start simultaneously, scanning their assigned lines one by one at a pixel clock (500MHz).

[0054] In terms of the algorithm flow, after the frame synchronization signal arrives, all scanning engines start simultaneously, starting from the first row they are responsible for, and traversing the 1024 pixels of that row from left to right at a pixel clock (500MHz). For the currently scanned pixel, if the value is "1", the event counter corresponding to that row is incremented by 1, and the current column coordinate (10 bits) is written into the coordinate FIFO corresponding to that row; if the pixel value is "0", scanning continues without any counting or writing operations. When a row is scanned (i.e., 1024 pixels of that row have been traversed), the scanning engine writes the value in the event counter of that row into the corresponding H_row[r] register in the global row density histogram register group. After writing, the event counter of that row is cleared to prepare for the next row scan. Since the 128 scanning engines work in parallel, each engine executes the above process sequentially for the 10 rows it is responsible for. The row density data H_row[0] to H_row

[1279] of all rows are written sequentially within the same frame time, finally forming a complete row density histogram. The event density refers to the number of event pixels with a value of "1" in each row of a binary sparse event frame. Specifically, in the first-level sparse acquisition and data warping, the event frame is scanned row by row. If the current pixel value is "1", the row event counter is incremented by 1. After a row is scanned, the value of the row event counter is written into the row density histogram register H_row[r]. Therefore, H_row[r] is the event density of the r-th row, and the row density histogram is composed of H_row[0] to H_row

[1279] . For an embodiment with a total of 1024 row pixels, the event density value ranges from 0 to 1024. After all rows are scanned, the coordinates of each row are concatenated into a complete coordinate list L_coords in order of row number, and output to the second level along with the filled row density histogram H_row[0...1279].

[0055] (ii) Second level: Intra-frame adaptive spatial compression coding processing

[0056] The second level is used to adaptively select the optimal encoding mode based on the event density of each row's coordinate list, eliminating intra-frame spatial redundancy.

[0057] In terms of hardware implementation, the front-end encoding chip 102 contains 1280 parallel line encoding units. Each unit receives the corresponding row coordinate list and row density H_row[r] from the first-level output. Each line encoding unit contains a mode decision unit, a sparse line encoder, a dense line encoder, a mixed-mode encoder, and an output bitstream assembler.

[0058] In terms of algorithm flow, the thresholds T_low=10 and T_high=100 are set. The first threshold T_low and the second threshold T_high are pre-set row event density thresholds, and T_low<T_high. For an embodiment where the total number of row pixels W=1024, T_low=10 and T_high=100 are preferred. They can be configured according to the sensor resolution, total number of row pixels, typical event sparsity rate and noise level, and are not limited to the above typical values. When H_row[r]<T_low, the sparse row mode is selected; when H_row[r]>T_high, the dense row mode is selected; when T_low≤H_row[r]≤T_high, the hybrid mode is selected. After the mode determiner reads H_row[r]: if H_row[r]<10, the sparse row mode is selected; if H_row[r]>100, the dense row mode is selected; otherwise, the hybrid mode is selected. Encoding is performed according to the selected mode: in the sparse row mode, the mode code 00 is output, followed by the column coordinates of each event (10 bits each); in the dense row mode, the mode code 01 is output, followed by the column coordinate of the first event (10 bits), and then the column interval between each subsequent event and the previous event is encoded; in the hybrid mode, 1024 columns are divided into 32 segments (32 columns per segment), the mode code 10 is output, and then only the segments with events are encoded. The encoded bitstream is temporarily stored in the row output buffer, waiting for reading by the third stage. At a 10% sparsity rate, the typical frame-level compression ratio is about 8:1, and the output data rate is reduced from about 5.24Gbps to about 655Mbps.

[0059] (3) Third stage: Inter-frame motion-aware temporal compression encoding processing

[0060] The third stage is configured to detect motion regions through row density variation by utilizing the strong correlation between adjacent frames, and perform differential encoding on static rows to eliminate inter-frame temporal redundancy.

[0061] In terms of hardware implementation, an on-chip SRAM is arranged in the front-end encoding chip 102, which is configured to buffer the second-stage encoding result of the previous frame and the row density histogram H_row_prev

[1280] of the previous frame, and 1280 parallel row differential units are provided, each unit being responsible for one row.

[0062] In terms of the algorithm flow, for each row, let r represent the index of the row (0≤r≤1279), denot the row density of the current frame as H_row_cur[r] (obtained from the first-level statistics), and denot the row density of the same row in the previous frame as H_row_prev[r] (read from the on-chip SRAM). Calculate the row density change ΔH = |H_row_cur[r] - H_row_prev[r]|. Set a motion threshold σ (σ is a configurable positive integer, typically 3~5). The motion threshold σ is the decision threshold for the absolute value of the event density difference of the same row in adjacent frames. σ is a configurable positive integer, typically 3~5. Specifically, calculate the absolute value of the event density difference ΔH = |H_row_cur[r] - H_row_prev[r]| between the current frame and the same row in the previous frame; if ΔH > σ, mark the row as a moving row; if ΔH ≤ σ, mark the row as a static row. σ can be dynamically adjusted according to the scene. The threshold can be dynamically adjusted by the adaptive control unit according to the scene: in high dynamic scenes, σ is appropriately increased to reduce false positives, and in low dynamic scenes, σ is appropriately decreased to improve motion detection sensitivity. The high and low dynamic scenes can be determined by the adaptive control unit based on the global event sparsity rate ρ: when ρ > 15%, it is determined to be a high dynamic scene, and σ is increased to 5-8 to reduce false positives; when ρ < 5%, it is determined to be a low dynamic scene, and σ is decreased to 1-3 to improve motion detection sensitivity; when 5% ≤ ρ ≤ 15%, a typical value of 3-5 can be maintained. ΔH is compared with σ: if ΔH > σ, the row is marked as a "moving row"; otherwise, it is marked as a "static row". The marking results of all rows (a total of 1280 marking bits) are compressed into a 1280-bit "moving row bitmap", where the r-th bit is 1 to indicate a moving row and 0 to indicate a static row. For each moving row, the second-level encoding result (full data) of the current frame is directly output. For each static line, calculate diff = C_spatial_cur[r] XOR C_spatial_prev[r] (bitwise XOR); if diff == 0, output 1 bit '0' to indicate that the line has no change; if diff ≠ 0, output 1 bit '1', followed by the compressed representation of the non-zero part of diff (since diff is usually very sparse, the index and value of the non-zero bytes can be directly output, or run-length encoding can be used). Finally, the frame header, moving line bitmap, moving line data block, and static line difference data block are concatenated in sequence to form the temporal compressed bitstream C_temporal output. In a typical scenario (20% moving lines, 80% static lines, 90% of the static lines are unchanged), the inter-frame compression ratio is approximately 5:1 to 8:1, and the total compression ratio can reach 40:1 after concatenation with the second stage, with the output data rate reduced to approximately 131 Mbps.

[0063] (iv) Level 4: Sparse Feature Extraction and Semantic Encoding Processing

[0064] The fourth stage is used to further utilize sparse convolutional neural networks to extract semantic features useful for downstream tasks, building on the compression of the second and third stages, and compressing the data rate to below 4 Mbps.

[0065] The sparse convolutional autoencoder consists of four sparse convolutional layers and one fully connected layer. For hardware acceleration, a "non-zero coordinate driven" architecture is adopted. It extracts the positions (coordinates and values) of all non-zero activation values ​​from the input sparse feature map. Sixty-four parallel processing units are allocated to process each non-zero position. Each parallel processing unit reads the non-zero values ​​within its 3×3 neighborhood (by querying the sparse map using a neighborhood address generator), performs a multiplication-accumulation operation with the corresponding convolutional kernel weights, and adds the result to the corresponding position in the output feature map. Since the input sparsity is approximately 10%, the actual computational cost is only 1 / 10 of that of dense convolutions. The fully connected layer maps the output feature map of the sparse convolutional layer to a 256-dimensional feature vector, and outputs this 256-dimensional feature vector after INT8 quantization, with a data volume of 256 bytes / frame. At 2000 FPS, the data rate = 256 × 2000 × 8 = 4.096 Mbps.

[0066] The back-end processing chip 103 receives the low-dimensional sparse feature vector sent by the front-end encoding chip 102 through a high-speed interface. Internally, it includes a feature decoding and mapping unit, an in-memory computing inference engine, and a post-processing unit.

[0067] (a) Feature Decoding and Mapping Unit

[0068] The feature decoding and mapping unit decodes and maps the received 256-dimensional INT8 feature vector into the input format of the in-memory computing inference engine. The specific process is as follows: Based on the system configuration, a mapping mode is selected. In lightweight mode, the 256-dimensional vector is directly used as the input to the inference network (suitable for fully connected classification networks). In high-precision mode, a lightweight decoder (a sparse convolutional transpose network symmetrical to the fourth-level encoder) reconstructs the 256-dimensional vector into a feature map, which is then fed into the detection or segmentation network. The feature decoding and mapping unit reads the global sparsity ρ provided by the adaptive control unit (this value is statistically determined by the first level and transmitted via the interface). Based on ρ, the inference precision is selected: INT4 precision (4-bit ADC resolution) is used when ρ > 15%, INT2 precision (2-bit ADC resolution) is used when 5% ≤ ρ ≤ 15%, and INT1 precision (1-bit ADC resolution) is used when ρ < 5%. Finally, the input feature vector is expanded row-wise into a column-parallel format and sent to the input register of the in-memory computing array.

[0069] (ii) In-memory computing inference engine

[0070] The core of the in-memory computing inference engine is a 256×256 SRAM in-memory computing array. Each computing unit stores 1 bit of weight (multiple bit weights are achieved by combining multiple 1-bit units or accumulating over time). Each row corresponds to an input channel, and each column corresponds to an output channel. Each row's input terminal is equipped with a gate switch controlled by a non-zero flag of the input feature value. Each column's end is equipped with a successive approximation register (SAR) ADC with programmable conversion accuracy (1 / 2 / 4 bits).

[0071] The inference process is as follows: During system initialization, the trained inference network weights are written into the in-memory computing array. These weights reside permanently and do not require repeated transfer. The feature decoding and mapping unit sequentially applies the decoded feature vector components to the 256 word lines of the in-memory computing array. Each component first passes through a zero-value detector: if the component is 0, an "open" signal is sent to the corresponding row's gate switch, and all in-memory units in that row do not participate in the calculation; if it is not 0, a "closed" signal is sent, and the value is converted into an analog voltage via a DAC to drive the word line of that row. For a closed row, the output current of each in-memory unit in that row is equal to the product of the input voltage and the stored weight (0 or 1). The currents of all closed rows in the same column are accumulated on the bit lines in an analog manner. The accumulated current of each column is converted into a digital value by an ADC, and the output accuracy is controlled by the inference accuracy regulator. For multi-layer networks, the output of the previous layer is used as the input of the next layer, and the above process is repeated. Since the in-memory computing array performs matrix multiplication, convolutional layers are first converted to matrix multiplication using the img2col transformation, or a dedicated convolutional in-memory computing array is used.

[0072] (III) Post-processing and output unit

[0073] The post-processing and output unit is used to convert the raw inference features output by the in-memory computing array into the final perception result. Taking object detection as an example, the detection head features are parsed from the output of the in-memory computing array: bounding box regression values ​​(dx, dy, dw, dh) and class confidence. The absolute coordinates are decoded, and non-maximum suppression (NMS) is performed on candidate boxes that exceed the confidence threshold to remove overlapping boxes. The formatted result is output, and each target includes a class ID, confidence, and normalized coordinates (x1, y1, x2, y2), which is sent to the upper-level system through an interface. The final output result may include one or more of the following: detection box, semantic mask, and target trajectory. In typical scenarios, the data rate of the final perception result output interface can be less than 1 Mbps; this output interface is the final result output interface, which is different from the high-speed interface for transmitting low-dimensional sparse feature vectors between the front-end encoding chip 102 and the back-end processing chip 103. The detection head features include bounding box regression values ​​dx, dy, dw, dh and class confidence. Let the anchor box be (x_a, y_a, w_a, h_a), the center of the predicted box be (x_c, y_c), and the width and height of the predicted box be (w, h). Then: x_c = x_a + dx·w_a; y_c = y_a + dy·h_a; w = w_a·exp(dw); h = h_a·exp(dh). Here, dx and dy represent the offsets of the predicted box center relative to the anchor box center in the x and y directions, respectively; dw and dh represent the logarithmic scaling of the predicted box width and height relative to the anchor box width and height, respectively. The class confidence can be the class probability obtained after applying softmax or sigmoid to the classification branch output, or it can be the product of the objectness confidence and the class probability. The normalized coordinates (x1, y1, x2, y2) are calculated as follows: x1=(x_c-w / 2) / W; y1=(y_c-h / 2) / H; x2=(x_c+w / 2) / W; y2=(y_c+h / 2) / H; where W and H are the width and height of the input image, respectively, and the normalized coordinates are within the range of [0,1].

[0074] It should be noted that the final perception result output interface is the system's final perception result output interface, and its data rate can be less than 1 Mbps in typical scenarios. This final perception result output interface is different from the high-speed interface for transmitting low-dimensional sparse feature vectors from the front-end encoding chip 102 to the back-end processing chip 103. The high-speed interface for transmitting low-dimensional sparse feature vectors from the front-end encoding chip 102 to the back-end processing chip 103 can still be approximately 4.096 Mbps or higher; for example, at 2000 FPS, each frame outputs a 256-dimensional INT8 feature vector, and its data rate is 256 × 2000 × 8 = 4.096 Mbps. If the frame rate, feature dimension, or quantization bit width changes, the data rate of this high-speed interface can be increased accordingly.

[0075] Taking a high-speed industrial part defect detection scenario as an example: a camera module integrating a dynamic visual frame sensor 101 and a front-end encoding chip 102 is aligned with the detection station. The back-end processing chip 103 preloads the INT4 quantization weights of a lightweight target detection network into its SRAM in-memory storage array. The dynamic visual frame sensor 101 captures moving parts at 2000 FPS (speed > 10 m / s, occupying approximately 5% of the screen area, sparse background, total sparsity of approximately 6%). The first stage of the front-end encoding chip 102 performs parallel scanning to generate a coordinate list and a row density histogram; the second stage identifies the part's location as a "dense row" using run-length encoding, and the background behavior as a "sparse row" using direct coordinate encoding; the third stage detects the part's behavior as a "moving row" and performs full encoding, and the background behavior as a "static row" and performs differential encoding (only 1 bit is used to represent the entire row when there is no change); the fourth stage compresses all information into a 256-dimensional feature vector through a sparse convolutional autoencoder. The entire process takes <180 μs. The 256-dimensional feature vector is transmitted to the back-end processing chip 103 via a high-speed interface. The feature decoding and mapping unit restores it to the format required by the network. The sparse sensing controller scans the input vector and finds that about 90% of the values ​​are zero. These columns are skipped in the multiply-accumulate operation. The inference precision adjuster automatically selects INT4 precision based on the 6% sparsity. The in-memory computing array completes the inference of the entire network in about 250μs and outputs the position, orientation, and defect classification of the part. The system outputs part information with detection boxes and classification labels in real time, meeting the needs of industrial real-time control.

[0076] Second embodiment:

[0077] Please refer to Figure 4 and Figure 5 This invention provides a high-throughput dynamic visual frame real-time encoding processing method, comprising the following steps:

[0078] S201: Acquire binary sparse event frames output by dynamic visual frame sensor 101 at a high frame rate;

[0079] Specifically, the dynamic visual frame sensor 101 outputs binary sparse event frames representing pixel-level brightness changes at a high frame rate (up to 2000 frames / second), with each frame data being a binary sparse matrix.

[0080] S202: In the front-end encoding chip 102, multi-level cascaded sparse compression coding is sequentially performed on the binary sparse event frame. The multi-level cascaded sparse compression coding includes sparse acquisition and data normalization processing, intra-frame adaptive spatial compression coding processing, inter-frame motion-aware temporal compression coding processing, and sparse feature extraction and semantic coding processing, and outputs a low-dimensional sparse feature vector.

[0081] Specifically, it includes the following sub-steps:

[0082] Sparse acquisition and data regularization processing. The front-end encoding chip 102 is equipped with 128 parallel scanning engines. Upon the arrival of the frame synchronization signal, all scanning engines start simultaneously, scanning each row they are responsible for line by line using a pixel clock. For the currently scanned pixel, if the value is "1", the event counter corresponding to that row is incremented by 1, and the current column coordinate is written into the coordinate FIFO corresponding to that row; if the pixel value is "0", scanning continues. After a row is scanned, the value in the row's event counter is written into the corresponding H_row[r] register in the global row density histogram register group, and then the row's event counter is cleared. After all rows are scanned, the coordinate FIFOs of each row are concatenated into a complete coordinate list L_coords in row number order, and output along with the filled row density histogram H_row[0...1279].

[0083] Intra-frame adaptive spatial compression coding is used. Thresholds T_low=10 and T_high=100 are set. For each row, the coding mode is selected based on the event density H_row[r] of the corresponding row in the row density histogram: if H_row[r]<10, a sparse row mode is selected, outputting mode code 00 followed by the column coordinates of each event (10 bits each); if H_row[r]>100, a dense row mode is selected, outputting mode code 01 followed by the column coordinates of the first event (10 bits), and then encoding each subsequent event with the column interval of the previous event; otherwise, a mixed mode is selected, dividing the 1024 columns into 32 segments (32 columns per segment), outputting mode code 10, and only encoding the segments containing events. The encoded bitstream is temporarily stored in the row output buffer.

[0084] Inter-frame motion-aware temporal compression coding processing. For each row r (0≤r≤1279), read the row density histogram H_row_prev[r] of the previous frame (read from on-chip SRAM), and calculate the absolute value of the event density difference ΔH = |H_row_cur[r] - H_row_prev[r]| between the row density histogram of the current frame and the row density histogram of the previous frame. Set a motion threshold σ (typically 3~5), and compare ΔH with σ: if ΔH>σ, mark the row as a "moving row"; otherwise, mark it as a "static row". For each moving row, output the intra-frame adaptive spatial compression coding processing result (full data) of the current frame. For each static line, calculate the bitwise XOR value of the intra-frame adaptive spatial compression coding result of the current frame and the intra-frame adaptive spatial compression coding result of the previous frame: diff = C_spatial_cur[r] XOR C_spatial_prev[r]. If diff == 0, output 1 bit '0' to indicate that the line has no change; if diff ≠ 0, output 1 bit '1', followed by the compressed representation of the non-zero part of diff. The frame header, the motion line bitmap consisting of the markers of all motion lines and static lines, the motion line data block, and the static line difference data block are sequentially concatenated to form a time-domain compressed bitstream output.

[0085] Sparse Feature Extraction and Semantic Encoding: A sparse convolutional autoencoder is used to encode the temporal compressed bitstream. The sparse convolutional autoencoder consists of four sparse convolutional layers and one fully connected layer. The sparse convolutional layers employ a non-zero coordinate-driven architecture, extracting the positions and values ​​of all non-zero activations in the input sparse feature map. Sixty-four parallel processing units are allocated to process each non-zero position. Each parallel processing unit reads the non-zero values ​​within its 3×3 neighborhood (by querying the sparse map using a neighborhood address generator), performs a multiplication-accumulation operation with the corresponding convolutional kernel weights, and adds the result to the corresponding position in the output feature map. The fully connected layer maps the output feature map of the sparse convolutional layers into a 256-dimensional feature vector, and then performs INT8 quantization on this 256-dimensional feature vector before outputting it.

[0086] S203: The low-dimensional sparse feature vector is transmitted to the back-end processing chip 103 through a high-speed interface;

[0087] S204: Decode the low-dimensional sparse feature vector in the feature decoding and mapping unit of the back-end processing chip 103, and map the decoded feature vector to the in-memory computing storage array.

[0088] Specifically, the feature decoding and mapping unit decodes the received 256-dimensional INT8 feature vector and maps it into the input format of the in-memory computing inference engine. The mapping mode is selected based on the system configuration: in lightweight mode, the 256-dimensional vector is directly used as the input to the inference network; in high-precision mode, the 256-dimensional vector is reconstructed into a feature map by a lightweight decoder before being fed into the detection or segmentation network. The global sparsity ρ provided by the adaptive control unit (this value is statistically determined by the first stage and transmitted via the interface) is read, and the inference precision is selected based on ρ: INT4 precision (4-bit ADC resolution) is used when ρ > 15%, INT2 precision (2-bit ADC resolution) is used when 5% ≤ ρ ≤ 15%, and INT1 precision (1-bit ADC resolution) is used when ρ < 5%. The input feature vector is expanded row-wise into a column-parallel format and sent to the input register of the in-memory computing array.

[0089] S205: Perform multiply-accumulate operations on the decoded feature vector within the in-memory computing array to complete neural network inference, and skip the corresponding multiply-accumulate operations based on the zero-value components in the decoded feature vector;

[0090] Specifically, during system initialization, the trained inference network weights are written into the in-memory computing array, where they reside long-term without repeated relocation. The decoded feature vector components are sequentially applied to the 256 word lines of the in-memory computing array. Each component first passes through a zero-value detector: if the component is 0, an "open" signal is sent to the corresponding row's gating switch, and all in-memory units in that row do not participate in computation; if it is not 0, a "closed" signal is sent, and the value is converted into an analog voltage via a DAC to drive the word line of that row. For a closed row, the output current of each in-memory unit in that row is equal to the product of the input voltage and the stored weight (0 or 1). The currents of all closed rows in the same column are accumulated on the bit lines in an analog manner. The accumulated current in each column is converted into a digital value by an ADC, and the output precision is dynamically selected by the inference precision regulator based on the real-time event sparsity, choosing between INT4, INT2, or INT1 precision. For multi-layer networks, the output of the previous layer is used as the input of the next layer, and the above process is repeated. Convolutional layers are first converted to matrix multiplication using the img2col transform, or a dedicated convolutional in-memory computing array is used.

[0091] S206: Output object detection results, semantic segmentation results, or object tracking results.

[0092] Specifically, the output results may include one or more of the following: detection bounding boxes, semantic masks, and target trajectories. The definitions of detection head features, category confidence, and normalized coordinates are the same as in the post-processing and output unit section of the specification. Taking target detection as an example, the detection head features (bounding box regression values ​​dx, dy, dw, dh and category confidence) are parsed from the output of the in-memory computing array. The absolute coordinates are decoded, and non-maximum suppression (NMS) is performed on candidate boxes exceeding the confidence threshold to remove overlapping boxes. A formatted result is output (each target includes a category ID, confidence, and normalized coordinates x1, y1, x2, y2), which is sent to the upper-level system via an interface. In a typical 10% sparsity scenario, the system employs pipelined parallel processing, with an equivalent processing frame rate greater than 2000 FPS; the data rate of the final perception result output interface can be less than 1 Mbps.

[0093] Taking an autonomous driving nighttime forward perception scenario as an example: In an autonomous driving nighttime driving scenario, the global sparsity is approximately 8%, with oncoming headlights and pedestrian areas generating localized dense events. After acquiring the binary sparse event frames output by the dynamic visual frame sensor 101 at a high frame rate, multi-level cascaded sparse compression encoding is sequentially performed within the front-end encoding chip 102: the third level analyzes the changes in the row density histogram, automatically marking the rows containing oncoming vehicles and pedestrians as "moving rows" for fine encoding, while significantly compressing "static rows" such as the road surface and dark sky. The output low-dimensional sparse feature vector is transmitted to the back-end processing chip 103 via a high-speed interface. The back-end processing chip 103 performs inference with INT4 precision based on the 8% sparsity. Due to the sparse perception skipping mechanism, the effective computational load is only 1 / 10 of that of a dense network. Finally, the system outputs the detection boxes for vehicles and pedestrians ahead and the segmentation mask for the drivable area within 530μs, leaving a margin of more than 4ms for the autonomous driving decision-making system (relative to the control cycle at a vehicle speed of 5m / s), demonstrating the real-time performance of this invention.

[0094] Depending on the specific requirements of the application regarding cost, power consumption, or accuracy, the parameters of the above system and method can be modified. For example, in ultra-low power scenarios, the inference precision adjuster can be forced to use INT1 or INT2 precision; in scenarios requiring higher precision, the third-level output can be directly sent to the in-memory computing inference engine instead of performing fourth-level deep compression. The physical partitioning of the front-end encoding chip 102 and the back-end processing chip 103 can be optimized according to specific hardware designs, but it does not deviate from the core logic architecture of this invention. The first sparse threshold and the second sparse threshold can also be flexibly configured according to the application scenario, and are not limited to typical values ​​of 15% and 5%.

[0095] This invention utilizes a front-end encoding chip 102 to sequentially perform four-stage cascaded processing on the binary sparse event frames output by the dynamic visual frame sensor 101: sparse acquisition and data normalization, intra-frame adaptive spatial compression coding, inter-frame motion-aware temporal compression coding, and sparse feature extraction and semantic coding. This process progressively compresses massive sparse event frames into low-dimensional sparse feature vectors, effectively eliminating intra-frame spatial redundancy and inter-frame temporal redundancy, achieving a total compression ratio exceeding 1000:1 and microsecond-level encoding latency. Simultaneously, the back-end processing chip 103 uses an in-memory computing array to directly perform multiply-accumulate operations on the decoded feature vectors within the storage array, avoiding the limitations of traditional von Neumann architectures. The power consumption overhead caused by data transfer is mitigated by a sparse sensing controller that skips corresponding multiply-accumulate operations based on zero-value components in the feature vector, and an inference precision regulator that dynamically selects INT4, INT2, or INT1 precision for inference based on the real-time event sparsity. This results in inference energy efficiency exceeding 50 TOPS / W and single-frame inference latency of less than 350μs. Ultimately, from data acquisition and multi-level compression coding to inference computation, the entire chain achieves extremely low latency and high energy efficiency, effectively solving the technical problems of high latency, high power consumption, and low energy efficiency caused by existing technologies such as high latency of coding schemes, high power consumption of data transfer in computing architecture and inability to utilize sparsity, and the separation of coding and inference modules.

[0096] The above-disclosed embodiments are merely one or more preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments and equivalent changes made in accordance with the claims of this application still fall within the scope of this application.

Claims

1. A high-throughput dynamic visual frame real-time encoding and processing system, characterized in that, include: A dynamic visual frame sensor, which outputs binary sparse event frames representing pixel-level brightness changes at a high frame rate. The front-end encoding chip is integrated with the dynamic visual frame sensor in the same module. The front-end encoding chip integrates a multi-level cascaded sparse compression encoder. The sparse compression encoder receives the binary sparse event frame and sequentially performs sparse acquisition and data normalization processing, intra-frame adaptive spatial compression encoding processing, inter-frame motion perception temporal compression encoding processing, and sparse feature extraction and semantic encoding processing to output a low-dimensional sparse feature vector. A back-end processing chip is connected to the front-end encoding chip via a high-speed interface. The back-end processing chip includes a feature decoding and mapping unit and an in-memory computing inference engine. The feature decoding and mapping unit receives and decodes the low-dimensional sparse feature vector. The in-memory computing inference engine includes an in-memory computing integrated storage array, in which the weights of the neural network are pre-stored. The in-memory computing inference engine performs multiply-add operations on the decoded feature vector within the in-memory computing integrated storage array to complete the neural network inference and outputs the target detection result, semantic segmentation result, or target tracking result.

2. The high-throughput dynamic visual frame real-time encoding and processing system as described in claim 1, characterized in that, Sparse acquisition and data normalization processing specifically include: The binary sparse event frame is scanned in parallel for each row, and the binary sparse event frame is converted into a list of coordinates sorted by row. During the scanning process, the event density of each row is calculated to generate a row density histogram.

3. The high-throughput dynamic visual frame real-time encoding and processing system as described in claim 2, characterized in that, Intra-frame adaptive spatial compression coding processing specifically includes: For each row, run-length encoding is adaptively selected based on the event density of the corresponding row in the row density histogram, choosing between sparse row mode, dense row mode, or mixed mode. When the event density is less than the first threshold, the sparse row mode is selected to directly encode the column coordinates of each event in the coordinate list; When the event density is greater than the second threshold, the dense row mode is selected to encode the column coordinates of the first event in the coordinate list and the column interval between each subsequent event and the previous event. When the event density is greater than or equal to the first threshold and less than or equal to the second threshold, the hybrid mode is selected, each row is divided into multiple column segments, and only the column segments containing events are encoded.

4. The high-throughput dynamic visual frame real-time encoding and processing system as described in claim 3, characterized in that, Inter-frame motion-aware temporal compression coding processing specifically includes: Read the row density histogram of the previous frame and calculate the absolute value of the difference in event density between each row of the row density histogram of the current frame and the row density histogram of the previous frame. Rows whose absolute value is greater than the motion threshold are marked as moving rows, and rows whose absolute value is less than or equal to the motion threshold are marked as static rows; For each of the motion lines, output the intra-frame adaptive spatial compression coding result of the current frame; For each static line, calculate the bitwise XOR value of the intra-frame adaptive spatial compression coding result of the current frame and the intra-frame adaptive spatial compression coding result of the previous frame. When the bitwise XOR value is zero, output 1 bit to indicate no change. When the bitwise XOR value is non-zero, output 1 bit flag followed by the compressed representation of the non-zero part of the bitwise XOR value. The frame header, the motion line bitmap consisting of markers for all the motion lines and the static lines, the motion line data block, and the static line difference data block are sequentially concatenated to form a time-domain compressed bitstream.

5. The high-throughput dynamic visual frame real-time encoding and processing system as described in claim 4, characterized in that, Sparse feature extraction and semantic encoding processing specifically include: The temporal compressed bitstream is encoded using a sparse convolutional autoencoder, which includes multiple sparse convolutional layers and one fully connected layer. The sparse convolutional layer adopts a non-zero coordinate driven architecture, extracts the position and value of all non-zero activation values ​​in the input sparse feature map, allocates multiple parallel processing units to process each non-zero position, each of the parallel processing units reads the non-zero values ​​in the neighborhood around the current non-zero position and performs a multiply-add operation with the corresponding convolutional kernel weights, and accumulates the result of the multiply-add operation to the corresponding position in the output feature map. The fully connected layer maps the output feature map of the sparse convolutional layer into a 256-dimensional feature vector, and outputs the 256-dimensional feature vector after INT8 quantization.

6. The high-throughput dynamic visual frame real-time encoding and processing system as described in claim 5, characterized in that, The feature decoding and mapping unit decodes and maps the 256-dimensional feature vector into the input format of the in-memory computing inference engine. The in-memory computing inference engine also includes a sparse perceptual controller and an inference precision adjuster. The sparse sensing controller identifies the zero-value components in the decoded feature vector and controls the in-memory computing array to skip the multiply-accumulate operations corresponding to the zero-value components. The inference precision adjuster selects INT4 precision, INT2 precision, or INT1 precision for inference based on the real-time event sparsity rate obtained from the sparse acquisition and data normalization processing. When the real-time event sparsity rate is greater than a first sparsity threshold, the INT4 precision is selected; when the real-time event sparsity rate is less than or equal to the first sparsity threshold but greater than a second sparsity threshold, the INT2 precision is selected; and when the real-time event sparsity rate is less than or equal to the second sparsity threshold, the INT1 precision is selected.

7. The high-throughput dynamic visual frame real-time encoding and processing system as described in claim 1, characterized in that, The front-end encoding chip is manufactured using logic technology and is directly connected to the dynamic visual frame sensor via a chip-level parallel bus. The back-end processing chip is manufactured using in-memory computing static random access memory technology. The front-end encoding chip and the back-end processing chip are connected through a high-speed serial interface and integrated into the same imaging computing module.

8. A high-throughput dynamic visual frame real-time encoding processing method, applicable to the high-throughput dynamic visual frame real-time encoding processing system as described in claim 1, characterized in that, Includes the following steps: Acquire binary sparse event frames output by a dynamic visual frame sensor at a high frame rate; In the front-end encoding chip, multi-level cascaded sparse compression coding is sequentially performed on the binary sparse event frame. The multi-level cascaded sparse compression coding includes sparse acquisition and data normalization processing, intra-frame adaptive spatial compression coding processing, inter-frame motion-aware temporal compression coding processing, and sparse feature extraction and semantic coding processing, outputting a low-dimensional sparse feature vector. The low-dimensional sparse feature vector is transmitted to the back-end processing chip via a high-speed interface; The low-dimensional sparse feature vector is decoded in the feature decoding and mapping unit of the back-end processing chip, and the decoded feature vector is mapped to the in-memory computing storage array. The decoded feature vector is multiplied and added within the in-memory computing array to complete neural network inference, and the corresponding multiplied and added operations are skipped based on the zero-value components in the decoded feature vector. Output object detection results, semantic segmentation results, or object tracking results.

Images