Vla edge processor, system, and method based on static token filtering and hybrid near-memory computation

Abstract

The application relates to the technical field of robot control, and discloses a VLA edge processor, a system and a method based on static token filtering and hybrid near-memory computing. The VLA edge processor based on static token filtering and hybrid near-memory computing comprises a task scheduling and configuration unit, a static token filter, a near-memory computing execution unit and a hybrid on-chip storage system. The task scheduling and configuration unit is connected with the static token filter, the near-memory computing execution unit and the hybrid on-chip storage system respectively. The static token filter is connected with the near-memory computing execution unit and the hybrid on-chip storage system respectively. The near-memory computing execution unit is connected with the hybrid on-chip storage system. The application solves the calculation redundancy through the static token filter, solves the memory bottleneck through the near-memory computing execution unit and the hybrid on-chip storage system, and thus improves the processing efficiency of model deployment on a robot edge device.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a VLA edge processor, system, and method based on static lexical filtering and hybrid near-memory computing. Background Technology

[0002] Vision-Language-Action (VLA) models have become the foundation of embodied AI, enabling autonomous robots to directly translate complex multimodal inputs, such as visual observations and natural language commands, into robot actions. A typical VLA model consists of a Vision-Language Model (VLM) for perception and an Action Expert (AE) based on a Diffusion Transformer (DiT) for action generation. The VLM passes extracted semantic features to the AE through a key-value cache to guide the generation of the robot's trajectory, such as... Figure 1 As shown.

[0003] Unlike natural language processing or image generation, deploying VLA models on real robots requires rigorous low-latency inference and real-time responses to environmental changes. However, current GPU-based systems operate at frequencies of only around 5 Hz, far below the frequencies (tens or hundreds of Hz) required for real-time robot control. While dedicated accelerators have been proposed to improve AE performance, they typically offload VLM to the cloud, introducing significant communication latency and hindering timely responses to dynamic environments. These limitations underscore the need for dedicated end-to-end VLM-AE accelerators to achieve low-latency, smooth, and responsive robot control.

[0004] Designing such accelerators remains challenging due to the different performance bottlenecks of VLM and AE (e.g., Figure 2(As shown). Specifically, VLM is limited by computational power, but much of its computation is redundant: over 50% of visual input terms remain static (e.g., background regions) between consecutive robot action steps and are repeatedly processed, leading to severe inefficiency. In contrast, AE is limited by memory bandwidth, with numerous expensive external memory accesses accounting for over 80% of total energy consumption, constituting a key bottleneck for real-time trajectory generation. Existing solutions mainly include GPU-based cloud inference systems, dedicated AE accelerators, and dedicated VLM accelerators, but they either lack real-time performance and have high communication latency, or only optimize a single module and cannot achieve end-to-end collaborative acceleration. Therefore, existing end-to-end edge accelerators cannot simultaneously overcome the VLM computational bottleneck and the AE memory bottleneck, and cannot achieve low-latency, high-energy-efficiency VLA model edge inference, thus making it difficult to support real-time, smooth, and responsive robot control.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] The main purpose of this application is to provide a VLA edge processor, system and method based on static lexical filtering and hybrid near-memory computing, which aims to solve the problem that in the prior art, accelerators are only optimized for VLM or AE, and cannot simultaneously improve the computing power of VLM and the memory bandwidth of AE, resulting in low processing efficiency of the model deployed on the edge device of the robot.

[0007] The first aspect of this application provides a VLA edge processor based on static lexical filtering and hybrid near-memory computing. The VLA edge processor includes a task scheduling and configuration unit, a static lexical filter, a near-memory computing execution unit, and a hybrid on-chip storage system. The task scheduling and configuration unit is connected to the static lexical filter, the near-memory computing execution unit, and the hybrid on-chip storage system. The static lexical filter is connected to the near-memory computing execution unit and the hybrid on-chip storage system. The near-memory computing execution unit is connected to the hybrid on-chip storage system. The task scheduling and configuration unit is used to obtain VLA inference tasks and configure the working parameters of the static lexical filter, the near-memory computation execution unit and the hybrid on-chip memory system according to the VLA inference tasks. The static lexical filter is used to acquire high-dimensional lexical data stream, filter the high-dimensional lexical data stream, generate dynamic data, send it to the near-memory computing execution unit, and send the key-value cache corresponding to the dynamic data to the hybrid on-chip storage system for selective updating; The hybrid on-chip storage system is used to store the target weights and send them to the near-memory computation execution unit; The near-memory computation execution unit is used to perform near-memory computation based on the dynamic data and the target weight, obtain the computation result, and send it to the hybrid on-chip memory system for storage. The task scheduling and configuration unit is also used to output robot joint motion commands through a peripheral interface when it detects that the calculation results have been stored in the hybrid on-chip memory system.

[0008] Optionally, in one embodiment of this application, the task scheduling and configuration unit includes a RISC-V core, a top-level controller, and a configuration register file. The RISC-V core is connected to the top-level controller and the configuration register file via a bus. The top-level controller and the configuration register file are controlled and connected to the static lexical filter, the near-memory computation execution unit, and the hybrid on-chip memory system.

[0009] Optionally, in one embodiment of this application, the static lexical filter includes a visual redundancy detector, a computation jumper, and a key-value updater. The visual redundancy detector is connected to the computation jumper and the key-value updater, respectively. The key-value updater is connected to the hybrid on-chip storage system and is used to selectively write the key-value cache corresponding to the dynamic data into the hybrid on-chip storage system.

[0010] Optionally, in one embodiment of this application, the near-memory computation execution unit includes a processing unit array, a SIMD vector kernel, a weight distributor, an activation value distributor, and an accumulation buffer. The computation jumper is connected to the activation value distributor, the weight distributor and the activation value distributor are respectively connected to the processing unit array, the processing unit array is connected to the accumulation buffer, and the accumulation buffer is connected to the SIMD vector kernel.

[0011] Optionally, in one embodiment of this application, the hybrid on-chip memory system includes an RRAM memory and a dynamic multi-array SRAM memory. The weight distributor is connected to the RRAM memory, the activation value distributor is connected to the dynamic multi-array SRAM memory, the KV updater is connected to the dynamic multi-array SRAM memory, and is used to selectively write the key-value cache corresponding to the dynamic data into the dynamic multi-array SRAM memory. The SIMD vector kernel is connected to the dynamic multi-array SRAM memory and is used to write the post-processed calculation results back to the dynamic multi-array SRAM memory.

[0012] Optionally, in one embodiment of this application, the VLA edge processor based on static lexical filtering and hybrid near-memory computing further includes a peripheral interface, which includes at least one of a JTAG interface, a UART interface, and a QSPI interface, and the RISC-V core is connected to the peripheral interface via a bus.

[0013] A second aspect of this application also provides a robot system, wherein the robot system includes a VLA edge processor based on static lexical filtering and hybrid proximity computing as described in any of the above embodiments and a robot body, the VLA edge processor based on static lexical filtering and hybrid proximity computing is disposed on the robot body, the robot body has a plurality of joints, and the plurality of joints move according to the action instructions output by the VLA edge processor based on static lexical filtering and hybrid proximity computing.

[0014] A third aspect of this application also provides a data processing method for a VLA edge processor based on static lexical filtering and hybrid near-memory computation as described in any of the above-mentioned schemes, wherein the data processing method includes: The task scheduling and configuration unit acquires the VLA inference task and configures the working parameters of the static lexical filter, the near-memory computation execution unit, and the hybrid on-chip memory system according to the VLA inference task. The static lexical filter acquires the high-dimensional lexical data stream, filters the high-dimensional lexical data stream, generates dynamic data, sends it to the near-memory computing execution unit, and sends the key-value cache corresponding to the dynamic data to the hybrid on-chip storage system for selective updates. The hybrid on-chip memory system stores the target weights and sends them to the near-memory computation execution unit; The near-memory computation execution unit performs near-memory computation based on the dynamic data and the target weight, obtains the computation result, and sends it to the hybrid on-chip memory system for storage. When the task scheduling and configuration unit detects that the calculation results have been stored in the hybrid on-chip memory system, it outputs motion commands for the robot joints.

[0015] Optionally, in one embodiment of this application, the step of filtering the high-dimensional terminology data stream to generate dynamic data specifically includes: The visual redundancy detector maps the high-dimensional word data stream to obtain the target fingerprint of each high-dimensional word element, and compares the target fingerprint with the historical fingerprint to generate a decision result; If the judgment result is static, then the jumper is calculated to block the data path of the high-dimensional word corresponding to the target fingerprint; If the decision result is dynamic, the calculation jumper allows the original data of the high-dimensional word corresponding to the target fingerprint and sends the original data of the high-dimensional word to the activation value distributor of the near-memory computation execution unit. The KV updater writes the new key-value pair corresponding to the high-dimensional word into the dynamic multi-array SRAM of the hybrid on-chip memory system and writes the target fingerprint into the fingerprint buffer of the KV updater itself.

[0016] Optionally, in one embodiment of this application, the dynamic data is a dynamic lexical activation value; The step of performing near-memory calculation based on the dynamic data and the target weight to obtain the calculation result specifically includes: The activation value distributor obtains the dynamic lexical activation values ​​allowed by the static lexical filter from the on-chip dynamic multi-array SRAM memory and sends them to the processing unit array. The processing unit array performs calculations based on the target weight and the dynamic lexical activation value to obtain the calculation result.

[0017] Beneficial effects: This application provides a VLA edge processor, system and method based on static lexical filtering and hybrid near-memory computing. This application solves computational redundancy through static lexical filtering and memory bottleneck through near-memory computing execution unit and hybrid on-chip storage system. At the same time, it improves VLM computing power and AE memory bandwidth, achieves low latency, low power consumption and reduces computing resource waste, thereby improving the processing efficiency of model deployment on robot edge devices. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram of a VLM model used for robot control; Figure 2 A diagram illustrating the different performance bottlenecks of VLM and AE; Figure 3 This is a processor block diagram of the VLA edge processor based on static lexical filtering and hybrid near-memory computing in this application; Figure 4 This is a diagram of the overall processor architecture in a preferred embodiment of the VLA edge processor based on static lexical filtering and hybrid near-memory computing in this application. Figure 5This is a schematic diagram of the static word filter in a preferred embodiment of the VLA edge processor based on static word filtering and hybrid near-memory computing in this application; Figure 6 This is a diagram of the hybrid RRAM-SRAM near-memory computing architecture in a preferred embodiment of the VLA edge processor based on static lexical filtering and hybrid near-memory computing in this application. Figure 7 This is a diagram of a dynamic multi-array SRAM architecture with online compression, as shown in a preferred embodiment of the VLA edge processor based on static lexical filtering and hybrid near-memory computing in this application. Figure 8 This is a diagram of a scalable multi-chip module supporting overlapping execution in a preferred embodiment of the VLA edge processor based on static lexical filtering and hybrid near-memory computing in this application. Figure 9 This is a flowchart of the data processing method of the VLA edge processor based on static lexical filtering and hybrid near-memory computation in this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and effects of this application clearer and more explicit, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only possible technical implementations of this application and not all possible implementations. Based on the embodiments in this application, those skilled in the art can obtain other embodiments without creative effort, and these embodiments are also within the protection scope of this application.

[0021] First, let's introduce the terms used in the embodiments of this application: VLA, Vision-Language-Action; VLM, Vision-Language Model; AE stands for Action Expert. DiT, Diffusion Transformer, Diffusion Transformer; KV Cache, Key-Value Cache; PNM, Processing Near Memory; SimHash, Similarity Hash; RRAM, Resistive Random Access Memory; SRAM, Static Random Access Memory; MCM stands for Multi-Chip Module. SIMD, Single Instruction, Multiple Data; RISC-V, Reduced Instruction Set Computer - V; PEA, Processing Element Array; SoC, System on Chip; JTAG, Joint Test Action Group (Debugging Interface Standard). UART stands for Universal Asynchronous Receiver / Transmitter. QSPI, Quad Serial Peripheral Interface.

[0022] EMA, External Memory Access.

[0023] This application applies to the fields of embodied artificial intelligence and real-time robot control. This application presents a real-time end-to-end vision-language-action (VLA) edge processor, aiming to solve the problems of high latency, high energy consumption, and wasted computing resources faced by current VLA models when deployed on robot edge devices. Specifically, this application focuses on solving the following three core technical problems: 1. Solving the serious computational redundancy problem in VLM inference process: In view of the defect that VLM has a large number of static visual terms (such as background regions) being repeatedly processed between consecutive robot action steps, resulting in wasted computing power, this application can identify and eliminate these static term redundant calculations in the time dimension.

[0024] 2. Solving the problems of limited memory bandwidth and high energy consumption during AE execution: In response to the bottleneck that AE is limited by memory bandwidth when generating execution trajectory and that frequent EMA accounts for the majority (more than 80%) of the total system energy consumption, this application can significantly reduce or even eliminate external storage dependence.

[0025] 3. Solving the communication latency and system-level collaborative deployment issues of end-to-end VLA systems: In response to the problem that existing solutions cause excessive communication latency by offloading VLM to the cloud, or that a single chip cannot simultaneously handle both compute-intensive VLM and memory-intensive AE, this application can achieve low latency and high throughput at the edge to meet the strict low-latency response requirements of robot real-time control.

[0026] This application aims to comprehensively solve the above-mentioned technical problems by using a hardware-friendly lexical filtering mechanism, a hybrid processing near-memory architecture, and a scalable multi-chip module design, so as to achieve low latency, high energy efficiency, and end-to-end edge VLA inference.

[0027] The technical solutions of this application will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0028] The VLA edge processor based on static lexical filtering and hybrid near-memory computing described in the preferred embodiment of this application, such as... Figure 3 As shown, the VLA edge processor based on static lexical filtering and hybrid near-memory computing includes a task scheduling and configuration unit, a static lexical filter, a near-memory computing execution unit, and a hybrid on-chip storage system. The task scheduling and configuration unit is connected to the static lexical filter, the near-memory computing execution unit, and the hybrid on-chip storage system. The static lexical filter is connected to the near-memory computing execution unit and the hybrid on-chip storage system. The near-memory computing execution unit is connected to the hybrid on-chip storage system. The task scheduling and configuration unit is used to obtain VLA inference tasks and configure the working parameters of the static lexical filter, the near-memory computation execution unit and the hybrid on-chip memory system according to the VLA inference tasks. The static lexical filter is used to acquire high-dimensional lexical data stream, filter the high-dimensional lexical data stream, generate dynamic data, send it to the near-memory computing execution unit, and send the key-value cache corresponding to the dynamic data to the hybrid on-chip storage system for selective updating; The hybrid on-chip storage system is used to store the target weights and send them to the near-memory computation execution unit; The near-memory computation execution unit is used to perform near-memory computation based on the dynamic data and the target weight, obtain the computation result, and send it to the hybrid on-chip memory system for storage. The task scheduling and configuration unit is also used to output robot joint motion commands through a peripheral interface when it detects that the calculation results have been stored in the hybrid on-chip memory system.

[0029] It should be noted that the VLA edge processor based on static token filtering and hybrid near-memory computation in this application is a real-time and end-to-end VLA edge processor. First, it adopts a static token filter to identify and skip static tokens to eliminate redundant computation. Second, it adopts a hybrid near-memory engine, utilizing high-density embedded resistive random access memory (RRAM) and dynamic multi-array static random access memory (SRAM) to eliminate external memory access. Third, it adopts scalable multi-chip modules and overlapping scheduling technology to achieve high-throughput VLM-AE collaborative execution.

[0030] Specifically, the overall processor architecture is as follows: Figure 4 As shown, the design includes a RISC-V core for top-level control; a 32×32 processing unit array for performing fixed-weight multiply-accumulate operations; a 32-way SIMD vector core for post-processing; and a hybrid on-chip memory system consisting of 16 Mb RRAM and 1 Mb SRAM. The embedded RRAM is implemented using foundry-sourced 1T1R cells, featuring high storage density (cell area of ​​53 F²) and supporting 512-bit parallel reads (providing 76.8 Gb / s bandwidth at 150 MHz). The SRAM buffer consists of 32 dual-port 256×128 memory arrays, supporting dynamic orchestration during inference to eliminate external memory access for activation values. Activation values ​​are transferred between connected chips via an inter-chip communication interface. This design supports scalable multi-chip modules (MCMs, implementable via PCB or individual chips), enabling the deployment of larger-scale models.

[0031] In one embodiment of this application, the task scheduling and configuration unit includes a RISC-V core, a top-level controller, and a configuration register file. The RISC-V core is connected to the top-level controller and the configuration register file via a bus. The top-level controller and the configuration register file are controlled and connected to the static lexical filter, the near-memory computation execution unit, and the hybrid on-chip memory system.

[0032] In one embodiment of this application, the static lexical filter includes a visual redundancy detector, a computation jumper, and a key-value updater. The visual redundancy detector is connected to the computation jumper and the key-value updater, respectively. The key-value updater is connected to the hybrid on-chip storage system and is used to selectively write the key-value cache corresponding to dynamic data into the hybrid on-chip storage system.

[0033] Specifically, a static term filter based on SimHash, such as... Figure 5As shown, to eliminate redundant computations in VLM, this application employs a SimHash-based lexical filter, which comprises a visual redundancy detector, a computation jumper, and a key-value updater. The detector converts high-dimensional visual tokens (e.g., 960 dimensions in SmolVLA) into 64-bit fingerprints using binary weight multiplication and symbol-based quantization, following the SimHash algorithm. Unlike existing methods that require buffering and comparing dense pixel-level activation values, resulting in high storage and bandwidth overhead, this application relies on compact fingerprints, reducing storage or bandwidth requirements by 86.7%. Subsequently, the computation jumper filters out static lexical tokens, distributing only dynamic lexical tokens to the processing unit array for computation and key-value updates. Overall, the lexical filter in this application reduces computation by 59.6% with a success rate loss of only 3%.

[0034] Understandably, the static lexical filter performs intelligent screening to filter redundant data. When a new visual image frame is converted into a high-dimensional lexical data stream, it first enters the visual redundancy detector. The detector immediately generates a 64-bit compact fingerprint for each lexical element (such as a small patch of an image). The detector compares this fingerprint with the fingerprint of the corresponding lexical element in the previous frame stored in the KV updater. If the two fingerprints are the same (e.g., a background wall), it means that this part of the data has not changed and is redundant, marked as static. If the two fingerprints are different (e.g., a moving hand), it means that this is new information that needs to be processed, marked as dynamic. The computation jumper only performs physical operations based on the markings, directly discarding "static" lexical elements and not passing them forward; only the data of dynamic lexical elements is allowed to pass and sent to the activation value distributor at the back end. At the same time, the KV updater also receives instructions to only write the new key-value pairs corresponding to the dynamic lexical elements into the dynamic multi-array SRAM as a new cache, while the static information of the previous frame is retained to avoid duplicate writing.

[0035] In one embodiment of this application, the near-memory computation execution unit includes a processing unit array, a SIMD vector kernel, a weight distributor, an activation value distributor, and an accumulation buffer. The computation jumper is connected to the activation value distributor. The weight distributor and the activation value distributor are respectively connected to the processing unit array. The processing unit array is connected to the accumulation buffer, and the accumulation buffer is connected to the SIMD vector kernel.

[0036] In one embodiment of this application, the hybrid on-chip memory system includes an RRAM memory and a dynamic multi-array SRAM memory. The weight distributor is connected to the RRAM memory, the activation value distributor is connected to the dynamic multi-array SRAM memory, the KV updater is connected to the dynamic multi-array SRAM memory, and is used to selectively write the key-value cache corresponding to the dynamic data into the dynamic multi-array SRAM memory. The SIMD vector kernel is connected to the dynamic multi-array SRAM memory and is used to write the post-processed calculation results back to the dynamic multi-array SRAM memory.

[0037] Specifically, hybrid near-memory processing engines that eliminate external memory access, such as... Figure 6 As shown, to eliminate expensive external memory access, this application employs a hybrid near-memory (PNM) computing architecture. All fixed VLM and AE weights are stored in on-chip high-density embedded RRAM, achieving non-volatile storage through 4 / 8-bit quantization and 2:4 sparse compression. As a supplement, a dynamic multi-array SRAM buffer is flexibly allocated to the KV cache, intermediate activation values, and compressed large tensor matrices, such as... Figure 7 As shown, the online compression within the SRAM buffer is coordinated by a RISC-V core, which performs real-time monitoring and dynamic configuration to maximize buffer utilization. By combining the aforementioned SimHash-based term-level filtering and element-level online zero compression, the SRAM data footprint is significantly reduced by 3.23 times. Furthermore, by leveraging dynamic data flow, including head-splitting attention and term-splitting feedforward networks, this architecture successfully achieves zero external memory access for all activation values.

[0038] Understandably, the on-chip closed-loop computation, which performs near-memory calculations, is completed within the chip, aiming to efficiently process the selected dynamic data. The on-chip convergence of weights and activation values ​​is key to zero external memory access. The activation value distributor sends the released dynamic term data to the 32×32 processing unit array. Simultaneously, the weight distributor directly reads the model's fixed weight parameters from the on-chip 16 Mb RRAM and also sends them to the array. Both data streams are on-chip, eliminating the need to access external DRAM. The 32×32 processing unit array performs the crucial multiply-accumulate operation. The intermediate results are temporarily stored in an accumulation buffer and then sent to a 32-way SIMD vector kernel for non-linear post-processing such as activation function and normalization.

[0039] After post-processing, according to the VLA model mechanism, this feature information will be used as a new key-value cache and needs to be passed to the subsequent AE module. At this time, the RISC-V core will coordinate to perform online zero compression on the newly calculated tensor before writing it to 1 Mb dynamic multiarray SRAM, removing invalid zero values ​​and maximizing the use of valuable on-chip cache space.

[0040] In a 16-chip system, each chip executes one layer of the model. Once a chip completes its task, its RISC-V core sends the compressed data out through the inter-chip communication interface. The next chip starts its own layer's computation almost simultaneously with receiving the data, forming a seamless overlapping execution pipeline.

[0041] After processing through all layers of the VLM and AE parts, the final chip outputs not image features, but specific motion commands that directly control the robot joints, such as the trajectory points of the robotic arm's end effector in space.

[0042] Furthermore, the visual redundancy detector includes a fingerprint generation module and a fingerprint comparison module, wherein the fingerprint comparison module is connected to the fingerprint generation module and the KV updater.

[0043] Furthermore, the KV updater includes an update control logic unit and a fingerprint storage buffer, the fingerprint storage buffer being connected to the visual redundancy detector, and the update control logic unit being connected to the dynamic multi-array SRAM.

[0044] The fingerprint generation module receives high-dimensional visual word vectors (e.g., 960-dimensional) as input. Internally, it stores the binary weight matrix of the SimHash algorithm. Its core function is to map complex high-dimensional words into a compact 64-bit binary fingerprint, reducing the hardware overhead of subsequent comparisons. The fingerprint comparison module connects the fingerprint generation module and the KV updater at its input. Internally, it contains a register set used to store the fingerprints of words at corresponding positions in the previous frame. Its function is to calculate the newly generated fingerprint against historical fingerprints (provided by the KV updater), quickly determining whether they are "same" or "different," and outputting a 1-bit "static / dynamic" control tag.

[0045] The computation jumper includes a control tag-driven multiplexer (MUX) or gated clock logic that simultaneously receives the raw term data stream and control tags from the visual redundancy detector. If the tag is "static," it physically blocks the data path for that term (does not send data to the backend) and gates the corresponding computation clock, eliminating redundant computation at the source. Only when the tag is "dynamic" will it pass the term data intact to the backend near-memory computation execution unit.

[0046] The KV updater consists of an update control logic unit and a fingerprint storage buffer. Its internal fingerprint buffer stores the fingerprint from the last inference, which is directly output to the visual redundancy detector for comparison; it also receives the "static / dynamic" label from the visual redundancy detector. Only when the label is "dynamic" will a write operation be performed, updating the newly generated KV cache and the new fingerprint to the backend hybrid on-chip storage system and its own fingerprint buffer, respectively. If the label is "static," the existing data remains unchanged.

[0047] Furthermore, the dynamic multi-array SRAM memory includes 32 dual-port SRAM storage subarrays, the input of each SRAM storage subarray is connected to the KV updater and the SIMD vector core, and the output of each SRAM storage subarray is connected to the activation value distributor.

[0048] The 16Mb RRAM memory comprises a high-density, non-volatile 1T1R resistive switching cell array and integrates high-bandwidth parallel readout circuitry. Serving as the sole weight repository, it stores all fixed weights for the VLM and AE models. These weights undergo 4 / 8-bit quantization and 2:4 sparse compression, allowing for read-only inference after a single write operation. Furthermore, a 512-bit wide parallel read interface continuously supplies weight data to the computing unit with a high bandwidth of 76.8 Gb / s, eliminating off-chip memory access for weights.

[0049] The dynamic multi-array SRAM memory (1Mb) comprises 32 dual-port 256×128 SRAM modules and integrates an online compression and dynamic orchestration control logic coordinated by the RISC-V core. As a flexible on-chip data buffer, it serves as the central hub for all dynamic data. Dynamically partitioned, it serves three key functions: a KV cache center storing key-value pairs generated by the VLM to guide AE ​​action generation and accepting selective updates from the static terminology filter; an activation value buffer storing all intermediate activation value tensors (feature maps) generated during inference for use in the next layer's computation; and a final result buffer temporarily storing action instruction data representing the final action trajectory after the AE module completes all iterations.

[0050] In one embodiment of this application, the VLA edge processor based on static lexical filtering and hybrid near-memory computing further includes a peripheral interface, which includes at least one of a JTAG interface, a UART interface, and a QSPI interface. The RISC-V core is connected to the peripheral interface via a bus.

[0051] In this embodiment, the RISC-V core is bidirectionally connected to the top-level controller and configuration register file via a bus. The RISC-V core issues tasks and configuration parameters, and the state of the registers is read by the RISC-V core. The top-level controller and configuration register file have control connections with all other modules, enabling the writing of configuration parameters into the local control registers of each module and the issuance of start / stop control signals.

[0052] External lexical data is transmitted to the visual redundancy detector, which is connected to the computation jumper (to transmit static or dynamic tags), and the visual redundancy detector is connected to the KV updater (to synchronously transmit tags). The computation jumper is connected to the activation value distributor (to only allow dynamic lexical data).

[0053] The RRAM is connected to the processing unit array via a weight distributor, the SRAM is connected to the processing unit array via an activation value distributor, the processing unit array is connected to an accumulation buffer, the accumulation buffer is connected to a SIMD vector kernel, and the SIMD vector kernel is connected to a dynamic multi-array SRAM (for writing back computation results). The KV updater is connected to the dynamic multi-array SRAM (for selectively writing new KV pairs).

[0054] The RISC-V core has bidirectional data connections with dynamic multi-array SRAM and peripheral interfaces (QSPI, UART) for online compression coordination and final result output.

[0055] This application supports Scalable Multi-Chip Modules (MCMs) with overlapping execution to achieve high-throughput end-to-end execution, such as... Figure 8 As shown in the diagram. In a specific embodiment, a 16-chip MCM is used, with each chip responsible for executing a corresponding layer of VLM and AE. Thanks to the aforementioned lexical filtering and inline compression techniques, the inter-chip data transfer volume of each layer is significantly compressed to 0.64-0.78 MB, thereby relaxing the bandwidth requirement of the MCM to 50 MB / s. Furthermore, the system adopts an overlapped execution scheduling scheme, enabling concurrent computation and inter-chip data transfer to occur simultaneously. Compared to the sequential execution method, the latency of VLM and AE is significantly reduced by 14.33 times and 4.13 times, respectively.

[0056] This application has the following technical effects: 1. Temporal lexical filtering eliminates visual redundancy computation. By using SimHash-based temporal lexical filtering, this invention identifies and skips static visual lexical units between consecutive action steps, reducing overall computation by 57.4%–60.9% while decreasing the task success rate by only about 3%. This mechanism effectively overcomes the computational redundancy bottleneck in VLM inference.

[0057] 2. Hybrid near-memory computing achieves zero external memory access. Through a hybrid RRAM-SRAM near-memory computing engine, this invention stores all weights and activation values ​​on-chip, achieving zero external memory access and completely eliminating the memory access bottleneck of the AE module. SRAM data footprint is reduced by 3.23 times, and energy consumption per inference is only 10.46–12.74 mJ.

[0058] 3. Scalable MCM for End-to-End VLM-AE Acceleration. Through a scalable multi-chip module architecture (supporting PCB or chiplet methods), the 16-chip prototype system of this invention achieves end-to-end VLM-AE inference using a 55nm CMOS process. On the Libero benchmark, the end-to-end inference latency is 5.31–6.66 ms, and the perception-action latency is 6.7 ms (corresponding to a 150Hz control frequency), significantly better than existing GPU systems (approximately 5 Hz). Compared to existing VLM or AE accelerators, this invention achieves a 1.87–88.45x reduction in VLM and AE latency and, for the first time, realizes end-to-end VLM-AE edge acceleration, meeting the stringent latency requirements of real-time robot control.

[0059] Based on the above embodiments, this application also provides a robot system, wherein the robot system includes a VLA edge processor based on static lexical filtering and hybrid proximity computing as described in any of the above schemes and a robot body, the VLA edge processor based on static lexical filtering and hybrid proximity computing is disposed on the robot body, the robot body has multiple joints, and the multiple joints move according to the action instructions output by the VLA edge processor based on static lexical filtering and hybrid proximity computing.

[0060] Based on the above embodiments, this application also provides a data processing method for a VLA edge processor based on static lexical filtering and hybrid near-memory computation as described in any of the above schemes, wherein, as Figure 9 As shown, the data processing method includes: Step S10: The task scheduling and configuration unit obtains the VLA inference task and configures the working parameters of the static lexical filter, the near-memory computation execution unit, and the hybrid on-chip memory system according to the VLA inference task.

[0061] Step S20: The static lexical filter obtains the high-dimensional lexical data stream, filters the high-dimensional lexical data stream, generates dynamic data, sends it to the near-memory computing execution unit, and sends the key-value cache corresponding to the dynamic data to the hybrid on-chip storage system for selective updating.

[0062] In one possible implementation, step S20 specifically includes: the visual redundancy detector maps the high-dimensional word data stream to obtain the target fingerprint of each high-dimensional word, and compares the target fingerprint with the historical fingerprint to generate a decision result; if the decision result is static, the computation jumper blocks the data path of the high-dimensional word corresponding to the target fingerprint; if the decision result is dynamic, the computation jumper allows the original data of the high-dimensional word corresponding to the target fingerprint, and sends the original data of the high-dimensional word to the activation value distributor of the near-memory computation execution unit; the KV updater writes the new key-value pair corresponding to the high-dimensional word into the dynamic multi-array SRAM of the hybrid on-chip memory system, and writes the target fingerprint into the fingerprint buffer of the KV updater itself.

[0063] Step S30: The hybrid on-chip memory system saves the target weight and sends it to the near-memory computing execution unit.

[0064] Step S40: The near-memory calculation execution unit performs near-memory calculation based on the dynamic data and the target weight, obtains the calculation result, and sends it to the hybrid on-chip storage system for storage.

[0065] In one possible implementation, the dynamic data is dynamic lexical activation values. Step S40 specifically includes: the activation value distributor obtains the dynamic lexical activation values ​​allowed by the static lexical filter from the on-chip dynamic multi-array SRAM memory and sends them to the processing unit array; the processing unit array performs calculations based on the target weight and the dynamic lexical activation values ​​to obtain the calculation result.

[0066] Step S50: When the task scheduling and configuration unit detects that the calculation results have been stored in the hybrid on-chip memory system, it outputs the motion commands for the robot joints.

[0067] In the description of this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0068] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0069] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0070] It should be noted that, in this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0071] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0072] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A VLA edge processor based on static lexical filtering and hybrid near-memory computation, characterized in that, The VLA edge processor based on static lexical filtering and hybrid near-memory computing includes a task scheduling and configuration unit, a static lexical filter, a near-memory computing execution unit, and a hybrid on-chip storage system. The task scheduling and configuration unit is connected to the static lexical filter, the near-memory computing execution unit, and the hybrid on-chip storage system. The static lexical filter is connected to the near-memory computing execution unit and the hybrid on-chip storage system. The near-memory computing execution unit is connected to the hybrid on-chip storage system. The task scheduling and configuration unit is used to obtain VLA inference tasks and configure the working parameters of the static lexical filter, the near-memory computation execution unit and the hybrid on-chip memory system according to the VLA inference tasks. The static lexical filter is used to acquire high-dimensional lexical data stream, filter the high-dimensional lexical data stream, generate dynamic data, send it to the near-memory computing execution unit, and send the key-value cache corresponding to the dynamic data to the hybrid on-chip storage system for selective updating; The hybrid on-chip storage system is used to store the target weights and send them to the near-memory computation execution unit; The near-memory computation execution unit is used to perform near-memory computation based on the dynamic data and the target weight, obtain the computation result, and send it to the hybrid on-chip memory system for storage. The task scheduling and configuration unit is also used to output robot joint motion commands through a peripheral interface when it detects that the calculation results have been stored in the hybrid on-chip memory system.

2. The VLA edge processor based on static lexical filtering and hybrid proximity computing according to claim 1, characterized in that, The task scheduling and configuration unit includes a RISC-V core, a top-level controller, and a configuration register file. The RISC-V core is connected to the top-level controller and the configuration register file via a bus. The top-level controller and the configuration register file are controlled by the static lexical filter, the near-memory computation execution unit, and the hybrid on-chip memory system.

3. The VLA edge processor based on static lexical filtering and hybrid proximity computing according to claim 2, characterized in that, The static lexical filter includes a visual redundancy detector, a computation jumper, and a key-value updater. The visual redundancy detector is connected to the computation jumper and the key-value updater, respectively. The key-value updater is connected to the hybrid on-chip storage system and is used to selectively write the key-value cache corresponding to dynamic data into the hybrid on-chip storage system.

4. The VLA edge processor based on static lexical filtering and hybrid proximity computing according to claim 3, characterized in that, The near-memory computation execution unit includes a processing unit array, a SIMD vector kernel, a weight distributor, an activation value distributor, and an accumulation buffer. The computation jumper is connected to the activation value distributor. The weight distributor and the activation value distributor are respectively connected to the processing unit array. The processing unit array is connected to the accumulation buffer, and the accumulation buffer is connected to the SIMD vector kernel.

5. The VLA edge processor based on static lexical filtering and hybrid proximity computing according to claim 4, characterized in that, The hybrid on-chip memory system includes an RRAM memory and a dynamic multi-array SRAM memory. The weight distributor is connected to the RRAM memory, the activation value distributor is connected to the dynamic multi-array SRAM memory, the KV updater is connected to the dynamic multi-array SRAM memory, and is used to selectively write the key-value cache corresponding to the dynamic data into the dynamic multi-array SRAM memory. The SIMD vector kernel is connected to the dynamic multi-array SRAM memory and is used to write the post-processed calculation results back to the dynamic multi-array SRAM memory.

6. The VLA edge processor based on static lexical filtering and hybrid proximity computing according to claim 5, characterized in that, The VLA edge processor based on static lexical filtering and hybrid near-memory computing also includes a peripheral interface, which includes at least one of a JTAG interface, a UART interface, and a QSPI interface. The RISC-V core is connected to the peripheral interface via a bus.

7. A robot system, characterized in that, The robot system includes a VLA edge processor based on static lexical filtering and hybrid proximity computing as described in any one of claims 1 to 6 and a robot body. The VLA edge processor based on static lexical filtering and hybrid proximity computing is disposed on the robot body. The robot body has multiple joints, and the multiple joints move according to the motion commands output by the VLA edge processor based on static lexical filtering and hybrid proximity computing.

8. A data processing method based on the VLA edge processor based on static lexical filtering and hybrid proximity computing as described in any one of claims 1 to 6, characterized in that, The data processing method includes: The task scheduling and configuration unit acquires the VLA inference task and configures the working parameters of the static lexical filter, the near-memory computation execution unit, and the hybrid on-chip memory system according to the VLA inference task. The static lexical filter acquires the high-dimensional lexical data stream, filters the high-dimensional lexical data stream, generates dynamic data, sends it to the near-memory computing execution unit, and sends the key-value cache corresponding to the dynamic data to the hybrid on-chip storage system for selective updates. The hybrid on-chip memory system stores the target weights and sends them to the near-memory computation execution unit; The near-memory computation execution unit performs near-memory computation based on the dynamic data and the target weight, obtains the computation result, and sends it to the hybrid on-chip storage system for storage. When the task scheduling and configuration unit detects that the calculation results have been stored in the hybrid on-chip memory system, it outputs motion commands for the robot joints.

9. The data processing method according to claim 8, characterized in that, The step of filtering the high-dimensional term data stream to generate dynamic data specifically includes: The visual redundancy detector maps the high-dimensional word data stream to obtain the target fingerprint of each high-dimensional word element, and compares the target fingerprint with the historical fingerprint to generate a decision result; If the judgment result is static, then the jumper is calculated to block the data path of the high-dimensional word corresponding to the target fingerprint; If the decision result is dynamic, the calculation jumper allows the original data of the high-dimensional word corresponding to the target fingerprint and sends the original data of the high-dimensional word to the activation value distributor of the near-memory computation execution unit. The KV updater writes the new key-value pair corresponding to the high-dimensional word into the dynamic multi-array SRAM of the hybrid on-chip memory system and writes the target fingerprint into the fingerprint buffer of the KV updater itself.

10. The data processing method according to claim 9, characterized in that, The dynamic data refers to dynamic lexical activation values. The step of performing near-memory calculation based on the dynamic data and the target weight to obtain the calculation result specifically includes: The activation value distributor obtains the dynamic lexical activation values ​​allowed by the static lexical filter from the on-chip dynamic multi-array SRAM memory and sends them to the processing unit array. The processing unit array performs calculations based on the target weight and the dynamic lexical activation value to obtain the calculation result.

Images