A Heterogeneous Acceleration Method and System for Mobile Retrieval Enhancement Generation Based on Fine-Grained Computational Pipeline and Speculative Loading

Abstract

This invention discloses a heterogeneous acceleration method and system for mobile retrieval enhancement generation based on fine-grained computational pipelines and speculative loading. First, it constructs an embedded model inference pipeline based on sub-tensor granularity, optimizing inference through direct memory access. Second, it constructs a semantically aware speculative I / O loading pipeline, triggering speculative I / O loading using intermediate results from heterogeneous computational units. Finally, it constructs a heterogeneous resource dynamic orchestration mechanism based on the RAG task lifecycle, ensuring full-load operation of the front-end inference pipeline while utilizing computation gaps to complete silent loading and memory reclamation of the back-end index. This invention solves the problem of idle computing power during embedded model inference; simultaneously, under a hierarchical storage architecture, it utilizes intermediate results from heterogeneous computational units to trigger speculative I / O loading, and combines vector processing units to perform in-situ decompression and format adaptation of the data stream, achieving end-to-end performance optimization of the RAG task on the client side.

Description

Technical Field

[0001] This application relates to the fields of mobile terminal architecture, edge AI inference optimization and high-performance heterogeneous storage management, and in particular to a mobile terminal retrieval enhancement generation heterogeneous acceleration method and system based on fine-grained computing pipeline and speculative loading. Background Technology

[0002] With the explosive growth of generative artificial intelligence (GenAI) technology, large language models (LLMs) are rapidly migrating from the cloud to mobile devices such as smartphones to meet users' stringent demands for privacy protection, real-time response, and offline usability. To overcome the knowledge illusion and information lag problems caused by the limited number of parameters in edge models, retrieval-enhanced generation (RAG) architecture has become a core paradigm for mobile intelligent assistants. This architecture requires the system to first transform unstructured input into a high-dimensional vector through an embedding model before generating a response, then retrieve relevant context from a local knowledge base, and finally synthesize the output by the generative model. However, deploying this complex "reasoning-retrieval-generation" end-to-end on mobile SoCs (System-on-a-Chip) with strictly limited power consumption and area faces deep-seated architectural bottlenecks and memory-computation mismatch challenges.

[0003] First, the embedded model inference process is severely constrained by the "memory wall" effect, leading to serious pipeline congestion. Mobile neural network processors (NPUs), in pursuit of extreme area efficiency, typically only have extremely limited on-chip caches, creating a stark physical contradiction with the hundreds of megabytes of weight parameters in modern Transformer architectures. Since the weights of a single layer far exceed the on-chip capacity threshold, existing inference frameworks are forced to break down data and computation into smaller granularities when handling such operators, but are generally limited by the synchronous dependency mechanism between data transmission and computation execution. Specifically, when scheduling each computational microkernel or data block, the system often follows a serialized timing logic: "computation is triggered immediately upon data arrival, and the next round of data transmission only begins after the computation is completely completed." While this mechanism adapts to limited storage space, it artificially creates time-domain mutual exclusion between computation cycles and bus transmission cycles, causing the NPU core to be forced into idle pauses during frequent data loading intervals. This not only results in fragmented waste of computing resources, but also directly slows down the edge response speed of the RAG system because it cannot mask bus latency.

[0004] Secondly, data transfer between heterogeneous computing units suffers from significant format impedance mismatch and transmission overhead. In a typical mobile RAG process, although the general-purpose processor (CPU) may be under low load, relying on the CPU for data decompression or format conversion forces the data flow to frequently traverse the system bus. Typically, CPU data processing involves loading data from main memory into the CPU cache, processing it, writing it back to main memory, and finally reading it from the NPU. This process not only introduces multiple redundant memory copies but also frequently triggers cache coherence protocol broadcasts in multi-core systems, causing significant fluctuations in bus utilization. Even if the NPU's matrix acceleration engine mandates half-precision (FP16) and tiled layout to maximize throughput, the CPU-based preprocessing scheme will negate the theoretical benefits of heterogeneous collaboration due to substantial data movement latency, limiting system performance to the memory wall rather than the computing power wall.

[0005] Furthermore, existing operating system-based hierarchical storage mechanisms suffer from significant semantic-physical mapping mismatches when handling large-scale vector indexes. Traditional virtual memory management relies on the principle of spatial locality, passively loading pages based on fixed page sizes. However, vector retrieval exhibits a high degree of semantic leap, meaning that logically similar vectors may not be contiguous in the linear logical address space of the storage device. When performing non-exhaustive searches based on inverted indexes, frequent random read requests not only lead to severe read amplification, but each I / O access also requires file system metadata parsing, kernel-mode data copying, and CPU-side serialization and decoding. This generic data loading path is unaware of the computational pace and semantic prediction results of the neural network processor, causing I / O throughput to fail to be converted into the effective tensor stream required by the computing engine in a timely manner, resulting in severe idle computing power of heterogeneous computing resources while waiting for data.

[0006] In summary, existing mobile inference and retrieval technologies generally suffer from pipeline fragmentation, low utilization of heterogeneous resources, and high I / O loading latency when facing RAG scenarios. Designing a system that can deeply adapt to the heterogeneous storage hierarchy of mobile devices, streamline the data flow from disk I / O to the matrix computation core, and achieve hardware acceleration across the entire inference and retrieval process has become a key technical challenge in the field of edge AI. Summary of the Invention

[0007] To address the technical challenges faced by mobile SoCs in processing large-scale RAG tasks—including inference blocking due to limited on-chip tightly coupled memory capacity, I / O latency caused by semantic mismatch between the general file system and vector retrieval, and bus bandwidth waste caused by impedance mismatch in data formats between heterogeneous computing units—this application provides a mobile-side retrieval enhancement generation heterogeneous acceleration method and system based on fine-grained computational pipelines and speculative loading. This invention solves the problem of idle computing power during embedded model inference through a double-buffered mechanism of sub-tensor direct memory access (DMA); simultaneously, under a hierarchical storage architecture, it utilizes intermediate results from heterogeneous computing units to trigger speculative I / O loading, and combines this with in-situ decompression and format adaptation of the data stream by the vector processing unit, achieving end-to-end performance optimization for edge-side RAG tasks.

[0008] In a first aspect, this application provides a heterogeneous acceleration method for mobile retrieval enhancement generation based on fine-grained computational pipeline and speculative loading, comprising the following steps:

[0009] Constructing an embedded model inference pipeline based on subtensor granularity: During initialization, the end-to-end pipeline controller cuts the giant weight tensor into semantically and physically aligned micro-tensor slices according to the NPU cache capacity and the input requirements of the matrix operation engine in the vector retrieval stage; during inference runtime, a dynamic pipeline interaction mechanism is adopted, in which the next slice is asynchronously prefetched to the backup buffer through a synchronization mechanism when the matrix calculation unit processes the slice of the current buffer, so as to achieve computation-transmission time domain overlap;

[0010] Constructing a semantically aware speculative I / O loading pipeline: Semantic data slices physically aligned with the NPU computing core are pre-built on Flash; during the vector retrieval stage, a computing power-guided I / O speculative mechanism is adopted. Utilizing the coarse-ranking results produced by the matrix computation unit, based on the Top-N candidate set, before fine-ranking begins, an asynchronous DMA descriptor linked list containing multiple non-contiguous physical block addresses is generated by the intelligent I / O scheduler, speculatively triggering asynchronous I / O prefetch instructions; and when the compressed data stream passes through the on-chip buffer, the vector processing unit is triggered to decompress and rearrange it, converting it into a dedicated format for matrix computation.

[0011] Construct a heterogeneous resource dynamic orchestration mechanism based on the RAG task lifecycle: Based on user voice or text input, the global orchestrator monitors bus traffic and NPU load in real time, driving the system to dynamically switch between different microarchitecture states. While ensuring the full-load operation of the foreground inference pipeline, the silent loading and memory reclamation of the background index are completed during the computation gap.

[0012] Secondly, this application also provides a mobile retrieval enhancement generation heterogeneous acceleration system based on fine-grained computational pipeline and speculative loading, including:

[0013] The embedded model inference module is used to build an embedded model inference pipeline based on sub-tensor granularity: During initialization, the end-to-end pipeline controller cuts the giant weight tensor into micro-tensor slices that are semantically and physically aligned, based on the NPU cache capacity and the input requirements of the matrix operation engine in the vector retrieval stage; During inference runtime, a dynamic pipeline interaction mechanism is adopted, and when the matrix calculation unit processes the slice of the current buffer, the next slice is asynchronously prefetched to the backup buffer through a synchronization mechanism to achieve computation-transmission time domain overlap;

[0014] The speculative I / O loading module is used to construct a semantically aware speculative I / O loading pipeline: semantic data slices physically aligned with the NPU computing core are pre-built on Flash; during the vector retrieval stage, a computing power-guided I / O speculation mechanism is adopted, utilizing the coarse ranking results produced by the matrix calculation unit, based on the Top-N candidate set, and before fine ranking starts, an asynchronous DMA descriptor linked list containing multiple non-contiguous physical block addresses is generated by the intelligent I / O scheduler, speculatively triggering asynchronous I / O prefetch instructions; and when the compressed data stream passes through the on-chip buffer, the vector processing unit is triggered to decompress and rearrange it, converting it into a special format for matrix calculation.

[0015] The heterogeneous resource dynamic orchestration module is used to build a heterogeneous resource dynamic orchestration mechanism based on the RAG task lifecycle: based on user voice or text input, the global orchestrator monitors bus traffic and NPU load in real time, driving the system to dynamically switch between different microarchitecture states. While ensuring that the foreground inference pipeline runs at full load, the silent loading and memory reclamation of the background index are completed during the computation gap.

[0016] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described mobile retrieval enhancement generation heterogeneous acceleration method based on fine-grained computing pipeline and speculative loading.

[0017] The beneficial effects of this application are:

[0018] By constructing a double-buffered inference pipeline with "subtensor" granularity, this application overcomes the physical limitation of on-chip memory capacity of mobile NPU on large model weights, eliminates the phenomenon of computing core idling under the traditional serial scheduling mechanism, and significantly reduces the first word generation latency of embedded model inference.

[0019] By introducing speculative I / O loading based on intermediate calculation results and in-situ streaming decompression technology for vector processing units, this application constructs an efficient link of "computing power guiding data", which effectively solves the semantic mismatch problem caused by passive loading of general file systems. While significantly reducing the system bus bandwidth usage, it achieves high-throughput retrieval of massive vector indexes in a limited memory environment.

[0020] By dynamically orchestrating heterogeneous resources across the entire chain, this application can accurately identify and utilize the micro-computation gaps in the RAG process, achieving parallel operation of high-performance front-end inference and low-power back-end maintenance, providing system-level energy efficiency optimization and stability assurance for large-scale mobile applications. Attached Figure Description

[0021] Figure 1 The overall system architecture diagram provided for this application;

[0022] Figure 2 This invention provides a speculative loading and in-situ decompression data flow diagram.

[0023] Figure 3 The RAG task state machine and resource orchestration diagram provided for this invention. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0025] like Figure 1As shown, this application provides a heterogeneous acceleration method for mobile retrieval enhancement generation based on fine-grained computing pipelines and speculative loading. It achieves pipeline-level fusion of embedded inference and vector retrieval by establishing a bypass copy data interconnect architecture that spans storage levels and computing units. From a hardware topology perspective, this application relies on a heterogeneous system-on-a-chip (SoC) integrating a multi-core general-purpose processor (CPU), a high-performance graphics processing unit (GPU), and a dedicated artificial intelligence neural network processor (NPU). The NPU, as the computing power hub of this architecture, is further subdivided into a command processor responsible for scalar control, a vector processing unit responsible for high-throughput data parallel processing, and a matrix operation engine dedicated to tensor matrix multiplication. These three components share a single on-chip cache with extremely limited capacity but high bandwidth. At the storage level, the system spans nanosecond-level on-chip cache, gigabyte-level system main memory (DRAM), and sub-terabyte-level high-speed general-purpose flash memory (NAND Flash), forming a typical pyramid-shaped storage structure where large capacity and low latency are mutually constrained.

[0026] Furthermore, based on the aforementioned hardware, this application embodiment deploys a RAG heterogeneous accelerated runtime environment. This runtime is not a simple collection of drivers, but an intelligent middleware residing between kernel mode and user mode. It shields the physical differences of the underlying heterogeneous storage media through a virtualization tensor context mechanism.

[0027] Specifically, the runtime environment consists of three tightly coupled logical subsystems: a full-link pipeline controller, a hierarchical storage intelligent I / O scheduler, and a heterogeneous resource global orchestrator.

[0028] Furthermore, the full-link pipeline controller primarily serves the inference process of the embedded model. It possesses static graph analysis capabilities, enabling it to pre-scan the computation graph structure of the deep neural network. By combining the physical boundaries of the NPU's on-chip cache, it automatically divides the massive weight tensors and activation values ​​into tiny computation graph tiles and generates a direct memory access (DMA) instruction list containing precise timing. This drives ping-pong parallel operations between the DMA controller and the matrix operation engine at runtime.

[0029] Furthermore, the hierarchical storage intelligent I / O scheduler is a key component for resolving the bottleneck of large-scale vector index loading. Unlike traditional file systems that passively wait for read requests, this scheduler takes over the data path during the vector retrieval stage, employing a direct I / O strategy that bypasses the operating system page cache, allowing physical page frames in flash memory to be directly mapped to the NPU's address space. More importantly, the scheduler embeds a microkernel injection mechanism for vector processing units, which can dynamically activate the NPU's vector units to perform in-situ format decompression, precision truncation (e.g., from FP32 to FP16), and memory layout reordering the instant the data flows through the transmission path. This design transforms the I / O path, which was originally a simple data transporter, into an "intelligent data stream" with computational capabilities, ensuring that once data is cached on the chip, it is in a ready state for direct computation by the matrix operation engine.

[0030] Furthermore, to coordinate the aforementioned complex concurrent tasks, the heterogeneous resource global orchestrator acts as the system's overall commander, monitoring the SoC's thermal design power margin, memory bandwidth utilization, and frame rate jitter of the foreground application in real time. Based on the characteristics of RAG tasks at different stages—namely, the computationally intensive inference stage and the I / O-intensive retrieval stage—the orchestrator dynamically adjusts the Quality of Service (QoS) weights of the system bus. For example, during model inference, the high priority of the DMA channel is locked to maintain pipeline saturation; while during retrieval, some NPU computing power is released to the I / O scheduler for data adaptation. When the system load is too high, some lightly loaded tasks are strategically rolled back to the GPU or CPU for execution, thereby achieving a dynamic balance between extreme RAG performance and the overall smoothness of the mobile device. This integrated hardware and software architecture design eliminates the drawbacks of module fragmentation and frequent data copying in traditional solutions, laying a solid system foundation for achieving millisecond-level retrieval enhancement generation experience on mobile devices.

[0031] Specifically, this application is achieved through the following technical solution:

[0032] S1: Addressing the challenges faced by embedding models in mobile inference—namely, the significant physical constraint that the volume of a single-layer weight tensor exceeds the on-chip cache capacity of a neural network processor, and existing solutions neglecting the data flow continuity of the "generation-retrieval" process in the RAG workflow—this embodiment proposes a forward-looking static slicing and dynamic pipeline transport collaborative mechanism oriented towards the retrieval dimension.

[0033] Because the inference process of the embedded model exhibits a highly deterministic memory access pattern and computation graph structure, static optimization at the pre-compilation level becomes possible. Specifically, during the model loading or initialization phase, the end-to-end pipeline controller intervenes first, not only based on the available on-chip cache physical space of the NPU, but also analyzing the optimal input tensor shape for the matrix operation engine in the subsequent vector retrieval phase. Using this as a constraint, the controller calculates the "optimal resident block" format that both fills the on-chip cache pipeline and ensures that the query vector output by the model naturally meets the requirements of the matrix operation engine. Based on this, the massive weight tensor is sliced ​​into a series of semantically and physically aligned micro-tensor slice sequences.

[0034] Furthermore, during the runtime data scheduling and execution phase, this embodiment abandons the common serial loading mode and instead adopts a forward-looking pipeline interaction mechanism oriented towards the retrieval dimension. Specifically, when processing micro-tensor slices based on RAG semantic alignment, this mechanism utilizes dynamically partitioned logical buffers to build a deep coupling between computation and transmission. Under this architecture, while the computation engine focuses on performing operations on the slices in the current buffer, the data transmission controller prefetches the next logical slice in parallel in the background. More importantly, relying on the semantic alignment slicing strategy at the input end, the embedded vector data produced by this pipeline at the output end naturally adapts to the special memory layout requirements of the subsequent NPU matrix operation engine.

[0035] Specifically, the forward-looking pipelined interaction mechanism for retrieval dimensions not only achieves temporal latency masking through underlying hardware synchronization primitives, but also realizes zero-overhead flow from "feature generation" to "distance calculation" in the data flow dimension, completely eliminating the intermediate data rearrangement overhead that must be performed due to task switching in traditional solutions. The most crucial breakthrough lies in its deep overlap effect in the "computation-transfer" temporal domain. Ideally, as long as the computation time of the NPU on the current slice is slightly greater than or equal to the DMA's time to move the next slice, the external memory access latency will be partially masked by the computation process, making the NPU's computation core "unaware" of the data loading process, thus maintaining a fully loaded state. This is fundamentally different from the double-buffering logic when processing vector indexes: vector index loading is limited by the randomness of user query content, often requiring real-time parsing and dynamic addressing, while the inference double buffering in this embodiment is based on deterministic streaming prefetching of model weights, aiming to resolve the contradiction between deterministic large model parameters and limited on-chip cache, partially eliminating the "pipeline bubble" in the embedded model inference process, and ensuring the rapid response of the RAG system in the first character generation stage.

[0036] S2: To address the challenge of maintaining massive vector indexes within the limited memory of mobile devices, this embodiment does not employ the common operating system virtual memory swapping mechanism. Instead, it constructs a semantically aware speculative I / O pipeline. This embodiment pre-builds semantic data slices on non-volatile storage media (Flash) that are strictly aligned with the physical size of the Neural Processing Unit (NPU) matrix computation core (Tile). Unlike the continuous byte stream storage of traditional file systems, each semantic data slice is physically encoded into a highly compressed quantized format and exists as an independent Direct Access Unit, thereby eliminating the overhead of file system metadata parsing.

[0037] Furthermore, during the runtime retrieval phase, this embodiment establishes a speculative execution mechanism based on multi-level segmented triggering of "computing power-guided I / O". Combined with... Figure 2 The data flow shown illustrates that when the NPU's matrix operation engine performs the coarse-ranking process for vector retrieval, once the first 20% of cluster centers are scanned, the scheduler calculates a confidence threshold based on the current preliminary Top-N score distribution. If the score of a candidate vector cluster exceeds this threshold, the system hardware logic does not need to wait for the entire coarse-ranking process to finish; it immediately intercepts these high-confidence indices and generates a prefetch request. Within a microsecond-level time window before the fine-ranking (Re-ranking) computation task begins, the intelligent I / O scheduler generates an asynchronous DMA descriptor list containing multiple non-contiguous physical block addresses and issues an asynchronous direct memory access (DMA) instruction list to the memory controller. This segmented triggering mode further overlaps I / O prefetching with the remaining coarse-ranking computation in the time domain, maximizing bus bandwidth utilization.

[0038] Furthermore, to address potential prefetching errors during speculative loading (i.e., prefetched vectors not entering the final fine-sorting list), this embodiment employs a fault-tolerance mechanism of "specular cache isolation" and "on-demand write-in." The on-chip buffer is logically divided into a formal buffer and a speculative buffer. Speculatively loaded data is preferentially stored in the speculative buffer, without polluting the currently active working set. When the fine-sorting engine starts, the system quickly compares the contents of the fine-sorting target list with those of the speculative buffer:

[0039] If a hit occurs: the data logic is directly mapped to the computing engine, eliminating waiting delays;

[0040] If a miss occurs: the "on-demand rollback" path is triggered, the missing data is filled in using high-priority DMA, and the streaming processing capability of the vector processing unit is used to decompress and calculate simultaneously to smooth out the performance jitter caused by the missing data.

[0041] Cleanup strategy: Once the fine sorting task is completed, redundant data that is not hit in the speculative buffer will be marked as "immediately invalidated", triggering memory reclamation to ensure sufficient storage space for the next round of RAG inference.

[0042] Furthermore, such as Figure 2 As shown in the vector processing unit node, to address the order-of-magnitude difference between I / O bandwidth and computation bandwidth, this embodiment incorporates streaming in-situ decompression logic along the data flow path into the NPU. Leveraging the wide instruction set of the NPU's internal vector processing unit, an online "decompression-rearrangement" microkernel is constructed. When compressed quantized data flows through the NPU's local memory, the vector processing unit intervenes in stream processing mode, performing table lookup decoding and precision expansion operations in parallel. This instantly restores the high-compression storage format to the half-precision floating-point tensors required by the matrix operation engine. The traditional serial link of "I / O -> CPU decoding -> NPU computation" is reconstructed into a three-stage multiplexed pipeline of "I / O stream -> vector processing unit decompression stream -> matrix operation engine computation stream." This not only significantly reduces the system bus bandwidth usage but also, through a computation-for-space approach, significantly improves the effective throughput of mobile devices when processing large-scale indexes.

[0043] S3: Given the highly dynamic nature and uncertain resource contention inherent in mobile heterogeneous computing environments, this application does not employ a static and rigid resource allocation strategy. Instead, it adopts a multimodal task state machine (Finite State Machine) and global quality of service (QoS) orchestration mechanism based on the RAG full lifecycle.

[0044] Furthermore, such as Figure 3 The state transition logic shown illustrates that a typical RAG workflow is a continuum encompassing three distinct load characteristics: "perceptual input (Embedding inference)," "cognitive search (vector retrieval)," and "representation generation (LLM decoding)." Each stage exhibits orthogonal or even conflicting demands on computational density, memory bandwidth, and I / O throughput. The global orchestrator drives the system to perform millisecond-level context switching between different microarchitectural states by real-time monitoring of traffic characteristics on the system bus and the load level of the NPU cores.

[0045] Specifically, when the user inputs a voice command, enter Figure 3In the inference state, the global orchestrator immediately locks the highest priority of the DMA channel to ensure weight flow. At this time, the NPU's matrix operation engine is in full-load mode, while the vector processing unit is in standby mode. Once the query vector is generated, the system transitions to the retrieval state. At this time, the resource bottleneck shifts to I / O. The orchestrator adjusts the DMA priority to medium and activates the vector processing unit to perform in-situ adaptation at full load. During the computation gap in the LLM generation stage after the retrieval is completed, the system automatically slides into the maintenance state, utilizing the low-priority DMA channel and the NPU's idle computing power to perform background index defragmentation and memory reclamation. This state-aware resource flow ensures that the heterogeneous SoC serves the most pressing bottleneck at any given time.

[0046] Furthermore, this orchestration mechanism possesses deep environmental awareness, effectively handling resource conflicts between RAG background tasks and high-load foreground applications. For example, during the intervals when LLM generates answers, or during periods of quiet user interaction, the system automatically enters a "maintenance state," utilizing the idle computing power and I / O bandwidth of the NPU to asynchronously degrade or reclaim cold index data residing in memory, and even perform defragmentation. Once a sudden surge in GPU or memory bandwidth demand from a foreground application is detected, the orchestrator immediately executes a "graceful degradation" strategy, dynamically reducing the DMA prefetch window or temporarily suspending background index building tasks, forcibly relinquishing system bus control. This ensures a seamless, intelligent experience on the client-side while maintaining an absolutely smooth user interface.

[0047] In summary, this embodiment constructs a retrieval-enhanced generation (RAG) acceleration closed loop that integrates fine-grained pipelined inference, near-memory in-situ streaming decompression, and dynamic orchestration throughout the entire lifecycle by deeply deconstructing and reorganizing the heterogeneous computing architecture of mobile devices. This solution breaks down the spatiotemporal barriers between data transport and computation tasks in traditional memory-compute separation architectures. It not only cleverly circumvents the physical capacity limits of tightly coupled on-chip memory using multiple buffering mechanisms of direct memory access, achieving bubble-free operation of embedded model inference, but also, by assigning the vector processing unit a new role as an "intelligent data gateway," constructs a transparent format adaptation layer on the computationally guided speculative I / O path, alleviating the bandwidth bottleneck of general-purpose processors when searching massive indexes. This system-level design, combining hardware and software, makes it possible to deploy large-scale edge-side model applications with long-term memory capabilities and extremely fast response characteristics on smartphone platforms with strictly limited power consumption and area. This provides crucial underlying computing power support and architectural paradigm for the evolution of next-generation mobile artificial intelligence from simple perceptual intelligence to cognitive intelligence.

[0048] In one exemplary embodiment, a mobile retrieval enhancement generation heterogeneous acceleration system based on fine-grained computational pipeline and speculative loading is provided, comprising:

[0049] The embedded model inference module is used to build an embedded model inference pipeline based on sub-tensor granularity: During initialization, the end-to-end pipeline controller cuts the giant weight tensor into micro-tensor slices that are semantically and physically aligned, based on the NPU cache capacity and the input requirements of the matrix operation engine in the vector retrieval stage; During inference runtime, a dynamic pipeline interaction mechanism is adopted, and when the matrix calculation unit processes the slice of the current buffer, the next slice is asynchronously prefetched to the backup buffer through a synchronization mechanism to achieve computation-transmission time domain overlap;

[0050] The speculative I / O loading module is used to construct a semantically aware speculative I / O loading pipeline: semantic data slices physically aligned with the NPU computing core are pre-built on Flash; during the vector retrieval stage, a computing power-guided I / O speculation mechanism is adopted, utilizing the coarse ranking results produced by the matrix calculation unit, based on the Top-N candidate set, and before fine ranking starts, an asynchronous DMA descriptor linked list containing multiple non-contiguous physical block addresses is generated by the intelligent I / O scheduler, speculatively triggering asynchronous I / O prefetch instructions; and when the compressed data stream passes through the on-chip buffer, the vector processing unit is triggered to decompress and rearrange it, converting it into a special format for matrix calculation.

[0051] The heterogeneous resource dynamic orchestration module is used to build a heterogeneous resource dynamic orchestration mechanism based on the RAG task lifecycle: based on user voice or text input, the global orchestrator monitors bus traffic and NPU load in real time, driving the system to dynamically switch between different microarchitecture states. While ensuring that the foreground inference pipeline runs at full load, the silent loading and memory reclamation of the background index are completed during the computation gap.

[0052] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above-described mobile retrieval enhancement generation heterogeneous acceleration method based on fine-grained computing pipeline and speculative loading.

[0053] The foregoing description of embodiments of the present invention, through which those skilled in the art are able to implement or use the present invention, will be readily apparent to those skilled in the art. Various modifications to these embodiments will be readily apparent to those skilled in the art. The general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novelty disclosed herein.

Claims

1. A heterogeneous acceleration method for mobile retrieval enhancement generation based on fine-grained computational pipeline and speculative loading, characterized in that, Includes the following steps: Constructing an embedded model inference pipeline based on subtensor granularity: During initialization, the end-to-end pipeline controller cuts the giant weight tensor into semantically and physically aligned micro-tensor slices according to the NPU cache capacity and the input requirements of the matrix operation engine in the vector retrieval stage; during inference runtime, a dynamic pipeline interaction mechanism is adopted, in which the next slice is asynchronously prefetched to the backup buffer through a synchronization mechanism when the matrix calculation unit processes the slice of the current buffer, so as to achieve computation-transmission time domain overlap; Constructing a semantically aware speculative I / O loading pipeline: Pre-constructing semantic data fragments physically aligned with the NPU compute core on Flash; In the vector retrieval stage, a computing power-guided I / O speculation mechanism is adopted. Utilizing the coarse ranking results produced by the matrix calculation unit, based on the Top-N candidate set, before the fine ranking starts, an asynchronous DMA descriptor linked list containing multiple non-contiguous physical block addresses is generated by the intelligent I / O scheduler, speculatively triggering asynchronous I / O prefetch instructions; and when the compressed data stream passes through the on-chip buffer, the vector processing unit is triggered to decompress and rearrange it, converting it into a special format for matrix calculation. Construct a heterogeneous resource dynamic orchestration mechanism based on the RAG task lifecycle: Based on user voice or text input, the global orchestrator monitors bus traffic and NPU load in real time, driving the system to dynamically switch between different microarchitecture states. While ensuring the full-load operation of the foreground inference pipeline, the silent loading and memory reclamation of the background index are completed during the computation gap.

2. The method according to claim 1, characterized in that, The semantic data fragments are encoded in a high-compression quantization format and exist as independent direct access units, eliminating the overhead of file system metadata parsing.

3. The method according to claim 1 or 2, characterized in that, When the computing power-driven I / O speculation mechanism fails to prefetch, it is handled through a speculative cache isolation and on-demand rewriting fault tolerance mechanism, and the speculatively loaded data is first stored in the speculative buffer to avoid polluting the current active working set.

4. The method according to claim 3, characterized in that, After the fine sorting is initiated, the system will quickly compare the contents of the fine sorting target list with the contents of the speculative buffer: Hit: Directly maps data logic to the matrix calculation engine, eliminating waiting delays; Miss: Trigger the on-demand rollback path, fill in the missing data with high-priority DMA, and complete the decompression calculation using the vector processing unit.

5. The method according to claim 4, characterized in that, After fine sorting, redundant data that is not hit in the speculative buffer is marked as immediately invalid and memory reclamation is triggered, reserving sufficient storage space for the next round of RAG inference.

6. The method according to claim 5, characterized in that, The vector processing unit intervenes in stream processing mode, performing table lookup decoding and precision expansion in parallel, converting compressed data into half-precision floating-point tensors required by the matrix calculation engine.

7. The method according to claim 1, characterized in that, The global orchestrator monitors bus traffic and NPU load in real time, driving the system to dynamically switch between different microarchitecture states, including: Inference state: During voice input, the global orchestrator locks the DMA with the highest priority, and the NPU matrix operation engine is fully loaded to process the embedding model; Retrieval State: After generating the query vector, the DMA priority is reduced, and the vector processing unit is fully loaded to perform index retrieval; Maintenance mode: After the retrieval is completed, the background index defragmentation and memory reclamation are performed using low-priority DMA channels and the idle computing power of the NPU.

8. The method according to claim 1, characterized in that, The heterogeneous resource dynamic orchestration mechanism based on the RAG task lifecycle has a deep environmental awareness capability. When a sudden high resource demand is detected in the foreground application, it immediately performs degradation by reducing DMA prefetching or pausing background tasks, releasing bus control, ensuring a smooth interface, and achieving a seamless intelligent experience on the end side.

9. A mobile retrieval enhancement generation heterogeneous acceleration system based on fine-grained computational pipeline and speculative loading, characterized in that, include: The embedded model inference module is used to build an embedded model inference pipeline based on sub-tensor granularity: During initialization, the end-to-end pipeline controller cuts the giant weight tensor into micro-tensor slices that are semantically and physically aligned, based on the NPU cache capacity and the input requirements of the matrix operation engine in the vector retrieval stage; During inference runtime, a dynamic pipeline interaction mechanism is adopted, and when the matrix calculation unit processes the slice of the current buffer, the next slice is asynchronously prefetched to the backup buffer through a synchronization mechanism to achieve computation-transmission time domain overlap; The speculative I / O loading module is used to build a semantically aware speculative I / O loading pipeline: pre-builds semantic data fragments on Flash that are physically aligned with the NPU computing core; In the vector retrieval stage, a computing power-guided I / O speculation mechanism is adopted. Utilizing the coarse ranking results produced by the matrix calculation unit, based on the Top-N candidate set, before the fine ranking starts, an asynchronous DMA descriptor linked list containing multiple non-contiguous physical block addresses is generated by the intelligent I / O scheduler, speculatively triggering asynchronous I / O prefetch instructions; and when the compressed data stream passes through the on-chip buffer, the vector processing unit is triggered to decompress and rearrange it, converting it into a special format for matrix calculation. The heterogeneous resource dynamic orchestration module is used to build a heterogeneous resource dynamic orchestration mechanism based on the RAG task lifecycle: based on user voice or text input, the global orchestrator monitors bus traffic and NPU load in real time, driving the system to dynamically switch between different microarchitecture states. While ensuring that the foreground inference pipeline runs at full load, the silent loading and memory reclamation of the background index are completed during the computation gap.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images