A task-aware hierarchical sparse screening method and system for npus

Abstract

The application discloses an NPU-oriented task-aware hierarchical sparse screening method and system, belonging to the field of artificial intelligence. The method comprises the following steps: acquiring mechanical arm joint state parameters and camera calibration parameters, and combining a two-way visual feature word sequence output by a visual coding network; determining a candidate region center point based on kinematics forward solution and camera projection, generating a candidate region window, and constructing a candidate block set; calculating prior weights and learning weights for the candidate blocks to obtain importance scores and sort them; selecting a sparse budget K according to the score distribution and end-side budget parameters, and correcting K* according to the alignment constraint; screening the top m candidate blocks and expanding them into a sparse word sequence to generate an index mapping and a mask set; and organizing the candidate blocks into continuous blocks in sequence and outputting a fixed-length sparse word sequence with a length of K* in length alignment. The application effectively reduces non-continuous memory access and dynamic shape scheduling overhead and improves end-side reasoning delay stability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and relates to edge neural network processor inference acceleration and robot vision-action decision-making technology, specifically to a feature sparse screening method and system for NPU. Background Technology

[0002] In recent years, embodied intelligence and robotic manipulation tasks have gradually shifted from "offline planning" to "end-to-end vision-action decision-making," where models typically need to output robotic arm control commands under continuous video observation. To achieve stronger temporal modeling and multimodal fusion capabilities, vision-action models based on the Transformer architecture have been widely adopted. However, the computational overhead of Transformer attention increases quadratically with the number of input tokens, and in edge-side inference scenarios, a low-parallelism execution mode with a batch size of 1 is typically used, making attention-related matrix operations one of the main performance bottlenecks in the edge-side real-time inference chain. Therefore, how to reduce the length of the attention input sequence while ensuring the preservation of effective task information, and enable the edge-side NPU to perform matrix operations efficiently and stably in a fixed shape, has become a key problem that urgently needs to be solved in the deployment of edge-side embodied intelligence.

[0003] To address the issue of excessive attention overhead, existing methods typically start with reducing the length of the input sequence, employing strategies such as token filtering, sparse attention, or candidate region pruning to decrease the number of tokens involved in attention computation. For example, Top-K selection of tokens can be performed using saliency or similarity scores, or local regions can be preserved in a window / block structure to approximate global attention. While these methods can reduce computational load to some extent in general vision tasks, they often introduce new engineering bottlenecks when deploying on-device NPUs: First, the number and distribution of tokens obtained from Top-K filtering vary with the samples, easily leading to dynamic changes in the input shape of subsequent matrix operations, thus triggering frequent padding, rearrangement, and dynamic scheduling; Second, irregular indexing (Gather / Scatter) causes discontinuous memory access, increasing data transfer overhead and offsetting the computational gains from sparsity; Third, when sparse selection lacks task constraints, the filtering results lack stability, easily introducing decision jitter in robot operation scenarios.

[0004] Especially in edge NPU inference scenarios (such as Ascend NPU), attention-related computations typically involve matrix operations such as matrix multiplication and linear transformations. Their execution efficiency is highly sensitive to the regularity and alignment of the input shape. When the number of input tokens varies or the index distribution is irregular, it easily leads to tail block processing, additional padding and data rearrangement overhead, and increased data transfer pressure from external storage to on-chip cache, ultimately resulting in edge latency jitter and decreased throughput. On the other hand, robot operation tasks have quantifiable prior information (such as joint states, end-effector pose, and camera calibration relationships). If this prior information can be integrated into the sparse selection process, constraining candidate regions from the source and improving selection stability, it is hoped that the continuity and interpretability of key visual information can be maintained while reducing input length, thus better matching the computational and storage constraints of real-time NPU inference. Summary of the Invention

[0005] The purpose of this invention is to provide a task-aware hierarchical sparse filtering method, apparatus, device, and storage medium for NPUs, to solve the following problems existing in edge-side sparse filtering schemes in edge-side NPU deployments: the sparse filtering result changes with the sample, causing the shape of the subsequent matrix operation input to change dynamically, triggering frequent padding and data rearrangement; irregular index access causes discontinuous memory access and increases data transfer overhead; and the lack of task prior constraints leads to unstable filtering positions and causes edge-side inference latency jitter.

[0006] To address the aforementioned issues, this invention proposes a task-aware hierarchical sparse filtering method for NPUs. First, a priori projection of the task is established using joint states and camera calibration, generating candidate regions and constructing a candidate tile set in units of tiles. Then, prior weights and learnable weights are combined to fuse, score, and rank the candidate tiles. Further, a gating strategy is used to select K from multiple discrete sparse budget levels and apply alignment constraints, generating a sparse token output sequence that meets fixed length and alignment requirements. Finally, sparse tokens are organized in a tile-major order to form contiguous blocks, thereby reducing memory access overhead caused by irregular indexes and enabling subsequent attention-related matrix operations to be efficiently executed in a fixed shape on the edge NPU matrix computation path. Through these technical means, this invention aims to balance task accuracy, latency, and execution stability in edge-side real-time inference scenarios.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A task-aware hierarchical sparse filtering method for NPUs is characterized in that the method is applied to the inference scenario of an edge neural network processor (NPU), and performs sparsification processing on visual feature inputs to generate fixed-length input sequences that satisfy alignment constraints. The method includes:

[0009] First, acquire the task prior information and visual feature input at the current moment; wherein, the task prior information includes at least the robot arm joint state parameters and camera calibration parameters, and the visual feature input includes at least the dual-channel visual feature token sequence output by the visual coding network; Second, perform forward kinematics based on the joint state parameters to obtain the end effector pose, and project the end effector pose based on the camera calibration parameters to obtain pixel coordinates, and then map the pixel coordinates to visual feature grid coordinates to determine the candidate region center point; Third, generate a candidate region window based on the candidate region center point, and divide the candidate region window into candidate tile sets according to tile blocks, and establish the candidate tile blocks and their internal features. The mapping relationship between lexical tokens; 4. For each candidate tile in the candidate tile set, calculate the prior weight based on the geometric relationship between the candidate tile and the center point of the candidate region; and perform pooling on the feature lexical tokens within the candidate tile to obtain a tile representation, and input the tile representation into a lightweight scoring network to obtain learnable weights; 5. Fuse the prior weights and the learnable weights to obtain the importance score of each candidate tile, and sort the candidate tile set according to the importance score; 6. Based on the distribution characteristics of the importance score and the edge budget parameters, select a sparse budget level K from a preset discrete level set, and modify K to K* according to alignment constraints, where M is a preset alignment factor parameter used to characterize the block execution granularity of the matrix operation execution unit of the target edge NPU; K is the effective token budget number participating in subsequent attention-related calculations; K* is the length of the fixed-length output sequence after alignment constraint correction, and satisfies K*≥K, so that K* satisfies the constraints of the matrix operation execution unit of the target edge NPU on the input shape and alignment, thereby enabling subsequent attention-related matrix operations to be executed in a fixed shape and reducing tail block processing and dynamic shape scheduling overhead;

[0010] 7. Determine the number of selected tiles m based on the sparse budget level K and the preset tile size, wherein m satisfies The T tileThe preset number of feature tokens contained in a single tile is defined. The top-m candidate tiles are selected from the sorting results, and these top-m candidate tiles are expanded into a sparse feature token sequence in tile-major order, while simultaneously generating an index mapping and / or a mask set. The mask set is used to indicate invalid pruning positions, budget truncation positions, and padding. Fill in the missing parts; 8. Organize the sparse feature token sequence into a continuous block sequence according to the tile-major order, and perform fixed-length alignment processing on the sparse feature token sequence to obtain a fixed-length sparse feature token output sequence of length K*. Input the fixed-length sparse feature token output sequence into the subsequent attention calculation or control strategy network; wherein, the importance score calculation, sorting, gating selection and index / mask generation are performed by the vector calculation path of the edge NPU, and the subsequent attention-related matrix operation is performed by the matrix operation execution unit of the edge NPU to reduce the overhead of non-continuous memory access and dynamic shape scheduling.

[0011] Furthermore, the third step of generating candidate region windows includes: generating a square window or a circular window based on a preset window radius parameter; and mapping the candidate region window to a candidate range on a visual feature grid, so that the candidate tile set is limited to the region related to the pose of the end effector.

[0012] Furthermore, the third step of dividing by tile includes: dividing the candidate range into a preset number of tiles according to a preset tile size, wherein each tile consists of a preset number of adjacent feature tokens, and the number of feature tokens in the same tile is not less than 2 and not more than 64, so that the feature tokens in the same tile are spatially continuous and form a continuous address block in storage, thereby reducing the memory access overhead caused by irregular index access.

[0013] Furthermore, in step 4, the prior weight is the output value of a distance decay function or a Gaussian function, and the smaller the distance between the candidate tile and the center point of the candidate region, the larger the prior weight.

[0014] Furthermore, the feature is that the pooling in step 4 includes average pooling, max pooling, or weighted pooling; the lightweight scoring network consists of at least one linear layer and a nonlinear activation function, used to output learnable weights for candidate tiles.

[0015] Furthermore, in step 5, the importance score s is obtained by weighted summation. i ,in Where i is the index of each tile in the candidate tile set, and the value of i ranges from 1 to the total number of tiles in the candidate tile set; w prior,i The prior weight of the candidate tile corresponding to index i, w learn,i The learnable weights of the candidate tile corresponding to index i are α∈[0,1], which are preset fusion coefficients or learnable fusion coefficients. α can be adaptively adjusted through training, and its value range is kept within [0,1] during training.

[0016] Furthermore, in step 6, the preset discrete gear set includes at least a number of sparse budget candidate values; the alignment constraint is characterized by a preset alignment multiple parameter M, where M is the block execution granularity of the target edge NPU matrix operation execution unit, and K* is an integer multiple of M, so as to adapt to the block execution granularity of the edge NPU matrix operation and reduce the tail block processing overhead.

[0017] Furthermore, the fixed-length alignment process in step 8 includes: when the number of sparse feature tokens obtained from the screening is insufficient to satisfy K*, padding is performed using at least one of the following methods: fixed-position feature token padding, zero-value feature token padding, or boundary tile repeated padding; and in subsequent attention-related matrix operations, the padding positions are masked according to the mask set to avoid the padding feature tokens from participating in matrix accumulation operations and to reduce invalid computational overhead.

[0018] A task-aware hierarchical sparse filtering system for NPU is characterized in that the system is applied in the inference scenario of edge neural network processor (NPU) to perform sparsification processing on visual feature input to generate a fixed-length input sequence that satisfies alignment constraints. The system includes: an input acquisition module, a candidate region generation module, a prior weight calculation module, a learnable weight calculation module, a score fusion and sorting module, a budget gating and alignment module, a Top-m filtering expansion module, a fixed-length alignment output module, and a parameter storage device.

[0019] The input end of the input acquisition module is connected to the external task prior information data source and the output end of the external visual coding network, respectively. The output end of the input acquisition module is connected to the input end of the candidate region generation module and the learnable weight calculation module, respectively.

[0020] The output of the candidate region generation module is connected to the input of the prior weight calculation module and the learnable weight calculation module, respectively.

[0021] The outputs of the prior weight calculation module and the learnable weight calculation module are both connected to the input of the score fusion and sorting module.

[0022] The output of the score fusion sorting module is connected to the input of the budget gating and alignment module;

[0023] The output of the budget gating and alignment module is connected to the input of the Top-m filtering and expansion module;

[0024] The output of the Top-m filtering and unfolding module is connected to the input of the fixed-length alignment output module;

[0025] The output of the fixed-length aligned output module is connected to the input of an external subsequent attention calculation network or control strategy network;

[0026] The parameter storage device is bidirectionally connected to the input acquisition module, candidate region generation module, prior weight calculation module, learnable weight calculation module, score fusion and sorting module, budget gating and alignment module, Top-m filtering and expansion module, and fixed-length alignment output module, providing parameter read and write support for each module, wherein:

[0027] The input acquisition module is used to acquire the task prior information and visual feature input at the current moment, and output the task prior information to the candidate region generation module and output the visual feature input to the learnable weight calculation module; the task prior information includes at least the robot arm joint state parameters and camera calibration parameters, and the visual feature input includes at least the dual-path visual feature token sequence output by the visual coding network;

[0028] The candidate region generation module receives joint state parameters and camera calibration parameters output by the input acquisition module, performs forward kinematics based on the joint state parameters to obtain the end effector pose, projects the end effector pose based on the camera calibration parameters to obtain pixel coordinates, and then maps the pixel coordinates to visual feature grid coordinates to determine the center point of the candidate region; generates a candidate region window based on the center point of the candidate region, and divides the candidate region window into candidate tile sets according to tiles, and establishes a mapping relationship between the candidate tile sets and their internal feature tokens; outputs the candidate tile set to the prior weight calculation module and the learnable weight calculation module respectively.

[0029] The prior weight calculation module receives the candidate tile set and candidate region center point output by the candidate region generation module, calculates the prior weight based on the geometric relationship between the candidate tile and the candidate region center point, and outputs the prior weight to the score fusion and sorting module.

[0030] The learnable weight calculation module is used to receive the candidate tile set output by the candidate region generation module and the visual feature input output by the input acquisition module, perform pooling on the feature tokens in the candidate tile to obtain the tile representation, input the tile representation into the lightweight scoring network to obtain learnable weights, and output the learnable weights to the score fusion and ranking module.

[0031] The score fusion and sorting module receives the prior weights output by the prior weight calculation module and the learnable weights output by the learnable weight calculation module, fuses the prior weights and the learnable weights to obtain the importance score of each candidate tile, sorts the candidate tile set according to the importance score, and outputs the sorted candidate tile set and importance score to the budget gating and alignment module.

[0032] The budget gating and alignment module receives the importance score output by the score fusion and sorting module. Based on the distribution characteristics of the importance score and the end-side budget parameters, it selects a sparse budget level K from a preset discrete level set and modifies K to K* according to alignment constraints. This ensures that K* satisfies the constraints on input shape and alignment imposed by the edge NPU matrix computation path. The alignment constraint is characterized by a preset alignment multiple parameter M, where M is the block execution granularity of the target edge NPU matrix operation execution unit. K* is made an integer multiple of M, thereby enabling subsequent attention-related matrix operations to be executed with a fixed shape and reducing tail block processing and dynamic shape scheduling overhead.

[0033] The Top-m filtering expansion module receives the K value output by the budget gating and alignment module, and the sorted candidate tile set output by the score fusion and sorting module. It determines the filtering quantity m based on K and the preset tile size, where m is [K / T]. tile The T tile The number of feature tokens contained in a single tile is preset, and the top-m candidate tiles are selected from the sorting results; the candidate tiles are expanded into a sparse feature token sequence, and an index mapping and / or mask set is generated; the mask set is used to indicate invalid pruning positions, budget truncation positions, and padding positions;

[0034] The fixed-length aligned output module receives the sparse feature token sequence, index mapping, and mask set output by the Top-m filtering and unpacking module, as well as the K value output by the budget gating and alignment module. It organizes the sparse feature token sequence into a continuous block sequence according to tile major order, performs fixed-length alignment processing on the sparse feature token sequence to obtain a fixed-length sparse feature token output sequence of length K*, and outputs the fixed-length sparse feature token output sequence to the subsequent attention calculation or control strategy network. The importance score calculation, sorting, gating selection, and index / mask generation are performed by the edge-side NPU vector calculation path, while the subsequent attention-related matrix operations are performed by the edge-side NPU matrix calculation path to reduce the overhead of non-continuous memory access and dynamic shape scheduling.

[0035] The parameter storage device includes a strategy parameter management module, a model and weight parameter module, and a statistics and configuration module. The strategy parameter management module stores strategy parameters such as the fusion coefficient α, alignment factor parameter M, sparse budget discrete tier set, and coverage threshold. The model and weight parameter module stores the weight parameters of the lightweight scoring network. The statistics and configuration module stores configuration parameters such as edge budget parameters, tile size, and window radius parameters. The parameter storage device provides parameter reading services to various functional modules within the system and receives parameter update data from each module. When the system adapts to new NPU hardware or task scenarios, each functional module can initiate a parameter update request to the parameter storage device.

[0036] The specific content involved in this invention and the meaning of the terms used are as follows:

[0037] To facilitate understanding and implementation of the technical solution of this invention, the terms used in this specification shall have the following meanings without ambiguity:

[0038] (1) Neural Processor (NPU): refers to the neural network processor hardware and its operating environment used for edge inference computation, which is used to perform computational loads such as vector computation and matrix computation.

[0039] (2) token: a word unit, which refers to the feature vector unit output by the visual encoding network and used for subsequent attention calculation.

[0040] (3) Tile: refers to a block unit composed of multiple adjacent tokens, which serves as the basic unit for candidate construction, sorting and filtering.

[0041] (4) Candidate region center point: refers to the reference point calculated from the joint state and camera calibration and mapped to the visual feature grid coordinates, used to generate candidate region windows.

[0042] (5) Prior weight: refers to the weight calculated based on the geometric relationship between the candidate tile and the center point of the candidate region, which is used to characterize the spatial relevance priority.

[0043] (6) Learnable weights: These are the weights output by the lightweight scoring network, used to characterize the degree of association between candidate tiles and the target task.

[0044] (7) K, K* and m: K is the effective token budget selected from the preset discrete tier set, used to represent the number of effective tokens participating in subsequent attention-related calculations; K* is the fixed-length output length obtained by K under alignment constraints, satisfying K*=[K / M]·M, where M is the alignment multiple parameter, and K*≥K; m is the number of tiles to be selected by converting the effective token budget K, for example T tile This represents the number of tokens contained in a single tile. For any portion less than K*, padding is used to fill the gaps, and a mask is used to mask them, thus ensuring a fixed output shape and not affecting the calculation results for valid tokens.

[0045] (8) Tile-major order: refers to the order in which tokens are organized by tile, so that tokens within the same tile are arranged consecutively in the output sequence.

[0046] (9) Vector computation path / matrix computation path: These refer to the execution units in the NPU used to perform vector / index / control operations, and the execution units used to perform matrix operations such as matrix multiplication, linear transformation, and attention-related matrix operations.

[0047] (10) Ascend NPU: An NPU implementation form, which is used as an example platform in some embodiments of this specification.

[0048] Advantages and positive effects of the present invention:

[0049] Compared with the prior art, the present invention has the following advantages and positive effects:

[0050] (1) Task constraints enhance screening stability. By using joint states and camera calibration to generate candidate regions and construct a candidate tile set, the candidate space of sparse screening is subject to task prior constraints, reducing the screening position drift caused by unconstrained saliency screening and improving the interpretability and time delay stability of end-side reasoning.

[0051] (2) Hierarchical screening reduces invalid computation and data transfer. By merging prior weights and learnable weights for sorting, visual information that is more relevant to task decision is prioritized under the same sparse budget, thereby increasing the proportion of "effective tokens" and reducing the proportion of invalid tokens participating in subsequent attention computation.

[0052] (3) Discrete K-level and alignment constraints ensure fixed shape execution. Discrete sparse budget levels are selected through gating, and the K-level is modified to K* according to the alignment constraints, where... This enables subsequent attention-related matrix operations to be performed in a fixed shape on the edge NPU matrix computation path, thereby reducing tail block processing, padding branches, and dynamic scheduling overhead.

[0053] (4) Tile-major contiguous block organization reduces irregular memory access overhead. Sparse tokens are organized in tile-major order and the index / mask is output so that the filtered tokens are distributed in contiguous blocks, reducing the non-contiguous memory access and rearrangement overhead caused by irregular Gather / Scatter, and improving the efficiency of edge-side handling and cache reuse.

[0054] (5) Improve edge real-time performance through division of labor and collaboration of vector / matrix paths. Control operations such as weight calculation, gating and index generation are completed in the vector calculation path, and subsequent attention-related matrix operations are constrained to fixed-shape inputs and executed by the matrix calculation path, thereby improving the execution efficiency and latency stability of edge inference. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments are briefly described below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 A flowchart illustrating a task-aware hierarchical sparse filtering method for edge-side NPUs provided in an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of a task-aware hierarchical sparse filtering system for edge-side NPUs provided in an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram illustrating the generation of candidate regions and the construction of a candidate tile set based on task prior projection in an embodiment of the present invention;

[0059] Figure 4This is a schematic diagram illustrating the gating selection of sparse budget levels based on score distribution and the execution of alignment correction to obtain K* in an embodiment of the present invention;

[0060] Figure 5 This is a schematic diagram of the division of labor and collaborative execution of the vector computing path and matrix computing path for the edge-side NPU in an embodiment of the present invention. The left side shows the conventional token mixed arrangement method and the right side shows the continuous block organization method of tile-major order, illustrating the core mechanism of the present invention to reduce non-contiguous memory access overhead.

[0061] Figure 6 This is a schematic diagram illustrating tile-major organization, fixed-length aligned output, and mask generation in an embodiment of the present invention. Detailed Implementation

[0062] The technical solutions provided by the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0063] like Figure 2 As shown, this embodiment provides a task-aware hierarchical sparse filtering system 100 for edge-side NPUs. The device 100 is deployed on an edge-side NPU platform and is used to perform hierarchical sparse filtering on visual feature token sequences in inference scenarios, and output a fixed-length sparse token sequence that satisfies alignment constraints, which can be used as input for subsequent attention-related matrix operations or control policy networks.

[0064] The device 100 includes: an input acquisition module 110, a candidate region generation module 120, a prior weight calculation module 130, a learnable weight calculation module 140, a score fusion and sorting module 150, a budget gating and alignment module 160, a Top-m filtering and expansion module 170, and a fixed-length alignment output module 180, as well as a parameter storage device 300; wherein the parameter storage device 300 includes a strategy parameter management module 310, a model and weight parameter module 320, and a statistics and configuration module 330, and the parameter storage device 300 is bidirectionally connected to the above 8 functional modules, providing parameter read and write support for each module.

[0065] The input acquisition module 110 has an input end that connects to an external task prior information data source and the output end of an external visual coding network, and is used to acquire the task prior information and visual feature input at the current moment. The task prior information includes at least the robot arm joint state parameters and camera calibration parameters, and the visual feature input includes at least the dual-channel visual feature token sequence output by the visual coding network. The output end is connected to the input ends of the candidate region generation module 120 and the learnable weight calculation module 140, respectively, outputting the task prior information to the candidate region generation module 120 and outputting the visual feature input to the learnable weight calculation module 140.

[0066] Candidate region generation module 120: Its input end is connected to the output end of the input acquisition module 110, and is used to receive the joint state parameters and camera calibration parameters output by the input acquisition module 110. Based on the joint state parameters, it performs forward kinematics to obtain the end effector pose, and projects the end effector pose to obtain pixel coordinates based on the camera calibration parameters, and then maps them to visual feature grid coordinates to determine the center point of the candidate region. A candidate region window is generated based on the center point of the candidate region. Within the candidate region window, a candidate tile set is formed by dividing the area into tiles, and a mapping relationship between tiles and tokens is established. Its output end is connected to the input end of the prior weight calculation module 130 and the learnable weight calculation module 140, respectively. The candidate tile set and the candidate region center point are output to the prior weight calculation module 130, and the candidate tile set is output to the learnable weight calculation module 140.

[0067] The prior weight calculation module 130 has its input end connected to the output end of the candidate region generation module 120. It receives the candidate tile set and candidate region center points output by the candidate region generation module 120, calculates prior weights based on the geometric relationship between the candidate tiles and the candidate region center points, and these prior weights characterize spatial relevance priority. Its output end is connected to the input end of the score fusion and ranking module 150, outputting the prior weights to the score fusion and ranking module 150.

[0068] The learnable weight calculation module 140 has its input ends connected to the output ends of the input acquisition module 110 and the candidate region generation module 120, respectively. It is used to receive the candidate tile set output by the candidate region generation module 120 and the visual feature input output by the input acquisition module 110. It performs pooling on the tokens within the candidate tiles to obtain tile representations, and inputs the tile representations into a lightweight scoring network to obtain learnable weights. The learnable weights are used to represent the degree of correlation between the candidate tiles and the target task. The output end is connected to the input end of the score fusion and ranking module 150, and outputs the learnable weights to the score fusion and ranking module 150.

[0069] The score fusion and sorting module 150 has its input terminals connected to the output terminals of the prior weight calculation module 130 and the learnable weight calculation module 140, respectively. It receives the prior weights and learnable weights, fuses them to obtain the importance score of each candidate tile, and sorts the candidate tile set according to the importance score. Its output terminal is connected to the input terminal of the budget gating and alignment module 160, and outputs the sorted candidate tile set and importance score sequence to the budget gating and alignment module 160.

[0070] Budget Gating and Alignment Module 160:

[0071] The input end connects to the output end of the score fusion and sorting module 150, and is used to select a sparse budget level K from a preset discrete level set based on the distribution characteristics of the importance score and the end-side budget parameters, and to modify K to K* according to alignment constraints, where This ensures that the input shape and alignment of subsequent matrix operations satisfy the execution constraints of the edge NPU matrix computation path, reducing tail block processing and dynamic scheduling overhead. In one example implementation, the distribution feature is obtained by coverage gating, that is, the importance score is normalized to a probability distribution by Softmax and the minimum number of tiles m is selected based on the coverage threshold, thereby obtaining K_raw and projecting it onto the discrete set of tiles to determine K (unrestricted). The output is connected to the input of the Top-m filtering expansion module 170 and the fixed-length alignment output module 180, respectively, outputting the sparse budget tile K to the Top-m filtering expansion module 170 and outputting the aligned K* to the fixed-length alignment output module 180.

[0072] Top-m filtering expansion module 170: The input ends are respectively connected to the output ends of the score fusion sorting module 150 and the budget gating and alignment module 160, and are used to filter based on m=[K / T tileThe top-m candidate tiles are selected from the sorting results and expanded into a sparse token sequence. At the same time, an index map and / or a mask set are generated. The mask set is further used to indicate the budget truncation and padding positions. The output is connected to the input of the fixed-length aligned output module 180 and outputs the sparse token sequence, index map, and mask set to the fixed-length aligned output module 180.

[0073] Fixed-length alignment output module 180: Its input ends are respectively connected to the output ends of Top-m filtering and expansion module 170 and budget gating and alignment module 160. It is used to organize the sparse token sequence into a continuous block sequence according to the tile major order, and perform fixed-length alignment processing on the sparse token sequence to obtain a fixed-length sparse token output sequence of length K*. Its output end is connected to the input end of the external subsequent attention calculation network or control policy network, and outputs the fixed-length sparse token sequence, index mapping, and mask set to the downstream network.

[0074] Parameter storage device 300: Strategy parameter management module 310: used to store strategy parameters such as fusion coefficient α, alignment multiple parameter M, sparse budget discrete tier set, and coverage threshold; Model and weight parameter module 320: used to store weight parameters of lightweight scoring network; Statistics and configuration module 330: used to store configuration parameters such as end-side budget parameters, tile size, and window radius parameters; The parameter storage device 300 communicates bidirectionally with the above 8 functional modules, providing each module with the parameters required for operation, and receiving parameter update data from each module.

[0075] like Figure 5 As shown, in one implementation, the system 100 of this embodiment is based on the hardware architecture of the edge NPU and adopts a hardware division of labor and cooperation mode of "vector computing path + matrix computing path" to perform the task, wherein:

[0076] The edge-side NPU includes two types of fixed hardware execution paths: a vector computation path, which is a hardware execution unit in the NPU used to perform vector operations, indexing operations, logic control, and sorting / filtering operations, possessing low latency and high flexibility in control-type computation capabilities; and a matrix computation path, which is a hardware execution unit in the NPU used to perform matrix multiplication, linear transformation, and attention-related matrix operations, possessing high parallelism and high computing power in matrix operations, whose execution efficiency is highly sensitive to the shape regularity and alignment of the input data. Hardware deployment mapping of the software flow: The sparse filtering method of this patent is deployed and executed according to the capabilities of the hardware units as follows: control and indexing operations such as candidate region construction, prior weight calculation, learnable weight calculation, score fusion and sorting, sparse budget level selection and alignment correction, Top-m filtering, and indexing and mask generation in the software flow are deployed and executed on the vector computation path; subsequent attention-related matrix operations in the software flow use the fixed-length sparse token sequence output by this system as input and are deployed and executed on the matrix computation path.

[0077] Through the coordinated adaptation of the aforementioned hardware and software, control and indexing operations are centralized within the vector computation path, and subsequent matrix operations are constrained to fixed-shape inputs. This embodiment reduces the overhead caused by dynamic shape scheduling and irregular memory accesses, thereby improving the execution efficiency and latency stability of edge inference.

[0078] It should be noted that the above module division is only an example of functional division. In actual implementation, some modules can be merged, split, or renamed according to the software and hardware architecture. As long as the technical solution of this invention can be achieved, it should fall within the protection scope of this invention.

[0079] like Figure 1 As shown, the task-aware hierarchical sparse filtering method for NPU in this embodiment includes at least steps 1 to 8:

[0080] Step 1: Input Acquisition (corresponding to the box in Figure 1: Input Acquisition 110).

[0081] Obtain the task prior information and visual feature input at the current moment; wherein, the task prior information includes at least the robot arm joint state parameters q. t Along with camera calibration parameters, the visual feature input includes at least a dual-path visual feature token sequence output by a visual coding network. This dual-path visual feature token sequence corresponds to feature outputs from both the global camera viewpoint and the wrist camera viewpoint.

[0082] In one implementation, the token sequence output by the visual encoding network can be represented as a set of vectors on a two-dimensional feature grid, denoted as... ,in These are the coordinates of the feature grid.

[0083] Step 2: Candidate Region Generation (Corresponding to the box in Figure 1: Candidate Region Generation 120)

[0084] Based on joint state parameter q t The end effector pose is obtained by performing forward kinematics; and the end effector pose is projected onto the image coordinate system based on camera calibration parameters to obtain pixel coordinates. This maps pixel coordinates to visual feature grid coordinates. To determine the center point of the candidate region.

[0085] like Figure 3 As shown, the center point of the candidate region is used to constrain the generation of subsequent candidate regions, so that the candidate space for screening is consistent with the task-related region, thereby improving the stability of sparse screening and reducing the number of irrelevant region tokens entering subsequent matrix operations.

[0086] Step 3: Hierarchical scoring and sorting (corresponding to the box in Figure 1: hierarchical scoring 130 / 140 / 150) Generate a candidate region window based on the center point of the candidate region, and form a candidate tile set by dividing the candidate region window into tile blocks; at the same time, establish the mapping relationship between the tile and its internal token.

[0087] In one implementation, the candidate region window can be generated by a preset window radius parameter; a tile can consist of multiple adjacent tokens, making tokens within the same tile spatially continuous. By using tiles as the filtering unit, index fragmentation caused by filtering granularity can be reduced while maintaining local continuity.

[0088] Step 4: Budget Gating and Alignment (corresponding to the box in Figure 1: Budget Gating and Alignment 160)

[0089] For each candidate tile in the candidate tile set, the prior weight is calculated based on the geometric relationship between the candidate tile and the center point of the candidate region. This is used to characterize spatial relevance priority; and pooling is performed on the tokens within the candidate tile to obtain the tile representation. The tile representation is input into a lightweight scoring network to obtain learnable weights. This is used to characterize the degree of association between candidate tiles and the target task.

[0090] In one implementation, the prior weights can be output using a distance decay function, and the closer the candidate tile is to the center point, the greater the prior weight; the tile representation can be obtained through average pooling or max pooling; the lightweight scoring network can be composed of linear layers and nonlinear activation functions to reduce additional overhead at the edge.

[0091] Step 5: Top-m selection and expansion (corresponding to the box in Figure 1: Top-m selection and expansion 170)

[0092] Importance scores are obtained by fusing prior weights and learnable weights. The candidate tile set is then sorted according to the importance score. In one implementation, a weighted summation method can be used for fusion, for example...

[0093] ,in This involves pre-set or learnable fusion coefficients. By fusing prior knowledge and data-driven information, key task regions (tiles) can be prioritized for retention within a limited budget on the device side.

[0094] An optional formula for calculating the score s (not limited)

[0095] Let the candidate tile be Prior score is Learnable score All of them can be normalized to [0,1].

[0096] Formula group (A) linear fusion

[0097]

[0098] Formula group (B) geometric fusion

[0099]

[0100] Formula group (C) rank fusion setting The rankings are sorted according to two different scores (lower rankings are better).

[0101]

[0102] Formula group (D) confidence level gating fusion setting confidence level ,in If it is a parameter, then

[0103]

[0104] Formula group (E) dual-path normalization and then fusion setting Let each be a value after Softmax / MinMax normalization.

[0105]

[0106] In one example implementation, the fusion coefficient α can take the values ​​shown in Example Parameter Table 1, and the fusion score s can be calculated using the linear fusion method shown in Example Parameter Table 1 (not limited).

[0107] In this numerical demonstration, the example uses formula group (A) / (D) to calculate s (unrestricted).

[0108] Step 6: Fixed-Length Alignment Preprocessing (corresponding to the box in Figure 1: Fixed-Length Alignment Output 180) Based on the distribution characteristics of importance scores and end-side budget parameters, select a sparse budget level K from the preset discrete level set; modify K to K* according to alignment constraints, where This ensures that K* satisfies the constraints on input shape and alignment imposed by the edge NPU matrix computation path, thereby reducing the overhead of tail block processing, padding branches, and dynamic scheduling.

[0109] like Figure 4 As shown, in one example implementation, the budget gating and alignment module 160 uses a coverage gating rule to determine the sparse budget. Specifically, the sorted importance score sequence s (202) is normalized by Softmax to obtain the probability distribution p, and a coverage threshold θ is set. cov Choose the smallest integer m such that the cumulative coverage rate satisfies Based on this, K is calculated. raw =m·T tile and K raw Projecting onto the preset discrete tier set yields the final sparse budget tier K (203); and updating the number of filtered tiles based on the final K. This ensures that the token capacity after Top-m expansion is no less than K. Then, based on the alignment factor parameter M, K is corrected to K*(204), with the following formula: This ensures that subsequent matrix operations are performed on the edge NPU matrix computation path with fixed-shape inputs, and reduces tail block processing and dynamic scheduling overhead. The above rules are non-limiting examples.

[0110] Step 7: Valid token count determination (corresponding to the diamond-shaped judgment box in Figure 1) based on K and T. tile Calculate the number of filters m ( The top-m candidate tiles are selected and expanded into a sparse token sequence to generate an index map and a mask set; the mask set is used to indicate invalid clipping positions, budget truncation positions and padding positions.

[0111] like Figure 6 As shown, the index mapping and mask set can be used to mask the padding position in subsequent attention calculations, avoiding invalid tokens from participating in matrix accumulation operations, thereby reducing the overhead of invalid computation.

[0112] Step 8: Output Results (corresponding to the output box in Figure 1) Organize the sparse token sequence into a continuous block sequence according to the tile major order, and perform fixed-length alignment processing on the sparse token sequence to obtain a fixed-length sparse token output sequence; the fixed-length alignment processing includes at least padding, the length of the padded sparse token output sequence is K*, and the padding position is indicated as an invalid position through a mask set.

[0113] In one implementation, tile major organization arranges tokens within the same tile consecutively in the output sequence, enabling sparse token sequences to be moved in a contiguous address manner and reducing the non-contiguous memory access overhead caused by irregular Gather / Scatter operations. At the same time, fixed-length K* outputs enable subsequent attention-related matrix operations to be performed in a regular and fixed shape on the Ascend NPU matrix computation path, improving the execution efficiency and latency stability of edge inference.

[0114] After completing step 8, the fixed-length sparse token output sequence is input into the subsequent attention calculation or control policy network to obtain the attention result or control output.

[0115] Application Examples

[0116] The following is an application example of the method of the present invention to illustrate its specific application in an edge-side NPU inference scenario. In this application example, Ascend NPU is used as a non-limiting example platform. It should be noted that this application example is used to illustrate the feasibility of the technical solution of the present invention and does not constitute a limitation on the scope of protection of the present invention.

[0117] In this application example, the edge system includes a robotic arm, a camera, and an Ascend NPU computing unit. The robotic arm performs tasks such as grasping, moving, and placing; the camera acquires visual information of the work area; and the Ascend NPU performs visual encoding network, task-aware hierarchical sparse filtering, and subsequent attention-related calculations or control strategy network inference. To illustrate the feasibility of this invention in an edge-side NPU inference scenario, a set of example parameter tables is provided below, followed by a complete numerical demonstration.

[0118] Example parameter table 1 (non-restricted)

[0119] To illustrate the feasibility of this invention in edge-side inference scenarios, a set of example parameters are provided below. It should be understood that these parameters are for illustrative purposes only, and those skilled in the art can adjust them according to model structure, input resolution, hardware constraints, and real-time requirements; they do not constitute a limitation of this invention.

[0120] (1) Dual-path token grid size: The example uses a 32×32 token grid (1024 tokens) output by the visual encoding network. (2) Tile size (block size): The example uses 4×4 tokens per tile, i.e., 16 tokens per tile. (3) Candidate window parameters: The example uses a candidate window covering 5×5 tiles (i.e., a maximum of 25 candidate tiles), with the window center determined by prior information. (4) Prior weight calculation parameters: The example uses a distance function from the tile center to the window center to obtain the prior weight w. prior =exp(-dist² / σ²), in the example, σ = 2.0 (tile unit distance). (5) Calculation of learnable weights: The example uses the output w of the lightweight scoring network ScoreNet. learn w learn Normalize to [0,1] (one example method). (6) Fusion coefficient: In the example, α = 0.6, and the fusion score is defined as s = α·w prior + (1-α)·w learn (7) Distribution characteristics and coverage gating (one example method): The example denotes the importance score sequence of candidate tiles as s={s1,s2,…,s_N} (sorted from largest to smallest), and uses Softmax normalization to obtain the probability distribution p: , where τ is the temperature parameter (in this example, τ = 0.15). Further, let the coverage threshold θ be... cov (Example: θ) cov =0.90), choose the smallest positive integer m that satisfies the constraints such that And calculate the continuous value of the token budget: K raw =m·T tile T tile This is the number of tokens for a single tile (16 in this example).

[0121] (7a) Discrete budget projection: Example of K raw Project onto the discrete budget set K∈{80, 96, 104, 120, 136}, the projection rule is to select values ​​strictly greater than K. rawThe minimum level is used as K (used for conservative upward floating to resist invalid tokens on the boundary tile and edge jitter; not limited). (7b) Alignment correction: The example takes the alignment multiple M = 16 and uses K* = [K / M]·M to obtain the alignment budget K*. (Explanation) The above coverage gating is used to directly associate the budget selection with the "concentration" of the score distribution, thereby improving the edge latency stability (one example explanation). (8) Discrete budget set: The example takes K to belong to the discrete set K∈{80, 96, 104, 120, 136} (unit: number of tokens). (9) Alignment multiple parameter: The example takes M = 16 (that is, K* must be an integer multiple of 16). (10) Alignment correction rule: The example uses K* = [K / M]·M. (11) Top-m filtering objects: The example selects the top m candidate tiles by sorting by tile score ( ), and expand the selected tile into a token sequence Z. raw Simultaneously generate index mapping I raw With Mask raw If the number of tokens after expansion is greater than K, the position exceeding K is marked as budget truncation and the Mask is set to 0; then, according to the fixed-length output requirement of K*, the insufficient part is padded and the Mask is updated. (12) Mask convention: The example convention is that Mask=1 represents a valid token and Mask=0 represents a padding placeholder token. (13) Tile-major rearrangement rule: The example organizes tokens according to "tile major order", that is, tokens in the same tile are arranged continuously in Z, and different tiles are arranged according to the tile score and spliced ​​in order. (14) Padding strategy: When the number of valid tokens is less than K*, the example uses a zero vector or a preset placeholder vector to pad and sets the corresponding Mask to 0.

[0122] The following describes the gating selection of K, alignment to obtain K*, Top-K* filtering, index / mask generation, and fixed-length padding output process of the present invention, using the example parameters described above. A complete numerical demonstration is also provided to facilitate reproduction and implementation by those skilled in the art.

[0123] Method and Flow

[0124] (1) Input preparation. For example... Figure 3 As shown, at time Obtain the joint state parameters q of the robotic arm t In addition to camera calibration parameters, the system also acquires a dual-channel visual feature token sequence output by the visual coding network. This dual-channel visual feature token sequence corresponds to the feature outputs from both the global camera viewpoint and the wrist camera viewpoint.

[0125] (2) Candidate region construction. Based on joint state parameter q t The end effector pose is obtained by performing forward kinematics; the end effector pose is projected onto the camera calibration parameters to obtain pixel coordinates, and then mapped to visual feature grid coordinates to determine the center point of the candidate region; a candidate region window is generated based on the center point of the candidate region, and a candidate tile set is formed by dividing the candidate region window into tile-based sections. A mapping relationship between tiles and tokens is established, such as... Figure 3 As shown. This candidate region constraint can reduce the number of task-irrelevant regions entering subsequent filtering and matrix operations.

[0126] (3) Hierarchical scoring and ranking. For each candidate tile in the candidate tile set, a prior weight is calculated based on the geometric relationship between the candidate tile and the center point of the candidate region; pooling is performed on the tokens within the candidate tile to obtain the tile representation, and the representation is input into a lightweight scoring network to obtain learnable weights; the prior weights and learnable weights are fused to obtain an importance score and complete the ranking. This step prioritizes retaining candidate tiles that are relevant to the task, providing a basis for subsequent fixed-budget screening.

[0127] (4) Gating selection K and alignment correction. When the terminal NPU is an Ascend NPU, such as Figure 4 As shown, based on the distribution characteristics of importance scores and end-side budget parameters, a sparse budget level K is selected from a preset discrete level set. K is then modified to K* according to alignment constraints, where... In one implementation, the alignment constraint is characterized by a preset alignment multiple parameter, such that K* is an integer multiple of the alignment multiple, thereby enabling subsequent attention-related matrix operations to be performed on the Ascend NPU matrix computation path with a fixed-shape input, reducing tail block processing and dynamic scheduling overhead.

[0128] (5) Top-m filtering, expansion, and mask generation. Based on m ( ) Filter the top-m candidate tiles from the sorted results, and expand the selected tiles into a token sequence Z. raw Simultaneously generate index mapping I raw With Mask raw If the number of tokens after expansion is greater than K, then the positions exceeding K are marked as budget truncation and the Mask is set to 0; subsequently, padding is applied to the insufficient portion according to the fixed-length output requirement of K * K, and the Mask is updated. For example... Figure 6 As shown.

[0129] (6) Tile major organization and fixed-length alignment output. For example... Figure 6As shown, the sparse token sequence is organized into a continuous block sequence according to tile major order, and fixed-length alignment is performed to obtain a fixed-length sparse token output sequence. When the number of selected tokens is insufficient to satisfy K*, padding is performed using fixed placeholder tokens or zero-value tokens, and the padding position is indicated as an invalid position through a mask set. By organizing the tokens into continuous blocks according to tile major, the non-contiguous memory access overhead caused by irregular Gather / Scatter operations can be reduced and data handling efficiency can be improved. By using a fixed-length K* output, subsequent matrix operations can be performed in a regular and fixed shape.

[0130] (7) NPU edge execution mode. When the edge NPU is an Ascend NPU, such as Figure 5 As shown, on the Ascend NPU edge, operations such as candidate construction, weight calculation, fusion sorting, gating selection K and alignment correction, Top-m filtering, and index / mask generation can be performed by the vector computation path; the fixed-length sparse token output sequence serves as the input for subsequent attention-related matrix operations or control strategy network operations, and is performed by the matrix computation path. Through this division of labor and collaboration, the overhead caused by dynamic shape scheduling and discontinuous memory access can be reduced, the stability of edge inference latency can be improved, and the real-time requirements of robot operation tasks can be met.

[0131] A demonstration

[0132] (1) Candidate tile set C(201) and basic settings In this example, the candidate window covers a maximum of 25 tiles. For simplicity, assume that the actual candidate tile set is 10 tiles, denoted as C(201) = {t1, t2, ..., t10}, and each tile contains 16 tokens by default (i.e., T). tile =16). Among them, considering the boundary effect, it is assumed that t7 is a boundary tile, containing only 12 valid tokens (the remaining 4 positions are invalid / outbound tokens).

[0133] (2) Calculate the prior weight w prior With learnable weights w learn The example gives the weights of 10 candidate tiles (all normalized to [0,1]) as follows (example only): • t1: w prior =0.90, w learn =0.80• t2:w prior =0.85, w learn =0.75•t3:w prior =0.70, w learn =0.82•t4:w prior=0.65, w learn =0.60• t5:w prior =0.60, w learn =0.58•t6:w prior =0.55, w learn =0.52• t7:w prior =0.50, w learn =0.50 (boundary tile) • t8:w prior =0.35, w learn =0.55• t9:w prior =0.30, w learn =0.40• t10:w prior =0.20, w learn =0.35

[0134] (3) Calculate the fusion score s(202) and sort them (Module 130 / 140 / 150). Example: Take α = 0.6, the fusion score is s = 0.6·w prior +0.4w learn The calculations are as follows (example only, rounded to three decimal places): • s(t1) = 0.6 × 0.90 + 0.4 × 0.80 = 0.860 • s(t2) = 0.6 × 0.85 + 0.4 × 0.75 = 0.810 • s(t3) = 0.6 × 0.70 + 0.4 × 0.82 = 0.748 • s(t4) = 0.6 × 0.65 + 0.4 × 0.60 = 0.630 • s(t5) = 0.6 × 0.60 + 0.4 × 0.58 = 0.592 • s(t6) = 0.6 × 0.55 + 0.4 × 0.52 = 0.538 • s(t7) = 0.6 × 0.50 + 0.4 × 0.50 = 0.500 s(t8) = 0.6 × 0.35 + 0.4 × 0.55 = 0.430 • s(t9) = 0.6 × 0.30 + 0.4 × 0.40 = 0.340 • s(t10) = 0.6 × 0.20 + 0.4 × 0.35 = 0.260 Sort by score from highest to lowest to get the sorting result (202). Example: t1 > t2 > t3 > t4 > t5 > t6 > t7 > t8 > t9 > t10.

[0135] (4) Coverage gating selection budget K(203) (Module 160) takes the sorted fusion score sequence s={0.860, 0.810, 0.748, 0.630, 0.592, 0.538, 0.500, 0.430, 0.340, 0.260} as input. For example, the temperature parameter τ=0.15 is taken, and the Softmax normalized probability distribution is calculated. Where N=10 is the total number of candidate tiles, and i is the tile index (1≤i≤10). The probability distribution p is calculated as follows (for example only, rounded to four decimal places):

[0136] p≈{0.3463,0.2481,0.1641,0.0747,0.0580,0.0405,0.0314,0.0197,0.0108,0.0063} (example only). Let the coverage threshold be θ. cov =0.90, calculate the cumulative coverage rate:

[0137] Therefore, the smallest integer m that satisfies the cumulative coverage rate is chosen. gate =6. Because each tile contains a T by default. tile =16 tokens, resulting in a continuous budget: K raw =m gate ·T tile =6·16=96. Let K raw Projecting onto the discrete budget set K∈{80, 96, 104, 120, 136}, the example uses "strictly greater than K". raw The minimum gear rule yields K = 104 (token) (unrestricted). Then, K*(204) is obtained by adjusting according to the alignment constraint, and its calculation is shown in the next step (5).

[0138] (5) Alignment yields K(204) (Module 160) Example: Take the alignment multiple M = 16, and the alignment rule is K* = [K / M]·M = [104 / 16]·16 = [6.5]·16 = 7·16 = 112. That is, after alignment, the fixed-length output length is K = 112 (token), which satisfies that K is an integer multiple of M and K* ≥ K.

[0139] (6) Update the number of filters m by K and perform Top-m filtering (Module 170). Since K is the effective token budget, and each tile contains T by default. tile = 16 tokens. To ensure that the token capacity after expansion is no less than K, the example updates the filtering quantity as follows: = [104 / 16] = [6.5] = 7. The tile sequence with the highest score, selected in order, is: t1, t2, t3, t4, t5, t6, t7 (7 tiles in total). These tiles are then expanded and concatenated sequentially according to the tile-major order to obtain Z. raw Simultaneously generate index mapping Iraw With Mask raw Among them, t1–t6 each contain 16 valid tokens, and t7 is the boundary tile containing only 12 valid tokens. Therefore, the total number of valid tokens after expansion is: N. valid =6 × 16 + 12 = 96 + 12 = 108. Meanwhile, since the remaining 4 positions in t7 are invalid / out-of-bounds tokens, the example marks these 4 positions as invalid and sets them to Mask in the expanded sequence. raw =0 (or not counted in the valid token count, not limited). In this example, N valid =108 ≥ K=104, therefore budget truncation needs to be performed to obtain exactly K valid tokens.

[0140] (7) Truncate the budget to K, padding to K, and update the Mask (module 180).

[0141] (7.1) Budget truncation (ensuring the number of valid tokens is K): Example: Keep the first K = 104 valid tokens in tile-major order to obtain the truncated sequence (still denoted as Z). raw / I raw / Mask raw , or denoted as Z trunc / I trunc / Mask trunc (Unrestricted); For valid tokens exceeding K (in this example, 10⁸ − 10⁴ = 4 valid tokens), mark them as budget truncation and set Mask to 0 (or directly discard the corresponding position and fill it with padding later, unrestricted).

[0142] Therefore, the number of valid tokens after budget truncation is: K = 104.

[0143] (7.2) Padding (to ensure fixed-length output of K): To satisfy the fixed-length output of K, the example pads the truncated sequence to a length of K* = 112: Number of padding tokens required = K* - K = 112 - 104 = 8. The example uses 8 placeholder vectors (e.g., all-zero vectors) to pad to a length of 112, and sets the mask corresponding to these 8 positions to 0, while setting the mask corresponding to the remaining 104 valid tokens to 1. Therefore, the output satisfies: • Z (205): a fixed-length sparse token sequence of length 112; • I (206): an index mapping of length 112, where the first 104 items correspond to the actual token positions (e.g., can be represented as "(tile number, position within the tile)" or "(grid row and column)"), and the last 8 items are placeholder indices; • Mask (207): a mask sequence of length 112, where the first 104 items are 1 and the last 8 items are 0.

[0144] (Example) For ease of understanding, the representation of the first few items of I / Mask can be given (unrestricted): • The first token comes from the first position of t1: I[1] = (t1, p1), Mask[1] = 1; • The second token comes from the second position of t1: I[2] = (t1, p2), Mask[2] = 1; • …• The 97th token comes from the first position of t7: I

[97] = (t7, p1), Mask

[97] = 1; • …• The 105th–112th tokens are padding placeholders: I[105..112] is the placeholder index, Mask[105..112] = 0.

[0145] (8) The output is used for subsequent fixed shape calculation. The Z (205), I (206), and Mask (207) obtained constitute a fixed-length input, which can be directly input into the subsequent attention calculation or control policy network. The Mask is used to shield the budget truncation and padding position from the subsequent calculation, thereby keeping the input shape fixed on the edge NPU and reducing the dynamic shape-related overhead (one example).

[0146] Therefore, this invention, through task prior constraints, hierarchical scoring and discrete budget gating, combined with alignment correction and fixed-length output, transforms the edge-side inference process from irregular sparse screening into a fixed-shape matrix operation process that can be efficiently executed by the Ascend NPU, demonstrating good engineering feasibility.

[0147] Examples of electronic devices and storage media

[0148] In one implementation, the task-aware hierarchical sparse screening system 100 described in this embodiment of the invention can be implemented by an electronic device. The electronic device includes at least a processor, a memory, and a bus, wherein the processor and the memory are connected via the bus. The processor is used to execute a computer program stored in the memory to implement the steps of the task-aware hierarchical sparse screening method described in this embodiment of the invention.

[0149] In this embodiment, the processor may include a general-purpose processor and a neural network processor (NPU). The NPU may include, but is not limited to, the Ascend NPU, used to execute the computational load related to the vector computation path and matrix computation path in this embodiment. Specifically, control and indexing operations such as candidate region construction, prior weight calculation, learnable weight calculation, fusion sorting, gating selection of sparse budget K and alignment correction to obtain K*, Top-m filtering, and index / mask generation can be executed by the vector computation path; a fixed-length sparse token sequence as input for subsequent attention-related matrix operations or control strategy networks can be executed by the matrix computation path, thereby reducing dynamic shape scheduling and non-continuous memory access overhead and improving edge inference latency stability.

[0150] In one optional implementation, the electronic device may further include a communication interface and an input / output interface. The communication interface is used to interact with a camera, a robotic arm controller, or a host computer, while the input / output interface is used to receive joint state parameters, camera calibration parameters, and visual feature inputs, and to output data such as sparse token sequences, index mappings, mask sets, and control strategy network outputs.

[0151] In one implementation, the present invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the task-aware hierarchical sparse screening method described in the present invention. The steps include at least: acquiring task prior information and visual feature input; generating candidate regions and constructing a candidate tile set based on task prior projection; calculating prior weights and learnable weights and fusing and sorting them; gating the sparse budget K and aligning and correcting it to obtain K*; filtering the Top K* and generating an index mapping and / or mask set; organizing by tile major and performing fixed-length alignment to output a fixed-length sparse token sequence.

[0152] Furthermore, in one implementation, the present invention also provides a computer program product that, when running on an electronic device, causes the electronic device to execute the steps of the task-aware hierarchical sparse filtering method described in the present invention.

[0153] In view of the description and exemplary embodiments of the invention disclosed herein, other embodiments of the invention will be apparent to those skilled in the art. These descriptions and embodiments are considered as examples only, and any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A task-aware hierarchical sparse filtering method for NPU, characterized in that, The method is applied in edge neural network processor (NPU) inference scenarios to perform sparsification processing on visual feature inputs to generate fixed-length input sequences that satisfy alignment constraints. The method includes: First, acquire the task prior information and visual feature input at the current moment; wherein, the task prior information includes at least the robot arm joint state parameters and camera calibration parameters, and the visual feature input includes at least the dual-channel visual feature token sequence output by the visual coding network; Second, perform forward kinematics based on the joint state parameters to obtain the end effector pose, and project the end effector pose based on the camera calibration parameters to obtain pixel coordinates, and then map the pixel coordinates to visual feature grid coordinates to determine the candidate region center point; Third, generate a candidate region window based on the candidate region center point, and divide the candidate region window into candidate tile sets according to tile blocks, and establish the candidate tile blocks and their internal features. The mapping relationship between lexical tokens; 4. For each candidate tile in the candidate tile set, calculate the prior weight based on the geometric relationship between the candidate tile and the center point of the candidate region; and perform pooling on the feature lexical tokens within the candidate tile to obtain a tile representation, and input the tile representation into a lightweight scoring network to obtain learnable weights; 5. Fuse the prior weights and the learnable weights to obtain the importance score of each candidate tile, and sort the candidate tile set according to the importance score; 6. Based on the distribution characteristics of the importance score and the edge budget parameters, select a sparse budget level K from a preset discrete level set, and modify K to K* according to alignment constraints, where M is a preset alignment factor parameter used to characterize the block execution granularity of the matrix operation execution unit of the target edge NPU; K is the effective token budget number participating in subsequent attention-related calculations; K* is the length of the fixed-length output sequence after alignment constraint correction, and satisfies K*≥K, so that K* satisfies the constraints of the matrix operation execution unit of the target edge NPU on the input shape and alignment, thereby enabling subsequent attention-related matrix operations to be executed in a fixed shape and reducing tail block processing and dynamic shape scheduling overhead; 7. Determine the number of selected tiles m based on the sparse budget level K and the preset tile size, wherein m satisfies The T tile The preset number of feature tokens contained in a single tile is defined. The top-m candidate tiles are selected from the sorting results, and these top-m candidate tiles are expanded into a sparse feature token sequence in tile-major order, while simultaneously generating an index mapping and / or a mask set. The mask set is used to indicate invalid pruning positions, budget truncation positions, and padding. Fill in the missing parts; 8. Organize the sparse feature token sequence into a continuous block sequence according to the tile-major order, and perform fixed-length alignment processing on the sparse feature token sequence to obtain a fixed-length sparse feature token output sequence of length K*. Input the fixed-length sparse feature token output sequence into the subsequent attention calculation or control strategy network; wherein, the importance score calculation, sorting, gating selection and index / mask generation are performed by the vector calculation path of the edge NPU, and the subsequent attention-related matrix operation is performed by the matrix operation execution unit of the edge NPU to reduce the overhead of non-continuous memory access and dynamic shape scheduling.

2. The method according to claim 1, characterized in that, The third step of generating candidate region windows includes: generating a square window or a circular window based on a preset window radius parameter; and mapping the candidate region window to a candidate range on a visual feature grid, so that the candidate tile set is limited to the region related to the pose of the end effector.

3. The method according to claim 1, characterized in that, The third step of dividing by tile includes: dividing the candidate range into a preset number of tiles according to a preset tile size, wherein each tile consists of a preset number of adjacent feature tokens, and the number of feature tokens in the same tile is not less than 2 and not more than 64, so that the feature tokens in the same tile are spatially continuous and form a continuous address block in storage, thereby reducing the memory access overhead caused by irregular index access.

4. The method according to claim 1, characterized in that, In step 4, the prior weight is the output value of the distance decay function or the Gaussian function, and the smaller the distance between the candidate tile and the center point of the candidate region, the larger the prior weight.

5. The method according to claim 1, characterized in that, The pooling in step 4 includes average pooling, max pooling, or weighted pooling; the lightweight scoring network consists of at least one linear layer and a nonlinear activation function, used to output learnable weights for candidate tiles.

6. The method according to claim 1, characterized in that, In step 5, the importance score s is obtained by weighted summation. i ,in Where i is the index of each tile in the candidate tile set, and the value of i ranges from 1 to the total number of tiles in the candidate tile set; w prior,i The prior weight of the candidate tile corresponding to index i, w learn,i The learnable weights of the candidate tile corresponding to index i are α∈[0,1], which are preset fusion coefficients or learnable fusion coefficients. α can be adaptively adjusted through training, and its value range is kept within [0,1] during training.

7. The method according to claim 1, characterized in that, In step 6, the preset discrete gear set includes at least a number of sparse budget candidate values; the alignment constraint is characterized by a preset alignment multiple parameter M, where M is the block execution granularity of the target edge NPU matrix operation execution unit, and K* is an integer multiple of M to adapt to the block execution granularity of the edge NPU matrix operation and reduce the tail block processing overhead.

8. The method according to claim 1, characterized in that, The fixed-length alignment process in step 8 includes: when the number of sparse feature tokens obtained from the screening is insufficient to satisfy K*, padding is performed using at least one of the following methods: fixed-position feature token padding, zero-value feature token padding, or boundary tile repeated padding; and in subsequent attention-related matrix operations, the padding positions are masked according to the mask set to avoid the padding feature tokens from participating in matrix accumulation operations and to reduce invalid computational overhead.

9. A task-aware hierarchical sparse filtering system for NPU, characterized in that, The system is applied in the inference scenario of the edge neural network processor (NPU) to perform sparsification processing on visual feature input to generate a fixed-length input sequence that satisfies alignment constraints. The system includes: an input acquisition module, a candidate region generation module, a prior weight calculation module, a learnable weight calculation module, a score fusion and sorting module, a budget gating and alignment module, a Top-m filtering and expansion module, a fixed-length alignment output module, and a parameter storage device. The input acquisition module's input end connects to an external task prior information data source and an external visual encoding network, while its output end connects to a candidate region generation module and a learnable weight calculation module, respectively. The output of the candidate region generation module is connected to the prior weight calculation module and the learnable weight calculation module, respectively. The outputs of the prior weight calculation module and the learnable weight calculation module are both connected to the score fusion and sorting module. The output of the score fusion sorting module is connected to the budget gating and alignment module; The output of the budget gating and alignment module is connected to the Top-m filtering and expansion module; The output end of the Top-m filtering expansion module is connected to the fixed-length alignment output module; The output of the fixed-length alignment output module is connected to an external subsequent attention calculation network or control strategy network. The parameter storage device is bidirectionally connected to the above-mentioned functional modules, providing parameter read and write support. in, The input acquisition module is used to acquire the task prior information and visual feature input at the current moment, and output the task prior information to the candidate region generation module and output the visual feature input to the learnable weight calculation module; the task prior information includes at least the robot arm joint state parameters and camera calibration parameters, and the visual feature input includes at least the dual-path visual feature token sequence output by the visual coding network; The candidate region generation module receives joint state parameters and camera calibration parameters output by the input acquisition module, performs forward kinematics based on the joint state parameters to obtain the end effector pose, projects the end effector pose based on the camera calibration parameters to obtain pixel coordinates, and then maps the pixel coordinates to visual feature grid coordinates to determine the center point of the candidate region; generates a candidate region window based on the center point of the candidate region, and divides the candidate region window into candidate tile sets according to tiles, and establishes a mapping relationship between the candidate tile sets and their internal feature tokens; outputs the candidate tile set to the prior weight calculation module and the learnable weight calculation module respectively. The prior weight calculation module receives the candidate tile set and candidate region center point output by the candidate region generation module, calculates the prior weight based on the geometric relationship between the candidate tile and the candidate region center point, and outputs the prior weight to the score fusion and sorting module. The learnable weight calculation module is used to receive the candidate tile set output by the candidate region generation module and the visual feature input output by the input acquisition module, perform pooling on the feature tokens in the candidate tile to obtain the tile representation, input the tile representation into the lightweight scoring network to obtain learnable weights, and output the learnable weights to the score fusion and ranking module. The score fusion and sorting module receives the prior weights output by the prior weight calculation module and the learnable weights output by the learnable weight calculation module, fuses the prior weights and the learnable weights to obtain the importance score of each candidate tile, sorts the candidate tile set according to the importance score, and outputs the sorted candidate tile set and importance score to the budget gating and alignment module. The budget gating and alignment module receives the importance score output by the score fusion and sorting module. Based on the distribution characteristics of the importance score and the end-side budget parameters, it selects a sparse budget level K from a preset discrete level set and modifies K to K* according to alignment constraints. This ensures that K* satisfies the constraints on input shape and alignment imposed by the edge NPU matrix computation path. The alignment constraint is characterized by a preset alignment multiple parameter M, where M is the block execution granularity of the target edge NPU matrix operation execution unit. K* is made an integer multiple of M, thereby enabling subsequent attention-related matrix operations to be executed with a fixed shape and reducing tail block processing and dynamic shape scheduling overhead. The Top-m filtering expansion module receives the K value output by the budget gating and alignment module, and the sorted candidate tile set output by the score fusion and sorting module. It determines the filtering quantity m based on K and the preset tile size, where m is [K / T]. tile The T tile The number of feature tokens contained in a single tile is preset, and the top-m candidate tiles are selected from the sorting results; the candidate tiles are expanded into a sparse feature token sequence, and an index mapping and / or mask set is generated; the mask set is used to indicate invalid pruning positions, budget truncation positions, and padding positions; The fixed-length aligned output module receives the sparse feature token sequence, index mapping, and mask set output by the Top-m filtering and unpacking module, as well as the K value output by the budget gating and alignment module. It organizes the sparse feature token sequence into a continuous block sequence according to tile major order, performs fixed-length alignment processing on the sparse feature token sequence to obtain a fixed-length sparse feature token output sequence of length K*, and outputs the fixed-length sparse feature token output sequence to the subsequent attention calculation or control strategy network. The importance score calculation, sorting, gating selection, and index / mask generation are performed by the edge-side NPU vector calculation path, while the subsequent attention-related matrix operations are performed by the edge-side NPU matrix calculation path to reduce the overhead of non-continuous memory access and dynamic shape scheduling. The parameter storage device includes a strategy parameter management module, a model and weight parameter module, and a statistics and configuration module. The strategy parameter management module stores strategy parameters such as the fusion coefficient α, alignment factor parameter M, sparse budget discrete tier set, and coverage threshold. The model and weight parameter module stores the weight parameters of the lightweight scoring network. The statistics and configuration module stores configuration parameters such as edge budget parameters, tile size, and window radius parameters. The parameter storage device provides parameter reading services to various functional modules within the system and receives parameter update data from each module. When the system adapts to new NPU hardware or task scenarios, each functional module can initiate a parameter update request to the parameter storage device.

Images