A deep neural network sparse compilation optimization system and method for mobile enforcement edge devices

Abstract

The application relates to a deep neural network sparse compilation optimization system and method for a mobile law enforcement edge device, relates to the fields of edge computing and artificial intelligence acceleration technology, and comprises a sparse mode analysis module, a hardware mapping module and a dynamic scheduling engine; the sparse mode analysis module is configured to receive a pre-trained deep neural network model, to perform sparse mode identification on weight tensors of each layer in the model, to generate a structured sparse mask and an unstructured sparse mask, and to obtain a structured sparse weight part and an unstructured sparse weight part according to the structured sparse mask and the unstructured sparse mask. The application can improve the deep neural network sparse compilation calculation efficiency of the mobile law enforcement edge device while reducing energy consumption.

Description

Technical Field

[0001] This invention relates to the fields of edge computing and artificial intelligence acceleration technology, and in particular to a deep neural network sparse compilation optimization system and method for mobile law enforcement edge devices. Background Technology

[0002] Mobile law enforcement scenarios typically require tasks such as target detection, facial verification, license plate recognition, and abnormal behavior detection to be performed on edge devices such as patrol cars, body cameras, handheld verification terminals, and drones. These devices are usually limited by computing power, power consumption, heat dissipation, storage capacity, and network connectivity, making it difficult to run deep neural network models with a large number of parameters for extended periods.

[0003] To reduce inference overhead, existing technologies typically employ model compression, pruning, quantization, lightweight network design, and dedicated accelerator deployment. Structured sparsity can form regular non-zero patterns, facilitating hardware parallel computation; unstructured sparsity can achieve higher compression ratios but requires additional indexing for memory access. Compiler technology can map computation graphs to operators or kernels on target hardware, but existing solutions are mostly geared towards server GPUs or general-purpose computing platforms.

[0004] In mobile law enforcement edge devices, existing solutions still have the following problems: First, there is a lack of matching between the sparse pattern and the operator capabilities, storage alignment rules, and index addressing methods of the target edge accelerator, resulting in the actual acceleration effect of the sparse model on the hardware being lower than expected. Second, the model pruning parameters are usually determined during the model training stage or the offline compression stage, without fully utilizing the latency, power consumption, and memory access characteristics of the target hardware for collaborative optimization. Third, the illumination, occlusion, target density, and device status vary greatly in the mobile law enforcement environment, making it difficult for static sparse configuration to simultaneously take into account accuracy, latency, and power consumption. Fourth, when dealing with sensitive data such as faces and license plates, the existing edge security computing schemes are not sufficiently integrated with sparse inference, making it difficult to balance security and computational efficiency.

[0005] Therefore, a deep neural network sparse compilation optimization scheme that combines sparse pattern recognition, hardware mapping, pre-compilation alignment, runtime scheduling, and secure execution is needed to enable mobile law enforcement edge devices to achieve stable, real-time, and secure artificial intelligence inference under limited power consumption and computing power conditions. Summary of the Invention

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a deep neural network sparse compilation optimization system and method for mobile law enforcement edge devices. Through dynamic sparse pattern recognition and hardware instruction co-mapping, the inherent contradiction between energy efficiency and computing efficiency of traditional edge devices is solved.

[0007] To achieve the above objectives, the main technical solutions adopted by the present invention include: A deep neural network sparse compilation and optimization system for mobile law enforcement edge devices includes: a sparse pattern analysis module, a hardware mapping module, and a dynamic scheduling engine. The sparse pattern analysis module is configured to receive a pre-trained deep neural network model, perform sparse pattern recognition on the weight tensors of each layer in the model, generate structured sparse masks and unstructured sparse masks, and obtain structured sparse weight parts and unstructured sparse weight parts based on the structured sparse masks and unstructured sparse masks. The hardware mapping module is connected to the sparse pattern analysis module and is configured to convert the structured sparse weight parts and unstructured sparse weight parts into sparse operator execution description information that can be called by the target edge accelerator, according to the operator capabilities, storage alignment rules, and index addressing rules of the target edge accelerator. The dynamic scheduling engine is configured to acquire the current input data features and device status at runtime, and select or switch among multiple pre-compiled sparse computing kernels according to the current input data features and device status to execute the inference task of the current frame or the current batch.

[0008] Furthermore, the configuration of the sparse pattern analysis module includes normalizing the first-layer weight tensor into a two-dimensional weight matrix, dividing the two-dimensional weight matrix into several non-overlapping blocks, performing flattening processing on each block and adding two-dimensional position encoding to form a block label sequence; encoding the block label sequence through a single-head self-attention encoder to obtain the global sparse feature vector of the layer weights; mapping the global sparse feature vector to a classification probability distribution of several predefined sparse pattern categories, including structured sparsity, block sparsity, diagonal sparsity, and unstructured sparsity; determining the target sparse pattern based on the classification probability distribution, and generating a binary mask matrix based on the target sparse pattern.

[0009] Furthermore, the target sparse pattern is determined based on the classification probability distribution, and a binary mask matrix is ​​generated based on the target sparse pattern as follows: When the target category is the first category, the 2:4 structured sparsity rule is adopted. Each row of the two-dimensional weight matrix is ​​divided into groups of four consecutive elements. For the four elements in each group, the mask values ​​of the two elements with the largest absolute values ​​are set to 1, and the mask values ​​of the other two elements are set to 0. When the target category is the second category, the block sparsity rule is adopted to divide the two-dimensional weight matrix into multiple square blocks. The side length of each square block is a preset second positive integer. The square root of the sum of the squares of all elements in each block is calculated to obtain the Frobenius norm of that block, and the median of the Frobenius norms of all blocks is calculated. For any block, if its Frobenius norm is greater than or equal to the product of the preset scaling factor and the median, the mask value of all positions in that block is set to 1, otherwise it is set to 0. When the target category is the third category, the diagonal sparsity rule is adopted. Based on the number of rows and columns of the current layer's two-dimensional weight matrix, the smaller value is taken and multiplied by a preset bandwidth ratio coefficient, and then rounded up to obtain the diagonal bandwidth. For any position in the matrix, if the absolute value of the difference between its row index and column index is less than or equal to the diagonal bandwidth, the mask value of that position is set to 1, otherwise it is set to 0. When the target category is the fourth category, an unstructured amplitude sparsity rule is adopted. The absolute value of each element in the two-dimensional weight matrix is ​​compared with a preset pruning threshold. If the absolute value is greater than or equal to the pruning threshold, the mask value at that position is set to 1; otherwise, it is set to 0. The pruning threshold is determined by the pre-compiled alignment module according to the hardware characteristics.

[0010] Furthermore, the configuration of the hardware mapping module includes, for the structured sparse weight portion, generating a sparse operator execution descriptor for the target edge accelerator. This sparse operator execution descriptor includes at least an operator identifier, weight storage address, index or mask storage address, alignment step size, and a calculation skip threshold. The operator identifier is determined using a lookup table based on the sparse pattern category index. The alignment step size is determined based on the non-zero rate, group size, index byte count, and the storage alignment granularity of the target edge accelerator. The calculation skip threshold is updated based on the hardware performance proxy model and the actual operating status. For the unstructured sparse weight portion, non-zero elements are stored using a compressed sparse row (CSR) format, and index-addressed memory access description information is generated.

[0011] Furthermore, it also includes a pre-compiled alignment module, which is configured to iteratively optimize the pruning parameter vector based on a hardware performance proxy model and finite difference estimation, so that the prediction inference latency of the sparse model on the target edge accelerator meets the target latency constraint; wherein, the hardware performance proxy model takes sparse configuration features as input and outputs prediction inference latency, and the sparse configuration features include at least the sparsity distribution of each layer, the memory access mode entropy value, and the effective floating-point operation statistics.

[0012] Furthermore, the objective function of the pre-compiled alignment module includes a prediction inference delay term and a delay exceeding penalty term, the delay exceeding penalty term being activated when the prediction inference delay is greater than the target delay; each component θ of the pruning parameter vector θ l Used to determine the pruning threshold ε of layer l prune,l , and ε prune,l equal to θ l The product of the absolute value of the weights in layer l and the maximum value of the weights in layer l.

[0013] Furthermore, the dynamic scheduling engine is modeled based on a Markov decision process, which includes a state space, an action space, and a reward function. The state space includes the one-hot encoding of the current sparse configuration, a device state vector, and an input feature vector. The device state vector includes the remaining power ratio, chip temperature, and available memory ratio, while the input feature vector includes the target density, image blur score, and motion amplitude. The action space includes sparse configuration adjustment actions and computational precision selection actions, with each action combination corresponding to a pre-compiled sparse computation kernel. The reward function is obtained based on the weighted result of the online precision estimation of the current frame, the actual inference latency, and the measured power consumption relative to their respective benchmark values.

[0014] Furthermore, it also includes a secure execution module, configured to perform homomorphic encryption inference based on the near-homomorphic encryption CKKS on the non-zero elements of the sparse weights in sensitive tasks; the secure execution module is further configured to: encode the non-zero elements of each layer of weights into a CKKS plaintext polynomial in batches, and encrypt them using a public key to generate sparse weight ciphertext; during inference, perform homomorphic multiplication-addition operations on the sparse weight ciphertext and the corresponding input feature map ciphertext, wherein the number of homomorphic multiplications is proportional to the number of non-zero elements; generate a hash commitment for the sparse index set, and verify whether the currently used sparse index set is consistent with the hash commitment after inference; if they are inconsistent, terminate the inference and output an alarm message.

[0015] The deep neural network sparse compilation optimization method for mobile law enforcement edge devices using the aforementioned system includes the following steps: Step 1, Model Input: Receive the pre-trained deep neural network model, read the weight tensors of each pruning layer, and normalize them into a two-dimensional weight matrix according to the layer type; Step 2, Sparse Pattern Recognition: Divide the two-dimensional weight matrix into multiple blocks, generate a block label sequence, obtain the global sparse feature vector through a lightweight self-attention encoder, and output the sparse pattern classification probability distribution π. Step 3, Mask Generation and Weight Decomposition: Based on the target sparse pattern category, generate structured sparse masks and unstructured sparse masks according to deterministic rules, and obtain the structured sparse weight part and the unstructured sparse weight part respectively. Step 4, Pre-compilation alignment: Construct the pruning parameter vector θ, use the hardware performance proxy model to predict the inference latency, and obtain θ* that meets the target latency through finite difference estimation and iterative optimization; Step 5, Hardware Mapping: Map the structured sparse weights to regular sparse operator execution descriptors, and map the unstructured sparse weights to CSR storage data and index addressing memory access description information, generating multiple pre-compiled kernels with different sparsity and precision. Step 6, Runtime Scheduling: Collect current input features and device status to form a state vector; select actions according to the scheduling strategy; switch to the corresponding pre-compiled kernel to complete the current frame or current batch inference; write the state transition data to the experience replay buffer and update the scheduling strategy during idle periods; Step 7, Secure Execution: In sensitive tasks, perform CKKS batch encoding and encrypted computation on the non-zero elements of sparse weights and their corresponding input features, and verify the sparse index set hash commitment after inference.

[0016] Furthermore, the scheduling strategy employs a dual-depth Q network, with both the online network and the target network using a three-layer fully connected structure.

[0017] The beneficial effects of this invention are as follows: The deep neural network sparse compilation optimization system for mobile law enforcement edge devices proposed in this invention can improve the computational efficiency of deep neural network sparse compilation in mobile law enforcement edge devices while reducing energy consumption, thereby improving the efficiency of intelligent inference on mobile law enforcement edge devices. This invention constructs a complete compilation chain from sparse pattern recognition to hardware instruction generation, solving the problem of insufficient support for hybrid sparseness in traditional edge accelerators. At the application level, this invention develops a collaborative mechanism for dynamic scheduling and secure execution, meeting the comprehensive requirements of law enforcement scenarios for real-time performance, reliability, and privacy protection. At the system level, this invention achieves deep collaboration between the compiler and edge hardware, feeding hardware characteristics back to the model optimization stage through a differentiable proxy model, forming a closed-loop design paradigm. Attached Figure Description

[0018] Figure 1 This is a diagram of the deep neural network sparse compilation optimization system architecture for mobile law enforcement edge devices according to the present invention; Figure 2 This is a schematic diagram of the method flow of the present invention; Figure 3 This is a flowchart illustrating the sparse pattern analysis module. Figure 4 This is a schematic diagram of the hardware instruction mapping and pre-compilation alignment process; Figure 5 This is a schematic diagram of the dynamic scheduling and safe execution process. Detailed Implementation

[0019] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The present invention will be described below with reference to specific embodiments. These embodiments are used to explain the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. To avoid confusion of symbols, the following symbols are used uniformly in this document: l represents the network layer index; L represents the total number of pruning layers; b represents the block side length; π represents the sparse pattern classification probability distribution; c represents the sparse pattern category index; M s M represents a structured sparse mask. u Represents an unstructured sparse mask; ρ l r represents the sparsity of the l-th layer. l =1-ρ l ε represents the non-zero rate of the l-th layer; prune,l T represents the pruning threshold of the l-th layer; g This represents the Gumbel-Softmax temperature parameter; T skip,l This indicates that the skip threshold is calculated for the l-th layer; ε explore ε represents the exploration probability in dynamic scheduling; adam N represents the numerically stable term in the Adam optimizer. ckks This represents the ring dimension of the CKKS polynomial.

[0021] I. System Overall Structure like Figure 1-2As shown, this invention provides a deep neural network sparse compilation and optimization system for mobile law enforcement edge devices. The system includes a sparse pattern analysis module, a hardware mapping module, a pre-compilation alignment module, a dynamic scheduling engine, and a secure execution module. The sparse pattern analysis module is configured to receive a pre-trained deep neural network model, perform sparse pattern recognition on the weight tensors of each layer in the model, generate structured sparse masks and unstructured sparse masks, and obtain structured sparse weight parts and unstructured sparse weight parts based on the structured and unstructured sparse masks. That is, the sparse pattern analysis module reads the weight tensors of the pre-trained model during the compilation phase and outputs the sparse patterns of each layer, the structured sparse masks, and the unstructured sparse masks. The hardware mapping module is connected to the sparse pattern analysis module and is configured to convert the structured sparse weight parts and unstructured sparse weight parts into sparse operator execution description information that can be invoked by the target edge accelerator, based on the operator capabilities, storage alignment rules, and index addressing rules of the target edge accelerator. The hardware mapping module generates sparse operator execution descriptors, compressed storage data, and pre-compiled computation kernels based on the sparse mask and the execution constraints of the target edge accelerator. The pre-compilation alignment module optimizes the pruning thresholds at each layer before deployment, ensuring that the model's inference latency on the target hardware meets the target constraints. The dynamic scheduling engine is configured to acquire the current input data features and device state at runtime, and select or switch among multiple pre-compiled sparse computation kernels based on these features to execute the inference task for the current frame or batch. The dynamic scheduling engine selects the pre-compiled kernel at runtime based on the input frame complexity and device state. The system also includes a pre-compilation alignment module, configured to iteratively optimize the pruning parameter vector based on a hardware performance proxy model and finite difference estimation, ensuring that the prediction and inference latency of the sparse model on the target edge accelerator meets the target latency constraints. The hardware performance proxy model takes sparse configuration features as input and outputs the prediction and inference latency; these sparse configuration features include at least the sparsity distribution of each layer, memory access mode entropy values, and effective floating-point operation statistics. It also includes a secure execution module, configured to perform CKKS-based homomorphic encryption inference on the non-zero elements of the sparse weights in sensitive tasks. The secure execution module is further configured to: batch-encode the non-zero elements of each layer of weights into a CKKS plaintext polynomial and encrypt it using a public key to generate sparse weight ciphertext; during inference, perform homomorphic multiplication-addition operations on the sparse weight ciphertext and the corresponding input feature map ciphertext, where the number of homomorphic multiplications is proportional to the number of non-zero elements; generate a hash commitment for the sparse index set, and verify whether the currently used sparse index set is consistent with the hash commitment after inference; if inconsistent, terminate the inference and output an alarm message. When sensitive data needs protection, the secure execution module performs homomorphic encryption calculations on the non-zero elements of the sparse weights and the input features, and verifies the integrity of the sparse index set.

[0022] In one embodiment, the input model is the object detection model YOLOv5s; in other embodiments, the input model can also be MobileNetV3, ResNet-18, or other deep neural network models suitable for deployment on mobile law enforcement edge devices. The deep neural network model may include, but is not limited to, object detection networks, image classification networks, face detection networks, face feature extraction networks, license plate recognition networks, abnormal behavior recognition networks, and multimodal perception networks. For deep neural network models containing convolutional layers, the weight tensors of the convolutional layers can be normalized into a two-dimensional weight matrix; for deep neural network models containing fully connected layers or other layers with pruned parameters, their weight tensors can be normalized into a two-dimensional weight matrix suitable for sparse pattern analysis based on their dimension.

[0023] The pre-trained model can be obtained using existing supervised learning methods, transfer learning methods, distillation training methods, or self-supervised pre-training followed by fine-tuning. For example, a deep neural network model can be trained on publicly available object detection datasets, publicly available face recognition datasets, publicly available vehicle or license plate datasets, or mobile law enforcement business datasets to acquire capabilities for object detection, image classification, face verification, license plate recognition, or abnormal behavior recognition. After training, a deep neural network model with a defined network structure and trained weight parameters is obtained. This invention does not limit the specific training algorithm, loss function, or training data source of the pre-trained model; the focus is on sparse pattern recognition, hardware mapping, pre-compilation alignment, and runtime scheduling optimization of the weights of the trained deep neural network model.

[0024] II. Sparse Pattern Analysis Module like Figure 3 As shown, the configuration of the sparse pattern analysis module includes normalizing the first-layer weight tensor into a two-dimensional weight matrix, dividing the two-dimensional weight matrix into several non-overlapping blocks, performing flattening processing on each block and adding two-dimensional position encoding to form a block label sequence; encoding the block label sequence through a single-head self-attention encoder to obtain the global sparse feature vector of the layer weights; mapping the global sparse feature vector to a classification probability distribution of several predefined sparse pattern categories, including structured sparsity, block sparsity, diagonal sparsity and unstructured sparsity; determining the target sparse pattern according to the classification probability distribution, and generating a binary mask matrix according to the target sparse pattern.

[0025] Specifically, the target sparse pattern is determined based on the classification probability distribution, and a binary mask matrix is ​​generated based on the target sparse pattern as follows: When the target category is the first category, the 2:4 structured sparsity rule is adopted. Each row of the two-dimensional weight matrix is ​​divided into groups of four consecutive elements. For the four elements in each group, the mask values ​​of the two elements with the largest absolute values ​​are set to 1, and the mask values ​​of the other two elements are set to 0. When the target category is the second category, the block sparsity rule is adopted to divide the two-dimensional weight matrix into multiple square blocks. The side length of each square block is a preset second positive integer. The square root of the sum of the squares of all elements in each block is calculated to obtain the Frobenius norm of that block, and the median of the Frobenius norms of all blocks is calculated. For any block, if its Frobenius norm is greater than or equal to the product of the preset scaling factor and the median, the mask value of all positions in that block is set to 1, otherwise it is set to 0. When the target category is the third category, the diagonal sparsity rule is adopted. Based on the number of rows and columns of the current layer's two-dimensional weight matrix, the smaller value is taken and multiplied by a preset bandwidth ratio coefficient, and then rounded up to obtain the diagonal bandwidth. For any position in the matrix, if the absolute value of the difference between its row index and column index is less than or equal to the diagonal bandwidth, the mask value of that position is set to 1, otherwise it is set to 0. When the target category is the fourth category, an unstructured amplitude sparsity rule is adopted. The absolute value of each element in the two-dimensional weight matrix is ​​compared with a preset pruning threshold. If the absolute value is greater than or equal to the pruning threshold, the mask value at that position is set to 1; otherwise, it is set to 0. The pruning threshold is determined by the pre-compiled alignment module according to the hardware characteristics.

[0026] More specifically, given the weight tensor W of the l-th layer l If the layer is a convolutional layer, then W l The dimension is C out,l ×C in,l ×k h,l ×k w,l C out,l C is the number of output channels. in,l k is the number of input channels. h,l and k w,l These represent the height and width of the convolution kernel, respectively. They are then normalized into a two-dimensional weight matrix. : ; If the layer is a fully connected layer, then directly let ∈R m_l×n_l Reshape represents merging the input channel dimension and spatial dimension of the convolution kernel into the column dimension of a two-dimensional matrix while keeping the weight values ​​unchanged.

[0027] Will The system is divided into multiple non-overlapping b×b blocks, where b is the side length of the block; in one embodiment, b=4. Boundary regions smaller than b×b are padded with zeros. The number of blocks in the l-th layer is P. l for: P l =ceil(m l / b)·ceil(n l / b); Where ceil(·) represents rounding up. For the i-th block B i l ∈R b×b Perform flattening to obtain the label vector f i l : f i l =Flatten(B i l )∈R b^2 ; Flatten means expanding the block matrix into a one-dimensional vector according to a preset order; in one embodiment, a row-major order is used. To preserve the block positions, the system adds a learnable two-dimensional positional code E to the i-th block. pos,i l , obtain the input marker x i l : x i l =f i l +E pos,i l E pos,i l ∈R b^2 ; Where E pos,i l It can be initialized as a zero vector and updated during the sparse pattern recognition training phase. The label matrix X consists of all the labels. l : X l =[x1 l x2 l ,...,x P _l l ]^T∈R P_l×b^2 ; For the label matrix X l Perform single-head self-attention encoding to obtain the attention output Y. l : Q=X l W Q, K=X l W K, V=Xl W V ; A=Softmax(QK T / sqrt(d h )),Y l =AV; Among them W Q W K W V The learnable projection matrices of the query matrix, key matrix, and value matrix, respectively, d h Let Q be the attention hidden layer dimension, Q, K, and V be the query matrix, key matrix, and value matrix, respectively, A be the attention weight matrix, and Softmax represent the normalization exponential function applied to each row.

[0028] Attention output Y l The output vectors of each block are averaged and pooled to obtain the global sparse feature vector h. l : h l =(1 / P l )·Σ i=1 P_l Y i l ; K is obtained through a linear classification layer. cls The unnormalized score of each sparse pattern category. l Then obtain the classification probability distribution π l : o l =LayerNorm(h l W c +b c ; π c l =exp(o c l ) / Σ r=1 K_cls exp(o r l ); Among them W c and b c These are the classification layer weight matrix and bias vector, respectively, K cls K represents the number of sparse pattern categories in one embodiment. cls =4, the four sparsity patterns are 2:4 structured sparsity, block sparsity, banded diagonal sparsity, and unstructured sparsity. Target category c l *Determine using the following formula: c l *=argmax c π cl .

[0029] According to target category c l * Generate a structured sparse mask M s l ∈{0,1}^(m l ×n l A mask element of 1 indicates that the corresponding weight is retained in the structured computation path, while a mask element of 0 indicates that it is not included in the structured computation path. The specific rules are as follows: When c l When *=1, a 2:4 structured sparsity is used. Each row is grouped into four consecutive elements, and the group containing the four elements at position (i,j) is denoted as G(i,j). If | If [i,j]| is one of the two largest absolute values ​​among the four values ​​of G(i,j), then M s l [i,j]=1, otherwise 0.

[0030] When c l When *=2, block sparsity is used. Divide into p blk ×p blk The block, in one embodiment p blk =8. Calculate the Frobenius norm for each block. If the block norm containing position (i,j) is not less than the product of β and the median of all block norms, then M... s l [i,j]=1, otherwise 0. In one embodiment, β=1.0.

[0031] M s l [i,j]=1,if||B block (i,j)||_F≥β·median({||B q ||_F});otherwiseM s l [i,j]=0.

[0032] When c l When *=3, a banded diagonal sparsity is adopted. Let the diagonal bandwidth be B. diag,l =ceil(0.1·min(m l,n _l)). If |i−j|≤B diag,l Then M s l [i,j]=1, otherwise 0.

[0033] M s l [i,j]=1,if|i-j|≤Bdiag,l ;M s l [i,j]=0,otherwise.

[0034] When c l When *=4, unstructured amplitude sparsity is adopted. If | [i,j]|≥ε prune,l Then M s l [i,j]=1, otherwise 0. Where ε prune,l Determined by the pre-compiled alignment module.

[0035] M s l [i,j]=1,if| [i,j]|≥ε prune,l ;M s l [i,j]=0,otherwise.

[0036] Unstructured sparse mask M u l Defined as M s l The complement: M u l =1-M s l ; Where 1 is related to M s l A matrix of all ones with the same dimension. The weight matrix is ​​decomposed into a structured sparse weight component W based on the mask. s l Unstructured sparse weights W u l : W s l = ⊙M s l W u l = ⊙M u l ; Where ⊙ denotes element-wise multiplication. W s l and W u l All parameters are derived from the original weights and corresponding masks, and are not independently trained. During deployment, the structured sparse weights are executed by the rule-based sparse computation kernel, while the unstructured sparse weights are stored in a compressed sparse row format and accessed via index addressing.

[0037] When training the sparse pattern classification layer, the Gumbel-Softmax pass-through estimator can be used to achieve approximate differentiability of discrete sparse pattern selection. Let u k Following a uniform distribution on the interval (0,1), g k For Gumbel noise, T g Temperature parameter: g k =﹣ln(﹣ln(u k )),u k ~Uniform(0,1); z soft,k =exp((lnπ k +g k ) / T g ) / Σ r exp((lnπ r +g r ) / T g ); z hard =OneHot(argmax k z soft,k ); Training forward computation uses z hard Choose the sparse mode and use z for inverse computation. soft The gradient approximation is obtained by disabling Gumbel noise during the deployment phase and directly using c. l *=argmax c π c l Choose the sparse mode.

[0038] The classification layer parameters can be jointly trained using task loss, sparsity constraint loss, and hardware efficiency proxy loss: L total =L task +λ sparse ·L sparse +λ hw ·L hw ; L task =L box +L obj +L cls ; L sparse =|ρ actual -ρ target |; L hw =F GBDT (feat(θ)) / T target +λ delay ·max(0,F GBDT(feat(θ))-T target ); Where L box L obj and L cls Let ρ represent the bounding box regression loss, target confidence loss, and class classification loss, respectively; actual ρ represents the actual sparsity corresponding to the current mask. target F represents the target sparsity. GBDT For hardware performance proxy model; feat(θ) is the sparse configuration feature constructed from the pruning parameter vector θ; T target Delay for the target; λ sparse , λ hw and λ delayThe weighting coefficients can be set by the deployer based on accuracy and latency requirements. The hardware performance proxy model can be constructed and trained as follows. In one embodiment, the hardware performance proxy model FGBDT adopts a gradient boosting decision tree model, i.e., GBDT (Gradient Boosting Decision Tree). The GBDT model consists of multiple regression decision trees, each used to fit the residual between the predicted inference latency and the measured inference latency of the previous stage. The outputs of multiple regression decision trees are summed to obtain the predicted inference latency on the target edge accelerator. In a specific embodiment, the number of decision trees in the GBDT model is 50 to 300, the maximum depth of a single decision tree is 3 to 8, the learning rate is 0.01 to 0.1, and the loss function is the mean squared error loss function. The above parameters are exemplary parameters, and the deployer can adjust them according to the computing power, storage capacity, and number of training samples of the target mobile law enforcement edge device. The training data of the hardware performance proxy model can be obtained through actual testing on the target edge accelerator or simulation using a high-precision time-series simulator. Specifically, multiple pruning parameter vectors θ are sampled within a preset range, and a corresponding sparse mask, sparse weights, compressed storage data, and pre-compiled computation kernel are generated based on each pruning parameter vector θ. Subsequently, the pre-compiled computation kernel is run on the target mobile law enforcement edge device or its high-precision timing simulator to perform inference on a preset number of input samples. The average inference latency, peak inference latency, power consumption, chip temperature, and memory usage corresponding to each sparse configuration are recorded, and the average inference latency is used as the training label for the GBDT model. In one embodiment, the number of training samples is 200 to 2000 sets, and each set of training samples corresponds to a sparse configuration and its measured or simulated performance results on the target edge accelerator. The sparse configuration feature feat(θ) is constructed from the pruning parameter vector θ and the statistics of the corresponding sparse mask. The sparse configuration features include at least the sparsity rate, non-zero rate, proportion of non-zero blocks, number of CSR indexes, compressed weight storage size, effective floating-point operations, average storage access address jump value, storage access mode entropy value, alignment padding overhead, storage alignment granularity of the target edge accelerator, and computational precision type for each prunable layer. For deep neural network models with multiple prunable layers, the sparsity distribution of each layer can be statistically represented as a histogram feature, and the mean, maximum, minimum, and variance of the effective floating-point operations for each layer can be calculated as global sparse configuration features. Therefore, the hardware performance proxy model can output the prediction and inference latency of the corresponding sparse model on the target mobile law enforcement edge device based on the sparse configuration features.

[0039] III. Hardware Mapping Module The configuration of the hardware mapping module includes, for the structured sparse weight portion, generating a sparse operator execution descriptor for the target edge accelerator. This sparse operator execution descriptor includes at least an operator identifier, weight storage address, index or mask storage address, alignment step size, and a calculation skip threshold. The operator identifier is determined using a lookup table based on the sparse mode category index. The alignment step size is determined based on the non-zero rate, group size, index byte count, and the storage alignment granularity of the target edge accelerator. The calculation skip threshold is updated based on the hardware performance proxy model and actual operating status. For the unstructured sparse weight portion, non-zero elements are stored using a compressed sparse row (CSR) format, and index-addressed memory access description information is generated.

[0040] The hardware mapping module converts the sparse pattern analysis results into execution description information that can be called by the target edge accelerator. This execution description information belongs to the operator configuration description within the compiler of this invention and does not require modification of the target hardware's native instruction set.

[0041] For the structured sparse weight part W s l The hardware mapping module generates the sparse operator execution descriptor D. l : D l ={OP ID, WEIGHT BASE, MASK BASE, STRIDE K, T skip,l, SHAPE l}; OP ID For operator identifier; WEIGHT BASE The address for storing sparse weight values; MASK BASE Encode storage addresses for indices, masks, or locations; STRIDE K T is the alignment step size of adjacent sparse weight groups; skip,l Calculate the skip threshold for layer l; SHAPE l Define the input and output shapes for this layer. OP ID It can be determined by looking up a table, OP ID The lookup table is shown in Table 1.

[0042] Table 1: OP ID Lookup table: .

[0043] Taking a 2:4 sparse approach as an example, in dense storage, every four FP16 weights occupy 8 bytes. During compressed storage, two non-zero FP16 values ​​are retained, occupying 4 bytes, along with a position mask representing the two non-zero positions. If the position masks are stored with 1-byte alignment, each group occupies approximately 5 bytes, with a theoretical compression ratio of approximately 8 / 5 = 1.6. If multiple position masks are packed bit-by-bit, the compression ratio can be further improved. The actual compression ratio is also related to cache line alignment and block size.

[0044] Alignment Step STRIDE K Determined based on non-zero rate and storage alignment granularity: STRIDE K =ceil((K grp ·r l ·B val +B idx ) / A align )·A align ; Where K grp For sparse grouping size, K is used when the sparsity is 2:4. grp =4;r l B is the non-zero rate of the l-th layer; val For a single weight value in bytes, B in FP16 val =2;B idx For each set of indices or masks, specify the number of bytes used; A align The required storage alignment granularity for the target edge accelerator.

[0045] Calculate the skip threshold T skip,l Used to determine at runtime whether a weighted block or activation block meets the conditions for skipping calculations. Its initial value can be set to: T skip,l,0 =0.1·max(| |).

[0046] During operation, current latency, power consumption, temperature, and the proportion of non-zero blocks are collected every preset number of frames, and the adjustment direction is estimated using a hardware performance proxy model. Since tree models such as GBDT are typically not analytically differentiable, this invention employs finite difference estimation to determine the update direction. ; Where η skip To update the step size; ΔF / ΔT skip,l This indicates that the output of the performance surrogate model obtained through finite difference estimation is related to T. skip,l The rate of change; Clip indicates that the update result will be restricted to [T] min,l,T Within the range of _max,l].

[0047] For the unstructured sparse weight part W ul It is stored using the Compressed Sparse Row (CSR) format. The CSR includes rows. ptr ,col idx There are three arrays: row, values, and row. ptr It is an array of row pointers with a length of m. l +1, row ptr [i] represents the cumulative number of non-zero elements in rows 0 to (i-1); col idx It is a column index array with length nnz l The column coordinates of each non-zero element are stored in row order; values ​​is an array of non-zero values ​​of length nnz. l Non-zero weight values ​​are stored in row order.

[0048] Storage CSR =(m l +1)·B row +nnz l ·B col +nnz l ·B val ; Among them nnz l B represents the number of non-zero elements in the l-th layer. row B col and B val These represent the number of bytes for the row pointer, column index, and weight value, respectively. In one embodiment, B... row =4, B col =2, B val =2.

[0049] Active region A of the input feature map x Based on the activation threshold α act Sure: A x ={(c,h,w)|ReLU(x[c,h,w])>α act}; Where x is the input feature map of the current layer, and c, h, and w represent the channel, height, and width indices, respectively. The index-addressing memory access module only accesses A. x The activation values ​​in the CSR are multiplied and added together with the non-zero weight values ​​in the CSR, and the result is written to the corresponding position in the output feature map.

[0050] IV. Precompiled Alignment Module like Figure 4As shown, it also includes a pre-compiled alignment module, which is configured to iteratively optimize the pruning parameter vector based on a hardware performance proxy model and finite difference estimation, so that the prediction inference latency of the sparse model on the target edge accelerator meets the target latency constraint; wherein, the hardware performance proxy model takes sparse configuration features as input and outputs prediction inference latency, and the sparse configuration features include at least the sparsity distribution of each layer, the memory access mode entropy value, and the effective floating-point operation statistics.

[0051] Specifically, the objective function of the pre-compiled alignment module includes a prediction inference delay term and a delay exceeding the limit penalty term, the delay exceeding the limit penalty term being activated when the prediction inference delay is greater than the target delay; each component θ of the pruning parameter vector θ l Used to determine the pruning threshold ε of layer l prune,l , and ε prune,l equal to θ l The product of the absolute value of the weights in layer l and the maximum value of the weights in layer l.

[0052] More specifically, the pre-compiled alignment module is used to determine the pruning thresholds for each layer before deployment. The pruning parameter vector θ = [θ...] is defined. 1, θ 2, ...,θ L ], where θ l The pruning threshold multiplier for the l-th layer is: The actual pruning threshold is: ε prune,l =θ l ·max(| |),θ l ∈[θ min, θ max ].

[0053] In one embodiment, θ min =0.01, θ max =0.5. The optimization objective of the pre-compiled alignment module is: J(θ)=F GBDT (feat(θ))+λ delay ·max(0,F GBDT (feat(θ))-T target ); Where F GBDT This is a hardware performance proxy model, with sparse configuration features feat(θ) as input and prediction inference latency as output; T target Delay for the target; λ delay This represents the penalty coefficient for exceeding the delay limit. The objective function aims to ensure that the system prioritizes meeting the target delay constraint while reducing inference latency.

[0054] The sparse configuration feature feat(θ) can include the following: First, the histogram H(θ) of the sparsity distribution of each layer, for example, ρ l The distribution is divided into 50 equal-width bins to obtain 50-dimensional features; second, the memory access pattern entropy value E(θ) is used to describe the dispersion of CSR index access; third, the effective floating-point operation statistics include the mean, maximum and minimum effective FLOPs of each layer.

[0055] ρ l =1-||M s l ||_0 / (m l ·n l ); E(θ) = -Σ a p(a)·log2p(a); FLOPs l =2·nnz l ·N act,l ; Where ||M s l ||_0 represents the number of non-zero elements in the mask; p(a) is the probability that address a appears in the sampled address sequence; N act,l The number of active inputs involved in the computation at layer l; factor 2 represents one multiplication and one addition.

[0056] Since the tree model does not directly provide analytical gradients, the pre-compiled alignment module uses finite difference estimation of J(θ) with respect to θ. l Direction of change: g l =[J(θ l +δ)-J(θ l -δ)] / (2δ); Where δ is the finite difference step size. We obtain g. l Then, θ can be updated using Adam or other iterative optimizers, and θ can be adjusted after each update. l Limited to [θ] min, θ max ]. When ||θ is in consecutive iterations t+1 -θ t When ||_2 is less than the preset convergence threshold, stop the optimization and output the final pruning parameter vector θ*.

[0057] The hardware performance proxy model can be trained offline based on the target hardware or a high-precision timing simulator during the initial deployment. If new performance profiling samples are continuously collected after deployment, the tree model can be periodically added or the proxy model can be retrained without affecting the main inference thread.

[0058] V. Dynamic Scheduling Engine like Figure 5 As shown, the dynamic scheduling engine is modeled based on a Markov decision process, which includes a state space, an action space, and a reward function. The state space includes the one-hot encoding of the current sparse configuration, a device state vector, and an input feature vector. The device state vector includes the remaining power ratio, chip temperature, and available memory ratio, while the input feature vector includes the target density, image blur score, and motion amplitude. The action space includes sparse configuration adjustment actions and computational precision selection actions, with each action combination corresponding to a pre-compiled sparse computation kernel. The reward function is obtained based on the weighted result of the online precision estimation of the current frame, the actual inference latency, and the measured power consumption relative to their respective baseline values.

[0059] The dynamic scheduling engine uses a Markov decision process to select the runtime kernel. (State vector s) t It consists of the current sparse configuration, device state, and input features. In one embodiment, the state vector has a dimension of 10: s t =[onehot(c t ), battery t, temp t, memory t, density t, blur t, motion t ]; Onehot(c t ) represents the 4D one-hot encoding of the current sparse configuration; battery t This represents the percentage of remaining battery power; temp t Normalized chip temperature value; memory t This represents the percentage of available memory; density t The target density for the current frame; blur t Image blur score; motion t The amplitude of motion.

[0060] Action a t It consists of sparse configuration adjustment step size and calculation precision selection: a t =(Δc t, q t ),Δc t ∈{﹣1,0,+1},q t ∈{INT8,FP16}; Where Δc t Indicates reducing, maintaining, or increasing sparsity; q tThis indicates the choice between quantization precision and half-precision computation paths. Each action combination corresponds to a pre-compiled sparse computation kernel; at runtime, only the kernel handle or operator configuration is switched, without online retraining.

[0061] The reward function is calculated based on online accuracy estimation, measured latency, and measured power consumption: R t =w acc ·(Acc t / Acc ref )-w lat ·(Lat t / Lat ref )-w pow ·(Pow t / Pow ref ); Among them Acc t For online accuracy estimation of the current frame, Lat t For actual inference delay, Pow t This represents the measured power consumption; Acc ref Lat ref and Pow ref These are the baseline values; w acc w lat and w pow These are weighting coefficients, and the sum of the three is 1. In one embodiment, the online accuracy estimate Acct for the current frame is determined based on the model output confidence, detection result stability, and inter-frame consistency. For object detection tasks, Acct can be calculated from the detection box category confidence, object confidence, intersection-union ratio of adjacent frame detection boxes, category consistency, and change in the number of objects; for face verification, license plate recognition, or abnormal behavior recognition tasks, Acct can be determined from feature matching similarity, recognition confidence, consistency of recognition results in consecutive frames, or the maximum classification probability. In another embodiment, the system pre-establishes a mapping relationship between the model output confidence and the actual recognition accuracy, and obtains Acct at runtime through lookup or interpolation. The online accuracy estimate does not require the actual labeled results of the current frame and is used to characterize the credibility of the current inference result.

[0062] In one embodiment, the dynamic scheduling strategy employs a dual-depth Q-network. Both the online network and the target network use a three-layer fully connected structure: the input dimension is 10, the first hidden layer has 64 neurons, the second hidden layer has 32 neurons, and the output layer contains the Q-values ​​of 6 actions. The number of parameters in this network is: 10×64+64+64×32+32+32×6+6=2982.

[0063] To reduce runtime overhead, experience replay and Q-network updates can be triggered during device idle periods, on low-priority threads, or every N frames; the main inference thread only reads the already obtained policy output and does not wait for training updates to complete. Target network parameters can be soft-updated using the following formula: θ target ←(1-η soft )·θ target +η soft ·θ online ; Where η soft θ is the soft update coefficient. target Let θ be the target network parameter. online For online network parameters. The exploration strategy uses ε. explore - Greedy strategy, where ε explore Represents the probability of random exploration, and the pruning threshold ε. prune,l and Adam's stable term ε adam different.

[0064] VI. Security Execution Module It also includes a secure execution module, which is configured to perform CKKS-based homomorphic encryption inference on the non-zero elements of the sparse weights in sensitive tasks. The secure execution module is further configured to: encode the non-zero elements of each layer of weights into a CKKS plaintext polynomial in batches, and encrypt them using a public key to generate sparse weight ciphertext; during inference, perform homomorphic multiplication-addition operations on the sparse weight ciphertext and the corresponding input feature map ciphertext, wherein the number of homomorphic multiplications is proportional to the number of non-zero elements; generate a hash commitment for the sparse index set, and verify whether the currently used sparse index set is consistent with the hash commitment after inference. If they are inconsistent, terminate the inference and output an alarm message.

[0065] The secure execution module is used for sensitive tasks or scenarios with high requirements for the chain of evidence. This module is based on the CKKS approximate homomorphic encryption scheme, performing encrypted multiplication and addition on non-zero elements of sparse weights and their corresponding input features. Sensitive tasks refer to edge intelligent reasoning tasks involving personal identity information, vehicle identity information, law enforcement evidence data, or verification of the integrity of the chain of evidence, including but not limited to face detection, face verification, face feature comparison, license plate recognition, vehicle trajectory verification, target retrieval, and law enforcement audio and video evidence analysis. In one embodiment, the system determines whether the current task is sensitive based on the task type identifier, input data type, or business security level. When a task involves faces, license plates, ID documents, law enforcement video frames, vehicle trajectory data, or is marked by the business system as requiring encrypted reasoning, it is determined to be a sensitive task. When the current task is determined to be sensitive, the secure execution module initiates an encrypted reasoning process based on the CKKS approximate homomorphic encryption scheme; otherwise, the system uses a normal sparse reasoning path to execute the reasoning task.

[0066] Specifically, let the set of non-zero elements of the sparse weights of the l-th layer be: S l ={(i k,j _k,w k )|M s l [i k,j _k]=1,k=1,2,...,nnz l}; Where i k and j k w is the row and column index of the non-zero element. k For the corresponding weight values, nnz l Let N be the number of non-zero elements. Let the dimension of the CKKS polynomial ring be N. ckks Then a single ciphertext can be packaged into N. ckks / 2 slots. Non-zero value sequences are batch-encoded as plaintext polynomials: Plain b =Encode([w b,1 ,w b,2 ,...,w b,N_ckks / 2 ]); Ctxt W,b =Encrypt(pk,Plain b ); Slots not included in a batch are filled with zeros and then encrypted together. The zero-filled bits remain encrypted within the ciphertext, making it impossible for external sources to directly distinguish between zero-filled and non-zero-filled slots. During inference, they will be combined with S... l The input features corresponding to the indices are packaged and encrypted in the same order, followed by homomorphic multiplication and homomorphic addition: Ctxt y =Σ b CKKS Mult (Ctxt W,b, Ctxt X,b ); Ctxt X,b CKKS is used to input ciphertext features. Mult This represents ciphertext multiplication. Since only batches with non-zero weights are processed, the number of homomorphic multiplications is proportional to the number of non-zero elements, nnz. l or non-zero rate r lThe number of homomorphic multiplications is proportional to the number of non-zero elements (nnzl) or the non-zero rate (rl) because only batches corresponding to non-zero weights are processed. This is lower than the number of computations required to perform homomorphic multiplication on all elements of dense weights. In one embodiment, the polynomial ring dimension Nckks of the CKKS scheme takes the value of 8192, 16384, or 32768, and a single ciphertext can be packed into Nckks / 2 approximate numerical slots; the scaling factor Δ takes the value of 2^30 to 2^40. The secure execution module batch-encodes the sparse weight non-zero elements and corresponding input features in the same index order to obtain the CKKS plaintext polynomial, and encrypts them into weight ciphertext and input feature ciphertext respectively. After performing homomorphic multiplication, the secure execution module performs a rescaling operation on the multiplication result to restore the ciphertext scaling factor to the preset range; when performing homomorphic addition on the multiplication results of multiple batches, the module first performs modulus level alignment, and then completes the accumulation. The inference result ciphertext is decrypted by the authorized decryption module using a private key to obtain the corresponding approximate inference result. The above CKKS parameters can be adjusted according to the target security level, equipment computing power, and inference accuracy requirements.

[0067] To ensure the integrity of the sparse index set in the law enforcement evidence chain, the secure execution module generates a hash commitment according to preset serialization rules. In one embodiment, the hash commitment generation process includes: during the compilation phase, the secure execution module sorts and serializes the sparse index set of each layer in a fixed order of layer number, batch number, row index, and column index to obtain the index sequence Serialize(S); after concatenating the model identifier, model version number, device identifier, timestamp, and random number nonce with Serialize(S), a hash commitment value hashS is generated using the SHA-256 or SM3 hash algorithm. The hashS is stored in a local immutable log, a trusted execution environment, or the evidence chain record of the law enforcement business system. During runtime, the secure execution module regenerates the hash value hashS' of the current sparse index set according to the same serialization rules and compares hashS' with the pre-stored hashS; if they match, the inference result and verification record are output; if they do not match, the inference is terminated and an index tampering alarm message is output.

[0068] The serialization rule can be: concatenate the sparse index sequentially according to the layer number, row index, column index, and batch order, excluding the weighted plaintext values; then calculate: hash S =SHA-256(Serialize(S1||S2||...||S L )); hash S Stored in a local, immutable log or trusted execution environment. After each inference, the system recalculates the hash value currently using the sparse index set and compares it with the hash. SThe inference process is terminated and an alarm message is output if the two inconsistencies are found. In one embodiment, the sparse index set itself can be stored in a trusted execution environment such as ARMTrustZone, and the non-zero weight values ​​participate in the calculation in encrypted form.

[0069] In one embodiment, the initial value of the Gumbel-Softmax temperature parameter Tg is 1.0 to 5.0, and it gradually decays to 0.1 to 1.0 during training; the numerical stability term εadam in the Adam optimizer is 10^-8 to 10^-6; the sparsity constraint loss weight coefficient λsparse is 0.01 to 10, the hardware efficiency proxy loss weight coefficient λhw is 0.01 to 10, and the latency exceedance penalty weight coefficient λdelay is 1 to 100; the target network soft update coefficient ηsoft in dynamic scheduling is 0.001 to 0.05; and the initial value of the exploration probability εexplore is 0.05 to 0.3, which can gradually decay to 0.01 to 0.05 with training rounds or running frames. These parameters can be adjusted according to the computing power of the target edge device, model size, target latency, and accuracy requirements.

[0070] VII. Methods and Procedures The method of the present invention includes the following steps: Step 1, Model Input. Receive the pre-trained deep neural network model and read the weight tensors W of each pruning layer. l And normalize it into a two-dimensional weight matrix according to the layer type. .

[0071] Step 2, sparse pattern recognition. The two-dimensional weight matrix is ​​divided into multiple blocks, generating a block label sequence. A lightweight self-attention encoder is used to obtain the global sparse feature vector, and the sparse pattern classification probability distribution π is output.

[0072] Step 3: Mask generation and weight decomposition. Based on the target sparse pattern category, a structured sparse mask M is generated according to deterministic rules. s Unstructured sparse mask M u And obtained W s and W u .

[0073] Step 4, pre-compilation alignment. Construct the pruning parameter vector θ, predict the inference latency using a hardware performance surrogate model, and obtain θ* that satisfies the target latency through finite difference estimation and iterative optimization.

[0074] Step 5, Hardware Mapping. Map W... s Mapping to a regular sparse operator execution descriptor, W u The data is mapped to CSR storage data and index addressing memory access description information, generating multiple pre-compiled kernels with different sparsity and precision.

[0075] Step 6, runtime scheduling. Collect current input features and device status to form a state vector s. t Select action a according to the scheduling strategy. t Switch to the corresponding pre-compiled kernel to complete the current frame or batch inference; write the state transition data to the experience replay buffer and update the scheduling policy during idle periods.

[0076] Step 7, Secure Execution. In sensitive tasks, CKKS batch encoding and encrypted computation are performed on the non-zero elements of the sparse weights and their corresponding input features, and the sparse index set hash commitment is verified after inference.

[0077] Example 1: License Plate Recognition for Police Patrol Vehicles In this embodiment, the mobile law enforcement edge device is a police patrol car's vehicle-mounted terminal, the input is video frames captured by the vehicle-mounted camera, and the model is a pre-trained license plate detection network. The system first normalizes the convolutional layer weights of the license plate detection network into a two-dimensional matrix, and generates a 2:4 structured sparse mask and a CSR unstructured sparse index, respectively.

[0078] During the pre-compilation stage, the system generates 2:4 sparse operator execution descriptors based on the storage alignment granularity of the vehicle terminal NPU, and generates rows for the unstructured part. ptr ,col idx The system generates three arrays: `values`, `null`, and `values`. It also generates three kernel versions: low-sparse, medium-sparse, and high-sparse.

[0079] During operation, when vehicles move at high speeds or the target density in the image increases, the dynamic scheduling engine selects a low-sparse, high-precision kernel; when device temperature rises or battery power decreases, it selects a high-sparse, low-power kernel. In this embodiment, the system reduces average inference latency and peak power consumption while maintaining usable license plate detection accuracy.

[0080] Example 2: Handheld Face Verification Terminal In this embodiment, the mobile law enforcement edge device is a handheld face verification terminal, and the model is a lightweight face detection and feature extraction network. The sparse pattern analysis module generates sparse masks for the face detection network and the feature extraction network respectively, so that the structured sparse part is executed through a regular convolutional kernel, and the unstructured sparse part is executed through CSR index addressing.

[0081] When ambient lighting is poor or the image blur score is high, the dynamic scheduling engine reduces sparsity and selects a higher-precision kernel; when the image is stable and the device battery is low, it increases sparsity to reduce power consumption. For sensitive comparison tasks involving face templates, the secure execution module performs homomorphic multiplication and addition on non-zero weights and corresponding feature fragments, and records the sparse index set hash commitment.

[0082] Example 3: Abnormal Behavior Detection by Law Enforcement Recorder In this embodiment, the mobile law enforcement edge device is a wearable law enforcement recorder, and the model is a lightweight target detection network for abnormal behavior detection. Before deployment, the system performs sparse pattern classification on each layer of the model and generates multiple pre-compiled kernels of different sizes based on the available memory limitations of the device.

[0083] During runtime, if a sudden event is detected by the microphone or inertial sensor, the dynamic scheduling engine sets the motion amplitude in the input features to a higher value and selects a low-sparse, high-precision kernel; in ordinary inspection recording scenarios, a medium-sparse or high-sparse kernel is selected. This embodiment demonstrates the process by which the present invention switches the sparse kernel according to the intensity of the input event and the device status.

[0084] Example 4: Unmanned Aerial Vehicle (UAV) Law Enforcement Patrol Platform In this embodiment, the mobile law enforcement edge device is an unmanned aerial vehicle (UAV)-borne edge computing module, and the model is a network for detecting road occupancy, pedestrian gathering, or vehicle targets. The sparse pattern analysis module processes the weights of the visible light branch and the infrared branch respectively, and generates corresponding sparse operator descriptors.

[0085] During drone flight, the device state vector includes battery level, chip temperature, and available memory percentage, while the input feature vector includes target density, image blur, and motion amplitude. The dynamic scheduling engine selects the kernel based on these states to avoid inference interruptions caused by high temperature or low battery.

[0086] Example 5: Multi-terminal collaborative law enforcement scenario In this embodiment, the system of the present invention is deployed on multiple law enforcement terminals. The command terminal distributes the same model structure and sparse configuration candidate set to each device, and each law enforcement terminal generates a localized pruning parameter vector based on its own hardware status through a pre-compiled alignment module.

[0087] During operation, each terminal independently selects a pre-compiled kernel based on its local input features and device status. For terminals with low battery or high temperature, the system prioritizes a high-sparse, low-power kernel; for terminals undertaking key identification tasks, the system selects a low-sparse, high-precision kernel. This embodiment illustrates that the present invention can maintain the differentiated deployment capability of a unified model under different hardware conditions.

[0088] Example 6: Sensitive Evidence Chain Verification Scenario In this embodiment, the mobile law enforcement edge device needs to perform local verification on sensitive images involving faces or license plates. During the compilation phase, the system generates a sparse index set S, and then serializes it according to the layer number, row index, and column index to generate a hash commitment.

[0089] During the inference phase, the secure execution module only performs CKKS batch encoding and encrypted multiplication-addition on non-zero weight values ​​and corresponding input features; after inference, it recalculates the hash value of the current sparse index set. If the hash value matches the pre-stored hash commitment, the inference result and verification record are output; if they do not match, the inference is terminated and an anomaly is recorded. This embodiment demonstrates the integrity verification capability of the present invention in sensitive law enforcement tasks.

[0090] The above embodiments are only used to illustrate the technical solutions of the present invention. Without departing from the concept of the present invention, those skilled in the art can make equivalent substitutions or modifications to the system modules, parameter values, target hardware types, model structures, and scheduling strategies, and all such substitutions or modifications should fall within the protection scope of the present invention.

[0091] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Any modifications, alterations, substitutions, and variations made by those skilled in the art to the above embodiments are within the scope of the present invention.

Claims

1. A deep neural network sparse compilation optimization system for mobile law enforcement edge devices, characterized in that, include: Sparse pattern analysis module, hardware mapping module, and dynamic scheduling engine; The sparse pattern analysis module is configured to receive a pre-trained deep neural network model, perform sparse pattern recognition on the weight tensors of each layer in the model, generate structured sparse masks and unstructured sparse masks, and obtain the structured sparse weight parts and unstructured sparse weight parts based on the structured sparse masks and unstructured sparse masks; the hardware mapping module is connected to the sparse pattern analysis module and is configured to convert the structured sparse weight parts and unstructured sparse weight parts into sparse operator execution description information that can be called by the target edge accelerator, according to the operator capabilities, storage alignment rules and index addressing rules of the target edge accelerator. The dynamic scheduling engine is configured to acquire the current input data characteristics and device state at runtime, and select or switch between multiple pre-compiled sparse computing kernels based on the current input data characteristics and device state to execute the inference task of the current frame or the current batch. The configuration of the sparse pattern analysis module includes normalizing the first layer weight tensor into a two-dimensional weight matrix, dividing the two-dimensional weight matrix into several non-overlapping blocks, performing flattening processing on each block and adding two-dimensional position encoding to form a block label sequence; and encoding the block label sequence through a single-head self-attention encoder to obtain the global sparse feature vector of the layer weights. The global sparse feature vector is mapped to a classification probability distribution of several predefined sparse pattern categories, including structured sparseness, block sparseness, diagonal sparseness, and unstructured sparseness. A target sparse pattern is determined based on the classification probability distribution, and a binary mask matrix is ​​generated based on the target sparse pattern. Specifically, determining the target sparse pattern based on the classification probability distribution and generating the binary mask matrix based on the target sparse pattern involves: When the target category is the first category, the 2:4 structured sparsity rule is adopted. Each row of the two-dimensional weight matrix is ​​divided into groups of four consecutive elements. For the four elements in each group, the mask values ​​of the two elements with the largest absolute values ​​are set to 1, and the mask values ​​of the other two elements are set to 0. When the target category is the second category, the block sparsity rule is adopted to divide the two-dimensional weight matrix into multiple square blocks. The side length of each square block is a preset second positive integer. The square root of the sum of the squares of all elements in each block is calculated to obtain the Frobenius norm of that block, and the median of the Frobenius norms of all blocks is calculated. For any block, if its Frobenius norm is greater than or equal to the product of the preset scaling factor and the median, the mask value of all positions in that block is set to 1, otherwise it is set to 0. When the target category is the third category, the strip diagonal sparsity rule is adopted. Based on the number of rows and columns of the current layer's two-dimensional weight matrix, the smaller value is taken and multiplied by a preset bandwidth ratio coefficient, and then rounded up to obtain the diagonal bandwidth. For any position in the matrix, if the absolute value of the difference between its row index and column index is less than or equal to the diagonal bandwidth, the mask value of that position is set to 1, otherwise it is set to 0. When the target category is the fourth category, an unstructured amplitude sparsity rule is adopted. The absolute value of each element in the two-dimensional weight matrix is ​​compared with a preset pruning threshold. If the absolute value is greater than or equal to the pruning threshold, the mask value at that position is set to 1; otherwise, it is set to 0. The pruning threshold is determined by the pre-compiled alignment module according to the hardware characteristics.

2. The deep neural network sparse compilation optimization system for mobile law enforcement edge devices according to claim 1, characterized in that, The configuration of the hardware mapping module includes, for the structured sparse weight portion, generating a sparse operator execution descriptor for the target edge accelerator. This sparse operator execution descriptor includes at least an operator identifier, weight storage address, index or mask storage address, alignment step size, and a calculation skip threshold. The operator identifier is determined using a lookup table based on the sparse mode category index. The alignment step size is determined based on the non-zero rate, group size, index byte count, and the storage alignment granularity of the target edge accelerator. The calculation skip threshold is updated based on the hardware performance proxy model and actual operating status. For the unstructured sparse weight portion, non-zero elements are stored using a compressed sparse row (CSR) format, and index-addressed memory access description information is generated.

3. The deep neural network sparse compilation optimization system for mobile law enforcement edge devices according to claim 1, characterized in that, It also includes a pre-compiled alignment module, which is configured to iteratively optimize the pruning parameter vector based on a hardware performance proxy model and finite difference estimation, so that the prediction inference latency of the sparse model on the target edge accelerator meets the target latency constraint; wherein, the hardware performance proxy model takes sparse configuration features as input and outputs prediction inference latency, and the sparse configuration features include at least the sparsity distribution of each layer, the memory access mode entropy value, and the effective floating-point operation statistics.

4. The deep neural network sparse compilation optimization system for mobile law enforcement edge devices according to claim 3, characterized in that, The objective function of the pre-compiled alignment module includes a prediction inference delay term and a delay exceeding penalty term, wherein the delay exceeding penalty term is activated when the prediction inference delay is greater than the target delay; each component θ of the pruning parameter vector θ l Used to determine the pruning threshold ε of layer l prune,l , and ε prune,l equal to θ l The product of the absolute value of the weights in layer l and the maximum value of the weights in layer l.

5. The deep neural network sparse compilation optimization system for mobile law enforcement edge devices according to claim 1, characterized in that, The dynamic scheduling engine is modeled based on a Markov decision process, which includes a state space, an action space, and a reward function. The state space includes the one-hot encoding of the current sparse configuration, a device state vector, and an input feature vector. The device state vector includes the remaining power percentage, chip temperature, and available memory percentage. The input feature vector includes the target density, image blur score, and motion amplitude. The action space includes sparse configuration adjustment actions and computational precision selection actions. Each action combination corresponds to a pre-compiled sparse computation kernel. The reward function is obtained based on the weighted result of the online precision estimation of the current frame, the actual inference latency, and the measured power consumption relative to their respective benchmark values.

6. The deep neural network sparse compilation optimization system for mobile law enforcement edge devices according to claim 1, characterized in that, It also includes a secure execution module, which is configured to perform homomorphic encryption inference based on the near-homomorphic encryption CKKS on the non-zero elements of the sparse weights in sensitive tasks; the secure execution module is further configured to: encode the non-zero elements of each layer of weights into a CKKS plaintext polynomial in batches, and encrypt them using a public key to generate sparse weight ciphertext; during inference, perform homomorphic multiplication-addition operations on the sparse weight ciphertext and the corresponding input feature map ciphertext, wherein the number of homomorphic multiplications is proportional to the number of non-zero elements; Generate a hash commitment for the sparse index set, and verify after inference whether the currently used sparse index set is consistent with the hash commitment. If they are inconsistent, terminate the inference and output an alarm message.

7. A deep neural network sparse compilation optimization method for mobile law enforcement edge devices using the system described in any one of claims 1-6, characterized in that, Includes the following steps: Step 1, Model Input: Receive the pre-trained deep neural network model, read the weight tensors of each pruning layer, and normalize them into a two-dimensional weight matrix according to the layer type; Step 2, Sparse Pattern Recognition: Divide the two-dimensional weight matrix into multiple blocks, generate a block label sequence, obtain the global sparse feature vector through a lightweight self-attention encoder, and output the sparse pattern classification probability distribution π. Step 3, Mask Generation and Weight Decomposition: Based on the target sparse pattern category, generate structured sparse masks and unstructured sparse masks according to deterministic rules, and obtain the structured sparse weight part and the unstructured sparse weight part respectively. Step 4, Pre-compilation alignment: Construct the pruning parameter vector θ, use the hardware performance proxy model to predict the inference latency, and obtain θ* that meets the target latency through finite difference estimation and iterative optimization; Step 5, Hardware Mapping: Map the structured sparse weights to regular sparse operator execution descriptors, and map the unstructured sparse weights to CSR storage data and index addressing memory access description information, generating multiple pre-compiled kernels with different sparsity and precision. Step 6, Runtime Scheduling: Collect current input features and device status to form a state vector; select actions according to the scheduling strategy; switch to the corresponding pre-compiled kernel to complete the current frame or current batch inference; write the state transition data to the experience replay buffer and update the scheduling strategy during idle periods; Step 7, Secure Execution: In sensitive tasks, perform CKKS batch encoding and encrypted computation on the non-zero elements of sparse weights and their corresponding input features, and verify the sparse index set hash commitment after inference.

8. The deep neural network sparse compilation optimization method for mobile law enforcement edge devices according to claim 7, characterized in that, The scheduling strategy employs a dual-depth Q network, with both the online network and the target network using a three-layer fully connected structure.

Images