A pedestrian detection method and system based on deep learning automatic pruning

Abstract

The present invention relates to the field of target detection technology, and more particularly to a pedestrian detection method and system based on deep learning automated pruning. The method comprises acquiring pedestrian image data; preprocessing the acquired pedestrian image data; constructing a deep neural network (DNN) model; pruning the DNN model based on a pattern pruning strategy; and deploying the pruned DNN model on an autonomous vehicle for pedestrian detection. Compared with traditional pruning methods, the proposed scheme significantly improves computational efficiency and hardware acceleration compatibility while minimizing or even partially recovering model accuracy.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a pedestrian detection method and system based on deep learning automatic pruning. Background Art

[0002] In recent years, with the continuous advancement of deep learning technology and the continuous improvement of GPU performance, deep neural networks (DNNs) have evolved towards deeper layers and more parameters, enabling them to learn higher-level and multi-dimensional data semantics. This has led to breakthroughs in image recognition, image classification, audio and video processing, and has also driven the development of smart transportation, e-commerce platforms, smart healthcare, and digital cities. However, in practical applications, especially on edge computing devices, there is often a conflict between limited hardware resources and limited memory and the large number of layers and parameters in deep neural network models. This is because edge devices generally use low-power processors to achieve a balance between performance and cost. For example, the VGG-16 network with 138 million parameters requires approximately 1.6 billion floating-point operations per second during inference and consumes approximately 528MB of memory, which poses a challenge for model deployment on edge computing devices. For example, to run the VGG-16 network model on an edge device equipped with a Hi3536AV100 SoC chip and 512MB of memory, model lightweighting techniques are required to adjust or design the network structure or increase memory size to facilitate deployment.

[0003] As the demand for deploying deep neural network models on edge devices continues to increase, various model lightweighting technologies have emerged (knowledge distillation, model compression, NAS, parameter quantization, tensor decomposition, etc.), and model compression, especially model pruning methods, has become an indispensable model lightweighting strategy. Model pruning methods can be divided into three types according to the granularity of pruning (such as Figure 1 As shown in the figure, there are three types of pruning: unstructured pruning (or fine-grained pruning), semi-structured pruning (or pattern-based pruning), and structured pruning (or coarse-grained pruning). For example, taking a convolutional layer containing k convolution kernels with a size of 3×3, unstructured pruning can delete weights at any position in the weight tensor of a filter, resulting in an irregular distribution of non-zero weights. This irregularity requires specialized software and hardware support for effective acceleration. In contrast, structured pruning deletes the entire filter, making it possible to achieve acceleration using conventional deep learning frameworks. However, due to its coarse granularity, structured pruning often has difficulty achieving the optimal balance between compression rate and accuracy.

[0004] To find the best pruning strategy, automated methods such as AutoML (automatic machine learning) have recently attracted attention. These methods use graph learning techniques, especially graph neural networks (GNNs), to represent or embed DNNs. The embedded data is then fed into a deep reinforcement learning (DRL) algorithm to find the optimal pruning strategy, thereby achieving higher compression efficiency. However, although existing automatic pruning methods have made significant progress, they mainly focus on structured and unstructured pruning, and pattern-based pruning methods are limited and face three major challenges: (1) They mainly deal with convolutional neural networks (CNNs) and lack the ability to generalize to various DNN architectures, such as the encoder and decoder in ViT-B / 16; (2) They are mainly designed for specific operators with fixed sizes (e.g., 3×3 convolutions) in CNNs, which may limit their effectiveness for other shapes; (3) Previous work has not considered integrating pruning pattern information and topology information into DNN models and combining it with automatic search to improve the final pruning effect. Summary of the Invention

[0005] In order to solve the above-mentioned problems, the present invention provides a pedestrian detection method and system based on deep learning automatic pruning.

[0006] In a first aspect, the present invention provides a pedestrian detection method based on deep learning automatic pruning, which adopts the following technical solutions:

[0007] A pedestrian detection method based on deep learning automatic pruning, comprising:

[0008] Obtain pedestrian image data;

[0009] Preprocessing the acquired pedestrian image data;

[0010] Build a deep neural network DNN model;

[0011] Prune the DNN model based on the pattern pruning strategy;

[0012] The pruned DNN model is deployed on a self-driving car for pedestrian detection.

[0013] Furthermore, the DNN model is pruned based on a pattern pruning strategy, including initializing a pruning pattern library and evaluating DNN model compression parameters; modeling the DNN model as a graph structure and obtaining a graph embedding representation; using DRL to generate a pruning strategy based on the graph embedding representation; and executing the pruning strategy on the DNN model and then performing compilation optimization using a compiler to obtain a pruned and optimized DNN model.

[0014] Furthermore, the pruning pattern library is initialized and the compression parameters of the DNN model are evaluated, including combining the distribution law of the weight parameters to design a pattern library graded by sparsity rate. The sparsity rate of the patterns in the pattern library decreases evenly from 100% to 0%. The sparsity interval Δ is set to control the sparsity granularity. Δ=9 generates 10 sparsity rates, corresponding to 10 types of patterns. The compression parameters of the DNN model are calculated to determine whether the constraints are met, and if the constraints are not met, the pruning strategy is continued to be searched.

[0015] Furthermore, the DNN model is modeled as a graph structure and a graph embedding representation is obtained, including extracting network topology information from the DNN model and integrating pruning pattern information into the graph structure, obtaining the graph embedding representation of the model through the GNN method, and Constructed graph Expressed as ,in is a node set, is an edge set, a subset represents the node associated with the weight tensor, is the set of input tensor nodes, is the set of output nodes, subset and Corresponding to the link The nodes in different layers of DNN are further divided into more fine-grained sets.

[0016] Furthermore, the method of using DRL to generate a pruning strategy based on the graph embedding representation includes: As the environment state is passed to the Agent, the direct output obtained by the Agent is the pattern in the pattern library The probability distribution of , so its action space Continuous is continuous, and then by calculating the action value, the selected target mode Sampling is performed to obtain the pruned DNN model; wherein, the function As a multi-layer perceptron, it is responsible for extracting graph representation information from graph representation embedding. The activation function will The output is mapped to interval, determine the selection probability of each mode, that is, the action space, and finally, apply the classification distribution function from the probability distribution Sampling, assigning patterns to different weight tensors in the DNN, producing a library of newly selected patterns Used for pruning.

[0017] Furthermore, the method of generating a pruning strategy based on the graph embedding representation using DRL also includes setting a reward function by combining the model compression metric and the model inference accuracy, which is expressed as:

[0018]

[0019] in is a learnable parameter. When it is greater than the set threshold, the agent is encouraged to adopt an aggressive compression strategy, giving priority to model compression rather than accuracy. On the contrary, when When it is less than the set value, the agent will be motivated to take a conservative approach.

[0020] Furthermore, after executing the pruning strategy on the DNN model, the compiler is used to perform compilation optimization, including performing a pruning operation on the model after the pruned DNN model meets the constraint conditions, replacing the model parameters with 0 according to the assigned pattern to obtain a pruned DNN model. At the same time, since the reasoning accuracy of the pruned DNN model will decrease, the pruned DNN model is fine-tuned to restore its reasoning accuracy.

[0021] Furthermore, after executing the pruning strategy on the DNN model, the compiler is used for compilation optimization, which also includes using the TVM open source compiler to perform graph optimization, perform operator fusion, memory optimization and parallelism adjustment; adopt an automatic tuning strategy to adjust the kernel size, parallel strategy and data caching method according to the target hardware platform to fully reduce the inference delay; compile and generate the final low-level code or executable file, and support hot update or online deployment mechanism to facilitate testing and optimization in various hardware environments.

[0022] Furthermore, the pruned model is deployed on the autonomous vehicle for pedestrian detection, including deploying the pruned DNN model in the perception module, which is responsible for extracting environmental information from sensor data and making detection results.

[0023] The second aspect is a pedestrian detection system based on deep learning automatic pruning, including:

[0024] The data acquisition module is configured to acquire pedestrian image data;

[0025] A preprocessing module is configured to preprocess the acquired pedestrian image data;

[0026] The model building module is configured to build a deep neural network DNN model;

[0027] The pruning module is configured to prune the DNN model based on the pattern pruning strategy;

[0028] The detection module is configured to deploy the pruned DNN model to the autonomous vehicle for pedestrian detection.

[0029] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a pedestrian detection method based on deep learning automatic pruning.

[0030] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded and executed by the processor to implement the pedestrian detection method based on deep learning automatic pruning.

[0031] In summary, the present invention has the following beneficial technical effects:

[0032] First, it can automatically prune a variety of DNN architectures, including CNN and Transformer, with good versatility. Second, by using graph structures to express network topology and weight information and embedding pruning patterns into edge features, it can finely control the pruning granularity and effectively delete local computing modules. Then, deep reinforcement learning is used to automatically search for pruning strategies, so that the pruning process can be adaptively adjusted according to the pruning reward function to achieve the optimal pruning effect, thereby significantly reducing FLOPs (for example, up to 90%) and greatly reducing inference latency (by more than 50% on some edge devices). Finally, compared with traditional pruning methods, the proposed scheme significantly improves computing efficiency and hardware acceleration compatibility while minimizing or even partially recovering the loss of model accuracy.

[0033] The present invention has the following advantages: 1) Using hierarchical sparsity rates can cover a larger search space, and the size of the search space can be controlled by adjusting Δ; 2) Unifying unstructured pruning and structured pruning simplifies the design of pruning strategies; 3) It is not limited to a specific model or a single operation structure and is applicable to a variety of complex networks (including residual networks, Transformers, etc.); 4) Using deep reinforcement learning for strategy search can automatically discover redundant parts in the network without the need for manual design of pruning schemes; 5) The pruned network can be optimized at the underlying level with the help of advanced compilers, which is particularly suitable for the requirements of low-latency operation on edge devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is the overall framework diagram of the present invention.

[0035] Figure 2 This is a framework diagram of the present invention that models DNN as a graph and obtains a graph embedding representation.

[0036] Figure 3 This is the framework diagram of the deep reinforcement learning pruning strategy search module.

[0037] Figure 4 This is a framework diagram for pruning and fine-tuning the DNN execution model.

[0038] Figure 5 This is a framework diagram for compiling, optimizing, and deploying the pruned model.

[0039] Figure 6 This is a flowchart of the automatic mode pruning method of deep learning models for heterogeneous edge devices of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described in detail below with reference to the accompanying drawings.

[0041] Example 1

[0042] Reference Figure 1 , a pedestrian detection method based on deep learning automatic pruning in this embodiment includes:

[0043] Obtain pedestrian image data;

[0044] Preprocessing the acquired pedestrian image data;

[0045] Build a deep neural network DNN model;

[0046] Prune the DNN model based on the pattern pruning strategy;

[0047] The pruned DNN model is deployed on a self-driving car for pedestrian detection.

[0048] Specifically:

[0049] S1. Get data,

[0050] Use the public pedestrian detection datasets COCO and Caltech Pedestrian Dataset to train the model, ensuring that the datasets contain a large number of pedestrian images and annotations.

[0051] COCO dataset: Contains approximately 330,000 images, 80 categories, including pedestrians (category ID is 1). Each object has a corresponding bounding box, and the annotations include category labels (pedestrians) and location coordinates.

[0052] Caltech Pedestrian Dataset: This dataset contains approximately 10,000 images of scenes containing pedestrians and backgrounds. Each image is annotated with the location and category of the pedestrian. Each pedestrian's location is annotated with a bounding box.

[0053] S2. Preprocess the acquired data.

[0054] Pedestrian detection models (such as YOLO and Faster R-CNN) usually require the input image size to be fixed. The following processing method is used:

[0055] Resizing: Resize the image to the input size required by the network. For example, the standard input size for YOLO is 416×416 or 608×608 pixels, and Faster R-CNN typically requires an input of 800×800 pixels.

[0056] Maintaining aspect ratio: When resizing an image, it is best to maintain its aspect ratio to avoid distortion that affects the image's features. This can usually be done by padding the image to a fixed size (by adding black borders).

[0057] Normalization: Scale the pixel values of an image to a uniform range, usually 0 to 1 or -1 to 1. This can speed up training and avoid the vanishing / exploding gradient problem. 0-1 normalization is used.

[0058] Data augmentation: A common technique in deep learning used to increase the diversity of the training set and improve the generalization ability of the model. This technique uses methods such as rotation, cropping, horizontal flipping, and color dithering.

[0059] Data annotation format conversion: YOLO format, each image corresponds to a text file, which contains the category ID, bounding box center coordinates (normalized relative to the image width and height), and aspect ratio of each target (pedestrian); PascalVOC format, each image corresponds to an XML file, which contains the category, bounding box coordinates, etc. of each target.

[0060] S3. Build a deep neural network DNN model;

[0061] Among them, 1. YOLOv3 model construction process:

[0062] (1) Network structure (Darknet-53 backbone + FPN),

[0063] Backbone network: Darknet-53 (with residual connections), outputting feature maps of three scales (e.g., 13×13, 26×26, and 52×52).

[0064] Feature Pyramid (FPN): Integrates multi-scale features to improve small target detection capabilities.

[0065] (2) Input and output,

[0066] Input: Image resized to a fixed size (e.g. 416×416).

[0067] Output: Each grid predicts B bounding boxes, each box contains:

[0068] coordinate : The offset of the center point relative to the grid, and the scaling of the width and height relative to the anchor box.

[0069] Confidence: .

[0070] Class Probability: (Softmax or Sigmoid output).

[0071] (3) Key formula,

[0072] Bounding Box Prediction:

[0073] (1)

[0074] in, : coordinates of the upper left corner of the grid; : Anchor box width and height; : Network prediction value; : Sigmoid function.

[0075] Loss function (multi-task loss):

[0076] (2)

[0077] in, : weight coefficient; : mean square error; : binary cross entropy; : Multi-classification cross entropy.

[0078] (4) Post-processing (NMS)

[0079] Filter low-scoring boxes by confidence and retain boxes with loU below the threshold.

[0080] 2. Faster R-CNN construction process:

[0081] (1) Network structure,

[0082] Backbone network: VGG16 / ResNet, etc., extract feature maps.

[0083] Region Proposal Network (RPN): Slides a window on the feature map to generate anchor boxes (such as 9 scales / aspect ratios). Output: category score (foreground / background) and bounding box offset of the anchor box.

[0084] RolPooling / RolAlign: Unify the proposed regions into fixed-size features.

[0085] Classification and regression head: predicts specific categories and precise box coordinates.

[0086] (2) Key formula,

[0087] RPN loss function:

[0088] (3)

[0089] in, : The probability that the anchor box is the foreground; : True label (0 / 1); : prediction offset; : The actual offset.

[0090] SmoothL1 loss:

[0091] (4)

[0092] Rol Pooling:

[0093] The proposed regions of different sizes are divided into H×W grids, and maximum pooling is performed in each grid.

[0094] Final detection loss:

[0095] (5)

[0096] (3) Training process,

[0097] End-to-end training: RPN and detection network share features and are optimized alternately.

[0098] Anchor box matching strategy: IoU with the ground-truth box > 0.7 is a positive sample, and < 0.3 is a negative sample.

[0099] S4. Prune the DNN model based on the pattern pruning strategy, including the following steps:

[0100] Step 1: Initialize the pruning pattern library.

[0101] The direct object of pattern pruning is the weight parameters of DNN. Parameters with large values have a greater impact on model performance, and vice versa. The parameter value distribution in different weight tensors is different, but the shape (pattern) formed by the parameters with larger weights is regular: certain specific shapes will appear repeatedly; multiple kernels in the same filter show similar patterns. The present invention combines the distribution law of weight parameters to design a pattern library graded by sparsity. The sparsity of the patterns in the pattern library drops evenly from 100% to 0%. Set a sparse interval Used to control the sparse granularity, for example, When , 10 kinds of sparse rates can be generated, and there are corresponding 10 types of modes.

[0102] Among them, when , the pattern library contains 10 pruning patterns with different sparsity rates. For each sparsity rate, the search starts with the minimum number of patterns (i.e., there is only one pattern for each sparsity rate at the beginning). Through multiple iterations, the number of patterns is gradually increased until a balance is reached between the number of patterns and the pruning effect, and the pattern shape is ensured to be consistent with the shape of the DNN weight tensor (for example, for a 3×3 convolution kernel, the pattern size is a 3×3 matrix). Multiple binary matrices with different pruning shapes are automatically generated. In the pattern library, each pattern is stored in the form of a binary matrix, with a value of 1 for retention and 0 for pruning. The maximum number of each pattern depends on the number of weights pruned by the sparsity rate. For example, for a 3×3 pattern with a sparsity rate of 55.56%, 5 weights need to be pruned, so the total number of patterns under this sparsity rate is indivual.

[0103] Step 2: Evaluate the DNN model compression parameters.

[0104] First, the compression parameters of the DNN model (such as inference accuracy, FLOPs, and pruning rate) are calculated to determine whether they meet the constraints. If not, the search for pruning strategies continues (jump to step 3); otherwise, pruning is performed on the DNN (jump to step 5).

[0105] The pruning rate constraint hyperparameter is set to 60%, and MACs (Multiply-Accumulate Operations) are used to calculate the computational cost of zero parameters and the computational cost of all parameters of the DNN model. The ratio of the computational cost of zero parameters to the computational cost of all parameters is calculated. When this ratio is greater than or equal to 60%, it means that the constraint condition is met, otherwise it is not met.

[0106] Step 3: Model the DNN as a graph and obtain the graph embedding representation.

[0107] The core of this method is to construct DNN into a graph structure and obtain graph embedding. First, the network topology information is extracted from the model, and the pruning pattern information is integrated into the graph structure. Then, the graph embedding representation of the model is obtained through the GNN method. Constructed graph It can be expressed as ,in is a node set, is an edge set. represents the node associated with the weight tensor, is the set of input tensor nodes, is the set of output nodes. and Corresponding to the link The nodes in different layers of DNN can be further divided into more fine-grained sets. For example, the nodes in layer 1 are contained in Next, we will use two typical DNN architectures, CNN and Transformer, as examples to illustrate the composition process.

[0108] (1) CNN composition process.

[0109] Given a regular CNN , using a set of weights To parameterize the CNN architecture, Indicates the The convolutional layer The weight tensor corresponding to the convolution kernel, is the number of convolutional layers, It is The number of convolution kernels in the layer. The input and output feature maps associated with the layer are and .

[0110] The specific composition process is as follows: First, The weight set of the layer Mapping to a node collection , where the nodes Indicates the The weight of the convolution kernel At the same time, The input and output feature maps of the layer are mapped to nodes and The dependencies of the convolution operation are then mapped to a set of nodes With node sets and The connection relationship between them forms a graph The edge set in After completing the above steps, we get the graph , the process is as follows Figure 3 As shown in Figure 2. Since the parameters of the CNN model are mainly located in the convolutional layer, and the final fully connected layer is connected to the output layer, the focus in the graph construction process is on the convolutional layer.

[0111] In constructing the graph After the basic structure of (like Figure 2 The PatternsLibrary derived from the Agent decision is integrated into Through research, it is found that fusing key information into edge features produces better results than fusing it into node features. Therefore, the following fusion method is adopted: Merge to edge set In the edge feature, the node set The corresponding weights are fused into their node features. In addition, the node set and Node features and edge sets in The edge feature embeddings in are assigned using random initialization.

[0112] Furthermore, in CNN architectures such as ResNet, there are residual connections. These residual connections establish additional paths between layers, allowing information to flow across layers without being interrupted by pruning. Therefore, when constructing the graph, residual connections are represented as additional edges that link the nodes corresponding to the layers involved in the residual path. This ensures that the graph Accurately reflecting the complete structure of CNN, including these key connections, is crucial for maintaining the performance of CNN models during pruning.

[0113] (2) Transformer composition process.

[0114] For the DNN structure containing the Transformer structure, take the Transformer encoder as an example. To model, assume that The input embedding of the encoder is ,in The input sequence is represented by Tokens, is the embedding dimension. Then, the Q, K, V matrices can be expressed as , and ,in is a learnable weight matrix, is the respective output dimension. The attention score is expressed as , the output is .

[0115] Although the Transformer model structure is significantly different from CNN, it is also possible to build a graph for it to achieve pattern pruning. An encoder will , , Mapped to nodes respectively . Similarly, The input and output of each encoder are mapped to and Then, the node set and related edge sets Together they form the target graph In addition, since the Transformer model contains multiple encoder and decoder blocks, each of which contains a feed-forward network (MLP) that holds a large proportion of the model's parameters, this also needs to be taken into account when building the graph.

[0116] In summary, DNN Constructing graphs and fusing patterns The process can be expressed as:

[0117] (6)

[0118] After the graph is constructed, we need to Encode and extract its representative embedding , this process can be expressed as:

[0119] (7)

[0120] Step 4: Use DRL to generate pruning strategies.

[0121] The reinforcement learning agent calculates the sampling probability of each pattern in the pattern library based on the graph embedding (State), assigns a new pruning pattern to the DNN, feeds the reallocated pruning pattern (Action) back to the environment (Environment), and re-evaluates whether the model meets the constraints; it updates the pruning pattern library and gradually increases the number of patterns at each sparsity rate.

[0122] Environmental status The DRL training goal is to combine the model parameters of DNN with the topological structure information to determine the pruning pattern of its operators. Therefore, we embed the graph obtained from the graph encoder The state of the environment is passed to the agent. Since the pruning pattern applied to the DNN changes after each iteration, the graph needs to be rebuilt. To update the embedding representation of DNN.

[0123] Action Space The direct output obtained by the agent is the pattern in the pattern library The probability distribution of , so its action space Continuous is continuous:

[0124] (8)

[0125] in, is the total number of predefined patterns. The action value is calculated using formula (9), and then the selected target pattern Sampling is performed to obtain the pruned DNN.

[0126] (9)

[0127] in Represents the obtained graph representation embedding. Function It is a multi-layer perceptron that is responsible for Extract graph representation information. The activation function will The output is mapped to interval, determine the selection probability of each mode (i.e., action space).

[0128] Finally, we apply the classification distribution function to get the probability distribution By sampling, we assign patterns to different weight tensors in the DNN. This will produce a library of newly selected patterns for pruning.

[0129] Specifically, assuming that there are 10 patterns in the current pattern library (10 sparsity rates, each sparsity rate contains one pattern), these patterns are sorted from small to large by sparsity rate and numbered (0-9). Traverse the weight tensor of the DNN, for each weight tensor, the agent applies the classification distribution function Categorical from the probability distribution Sampling (as shown in formula 10) to obtain a mode number , that is, for the current weight tensor, the number is applied The pruning pattern library is updated after the traversal is completed. The sparse rate with the largest number of patterns is counted, and a pattern with that sparse rate that is not currently in the current pattern library is added. If the number of patterns with that sparse rate in the pattern library has reached the upper limit, the pattern with the next largest number of patterns is added, and so on.

[0130] (10)

[0131] Reward Function The design of the reward function is crucial for DRL training. Generally speaking, a higher pruning rate leads to a more severe degradation in the model's inference accuracy. As training progresses, the agent tends to reduce the pruning rate to achieve better inference accuracy. Therefore, we design the reward function by combining the model compression metric (FLOPs as an example, but other metrics such as multiply-accumulate operations (MAC) can also be applied similarly) and the model's inference accuracy, as shown below:

[0132] (11)

[0133] in is a learnable parameter. When is large, the agent is encouraged to adopt a more aggressive compression strategy, prioritizing model compression over accuracy. When is smaller, the agent is incentivized to take a more conservative approach.

[0134] Furthermore, formula (11) can be replaced by To optimize , because FLOPs and inference latency are both pruning constraints and can be used as part of the reward function.

[0135] Step 5: Perform pattern pruning and fine-tuning on the DNN.

[0136] Once the pruned model satisfies the constraints, it is pruned (i.e., model parameters are replaced with zeros according to the assigned pattern), resulting in a pruned DNN. The pruned model's inference accuracy decreases, so the pruned DNN is fine-tuned to restore its inference accuracy.

[0137] Among them, according to the pruning mode output by the policy module, the corresponding binary matrix is used for local pruning for each convolutional layer or operation unit, and the Hadamard product operation is directly performed on the weight matrix to achieve local parameter zeroing; after the pruning operation is completed, the entire model is retrained using the preset fine-tuning training scheme (such as SGD or Adam optimizer, setting appropriate learning rate and decay strategy) to ensure that the performance of the pruned model is restored to a level close to or exceeding the original level.

[0138] The specific steps are as follows:

[0139] (1) Application of pruning mode,

[0140] Input: Original DNN model weight matrix , the pattern library output by the pruning strategy module (Contains a binary mask matrix).

[0141] operate:

[0142] 1) Traverse each weight tensor in the DNN (such as convolution kernel or fully connected layer weight), according to the mode number assigned by the agent , select the corresponding binary mask matrix from the pattern library (For convolution kernel size situation).

[0143] 2) For each weight tensor , perform element-wise Hadamard product operation, the formula is as follows:

[0144] (12)

[0145] in It represents element-by-element multiplication. The weights corresponding to the positions with values of 0 in the mask matrix are set to zero to achieve pruning.

[0146] (2) Global sparse constraint checking,

[0147] Calculate the overall sparsity rate of the pruned model:

[0148] (13)

[0149] If the sparsity ratio does not reach the preset threshold (for example, 60%), return to step 4 to readjust the pruning strategy; otherwise, enter the fine-tuning stage.

[0150] (3) Fine-tune the recovery accuracy,

[0151] Optimizer settings: Adam optimizer is used, and the initial learning rate is set to , it decays to 0.5 times of the original value every 10 epochs.

[0152] Loss function: Use a multi-task loss function that combines classification cross entropy (CE) and bounding box regression loss (SmoothL1). The formula is as follows:

[0153] (14)

[0154] Training strategy:

[0155] 1) Freeze the sparse weight structure after pruning and only fine-tune the retained weights.

[0156] 2) Use a subset of the original training data (20%-30%) for quick fine-tuning, iterating for 5-10 epochs.

[0157] 3) Monitor the accuracy of the validation set. If the accuracy is restored to above 98% of the original model, terminate fine-tuning; otherwise, extend the training cycle.

[0158] (4) Pruning stability verification,

[0159] The fine-tuned model is subjected to robustness tests, including adding noise and occlusion simulation, to ensure that pruning does not introduce sensitive vulnerabilities.

[0160] Step 6: Compile and optimize the pruned model.

[0161] After going through the above steps, the DNN still relies on large deep learning frameworks such as PyTorch or TensorFlow for inference. However, deploying the pruned model on MCU-equipped smartphones or edge devices requires an AI compilation framework. This framework converts the deep learning model into efficient, low-level executable code that can run on different hardware platforms. This compilation process applies various optimization strategies to improve execution performance. The proposed pattern design approach can better facilitate the effectiveness of AI compilation frameworks and achieve better application in practice.

[0162] The intermediate representation of the pruned model is exported, and the TVM open-source compiler is used for graph optimization, operator fusion, memory optimization, and parallelism adjustment. An automatic tuning strategy (AutoScheduler) is used to adjust the kernel size, parallelization strategy, and data caching method for the target hardware platform to fully reduce inference latency. The final low-level code or executable file is compiled and supported with hot update or online deployment mechanisms to facilitate testing and optimization in various hardware environments.

[0163] The specific steps are as follows:

[0164] (1) Intermediate Representation (IR) derivation,

[0165] Convert the fine-tuned DNN model into a universal intermediate representation (ONNX format) to remove framework dependencies.

[0166] (2) TVM compiler graph optimization,

[0167] 1) Operator fusion:

[0168] Identify consecutive operations in the computation graph (such as Conv-BN-ReLU) and merge them into a single composite operator to reduce memory access overhead.

[0169] 2) Memory optimization:

[0170] Analyze the tensor lifecycle, reuse memory space, and reduce peak memory usage.

[0171] 3) Parallelism adjustment:

[0172] Automatically partition data parallel dimensions (such as batch size or number of channels) based on the number of parallel computing units of the target hardware (such as GPU or NPU).

[0173] (3) Automatic tuning (AutoTVM),

[0174] 1) Parameter search space definition:

[0175] Kernel Size: Sample candidate configurations within a legal range (e.g., 3×3, 5×5).

[0176] Tiling: Adjusts the data block division method to match the hardware cache hierarchy.

[0177] 2) Cost model-based tuning:

[0178] Use reinforcement learning or genetic algorithms to evaluate the inference latency and resource consumption of different configurations and select the Pareto optimal solution.

[0179] (4) Low-level code generation,

[0180] Compile the optimized computation graph into efficient low-level code (such as CUDA kernels or ARM assembly) for the target hardware platform.

[0181] Supports dynamic library (.so) or executable file (.bin) format output to adapt to the runtime environment of edge devices.

[0182] (5) Deployment and hot update mechanism,

[0183] 1) Online deployment: Push the compiled model to the computing unit of the autonomous vehicle through OTA (Over-the-Air) technology.

[0184] 2) Hot update:

[0185] A lightweight incremental update protocol is designed to replace only the weight blocks that have changed in the pruned model, reducing transmission overhead.

[0186] 3) Performance Monitoring:

[0187] In real-time collection of inference latency, memory usage and other indicators on the vehicle terminal, and feedback to the cloud for subsequent optimization and iteration,

[0188] The present invention generally converts a deep neural network model into a graph structure, integrates the neural network's topological structure information and pruning pattern information into the graph structure, and then automatically generates a pruning pattern that adapts to the network structure through a graph encoder that integrates the pruning pattern and a policy search based on deep reinforcement learning. The pattern is then applied to the original model. After multiple iterations, when the pruning strategy can meet the pruning constraints, the model is pruned and fine-tuned to obtain a pruned model that meets the accuracy requirements while significantly reducing the amount of computation. Finally, the model is optimized and deployed to the edge device through the compiler.

[0189] S5. Use the pruned model to deploy on the self-driving car.

[0190] Among them, the hardware platform selection:

[0191] Select a computing platform suitable for autonomous vehicles (using NVIDIA Jetson Orin NX 8GB as an example) and use its supported GPU acceleration to perform optimized inference.

[0192] Deployment process:

[0193] 1) Load the quantized DNN model onto the vehicle computing platform.

[0194] 2) Collect image data through sensors such as cameras and input the images into the pedestrian detection model.

[0195] 3) The model outputs the pedestrian detection box and category information and passes it to other parts of the perception module (such as data fusion and decision module).

[0196] test:

[0197] Verify the accuracy, real-time performance, and robustness of the model in a test environment (simulator or actual road test) to ensure that it can work stably in various complex scenarios (such as lighting changes, occlusions, dynamic pedestrians, etc.).

[0198] Continuous optimization and maintenance:

[0199] 1) Model update: The model can be updated regularly by collecting new data and performing online learning to ensure high accuracy of pedestrian detection in different environments.

[0200] 2) Performance monitoring: Performance monitoring is performed through the on-board computing platform to ensure that the model's inference time and resource consumption are within an acceptable range.

[0201] Example 2

[0202] This embodiment provides a pedestrian detection system based on deep learning automatic pruning. Figure 1-5 The present invention will be further described with reference to the following embodiments:

[0203] (1) System architecture,

[0204] The deep learning model automatic pattern pruning and deployment system for heterogeneous edge devices includes a pruning pattern library generation module 100, a graph construction and encoding module 101, a deep reinforcement learning pruning strategy search module 102, an execution pruning and fine-tuning module 103, and a compilation and deployment module 104. Figure 1 , each part is described in detail below:

[0205] Pruning pattern library module 100: Based on the statistical information of network parameters and the law of weight distribution, a variety of pruning patterns are formed, covering multi-level pruning strategies from high sparsity to low sparsity, aiming to strike a balance between pruning granularity and retaining key network structures.

[0206] Graph construction and encoding module 101: Convert the pre-trained DNN model into a graph structure representation, abstractly describe the relationship between each layer in the model, weight dependency and operation process; encode the constructed graph, extract global topology information and local structural features, and provide accurate environment status information for subsequent pruning strategy decisions, such as Figure 2 .

[0207] Deep reinforcement learning pruning strategy search module 102: Based on the graph embedding information obtained from graph construction and encoding, it conducts a strategy search for pruning modes and automatically determines which pruning mode to adopt for each layer. It uses deep reinforcement learning to continuously adjust the pruning strategy based on the feedback of the model performance after pruning (such as FLOPs reduction rate, accuracy retention, and inference latency) to achieve an automatic balance between compression effect and model performance, such as Figure 3 .

[0208] Execute pruning and fine-tuning module 103: According to the best pruning strategy obtained by deep reinforcement learning search, the predefined pruning mode is actually applied to the DNN weights to form a pruned network structure; the pruned network is fine-tuned to restore or further improve the model accuracy to ensure that the performance of the compressed model in actual tasks does not decrease, such as Figure 4 .

[0209] Compilation and deployment module 104: compiles and optimizes the pruned and fine-tuned deep neural network model at a low level to generate efficient execution code suitable for the target hardware (e.g., edge devices, embedded systems); enables fast inference of the model on the actual deployment platform, reduces latency, and fully utilizes the acceleration characteristics of the hardware, such as Figure 5 .

[0210] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, for a pedestrian detection method based on deep learning automatic pruning.

[0211] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded and executed by the processor to implement a pedestrian detection method based on deep learning automatic pruning.

[0212] The above are all preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A pedestrian detection method based on deep learning automatic pruning, characterized in that: include: Obtain pedestrian image data; Preprocessing the acquired pedestrian image data; Build a deep neural network DNN model; Prune the DNN model based on the pattern pruning strategy; Deploy the pruned DNN model to a self-driving car for pedestrian detection; The DNN model is pruned based on the pattern pruning strategy, including initializing a pruning pattern library and evaluating DNN model compression parameters; Model the DNN model as a graph structure and obtain graph embedding representation; Use DRL to generate pruning strategies based on graph embedding representations; After executing the pruning strategy on the DNN model, the compiler is used to perform compilation optimization to obtain the pruned and optimized DNN model; Initializing the pruning pattern library and evaluating the DNN model compression parameters includes designing a pattern library graded by sparsity rate in combination with the distribution law of weight parameters, wherein the sparsity rate of the patterns in the pattern library decreases evenly from 100% to 0%, setting a sparsity interval Δ to control the sparsity granularity, Δ=9, which generates 10 sparsity rates corresponding to 10 types of patterns, and calculating the compression parameters of the DNN model to determine whether the constraints are met. If the constraints are not met, the search for pruning strategies continues; The DNN model is modeled as a graph structure and a graph embedding representation is obtained, including extracting network topology information from the DNN model and integrating pruning pattern information into the graph structure, and obtaining the graph embedding representation of the model through the GNN method. Constructed graph Expressed as ,in is a node set, is an edge set, a subset represents the node associated with the weight tensor, is the set of input tensor nodes, is the set of output nodes, subset and Corresponding to the link The nodes in different layers of DNN are further divided into more fine-grained sets. The DRL is used to generate a pruning strategy based on the graph embedding representation, including the graph embedding obtained from the graph encoder As the environment state is passed to the Agent, the direct output obtained by the Agent is the pattern in the pattern library The probability distribution of , so its action space Continuous is continuous, and then by calculating the action value, the selected target mode Sampling is performed to obtain the pruned DNN model; wherein, the function As a multi-layer perceptron, it is responsible for extracting graph representation information from graph representation embedding. The activation function will The output is mapped to interval, determine the selection probability of each mode, that is, the action space, and finally, apply the classification distribution function from the probability distribution Sampling, assigning patterns to different weight tensors in the DNN, producing a library of newly selected patterns Used for pruning; After executing the pruning strategy on the DNN model, the compiler is used to perform compilation optimization, including performing a pruning operation on the model after the pruned DNN model meets the constraint conditions, replacing the model parameters with 0 according to the assigned pattern to obtain a pruned DNN model. At the same time, since the reasoning accuracy of the pruned DNN model will decrease, the pruned DNN model is fine-tuned to restore its reasoning accuracy.

2. The pedestrian detection method based on deep learning automatic pruning according to claim 1 is characterized in that: The method of using DRL to generate a pruning strategy based on the graph embedding representation also includes setting a reward function by combining the model compression metric and the model inference accuracy, which is expressed as: ， in is a learnable parameter. When it is greater than the set threshold, the agent is encouraged to adopt an aggressive compression strategy, giving priority to model compression rather than accuracy. On the contrary, when When it is less than the set value, the agent will be motivated to take a conservative approach.

3. The pedestrian detection method based on deep learning automatic pruning according to claim 2 is characterized in that: After executing the pruning strategy on the DNN model, the compiler is used for compilation optimization, which also includes using the TVM open source compiler to perform graph optimization, perform operator fusion, memory optimization, and parallelism adjustment; an automatic tuning strategy is used to adjust the kernel size, parallelization strategy, and data caching method according to the target hardware platform to fully reduce inference latency; Compilation generates the final low-level code or executable file, and supports hot update or online deployment mechanism, which facilitates testing and optimization in various hardware environments.

4. The pedestrian detection method based on deep learning automatic pruning according to claim 3 is characterized in that: The pruned model is deployed on the autonomous vehicle for pedestrian detection, including deploying the pruned DNN model in the perception module, which is responsible for extracting environmental information from sensor data and making detection results.

5. A pedestrian detection system based on deep learning automatic pruning, which executes the pedestrian detection method based on deep learning automatic pruning according to claim 1, characterized in that: include: The data acquisition module is configured to acquire pedestrian image data; A preprocessing module is configured to preprocess the acquired pedestrian image data; The model building module is configured to build a deep neural network DNN model; The pruning module is configured to prune the DNN model based on the pattern pruning strategy; The detection module is configured to deploy the pruned DNN model to the autonomous vehicle for pedestrian detection.

Images