A rail transit obstacle identification method and device based on model lightening

Abstract

The application relates to a track traffic obstacle identification method and device based on model lightening, which comprises the following steps: obtaining a track traffic obstacle identification model as a network model to be pruned, pre-training, updating a model parameter set and a cost function, and obtaining output activation values and gradient values of all filters; constructing an initial importance expression of the filter, multiplying the Taylor expansion with a Hessian matrix to construct a final importance expression of the filter; averaging the final importance according to a preset dimension, normalizing the filter weight after obtaining the filter weight, sorting the normalized filter weight, constructing a filter set to be pruned based on a preset weight; starting pruning from a first layer convolution layer of the network model to be pruned, updating the model parameter set until a preset pruning number is reached, and obtaining a lightened track traffic obstacle identification model; inputting real-time collected track traffic images, obtaining the position and size of obstacles on the track traffic, and completing track traffic obstacle identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model pruning, in particular to a rail transit obstacle identification method and device based on model lightweight. BACKGROUND

[0002] With the automation and intelligentization of rail transit technology, the real-time performance of front obstacle identification is crucial. If there is a delay in model detection of obstacles, the safe operation of trains cannot be guaranteed. At present, deep neural network models are mostly used for obstacle identification. With the increasing size and complexity of the model, although the performance of the model has been improved, the huge number of parameters and computing requirements have brought challenges to deployment and inference, making it difficult to deploy the model in resource-limited environments, especially on mobile and edge computing devices with limited computing resources and storage space, and the model computing rate is also limited, which leads to the inability to identify unknown obstacles in time during rail transit intelligent driving, endangering the safety of rail transit operation.

[0003] To solve this problem, researchers have begun to seek various model lightweight and compression techniques to reduce the size of the rail transit obstacle identification model, thereby accelerating the obstacle identification process. Among them, model pruning technology, as an effective model optimization method, has received widespread attention and research. The basic idea of model pruning is to remove redundant and unimportant weights or neurons in neural networks to reduce the size of the model and speed up inference while maintaining the performance of the model as much as possible. Model pruning technology can be divided into two categories: unstructured pruning and structured pruning. Unstructured pruning focuses on removing individual weight values, and the result is that the model becomes sparse, but the structure of the matrix remains unchanged. This method can effectively reduce the size of the model, but due to the introduction of sparsity, special hardware or software support may be needed to achieve acceleration. In contrast, structured pruning operates at the network level, removing entire neurons or convolution kernels to simplify the model. This method can directly reduce the computational load of the model and is easier to implement on general hardware for acceleration, but it may have a greater impact on the performance of the model.

[0004] As model pruning techniques continue to evolve, more and more pruning strategies and methods are being proposed. Among them, pruning methods based on weight importance are the most common and widely used. This method assesses the importance of each weight in the neural network to determine which weights can be safely pruned with minimal impact on model performance. Common weight importance assessment indicators include weight size, gradient information, etc. Taylor expansion pruning technique is one of the methods based on weight importance assessment. It uses Taylor series expansion to estimate the importance of weights by evaluating the impact of weight changes on model output. This method has received widespread attention due to its simplicity and effectiveness. However, as deep learning models become larger and larger, the computational complexity of the Taylor expansion pruning technique also increases. To overcome these challenges, many improvements and optimization methods have been proposed in recent years, which aim to improve the efficiency of the pruning process, reduce the required computational resources, and maintain or even improve the performance of the pruned model. Among them, some methods use approximation or low-rank decomposition to reduce the complexity of the Hessian matrix calculation; others introduce regularization terms or use prior knowledge to guide the pruning process to achieve better performance. Some commonly used network pruning techniques in recent years include:

[0005] Iterative Pruning and Fine-tuning: Iterative pruning is a progressive pruning method that alternates between multiple rounds of pruning and fine-tuning to gradually reduce the complexity of the model while maintaining its performance as much as possible. After each round of pruning, fine-tuning is used to restore the model's performance. This method usually achieves better pruning results because it takes into account the cumulative impact of pruning on model performance. Han et al. detailed this method in their 2015 paper and demonstrated its effectiveness through extensive experiments. Their research showed that iterative pruning and fine-tuning not only significantly reduced the size of deep neural networks, but also maintained their performance to some extent. Some other work has tried to address these issues by introducing more advanced pruning strategies and optimization techniques. For example, some research has proposed methods based on Bayesian optimization and reinforcement learning to automatically determine the optimal pruning strategy, while other work has explored the use of knowledge distillation and network pruning in combination to further improve model performance. However, this method is computationally expensive and requires multiple iterations of the training process, which not only consumes a lot of time but also requires a large amount of computational resources, which may not be acceptable in practical applications, especially for small businesses or individual developers with limited resources.

[0006] Automatic Pruning: Automatic pruning techniques aim to automatically determine the pruning strategy through algorithms, without the need for manually setting the pruning proportion or threshold. This is usually achieved by transforming the pruning problem into an optimization problem, such as network sparsity regularization. The advantage of automatic pruning is that it can adapt to the importance of different layers and allocate different degrees of pruning for different layers. Liu et al. proposed a network pruning method based on sparsity learning in their 2017 paper, which introduced L1 norm regularization during training to encourage the weight matrix to become more sparse. However, automatic pruning often relies on complex optimization algorithms and a large number of hyperparameter adjustments, which not only increases the implementation complexity, but also makes the model training more difficult and unstable.

[0007] Sparse Training: Sparse training is a method that directly introduces sparsity constraints during training, aiming to train a sparse model to reduce storage and computational requirements. This is usually achieved by adding a sparsity regularization term to the loss function. Unlike traditional pruning methods, sparse training considers model sparsity during training, rather than pruning after training. In recent years, some work has focused on developing specialized hardware to accelerate the inference of sparse neural networks, such as the Cerebras system and NVIDIA's Sparse Tensor Core. The effectiveness of sparse training often depends on the details of initialization and training, making it difficult to predict and control the performance and sparsity of the model.

[0008] L2 Norm Pruning: L2 norm pruning is a method of pruning based on weight size, which evaluates the importance of weights according to their L2 norm. The smaller the L2 norm of a weight, the less important it is considered, and thus it is more likely to be pruned. This method is simple and intuitive, but may overlook the actual role of weights in the network, leading to unnecessary performance loss. For example, some methods consider the positional information of weights, considering that weights close to the input or output layer may be more important; other methods combine other pruning criteria, such as weight gradient information, to more comprehensively evaluate the importance of weights. In addition, this method also depends on the selection of appropriate pruning thresholds, which requires a large number of experiments and adjustments, increasing the difficulty in practice.

[0009] In summary, the existing model lightening method has the problems of high calculation cost, complex pruning process, unstable training process, low model lightening efficiency and limitation of pruning when pruning based on weight, and cannot be directly applied to the field of rail transit obstacle identification. The existing method has high time cost and unstable identification result, and is not suitable for obstacle identification in rail transit with high speed; rail transit obstacle identification needs a more efficient obstacle identification strategy to avoid collision between rail transit vehicles and obstacles due to failure to identify obstacles in time. SUMMARY

[0010] To this end, the technical problem to be solved by the present application is to overcome the problems of complex model pruning process, high calculation cost, low model lightening efficiency and limitation of pruning in the prior art, resulting in low rail transit obstacle identification efficiency and inaccurate identification.

[0011] To solve the above technical problems, the present application provides a rail transit obstacle identification method based on model lightening, comprising:

[0012] Obtain a rail transit obstacle identification model as a network model to be pruned;

[0013] Obtain the network model to be pruned and its initial parameter set, pre-train it using a rail transit image sample data set, update the model parameter set and the cost function, and obtain the output activation value and gradient value of all filters in each convolution layer of the pre-trained network model to be pruned; the gradient is the first derivative of the cost function with respect to the output activation value;

[0014] For each filter, based on the Oracle_Abs criterion, calculate the cost function change value after each filter is pruned, construct an initial importance expression, and obtain the initial importance of each filter;

[0015] After Taylor expansion of the initial importance expression, multiply it by the Hessian matrix, based on the first derivative of the cost function with respect to the output activation value and the first partial derivative of the gradient, construct the final importance expression of each filter, and obtain the final importance of each filter; the final importance is a four-dimensional vector;

[0016] Average the final importance according to the preset dimension, obtain the filter weight, and normalize it to obtain the normalized filter weight;

[0017] Sort all normalized filter weights of filters in the model to be pruned from small to large, obtain filters whose normalized filter weights are less than a preset weight, and construct a set of filters to be pruned;

[0018] Prune the filter existing in the filter set to be pruned from a first convolutional layer of the network model to be pruned, and update the model parameter set until the number of pruned filters in the network model to be pruned reaches a preset pruning number, to obtain a target model parameter set;

[0019] Based on the target model parameter set, a lightweight network model is obtained, and a lightweight rail transit obstacle identification model is obtained.

[0020] Real-time rail transit images are collected and input into the lightweight rail transit obstacle identification model to obtain the position and size of obstacles on the rail transit, and rail transit obstacle identification is completed.

[0021] In an embodiment of the present application, the initial importance expression is represented as:

[0022]

[0023] Wherein, the initial importance |ΔC(h i ) represents the change value of the cost function after pruning the filter i; h i represents the output activation value of the i th filter; represents the cost function of the network model to be pruned after pruning the i th filter, represents the cost function of the network model to be pruned without pruning the filter, D represents a training sample set including an input set X and a correct label set Y, represented as (X={x0, x1, …, x i ,…,x N}, Y={y0, y1, …, y i ,…,y N}), and N represents the total number of filters in the network model to be pruned; the expression of the cost function C is represents the probability of predicting the correct label y i under the given input x i and the model parameter set after pruning the i th filter.

[0024] In an embodiment of the present application, after Taylor expansion of the initial importance expression, the final importance expression of each filter is obtained by multiplying the initial importance expression by the Hessian matrix, including:

[0025] The initial importance expression is represented by Taylor expansion, represented as:

[0026]

[0027] The final importance expression is represented as:

[0028]

[0029] where the gradient of the filter represents the first derivative of the cost function C with respect to the i-th filter output activation value, R1(h i = 0) is the residual term of the Taylor expansion; represents the first partial derivative of the cost function C with respect to the gradient .

[0030] In an embodiment of the present application, the obtaining of the pre-trained network model to be pruned comprises:

[0031] obtaining an initial cost function value of the network model to be pruned on the training sample set D under the initial parameter set W, denoted as C(D|W);

[0032] obtaining an updated cost function value of the network model to be pruned on the training sample set D under the updated parameter set W', denoted as C(D|W');

[0033] constructing an optimization objective function with the difference between the updated cost function value and the initial cost function value being minimum, obtaining the parameter set when the optimization objective function converges, and obtaining the pre-trained network model to be pruned;

[0034] The optimization objective function is denoted as:

[0035] where H(f) is a Hessian matrix, and the expression is:

[0036] ||W'||0represents the number of non-zero parameters in the model parameters of the network model to be pruned, and B represents a preset non-zero parameter threshold.

[0037] In an embodiment of the present application, the obtaining of the filter weight by averaging the final importance according to the preset dimension comprises:

[0038] The final importance is denoted as a four-dimensional vector, including batch size J, channel number K, height L and width M, denoted as (J, K, L, M).

[0039] The final importance is averaged according to the channel number K as the preset dimension to obtain the filter weight mean K , denoted as:

[0040]

[0041] where T j,k,l,m represents the Taylor expansion formula of the cost function C; j = 0, 1, 2,..., J-1, k = 0, 1, 2,..., K-1, l = 0, 1, 2,..., L-1, and m = 0, 1, 2,..., M-1.

[0042] In an embodiment of the present application, the obtaining of the normalized filter weight comprises:

[0043] calculating the average value of the filter weights of all filters in each convolutional layer to obtain a layer average value;

[0044] for each filter in each convolutional layer, calculating the quotient of the filter weight of each filter and the layer average value of the convolutional layer to which the filter belongs as the normalized filter weight corresponding to each filter.

[0045] In an embodiment of the present application, the pruning of the filters existing in the set of filters to be pruned comprises:

[0046] obtaining a current filter to be pruned and a current convolutional layer to which the current filter to be pruned belongs;

[0047] creating a new convolutional layer and obtaining the weight matrix of the current convolutional layer after the current filter to be pruned is pruned and copying the weight matrix to the new convolutional layer;

[0048] updating the input channel number and the weight matrix of the next convolutional layer of the current convolutional layer to obtain an updated next convolutional layer;

[0049] replacing the current convolutional layer and the next convolutional layer of the current convolutional layer in the network model to be pruned with the new convolutional layer and the updated next convolutional layer to complete the pruning of the current filter to be pruned.

[0050] In an embodiment of the present application, after the pruning of the filters in the set of filters to be pruned in the network model to be pruned each time, the method further comprises fine-tuning the current pruned network model to be pruned by using the stochastic gradient descent method, which comprises:

[0051] constructing a training data set to train the current pruned network model to be pruned;

[0052] updating the model parameter set by using the gradient descent method, optimizing the loss function, until the loss function value converges, obtaining the current model parameter set, and completing the fine-tuning of the current pruned network model to be pruned.

[0053] In an embodiment of the present application, the loss function is expressed as:

[0054]

[0055] wherein, Loss total represents the loss function value, Let X represent the cost function of the network model to be pruned without a pruning filter, and D represent the training sample set including the input set X and the correct label set Y, denoted as (X={x0,x1,…,x...}). i ,…,x N},Y={y0,y1,…,y i ,…,y N}), where N represents the total number of filters in the network model to be pruned; h i Let represent the output activation value of the i-th filter; the expression for the cost function C is: This indicates that given input x i and the set of model parameters after pruning the i-th filter Below, predict the correct label y. i The probability of; W represents the regularization coefficient. i 2 This represents the square of the total weights of the network model to be pruned after pruning the i-th filter.

[0056] This invention also provides a model-based lightweight obstacle recognition device for rail transit, comprising:

[0057] The model pre-training module is used to acquire a rail transit obstacle recognition model as a network model to be pruned; it acquires the network model to be pruned and its initial parameter set, pre-trains it using a rail transit image sample dataset, updates the model parameter set and cost function, and acquires the output activation values ​​and gradient values ​​of all filters in each convolutional layer of the pre-trained network model to be pruned; the gradient is the first derivative of the cost function with respect to the output activation value.

[0058] The importance acquisition module is used to calculate the change in cost function of each filter after pruning based on the Oracle_Abs criterion, construct an initial importance expression, and obtain the initial importance of each filter; after Taylor expansion of the initial importance expression, it is multiplied with the Hessian matrix, and based on the first derivative of the cost function with respect to the output activation value and the first partial derivative with respect to the gradient, a final importance expression is constructed for each filter, and the final importance of each filter is obtained; the final importance is a four-dimensional vector;

[0059] The filter to be pruned module is used to average the final importance according to a preset dimension to obtain filter weights, and then normalize them to obtain normalized filter weights; the normalized filter weights of all filters in the model to be pruned are sorted from small to large to obtain all filters whose normalized filter weights are less than the preset weights, thus forming a set of filters to be pruned.

[0060] The pruning module is configured to prune filters existing in a filter set to be pruned from a first layer convolutional layer of a network model to be pruned, and update a model parameter set until a number of pruned filters in the network model to be pruned reaches a preset pruning number, and obtain a target model parameter set; and based on the target model parameter set, a lightweight network model is obtained, and a lightweight rail transit obstacle identification model is obtained.

[0061] The obstacle identification module is configured to collect rail transit images in real time, input the lightweight rail transit obstacle identification model, obtain positions and sizes of obstacles on the rail transit, and complete rail transit obstacle identification.

[0062] The above technical solution of the present application has the following advantages compared with the prior art:

[0063] The model-based lightweight rail transit obstacle identification method provided by the present application obtains the final importance of the filter by performing Taylor expansion on the initial importance expression based on the cost function and adding a Hessian matrix for fusion to obtain deep information of the filter itself and surrounding filters. The present application introduces the Hessian matrix and further introduces the second-order derivative information. By considering the second-order influence of the final importance of the filter on the model output, the importance of the weight is more comprehensively evaluated, the limitations of L2 norm pruning are overcome, unnecessary performance loss is reduced, the final importance of the filter is more accurately evaluated, a more stable pruning standard is provided, the number of hyperparameters is reduced, the pruning process is simplified, efficient pruning can be achieved in fewer iteration rounds, the computational cost is significantly reduced, and the stability and effect of automatic pruning are improved. The more stable pruning standard based on the Hessian matrix enhances the predictability of sparse training, ensures that the model maintains sparsity while maintaining high performance. The present application combines Taylor expansion and Hessian matrix to accurately evaluate the importance of the filter, thereby providing a more high-performance rail transit obstacle identification model, and reducing the model parameters, thereby speeding up the obstacle identification efficiency.

[0064] The present application reduces the internal covariant bias in the updating process of the network model to be pruned by layer normalization of the filter weights, makes the gradient flow more stable, and speeds up the convergence speed. Layer normalization is independent of batch size and is suitable for various batch size scenarios. The normalized weights also help the model to better generalize to new data sets, reduce the risk of overfitting, and improve the applicability of the model lightweight method of the present application, which is suitable for various rail transit obstacle identification models containing convolutional layers.

[0065] The application allows the user to prune any convolutional layer in the model and the specific filter in the layer after obtaining the set of filters to be pruned, and after pruning, the application reconstructs the weights and biases, adopts a modular layer replacement strategy, adjusts the weights and biases of the current pruned convolutional layer, and adaptively modifies the input channel number of the subsequent convolutional layer, thereby reducing the number of hyperparameters, simplifying the pruning process, significantly reducing the calculation cost of the rail transit obstacle recognition model, and thereby speeding up the recognition process.

[0066] The application uses iterative pruning, removes a certain proportion of filters at each iteration, and fine-tunes the remaining network to be pruned; uses cumulative pruning effect processing technology to constrain the cumulative changes of the current pruned network structure after each pruning, ensuring the correct index of subsequent filters; uses dynamic pruning adjustment technology to allow dynamic adjustment of the number of filters to be pruned at each iteration in the pruning process, improving the stability and effect of automatic pruning, ensuring the model performance of the lightweight rail transit obstacle recognition model, and thereby ensuring the accuracy of obstacle recognition. BRIEF DESCRIPTION OF DRAWINGS

[0067] In order to make the content of the application easier to be clearly understood, the application will be further described in detail below according to specific embodiments of the application and in conjunction with the drawings, in which

[0068] Figure 1 is a step flow chart of the rail transit obstacle recognition method based on model lightweight pruning provided by the application;

[0069] Figure 2 is a comparison diagram of inference accuracy of the model lightweight method and the traditional pruning method provided by the application;

[0070] Figure 3 is a comparison diagram of inference speed of the model lightweight method and the traditional pruning method provided by the application. DETAILED DESCRIPTION

[0071] The application will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the application and implement it, but the embodiments are not limiting the application.

[0072] Referring to Figure 1 the step flow chart of the rail transit obstacle recognition method based on model lightweight pruning provided by the application, the specific recognition steps include:

[0073] S101: Obtain a rail transit obstacle recognition model as a network model to be pruned;

[0074] S102: Obtain a network model to be pruned and its initial parameter set, pre-train it using a rail transit image sample dataset, update the model parameter set and the cost function, and obtain the output activation value and gradient value of all filters in each convolution layer of the pre-trained network model to be pruned; the gradient is the first derivative of the cost function with respect to the output activation value;

[0075] S103: For each filter, based on the Oracle_Abs criterion, calculate the cost function change value after each filter is pruned, construct an initial importance expression, and obtain the initial importance of each filter;

[0076] The initial importance expression is represented as: Wherein, the initial importance |ΔC(h i )| represents the cost function change value after pruning the filter i; h i represents the output activation value of the i-th filter; represents the cost function of the network model to be pruned after pruning the i-th filter, represents the cost function of the network model to be pruned without pruning the filter, D represents a training sample set including an input set X and a correct label set Y, represented as (X={x0,x1,…,x i ,…,x N},Y={y0,y1,…,y i ,…,y N}), and N represents the total number of filters in the network model to be pruned; the expression of the cost function C is represents the probability of predicting the correct label y i under the given input x and the model parameter set i after pruning the i-th filter.

[0077] S104: After Taylor expansion of the initial importance expression, multiply it by the Hessian matrix, construct the final importance expression of each filter based on the first derivative of the cost function with respect to the output activation value and the first partial derivative of the gradient, and obtain the final importance of each filter, including:

[0078] Taylor expansion of the initial importance expression:

[0079] The final importance expression is represented as:

[0080]

[0081] Wherein, the gradient of the filter represents the first derivative of the cost function C with respect to the output activation value of the i-th filter, and R1(hi =0) is a residual remainder of Taylor expansion; denotes the first-order partial derivative of the cost function C with respect to the gradient ; the final importance is a four-dimensional vector;

[0082] S105: average the final importance according to the preset dimension, obtain the filter weight, and normalize to obtain the normalized filter weight;

[0083] S106: sort the normalized filter weights of all filters in the to-be-pruned model from small to large, obtain the filters whose normalized filter weights are less than the preset weight, and constitute a to-be-pruned filter set;

[0084] S107: starting from the first convolutional layer of the to-be-pruned network model, prune the filters existing in the to-be-pruned filter set, and update the model parameter set until the number of pruned filters in the to-be-pruned network model reaches the preset pruning number, and obtain a target model parameter set;

[0085] S108: based on the target model parameter set, obtain a light-weight network model to obtain a light-weight rail transit obstacle identification model;

[0086] S109: real-time acquisition of rail transit images, input into the light-weight rail transit obstacle identification model, obtain the position and size of the obstacles on the rail transit, and complete the rail transit obstacle identification.

[0087] The model lightening-based rail transit obstacle identification method provided by the application, when obtaining the final importance of the filter, after Taylor expansion of the initial importance expression based on the cost function, the Hessian matrix is added for fusion to obtain the deep information of the filter itself and the surrounding filters. The application introduces the Hessian matrix and further introduces the second-order derivative information. By considering the second-order influence of the final importance of the filter on the model output, the importance of the weight is more comprehensively evaluated, the limitations of L2 norm pruning are overcome, unnecessary performance loss is reduced, the final importance of the filter can be more accurately evaluated, a more stable pruning standard is provided, the number of hyperparameters is reduced, the pruning process is simplified, efficient pruning can be realized in fewer iteration rounds, the calculation cost is significantly reduced, and the stability and effect of automatic pruning are improved. The more stable pruning standard based on the Hessian matrix enhances the predictability of sparse training, ensures that the model can maintain sparsity while maintaining high performance. The application combines Taylor expansion and Hessian matrix to accurately evaluate the importance of the filter, thereby providing a more high-performance rail transit obstacle identification model, and reducing the model parameters, thereby accelerating the obstacle identification efficiency.

[0088] Specifically, in the embodiment of the application, the obtaining of the pre-trained network model to be pruned comprises:

[0089] The initial cost function value of the network model to be pruned under the initial parameter set W for the training sample set D is denoted as C(D|W).

[0090] The updated cost function value of the network model to be pruned under the updated parameter set W' for the training sample set D is denoted as C(D|W').

[0091] The optimization objective function is constructed by taking the difference between the updated cost function value and the initial cost function value as the minimum, the parameter set when the optimization objective function converges is obtained, and the pre-trained network model to be pruned is obtained.

[0092] The optimization objective function is denoted as:

[0093] wherein H(f) is a Hessian matrix, and the expression is:

[0094] ||W'||0 represents the number of non-zero parameters in the model parameters of the network model to be pruned, and B represents a preset non-zero parameter threshold.

[0095] Layer Normalization is a common technique in deep learning, mainly used to improve the training process of neural networks. Traditional normalization techniques, such as Batch Normalization, mainly target the distribution of input data, while Layer Normalization targets the activation distribution within a single layer. This technique involves normalizing activations in each layer of the network to keep the activation distribution stable during training. In step S105 of the embodiment of the application, the process of obtaining the normalization filter weight based on the final importance comprises:

[0096] S105-1: The final importance is represented as a four-dimensional vector, including batch size J, channel number K, height L and width M, denoted as (J, K, L, M).

[0097] S105-2: The final importance is averaged with the channel number K as the preset dimension to obtain the filter weight mean K , denoted as:

[0098]

[0099] wherein T j,k,l,m represents the Taylor expansion formula of the cost function C; j=0,1,2,…,J-1, k=0,1,2,…,K-1, l=0,1,2,…,L-1, m=0,1,2,…,M-1.

[0100] S105-3: Calculate the average value of the filter weights of all filters in each convolutional layer to obtain a layer average value;

[0101] S105-4: For each filter in each convolutional layer, calculate the quotient of the filter weight of each filter and the layer average value of the convolutional layer to which the filter belongs as the normalized filter weight corresponding to each filter.

[0102] The present application reduces the internal covariate shift in the network model pruning process, makes the gradient flow more stable, and speeds up the convergence speed by layer normalization of the filter weight to obtain the normalized filter weight. The layer normalization is not dependent on the batch size and is suitable for various batch size scenarios. The normalized weight also helps the model to better generalize to new data sets, reduces the risk of overfitting, improves the applicability of the model lightweight method, and is suitable for various rail transit obstacle recognition models containing convolutional layers.

[0103] The present application realizes a layer normalization variant, which uses L2 norm to normalize the weights within the layer. This normalization process can not only be applied to activation values, but also be used to adjust the weights of neural network layers. Specifically, the technology is realized by the following steps: first, taking the absolute value of the weight, which can eliminate the influence of the weight sign and focus on the size of the weight; then, by dividing by the L2 norm, the length of each weight vector is ensured to be 1, which is the key to standardization and can reduce the inconsistency caused by different weight scales and improve the numerical stability of the model; finally, by transferring the data to the CPU, the normalized weights are compatible with other non-GPU environments, facilitating cross-platform operation and calculation. Through layer normalization, the present application embodiment can reduce the internal covariate shift in the training process, make the gradient flow more stable, and speed up the convergence speed. The normalized weight helps the model to better generalize to new data sets, increases the model generalization ability, and reduces the risk of overfitting. Unlike batch normalization, layer normalization does not depend on the batch size, so it can work effectively on small batch data and is suitable for more training scenarios, such as online learning or varying batch size.

[0104] In the present application embodiment, the filters existing in the set of filters to be pruned are pruned, including:

[0105] Obtaining the current filter to be pruned and the current convolutional layer to which it belongs;

[0106] Creating a new convolutional layer and obtaining the weight matrix of the current convolutional layer after the current filter to be pruned is pruned, and copying it to the new convolutional layer;

[0107] The input channel number of the next convolutional layer of the current convolutional layer and the weight matrix are updated to obtain an updated next convolutional layer.

[0108] The current convolutional layer and the next convolutional layer of the current convolutional layer in the network model to be pruned are replaced by the new convolutional layer and the updated next convolutional layer, and the pruning of the current filter to be pruned is completed.

[0109] Specifically, after the current filter to be pruned is pruned, the next filter is obtained from the set of filters to be pruned as a new current filter to be pruned, and the above operations are repeated for pruning until the number of pruned filters reaches a preset pruning threshold, and the pruning is ended.

[0110] After obtaining the set of filters to be pruned, the present application starts pruning from the first convolutional layer of the network model to be pruned, allowing the user to prune any convolutional layer in the model and a specific filter in the layer; and after pruning, the present application reconstructs the weights and biases, adopts a modular layer replacement strategy, adjusts the weights and biases of the current pruned convolutional layer, and adaptively modifies the input channel number of the next convolutional layer following it, thereby reducing the number of hyperparameters, simplifying the pruning process, and significantly reducing the computational cost of the rail transit obstacle recognition model, thereby speeding up the recognition process.

[0111] Overfitting refers to the situation where a model may lose its ability to generalize because it has learned the training data "too well". When a model overfits, it may perform very well on the training set, but its performance on unseen data will drop dramatically. To prevent overfitting, regularization techniques are often used to constrain the complexity of the model. L2 regularization, also known as weight decay, is a form of regularization. In this regularization, an additional term is added to the loss function, which is proportional to the square of the model weights. In other words, in addition to minimizing the original loss function (such as mean squared error), the model training process also tries to minimize the sum of the squares of the weights. L2 regularization encourages the model to learn smaller weights, which usually means that the model has lower complexity and can improve its ability to generalize to new data. Therefore, in the embodiments of the present application, after pruning a filter in the set of filters to be pruned in the network model to be pruned, the current pruned network model to be pruned is fine-tuned using the stochastic gradient descent method, which includes:

[0112] A training data set is constructed, and the current pruned network model to be pruned is trained.

[0113] The model parameter set is updated using the gradient descent method to optimize the loss function until the loss function value converges, and the current model parameter set is obtained to complete the fine-tuning of the current pruned network model to be pruned.

[0114] wherein the loss function is denoted as Loss total denotes the loss function value, denotes the cost function of the network model to be pruned without pruning the filter, D denotes a training sample set including an input set X and a correct label set Y, denoted as (X={x0, x1, …, x i ,…,x N}, Y={y0, y1, …, y i ,…,y N}), and N denotes the total number of filters in the network model to be pruned; h i denotes the output activation value of the i-th filter; the expression of the cost function C is denotes the probability of predicting the correct label y i after pruning the i-th filter under the given input x and the model parameter set i . denotes a regularization coefficient, which is a hyperparameter, and is used to adjust the degree of weight decay; W i 2 denotes the square of the total weight of the network model to be pruned after pruning the i-th filter, and is used to punish large weight values.

[0115] The present application uses iterative pruning, removes a certain proportion of filters at each iteration, and fine-tunes the remaining network to be pruned; uses cumulative pruning effect processing technology to constrain the cumulative changes of the current pruned network structure after each pruning, ensuring the correct index of subsequent filters; uses dynamic pruning adjustment technology to allow dynamic adjustment of the number of filters to be removed at each iteration in the pruning process, improving the stability and effect of automatic pruning, ensuring the model performance of the lightweight rail transit obstacle recognition model, thereby ensuring the accuracy of obstacle recognition.

[0116] Based on the above embodiment, the present application provides a model lightweight method applied in a rail transit obstacle recognition model, and the specific lightweight steps include network pre-training and fine-tuning, and pruning and fine-tuning iteration.

[0117] Network pre-training and fine-tuning collectively process the filters in the convolutional layers of the convolutional neural network. The core idea is to obtain the activation values of all filters in each convolutional layer of the convolutional neural network and the backpropagation high-order gradient through a round of training, then confirm the importance ranking of the filters based on the Taylor pruning judgment standard of the Hessian matrix fusion, and prune the filters from low to high based on the importance. After pruning, fine-tune and train on the target task until convergence to ensure that the network can properly complete its designed task; repeat the pruning and fine-tuning multiple times until the preset pruning upper limit is reached.

[0118] The present application is based on the Oracle Pruning idea, which represents an ideal network pruning method, the core goal of which is to achieve model compression and inference speed improvement while minimizing the difference in model accuracy before and after pruning. The original design of this method is based on a detailed and accurate assessment of the importance of network parameters, ensuring that the parameters essential to the performance of the model are retained to the maximum extent during the pruning process, while those with less impact on the final output are removed. In the implementation process of Oracle Pruning, the network is first fully pre-trained to ensure that the model has reached a state of convergence on the target task; pre-training not only provides a stable and optimized starting point for pruning, but also ensures the maximum preservation of performance during the pruning process. After pre-training, enter the iterative cycle stage of pruning and fine-tuning. In each iteration, the pruning stage starts with evaluating the importance of each non-zero parameter in the model. This evaluation is done by temporarily setting each parameter to zero and observing the impact on the overall loss of the model. According to the design of Oracle Pruning, the present application uses two different methods to quantify parameter importance, including Oracle-Loss and Oracle-Abs. Oracle-Loss uses the signed value of the loss function difference as an importance indicator, while Oracle-Abs uses the absolute value of the loss function difference. Both methods aim to identify and eliminate parameters with minimal impact on model performance, but they differ in their strategies for handling loss reduction; Oracle-Loss tends to encourage the removal of parameters that may improve model performance, while Oracle-Abs is cautious about any pruning operation that may cause loss changes, regardless of whether the change is positive or negative. That is, Oracle-Abs considers any change in the loss function as a penalty, regardless of whether the loss is increasing or decreasing.

[0119] After determining the parameters to be pruned, they are removed from the model and enter the fine-tuning stage. Fine-tuning is a critical step because it allows the model to adjust the remaining parameters after pruning to compensate for potential performance degradation due to the absence of parameters.

[0120] During the fine-tuning stage, the model continues to train on the training data, and the parameters are updated through gradient descent to optimize the loss function. In this way, even after pruning some parameters, the model can gradually recover its performance, ensuring that the impact of the pruning process on the final performance is minimized.

[0121] This pruning and fine-tuning iteration process will continue until a balance point between the preset accuracy and model size is reached. In the embodiments of the present application, the balance point stops when 50% of the weights in all convolutional layers are subtracted, focusing on achieving higher model compression rate and faster inference speed. That is, in the embodiments of the present application, the iteration process continues until a predetermined trade-off point between the accuracy and the pruning target (the number of floating-point operations or the memory utilization) is reached.

[0122] Specifically, in the embodiments of the present application, the parameter pruning problem of the neural network is regarded as an optimization problem, and the goal is to find a weight matrix in which the number of non-zero elements is limited and the absolute value of the loss function change is minimized.

[0123] S201: In this embodiment, the initial importance expression of the filter is obtained based on the Oracle_Abs criterion and the loss function change value after each filter is pruned, which is represented as:

[0124]

[0125] wherein, |ΔC(h i ) | represents the influence value of pruning the current filter i on the loss function; and the right side of the equation represents the influence of pruning the i-th filter on the original loss function. i represents the output activation value of the i-th filter; represents the cost function of the network model to be pruned after pruning the i-th filter, represents the cost function of the network model to be pruned without pruning the filter, D represents a training sample set including an input set X and a correct label set Y, which is represented as (X = {x0, x1, …, x i ,…,x N}, Y = {y0, y1, …, y i ,…,y N}), and N represents the total number of filters in the network model to be pruned; the expression of the cost function C is represents the probability of predicting the correct label y i under the given input x i and the model parameter set after pruning the i-th filter.

[0126] S202: Taylor formula-based expansion is used to reduce the calculation amount. The Taylor formula is represented as:

[0127]

[0128] wherein, is a series term, and R p (x) is a remainder term.

[0129] In this embodiment, during pruning, all filters in the convolutional layers are traversed to evaluate their impact after pruning. At this point, the importance of the i-th filter selected in this embodiment to the resulting loss function can be expressed using a Taylor expansion as follows:

[0130]

[0131] in, Let be the first and second terms of the series, and let the third term of the series involve (xa). p That is, the higher-order squares of the activation are discarded due to the large amount of computation required. The remaining terms are also discarded.

[0132] S203: Multiplying the Taylor expansion formula by the Hessian matrix, the final importance formula is expressed as:

[0133]

[0134] The addition of the Hessian matrix allows for the extraction of higher-order information from the filter with relatively low computational cost, revealing the filter's implicit relationships with other filters. The Hessian matrix formula is expressed as:

[0135]

[0136] The Hessian matrix, a matrix composed of the second-order partial derivatives of a multivariable function, provides an important tool for optimizing and analyzing the local properties of functions. In the technical field, particularly in machine learning and deep learning, the Hessian matrix can describe the curvature of a function, helping to determine its concavity or convexity and locate saddle points, thereby optimizing algorithm performance. Its eigenvalues ​​and eigenvectors reveal the behavior of the function at a certain point, such as local minima, maxima, or saddle points. This embodiment utilizes these properties of the Hessian matrix to improve and optimize related technical solutions, making them more stable and efficient. Each small unit in the matrix represents a higher-order partial derivative of the original filter in one direction.

[0137] The Hessian matrix H(f) is a square matrix, and (C(D|W')-C(D|W)) is a one-dimensional vector with the same rank as the one-dimensional square matrix. Multiplying them together yields a vector that is exactly the same as the original formula. The new judgment criteria with the same format are expressed as follows:

[0138]

[0139] This includes the implicit information of the filter, and pruning ends when the total weights are reduced by 50%. After simplification, the final importance expression is obtained.

[0140] S204: The ranking of the importance of filters in a convolutional layer is a four-dimensional vector, specifically represented as (J,K,L,M);

[0141] Batch size J refers to the number of data samples passed through the network at a time. In deep learning, multiple samples are usually packed into a batch for processing together in order to effectively utilize hardware resources and improve computational efficiency. Each sample in this batch will produce a set of filter activations.

[0142] Channel number K refers to the depth of the input data, for example, a color image has three color channels RGB. In a convolutional layer, each filter will also correspond to multiple channels, which correspond to the output of the previous layer or the color channels of the original input image.

[0143] Height L refers to the size of the filter activation in the vertical direction, or the number of rows of the two-dimensional feature map.

[0144] Width M refers to the size of the filter activation in the horizontal direction, or the number of columns of the two-dimensional feature map.

[0145] Therefore, the importance weight of the filter is a four-dimensional array, which contains the two-dimensional feature map generated by each filter for each channel of each sample in the batch. Each filter slides over the input data, calculates the dot product of the points in its coverage area, and produces a two-dimensional activation map (feature map). These activation maps are superimposed together to form a complete four-dimensional activation array for each channel of each filter. It is a high-order expansion of the two-dimensional array, which averages the values of other dimensions on the C dimension to obtain a scalar representing the filter weight, denoted as:

[0146]

[0147] Finally, the scalar importance weight of each filter is stored in a dictionary, and the key of the key-value pair in the dictionary is the convolutional layer of each layer of the model, and the value is the cumulative weight of all filters in the convolutional layer. After each pruning fine-tuning, the dictionary is cleared and ready for the next stage of pruning.

[0148] S205: Since finding the optimal subset of parameters is a combinatorial optimization problem, the calculation cost is very high for larger networks, so the embodiment of the present application uses a greedy algorithm for approximate solution. Starting from the complete set of model parameters, iteratively identify and remove the least important parameters until the pruning target is met.

[0149] The lightweight method provided by the embodiment is based on the improved Taylor pruning model of the Hessian matrix, aiming to solve a series of problems in existing pruning techniques and improve the efficiency and effectiveness of model pruning. Specifically, the technical problems to be solved by the present application include:

[0150] Computational cost problem: Although the existing iterative pruning and fine-tuning method can gradually optimize the pruning results and ensure the model performance, this method requires multiple rounds of training and fine-tuning, which has a very high computational cost. The Taylor pruning method based on the Hessian matrix can more accurately evaluate the importance of weights by introducing second-order derivative information, thereby achieving efficient pruning in fewer iterations and significantly reducing the computational cost.

[0151] Complexity problem of automatic pruning: Traditional automatic pruning methods rely on complex optimization algorithms and a large number of hyperparameter adjustments, which are complex to implement and unstable in the training process. The embodiments of the present invention provide a more intuitive and stable pruning standard by introducing the Hessian matrix, reducing the number of hyperparameters, simplifying the pruning process, and improving the stability and effectiveness of automatic pruning.

[0152] Unpredictability problem of sparse training: The effectiveness of sparse training often depends on multiple factors, making it difficult to predict and control model performance and sparsity. The embodiments of the present invention provide a more accurate weight importance evaluation method using the Hessian matrix, enhancing the predictability of sparse training and ensuring that the model maintains high performance while maintaining sparsity.

[0153] Limitation problem of L2 norm pruning: Although L2 norm pruning is simple and intuitive, it may ignore the actual role of weights, leading to performance loss. The embodiments of the present invention more comprehensively evaluate the importance of weights by considering the second-order impact of weights on model output, overcoming the limitations of L2 norm pruning and reducing unnecessary performance loss.

[0154] Direct pruning problem of knowledge distillation: Although knowledge distillation is an effective model lightweight method, it does not involve weight pruning itself. The embodiments of the present invention combine the improved Taylor pruning method of the Hessian matrix with knowledge distillation, enabling direct pruning of model weights while maintaining model lightweight, thereby improving the efficiency of model compression.

[0155] In the embodiments of the present invention, the rail transit obstacle recognition model is a network model with convolutional layers, including LeNet-5, AlexNet, VGGNet, and ResNet.

[0156] Based on the above embodiment, in an embodiment of the present application, a pruning experiment is performed on a pre-trained VGG16 model. After determining the filters to be removed, first determine the next convolutional layer, and try to find the next convolutional layer after the given convolutional layer. This is because pruning the filters of the current layer will affect the input channels of the next layer; then create a new convolutional layer with one less output channel. Because one filter is to be pruned, weight migration is then performed, copying the weights of the old convolutional layer to the new convolutional layer, but excluding the weights of the pruned filter. Then process the next convolutional layer, and if there is a next convolutional layer, create a new convolutional layer for it with one less input channel. The weights are migrated in a similar manner, but excluding the weights of the input channels corresponding to the pruned filters. Finally, update the model, replacing the old convolutional layer in the original model with the new convolutional layer according to whether there is a next convolutional layer. If the last convolutional layer is pruned, the linear layer immediately after it, i.e., the first linear layer of the classifier, needs to be adjusted. This involves creating a new linear layer with slightly fewer input features and migrating the weights. The function returns the updated model, which contains the pruned structure.

[0157] Based on the above embodiment, in an embodiment of the present application, the model lightweight method includes filter importance judgment and importance-based pruning operation, and the specific model lightweight steps include:

[0158] S301: In an embodiment of the present application, a pre-trained improved model ModifiedVGG16Model based on the VGG16 architecture is used as a pre-trained network model to be pruned;

[0159] S302: Before starting the pruning process, perform another training to obtain the activation values and gradients of the filters output by each convolutional layer in the network model to be pruned. Unlike the classical importance sorting method for filter weights themselves, this embodiment evaluates each filter based on the intermediate values generated by the filter during data processing; the intermediate values include gradients, activations, and higher-order derivatives of gradients and activations; based on the formula Each filter kernel is evaluated, the layer is normalized, and the average value of the channel dimension of the filter of interest is taken, and then this importance judgment is sorted from large to small to determine its contribution to the model performance, so as to perform accurate pruning.

[0160] is the first-order derivative of the cost function C with respect to h i . That is, the gradient transmitted to the filter for weight update during the backpropagation process. is the first-order derivative of the cost function C with respect to the gradient the first-order partial derivative of the cost function with respect to the filter gradient, which reflects the current cost function's change to the current filter gradient, and the final change trend is the filter importance judgment standard. For example, if a filter currently has a large impact on the cost function but the gradient value is small, it means that the change in the future time has little effect on the model; if a filter currently has a small impact on the cost function but the gradient value is large, it means that the change in the future time has a large effect on the model, and then the importance of this filter will rise, thereby comprehensively considering the importance of the filter at the current time and in the future.

[0161] S303: Further evaluate the final importance of each filter based on the initial importance using Taylor expansion and Hessian matrix;

[0162] In order to compare the importance of filters fairly between different layers, the scores of each layer need to be normalized. This is usually achieved by dividing the scores of all filters in each convolutional layer by some form of sum or average of the scores of that layer. This normalization ensures that the filter importance scores within each layer are on the same scale and allows for cross-layer comparisons.

[0163] S304: First, sort the filters according to their importance calculated earlier. Filters with lower importance contribute relatively less to the overall network performance, so they can be prioritized for pruning. Based on the sorting results, the filter with the lowest importance is selected for pruning. These filters are those that can be removed without significantly affecting the network performance. Starting from the first convolutional layer of the model, the layers are pruned in order of filter importance. This process follows a shallow-to-deep order, gradually delving deeper into the network.

[0164] Specifically, the pruning process includes:

[0165] S304-1: Determine the filters to be removed: based on the previous importance evaluation, determine which filters are the least important, and these filters will be removed.

[0166] S304-2: Adjust the convolutional layer weights: in a convolutional layer, each filter corresponds to a set of weights, and removing a certain filter means removing the corresponding weight set at the same time; this usually involves the following operations:

[0167] Create a new convolutional layer with one less output channel than the original layer (because one filter is removed);

[0168] Copy the weights of the original convolutional layer (except for the weights corresponding to the filter to be removed) to the new convolutional layer;

[0169] If the original convolutional layer has a bias term, the bias also needs to be adjusted to match the new number of output channels.

[0170] S304-3: Adjusting subsequent layers: Adjust the subsequent convolutional layers.

[0171] Reducing input channels: When a filter is removed from a convolutional layer, the number of output feature maps (i.e., the number of output channels) is reduced. Therefore, the subsequent convolutional layers need to reduce their input channels accordingly to match the output of the previous layer.

[0172] For the next convolutional layer, the weight matrix needs to be modified to reflect the reduced input channel number. Specifically, since the input of the next layer's filter is reduced by one channel (because the previous layer's filter was pruned), a new filter is still constructed with one less input channel than the original filter, and then the original filter's output slices corresponding to the other previous layer's filter outputs are transferred and stored in the new filter.

[0173] If the ith channel is removed, all weights corresponding to this channel need to be removed from the weight matrix of the subsequent convolutional layer. Specifically, if the weight matrix of the next layer is four-dimensional (e.g., in PyTorch, the format is usually [output channels, input channels, height, width][output channels, input channels, height, width]), the ith column of the weight matrix needs to be removed.

[0174] S304-4: Adjusting fully connected (linear) layers

[0175] Reducing input features: If the pruned is the last convolutional layer in the network, the input features of the subsequent fully connected layers will be affected. Because each filter's output contributes to the input of the fully connected layer, removing a filter means reducing the input features of the fully connected layer.

[0176] Adjusting weight matrix: The weight matrix of the fully connected layer needs to be updated to reflect the change in the number of input features. This usually involves removing the weights corresponding to the output of the pruned convolutional layer.

[0177] For example, if each feature map output by the convolutional layer corresponds to a group of weights in the fully connected layer, when a feature map is removed, the corresponding group of weights should also be removed.

[0178] S304-5: Updating model structure: Finally, the original model needs to be replaced with the newly created and adjusted layers to complete the model structure update.

[0179] Repeat the pruning process: According to the predetermined pruning strategy, the above process may need to be repeated multiple times to gradually remove multiple low-importance filters and gradually achieve network lightweight.

[0180] S305: After successive iterations of pruning, the structure of the model becomes more streamlined, i.e., a pruned network model after lightening is obtained.

[0181] S306: Fine-tuning the pruned model as necessary to restore or improve model performance. Finally, save the lightened model for subsequent use or further research.

[0182] Throughout the process, special attention should be paid to the monitoring and evaluation of model performance to ensure that the pruning process does not excessively harm the accuracy and efficiency of the model. Also, pay attention to the use of appropriate hardware resources (such as GPU acceleration), especially when dealing with large network models.

[0183] The embodiment of the application realizes an automatic method to prune convolutional layers in deep learning models, specifically the VGG16 network model. This process can automatically identify and prune specific filters in specified layers. The embodiment of the application proposes a selective filter pruning strategy that allows users to specify any convolutional layer in the model and the specific filter within that layer for pruning. It can also reconstruct weights and biases. After pruning, this method not only adjusts the weights and biases of the pruned convolutional layer, but also adaptively modifies the input channel number of the convolutional layer that follows.

[0184] It is also very efficient in handling consecutive convolutional layers. When there are two consecutive convolutional layers, the code can correctly handle the changes in weights, ensuring the integrity of the network structure and the performance is not affected. It has special processing for the final layer, such as when processing the last convolutional layer of the VGG16 network model. This technology also involves adjusting the first fully connected layer, which is crucial to ensure the model continues to work and is excellent in maintaining model functionality. Despite pruning, the technology ensures that the output function of the model remains unchanged, i.e., the pruned model can still perform the same task as the original model. Finally, a modular layer replacement strategy is adopted, which automatically replaces the original layer through functions, improving the universality and scope of the code. The embodiment of the application takes VGG16 as an example, but the proposed intelligent pruning technology is theoretically applicable to any deep convolutional network with similar structure, and has high promotional value.

[0185] Based on the above embodiment, in the embodiment of the application, the "Pytorch" framework based on Python is used to build a pruning fine-tuning process based on Taylor pruning, which specifically includes:

[0186] S401: Initial training and saving of deep learning model;

[0187] Model initialization: instantiate a ModifiedVGG16Model class, which is based on the VGG16 architecture but modifies the classifier to adapt to the binary classification problem;

[0188] Data loading: Use dataset.loader function to load training dataset and dataset.tese_loader function to load testing dataset; datasets have been pre-processed into appropriate format and size;

[0189] Model training: Use PrunningFineTuner_VGG16 class's train method to train the model using Stochastic Gradient Descent (SGD) as the optimizer; training includes forward propagation and backpropagation processes, lasting for several epochs until a certain accuracy or epoch limit is reached;

[0190] Model saving: After training, use torch.save function to save the model state dictionary to the file "model".

[0191] S402: Model pruning and fine-tuning;

[0192] Load model: Load the trained model "model" through the torch.load function;

[0193] Pruning preparation: Use the get_candidates_to_prune method in the PrunningFineTuner_VGG16 class to determine the filters (neurons) that need to be pruned;

[0194] Iterative pruning: For each iteration, calculate the least important filters and prune them, then use Stochastic Gradient Descent (SGD) for fine-tuning to recover the performance loss caused by pruning. The pruning ratio and number of iterations are set according to the model complexity and performance loss;

[0195] Performance evaluation: After each pruning and fine-tuning, evaluate the model's accuracy on the test dataset and compare it with the accuracy before pruning;

[0196] Final model saving: After completing all pruning and fine-tuning steps, use the torch.save function again to save the pruned model state dictionary to the file "model_prunned";

[0197] S403: Model deployment and use on specific hardware;

[0198] Hardware configuration: Ensure that the system has an NVIDIA GPU and is installed with a CUDA environment;

[0199] Deploy the model: Use the torch.load function and model.cuda() method to load the pruned model onto the GPU;

[0200] Real-time application: apply the model to real-time data streams, classify input images; each time an input is received, forward propagation is performed through GPU acceleration, and the classification result is returned in real time;

[0201] Performance monitoring: monitor the performance of the model in the actual environment, record the delay and accuracy indicators, and fine-tune if necessary.

[0202] In this embodiment, by using deep learning and the model lightweight method provided by the application, the complexity of the model is reduced by removing unimportant filters, thereby improving the running efficiency and response speed of the model without significantly affecting the accuracy.

[0203] The application uses Taylor series expansion to approximate the importance evaluation of each filter in the neural network, calculates based on the product of activation values and gradients, and adds a Hessian matrix for fusion to obtain deep information of the filter itself and surrounding filters. In the filter sorting stage, the filters are sorted by calculating the contribution of the filters to the output error, and the filter with the smallest contribution is found. Using non-training parameter fixation, the original pre-training model parameters remain unchanged when pruning the feature extraction layer of the model. Using iterative pruning, an iterative pruning mechanism is realized, in which a certain proportion of filters are removed each iteration, and the remaining network is fine-tuned. Using cumulative pruning effect processing technology, the cumulative changes of the network structure after pruning are managed to ensure the correct indexing of subsequent filters. Using dynamic pruning adjustment technology, the number of filters to be pruned in each iteration during the pruning process is dynamically adjusted.

[0204] Referring to Figure 2 Fig. 1 is a comparison diagram of inference accuracy of the model lightweight method provided by the application and the traditional pruning method; referring to Figure 3 Fig. 2 is a comparison diagram of inference speed of the model lightweight method provided by the application and the traditional pruning method; based on Figure 2 and Figure 3 , the horizontal coordinate in the figure is the training round, and the vertical coordinate is the model and accuracy and inference speed respectively. Because of the large amount of filter information obtained by the embodiment of the application, the effect is obviously better than that of the traditional pruning method; because of the cyclic iterative pruning fine-tuning, it can be seen that the inference time and accuracy drop periodically with the round, but with the intervention of fine-tuning, the accuracy is gradually restored to a high level while the inference speed is maintained, and the model inference speed of the model lightweight method provided by the application is 6.2% faster than before.

[0205] Based on the above embodiment, in the embodiment of the application, a rail transit obstacle identification device based on model lightweight is also provided, comprising:

[0206] The model pre-training module 100 is configured to obtain a rail transit obstacle identification model as a network model to be pruned, obtain the network model to be pruned and an initial parameter set of the network model to be pruned, pre-train the network model to be pruned by using a rail transit image sample data set, update a model parameter set and a cost function, and obtain output activation values and gradient values of all filters in each convolutional layer of the pre-trained network model to be pruned; the gradient is a first derivative of the cost function with respect to the output activation values;

[0207] The importance obtaining module 200 is configured to, for each filter, calculate a cost function change value of each filter after being pruned based on an Oracle_Abs criterion, construct an initial importance expression, and obtain an initial importance of each filter; after Taylor expansion of the initial importance expression, multiply the Taylor expansion by a Hessian matrix, construct a final importance expression of each filter based on a first derivative of the cost function with respect to the output activation values and a first partial derivative of the gradient, and obtain a final importance of each filter; the final importance is a four-dimensional vector.

[0208] The filter to be pruned obtaining module 300 is configured to average the final importance according to a preset dimension, obtain filter weights, normalize the filter weights, and obtain normalized filter weights; sort the normalized filter weights of all filters in the network model to be pruned from small to large, obtain filters with normalized filter weights less than a preset weight, and construct a filter set to be pruned.

[0209] The pruning module 400 is configured to start from a first convolutional layer of the network model to be pruned, prune filters existing in the filter set to be pruned, and update a model parameter set until a number of pruned filters in the network model to be pruned reaches a preset pruning number, and obtain a target model parameter set; based on the target model parameter set, obtain a lightweight network model, and obtain a lightweight rail transit obstacle identification model.

[0210] The obstacle identification module 500 is configured to collect rail transit images in real time, input the lightweight rail transit obstacle identification model, obtain positions and sizes of obstacles on the rail transit, and complete rail transit obstacle identification.

[0211] The model-based lightweight rail transit obstacle recognition device of the embodiment is used to implement the aforementioned model-based lightweight rail transit obstacle recognition method, and therefore the specific embodiments in the model-based lightweight rail transit obstacle recognition device can be seen from the embodiment part of the model-based lightweight rail transit obstacle recognition method in the foregoing, for example, the model pre-training module 100 is used to implement steps S101 and S102 in the aforementioned model-based lightweight rail transit obstacle recognition method; the importance obtaining module 200 is used to implement steps S103 and S104 in the aforementioned model-based lightweight rail transit obstacle recognition method; the to-be-pruned filter obtaining module 300 is used to implement steps S105 and S106 in the aforementioned model-based lightweight rail transit obstacle recognition method; the pruning module 400 is used to implement steps S107 and S108 in the aforementioned model-based lightweight rail transit obstacle recognition method; and the obstacle recognition module 500 is used to implement step S109 in the aforementioned model-based lightweight rail transit obstacle recognition method. Therefore, the specific embodiments can refer to the description of the corresponding respective embodiment part, and details are not described herein again.

[0212] The model-based lightweight rail transit obstacle recognition method and device provided by the application, when obtaining the final importance of the filter, perform Taylor expansion on the initial importance expression based on the cost function, add a Hessian matrix for fusion, and obtain deep information of the filter itself and surrounding filters. The application introduces the Hessian matrix and further introduces second-order derivative information. By considering the second-order influence of the final importance of the filter on the model output, the importance of the weight is more comprehensively evaluated, the limitations of L2 norm pruning are overcome, unnecessary performance loss is reduced, the final importance of the filter can be more accurately evaluated, a more stable pruning standard is provided, the number of hyperparameters is reduced, the pruning process is simplified, efficient pruning can be realized in fewer iteration rounds, the calculation cost is significantly reduced, the stability and effect of automatic pruning are improved, the more stable pruning standard based on the Hessian matrix enhances the predictability of sparse training, and ensures that the model can maintain high performance while maintaining sparsity. The application combines Taylor expansion and the Hessian matrix to accurately evaluate the importance of the filter, thereby providing a more high-performance rail transit obstacle recognition model, and reducing the model parameters, thereby accelerating the obstacle recognition efficiency.

[0213] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0214] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0215] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0216] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0217] Obviously, the above-described embodiments are only examples for clarity and are not limiting on the implementation. Other different forms or variations of the above-described embodiments can be made on the basis of the above description for those skilled in the art. Here, it is not necessary or possible to exhaust all the embodiments. The obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A model-lightweight obstacle recognition method for rail transit, characterized in that, include: Obtain the rail transit obstacle recognition model as the network model to be pruned; Obtain the network model to be pruned and its initial parameter set, pre-train it using a rail transit image sample dataset, update the model parameter set and cost function, and obtain the output activation value and gradient value of all filters in each convolutional layer of the pre-trained network model to be pruned; the gradient is the first derivative of the cost function with respect to the output activation value. For each filter, based on the Oracle_Abs criterion, calculate the change in cost function after pruning, construct an initial importance expression, and obtain the initial importance of each filter, expressed as: Among them, initial importance This represents the change in the cost function after pruning filter i; Indicates the first The output activation value of each filter; Indicates the first pruning The cost function of the network model to be pruned after filtering. This represents the cost function of the network model to be pruned without a pruning filter. This indicates that the input set is included. With the correct tag set The training sample set is denoted as , Represents the total number of filters in the network model to be pruned; cost function The expression is , Indicates that in the given input And pruning The set of model parameters after each filter Below, predict the correct label. The probability of; After performing a Taylor expansion of the initial importance expression, multiplying it with the Hessian matrix, and constructing the final importance expression for each filter based on the first derivative of the cost function with respect to the output activation value and the first partial derivative with respect to the gradient, the final importance of each filter is obtained; the final importance is a four-dimensional vector; wherein, the Taylor expansion of the initial importance expression is expressed as: The final importance expression is: ; where the filter gradient Representing the cost function The first derivative of the activation value of the i-th filter output, The remaining term in the Taylor expansion; Representing the cost function Relative to gradient The first-order partial derivative; The final importance is averaged according to a preset dimension to obtain the filter weights, and then normalized to obtain the normalized filter weights. The normalized filter weights of all filters in the pruning model are sorted from smallest to largest, and all filters with normalized filter weights less than the preset weights are obtained to form a set of filters to be pruned. Starting from the first convolutional layer of the network model to be pruned, the filters in the set of filters to be pruned are pruned, and the set of model parameters is updated until the number of pruned filters in the network model to be pruned reaches the preset number of prunes, and the target set of model parameters is obtained. Based on the target model parameter set, a lightweight network model is obtained, resulting in a lightweight rail transit obstacle recognition model. Real-time acquisition of rail transit images is input into a lightweight rail transit obstacle recognition model to obtain the location and size of obstacles on the rail transit, thus completing the rail transit obstacle recognition.

2. The model-lightweight obstacle recognition method for rail transit according to claim 1, characterized in that, Obtaining the pre-trained network model to be pruned includes: Obtain the network model to be pruned in the initial parameter set Below, on the training sample set The initial cost function value is expressed as: ; Obtain the updated parameter set of the network model to be pruned. Below, on the training sample set The update cost function value is expressed as: ; To minimize the difference between the updated cost function value and the initial cost function value, an optimization objective function is constructed. The parameter set at which the optimization objective function converges is obtained, and a pre-trained network model to be pruned is obtained. The optimization objective function is expressed as: ; in, The Hessian matrix; This indicates the number of non-zero parameters in the model parameters of the network model to be pruned. This indicates a preset non-zero parameter threshold.

3. The model-lightweight obstacle recognition method for rail transit according to claim 1, characterized in that, The step of averaging the final importance according to a preset dimension to obtain filter weights includes: The final importance is represented as a four-dimensional vector, including the batch size. Number of channels ,high With width , represented as ( , , , ); By number of channels The filter weights are obtained by averaging the final importance values ​​across a preset dimension. , represented as: ； in, The Taylor expansion formula for the cost function C; , , , .

4. The model-lightweight obstacle recognition method for rail transit according to claim 1, characterized in that, Obtaining the normalized filter weights includes: Calculate the average value of the filter weights of all filters in each convolutional layer to obtain the layer average value; For each filter in each convolutional layer, calculate the quotient of the filter weight of each filter and the average value of the convolutional layer to which it belongs, and use this as the normalized filter weight for each filter.

5. The model-lightweight obstacle recognition method for rail transit according to claim 1, characterized in that, The pruning of filters existing in the set of filters to be pruned includes: Obtain the current filter to be pruned and its corresponding current convolutional layer; Create a new convolutional layer, and obtain the weight matrix of the current convolutional layer after the current filter to be pruned is pruned, and copy it into the new convolutional layer; Update the number of input channels and weight matrix of the next convolutional layer after the current convolutional layer to obtain the updated next convolutional layer; By replacing the current convolutional layer and the next convolutional layer in the network model to be pruned with the new convolutional layer and the updated next convolutional layer, the pruning of the current filter to be pruned is completed.

6. The model-lightweight obstacle recognition method for rail transit according to claim 1, characterized in that, After pruning filters in the filter set to be pruned in the pruned network model each time, the process also includes fine-tuning the pruned network model using stochastic gradient descent, which includes: Construct a training dataset and train the current pruned network model to be pruned. The model parameter set is updated using gradient descent, the loss function is optimized until the loss function value converges, the current model parameter set is obtained, and the fine-tuning of the current pruned network model is completed.

7. The model-lightweight obstacle recognition method for rail transit according to claim 6, characterized in that, The loss function is expressed as: ； in, This represents the value of the loss function. Represents the regularization coefficient. This represents the square of the total weights of the network model to be pruned after pruning the i-th filter.

8. A model-based lightweight obstacle recognition device for rail transit, characterized in that, include: The model pre-training module is used to acquire a rail transit obstacle recognition model as a network model to be pruned; it acquires the network model to be pruned and its initial parameter set, pre-trains it using a rail transit image sample dataset, updates the model parameter set and cost function, and acquires the output activation values ​​and gradient values ​​of all filters in each convolutional layer of the pre-trained network model to be pruned; the gradient is the first derivative of the cost function with respect to the output activation value. The importance acquisition module, based on the Oracle_Abs criterion, calculates the change in cost function after pruning for each filter, constructs an initial importance expression, and obtains the initial importance of each filter, expressed as: Among them, initial importance This represents the change in the cost function after pruning filter i; Indicates the first The output activation value of each filter; Indicates the first pruning The cost function of the network model to be pruned after filtering. This represents the cost function of the network model to be pruned without a pruning filter. This indicates that the input set is included. With the correct tag set The training sample set is denoted as , Represents the total number of filters in the network model to be pruned; cost function The expression is , Indicates that in the given input And pruning The set of model parameters after each filter Below, predict the correct label. The probability; after Taylor expansion of the initial importance expression, multiplying it with the Hessian matrix, and constructing the final importance expression for each filter based on the first derivative of the cost function with respect to the output activation value and the first partial derivative with respect to the gradient, thus obtaining the final importance of each filter; the final importance is a four-dimensional vector; wherein, the Taylor expansion of the initial importance expression is expressed as: The final importance expression is: ; where the filter gradient Representing the cost function The first derivative of the activation value of the i-th filter output, The remaining term in the Taylor expansion; Representing the cost function Relative to gradient The first-order partial derivative; The filter to be pruned module is used to average the final importance according to a preset dimension to obtain filter weights, and then normalize them to obtain normalized filter weights; the normalized filter weights of all filters in the model to be pruned are sorted from small to large to obtain all filters whose normalized filter weights are less than the preset weights, thus forming a set of filters to be pruned. The pruning module is used to prune the filters in the filter set to be pruned, starting from the first convolutional layer of the network model to be pruned, and update the model parameter set until the number of pruned filters in the network model to be pruned reaches the preset number of prunings, and obtain the target model parameter set; based on the target model parameter set, a lightweight network model is obtained, resulting in a lightweight rail transit obstacle recognition model; The obstacle recognition module is used to acquire real-time images of rail transit, input them into a lightweight rail transit obstacle recognition model, obtain the location and size of obstacles on the rail transit, and complete the rail transit obstacle recognition.

Images