Image foreign matter detection method and device, electronic equipment and storage medium

Abstract

The application provides an image foreign matter detection method and device, electronic equipment and storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring a target image to be detected; inputting the target image to be detected into a foreign matter detection model to output a foreign matter detection result used for representing a category probability and a position coordinate of a predicted target; wherein the foreign matter detection model is obtained by network compression of a foreign matter detection teacher model based on sparse coefficients, and knowledge distillation of the compressed sparse foreign matter detection model; and a distillation loss function corresponding to the foreign matter detection model is determined based on a teacher loss function corresponding to the foreign matter detection teacher model, a first output result, and a second output result corresponding to an iterative sparse foreign matter detection model in different iterative training periods. The application can ensure detection accuracy while improving the detection speed and real-time performance of foreign matter detection.

Description

Image foreign object detection methods, devices, electronic equipment and storage media Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and storage medium for detecting foreign objects in images. Background Technology

[0002] Airport runways may contain foreign object debris (FOD) that could damage aircraft, such as metal parts, waterproof plastic sheeting, gravel, newspapers, bottles, and luggage tags. Therefore, frequent foreign object detection is necessary for airport runways.

[0003] In existing technologies, when detecting foreign objects in airport images, the YOLOv7-tiny deep neural network for object detection is typically trained using the airport's FOD dataset to obtain the FODNET network model, which has high detection accuracy. The FODNET network model is then used for foreign object detection. However, while the trained FODNET network model has high detection accuracy, its computational cost and parameter count are large, resulting in low detection speed and difficulty in meeting real-time requirements. Summary of the Invention

[0004] This invention provides an image foreign object detection method, apparatus, electronic device, and storage medium to address the shortcomings of low detection speed in existing technologies. While ensuring detection accuracy, it reduces the computational load of the foreign object detection model and improves the detection speed and real-time performance of foreign object detection.

[0005] This invention provides a method for detecting foreign objects in an image, comprising:

[0006] Acquire the image of the target to be detected;

[0007] The image of the target to be detected is input into the foreign object detection model, and the output is the foreign object detection result, which characterizes the predicted target's category probability and location coordinates.

[0008] The foreign object detection model is obtained by compressing the foreign object detection teacher model based on sparse coefficients and then performing knowledge distillation on the compressed sparse foreign object detection model. The distillation loss function corresponding to the foreign object detection model is determined based on the teacher loss function and the first output result of the foreign object detection teacher model, as well as the second output result of the iterative sparse foreign object detection model for different iteration training periods. The computational complexity of the foreign object detection model is different from that of the foreign object detection teacher model, and the sparse coefficients are different for different iteration training periods.

[0009] According to the image foreign object detection method provided by the present invention, the foreign object detection model is trained based on the following steps:

[0010] Obtain an image sample set;

[0011] Based on the image sample set, the initial foreign object detection teacher model is trained to obtain the foreign object detection teacher model;

[0012] Based on the sparse coefficients, the foreign object detection teacher model is compressed to determine the sparse foreign object detection model, wherein the sparse coefficients are different for different iteration training cycles.

[0013] The image sample set is input into the sparse foreign object detection model for knowledge distillation to obtain the foreign object detection model. The distillation loss function corresponding to the foreign object detection model is determined based on the teacher loss function and the first output result of the foreign object detection teacher model, as well as the second output result of the iterative sparse foreign object detection model for different iterative training cycles.

[0014] According to the image foreign object detection method provided by the present invention, the step of performing network compression on the foreign object detection teacher model based on sparse coefficients includes:

[0015] Determine the previous iteration sparse foreign object detection model corresponding to the previous training cycle;

[0016] Based on the sparse coefficients corresponding to the current iteration training cycle, the previous iteration sparse foreign object detection model is sparsely trained to obtain the current iteration sparse foreign object detection model.

[0017] Traverse at least two convolutional layers in the current iterative sparse foreign object detection model, prune each convolutional layer, and obtain the updated current iterative sparse foreign object detection model.

[0018] The updated sparse foreign object detection model is subjected to the next round of network compression, and the process stops when the total number of iterations is reached.

[0019] According to the image foreign object detection method provided by the present invention, the sparse coefficients corresponding to the current iterative training cycle are determined based on the following steps:

[0020] Determine the total number of iterations, the stage boost factor, and the upper limit of the sparsity coefficient;

[0021] When the current iteration training cycle is less than the stage threshold of the total number of iterations, the sparse coefficient corresponding to the current iteration training cycle is determined based on the current iteration training cycle, the total number of iterations, the stage boosting factor, and the upper limit of the sparse coefficient.

[0022] If the current iteration training cycle is greater than or equal to the stage threshold of the total number of iterations, the upper limit of the sparse coefficient is determined as the sparse coefficient corresponding to the current iteration training cycle.

[0023] According to the image foreign object detection method provided by the present invention, the pruning of each of the convolutional layers includes:

[0024] Iterate through each of the convolutional layers, determine the L2 norm of the weights corresponding to each convolutional kernel in the current convolutional layer, and determine the importance evaluation value corresponding to each convolutional kernel based on each L2 norm.

[0025] Based on the importance evaluation values, the convolutional kernels are sorted in ascending order;

[0026] Based on the sparse coefficients corresponding to the current iteration training cycle and the importance evaluation value, a convolution kernel retention vector is determined. The convolution kernel retention vector is obtained by pruning the convolution kernels that are ranked lower by the importance evaluation value based on the sparse coefficients.

[0027] Traverse the convolution kernel retention vectors, prune the number of channels of each retained convolution kernel, and determine the number of channel updates corresponding to each retained convolution kernel. The number of channel updates for each retained convolution kernel is determined based on the convolution kernel retention vector corresponding to the previous convolutional layer.

[0028] According to the image foreign object detection method provided by the present invention, determining the number of channel updates corresponding to each of the retained convolutional kernels includes:

[0029] Determine the kernel-preserving vector of at least one previous convolutional layer connected to the current convolutional layer;

[0030] When the number of the previous convolutional layers is 1, the number of the previous convolutional kernels in the previous convolutional kernel retention vector is determined as the number of channel updates corresponding to each of the retained convolutional kernels in the current convolutional layer;

[0031] If the number of previous convolutional layers is greater than 1, determine the union vector of the vectors retained by each previous convolutional kernel, and determine the number of previous convolutional kernels in the union vector as the number of channel updates corresponding to each retained convolutional kernel in the current convolutional layer.

[0032] According to the image foreign object detection method provided by the present invention, the distillation loss function corresponding to the foreign object detection model is determined based on the following steps:

[0033] Determine the confidence scaling factor in the first output result, the confidence scaling factor being used as the weight of the negative sample in the distillation loss function;

[0034] Based on the mean square and error functions corresponding to the first and second output results, and the second confidence loss function in the teacher loss function, the first confidence loss function is determined;

[0035] Based on the confidence scaling factor, the KL discrepancy function corresponding to the first output result and the second output result, and the second category loss function in the teacher loss function, the first category loss function is determined;

[0036] Based on the confidence scaling factor, the CIoU loss function corresponding to the first output result and the second output result, and the second position coordinate loss function in the teacher loss function, the first position coordinate loss function is determined;

[0037] The distillation loss function is determined based on the first confidence loss function, the first category loss function, and the first location coordinate loss function.

[0038] The present invention also provides an image foreign object detection device, comprising:

[0039] The acquisition module is used to acquire the image of the target to be detected;

[0040] The output module is used to input the image of the target to be detected into the foreign object detection model and output the foreign object detection result, which characterizes the predicted target's category probability and location coordinates.

[0041] The foreign object detection model is obtained by compressing the foreign object detection teacher model based on sparse coefficients and then performing knowledge distillation on the compressed sparse foreign object detection model. The distillation loss function corresponding to the foreign object detection model is determined based on the teacher loss function and the first output result of the foreign object detection teacher model, as well as the second output result of the iterative sparse foreign object detection model for different iteration training periods. The computational complexity of the foreign object detection model is different from that of the foreign object detection teacher model, and the sparse coefficients are different for different iteration training periods.

[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image foreign object detection method as described above.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image foreign object detection method as described above.

[0044] The image foreign object detection method, apparatus, electronic device, and storage medium provided by this invention input the acquired target image to be detected into a foreign object detection model. Since the foreign object detection model is based on sparse coefficients that change dynamically with the iterative training cycle, the network compression is performed on the foreign object detection teacher model with high accuracy and large scale obtained through training. The distillation loss function is determined by combining the teacher loss function and the first output result corresponding to the foreign object detection teacher model, as well as the second output result corresponding to the iterative sparse foreign object detection model with different iterative training cycles. This results in a knowledge distillation of the network-compressed sparse foreign object detection model, which is smaller in scale but higher in accuracy. This ensures the accuracy of the image foreign object detection results while improving the detection speed and real-time performance of image foreign object detection by reducing the computational load of the foreign object detection model. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 is a flowchart illustrating the image foreign object detection method provided by the present invention;

[0047] Figure 2 is a schematic diagram of one of the example network structures of the initial foreign object detection teacher model provided by the present invention;

[0048] Figure 3 is a second example network structure diagram of the initial foreign object detection teacher model provided by the present invention;

[0049] Figure 4 is a schematic diagram of the third example network structure of the initial foreign object detection teacher model provided by the present invention;

[0050] Figure 5 is a fourth example network structure diagram of the initial foreign object detection teacher model provided by the present invention;

[0051] Figure 6 is a fifth example network structure diagram of the initial foreign object detection teacher model provided by the present invention;

[0052] Figure 7 is a schematic diagram of determining the CIoU loss function provided by the present invention;

[0053] Figure 8 is a schematic diagram of the sparse training and pruning process provided by the prior art;

[0054] Figure 9 is a schematic diagram of the soft pruning process provided by the prior art;

[0055] Figure 10 is one of the schematic diagrams of the loss curve of the benchmark model YOLOv7-tiny provided by the prior art;

[0056] Figure 11 is a second schematic diagram of the loss curve of the benchmark model YOLOv7-tiny provided by the prior art;

[0057] Figure 12 is one of the performance curves of the YOLOv7-tiny benchmark model provided by the prior art;

[0058] Figure 13 is a second schematic diagram of the performance curves of the existing benchmark model YOLOv7-tiny;

[0059] Figure 14 is the third schematic diagram of the performance curves of the existing benchmark model YOLOv7-tiny;

[0060] Figure 15 is the fourth schematic diagram of the performance curves of the existing benchmark model YOLOv7-tiny;

[0061] Figure 16 is one of the performance curves of the constant training strategy provided by the present invention;

[0062] Figure 17 is a second schematic diagram of the performance curve of the constant training strategy provided by the present invention;

[0063] Figure 18 is a schematic diagram of the performance curve of the constant training strategy provided by the present invention (Part 3).

[0064] Figure 19 is one of the performance curve diagrams of the dynamic training strategy provided by the present invention;

[0065] Figure 20 is a second schematic diagram of the performance curve of the dynamic training strategy provided by the present invention;

[0066] Figure 21 is a third schematic diagram of the performance curve of the dynamic training strategy provided by the present invention;

[0067] Figure 22 is one of the performance curves of the direct fine-tuning training provided by the present invention;

[0068] Figure 23 is a second schematic diagram of the performance curve of the direct fine-tuning training provided by the present invention;

[0069] Figure 24 is a schematic diagram of the performance curve of the direct fine-tuning training provided by the present invention (the third one).

[0070] Figure 25 is a fourth schematic diagram of the performance curve of the direct fine-tuning training provided by the present invention;

[0071] Figure 26 is one of the performance curves of the knowledge distillation training provided by the present invention;

[0072] Figure 27 is a second schematic diagram of the performance curve of the knowledge distillation training provided by the present invention;

[0073] Figure 28 is a schematic diagram of the third performance curve of the knowledge distillation training provided by the present invention;

[0074] Figure 29 is a schematic diagram of the performance curve of the knowledge distillation training provided by the present invention (the fourth one).

[0075] Figure 30 is a schematic diagram of the image foreign object detection device provided by the present invention;

[0076] Figure 31 is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0078] While existing FODNET network models offer high detection accuracy, their large computational and parameter counts result in slow detection speeds, hindering real-time performance and making them unsuitable for practical FOD detection systems. This invention provides an image foreign object detection method. Figure 1 is a flowchart illustrating the method, which includes:

[0079] Step 110: Obtain the image of the target to be detected;

[0080] Optionally, embodiments of the present invention can be applied to foreign object detection in scenarios such as airport roads, highways, and racetracks. Taking airport road foreign object detection as an example, the target image to be detected uses the airport road as a background, and the airport road background can include: offset printing background, non-adhesive background, cement background, asphalt background, etc. The target image to be detected can contain typical airport road foreign objects at different image scales, such as: screws, nuts, stones, rubber, etc., and the material type of the airport road foreign objects can include: metal, stone, plastic, and glass, which is not limited in the embodiments of the present invention.

[0081] Step 120: Input the image of the target to be detected into the foreign object detection model, and output the foreign object detection result to characterize the category probability and location coordinates of the predicted target;

[0082] The foreign object detection model is obtained by compressing the foreign object detection teacher model based on sparse coefficients and then performing knowledge distillation on the compressed sparse foreign object detection model. The distillation loss function corresponding to the foreign object detection model is determined based on the teacher loss function and the first output result of the foreign object detection teacher model, as well as the second output result of the iterative sparse foreign object detection model for different iteration training periods. The computational complexity of the foreign object detection model is different from that of the foreign object detection teacher model, and the sparse coefficients are different for different iteration training periods.

[0083] Specifically, after inputting the acquired target image into the foreign object detection model, the sparsity coefficients dynamically change with the iterative training cycle. Network compression is performed on the trained, highly accurate, and large-scale foreign object detection teacher model to balance the contradiction between detection accuracy and network size. Furthermore, by combining the teacher loss function and first output result of the foreign object detection teacher model with the distillation loss function determined by the second output result of iterative sparse foreign object detection models with different training cycles, knowledge distillation is performed on the compressed sparse foreign object detection model. This results in a smaller but more accurate foreign object detection model. When using the foreign object detection model to detect the target image, it ensures both the accuracy of the foreign object detection results and improves the detection speed, meeting the timeliness requirements of foreign object detection. Based on the foreign object detection results, the location of the foreign object in the target image can be further determined, facilitating timely handling of the foreign object and preventing damage to roads and other equipment in the scene.

[0084] Optionally, the foreign object detection model is trained based on the following steps:

[0085] Obtain an image sample set;

[0086] Based on the image sample set, the initial foreign object detection teacher model is trained to obtain the foreign object detection teacher model;

[0087] Based on the sparse coefficients, the foreign object detection teacher model is compressed to determine the sparse foreign object detection model, wherein the sparse coefficients are different for different iteration training cycles.

[0088] The image sample set is input into the sparse foreign object detection model for knowledge distillation to obtain the foreign object detection model. The distillation loss function corresponding to the foreign object detection model is determined based on the teacher loss function and the first output result of the foreign object detection teacher model, as well as the second output result of the iterative sparse foreign object detection model for different iterative training cycles.

[0089] Specifically, in existing technologies, while directly training a foreign object detection teacher model yields high detection accuracy, it also results in a large network size. This requires significant resources for deployment, places high demands on equipment, and fails to meet the real-time requirements of practical applications. Therefore, in this embodiment of the invention, an initial foreign object detection teacher model is trained using an image sample set. The trained, high-accuracy but large-scale model is then compressed. The sparsity coefficient is dynamically adjusted for different training iterations to balance the conflict between detection accuracy and model size. By compressing the trained foreign object detection teacher model, a sparse foreign object detection model with fewer network parameters, faster computation, and less impact on detection performance is obtained, requiring fewer resources for deployment. However, while network compression significantly reduces the network size, it inevitably affects detection accuracy, leading to a decrease in model detection precision. Therefore, after network compression, the foreign object detection teacher model is used as the teacher model, and the compressed sparse foreign object detection model is used as the student model. Knowledge distillation is then performed on the sparse foreign object detection model to improve its detection accuracy.

[0090] Optionally, before obtaining the image sample set, a foreign object dataset needs to be constructed first, and the foreign object dataset is divided into an image sample set and a test dataset according to a certain data ratio. The above data ratio can be 4:1. The initial foreign object detection teacher model is trained using the image sample set, and after the model training is completed, the training effect of the trained foreign object detection teacher model is tested using the test dataset.

[0091] Optionally, to improve sample diversity, after determining the foreign object dataset, the dataset can be expanded through data augmentation. Data augmentation can include translation, rotation, and Mosaic data augmentation. Mosaic data augmentation involves stitching together four foreign object datasets. When training with the new stitched dataset, it's equivalent to inputting four training datasets into the initial foreign object detection teacher model at once, greatly enriching the background for foreign object detection.

[0092] Optionally, in the aforementioned image sample set, foreign objects on airport roads can be manually labeled using preset anchor boxes, or automatically or semi-automatically labeled using machine learning models. The image proportion β of foreign objects on airport roads in the aforementioned image sample set can be determined by measuring the ratio of the area of ​​the foreign objects on airport roads to the area of ​​the training data, and then taking the square root of this ratio. Furthermore, based on the size of the image proportion β, foreign objects on airport roads in the image sample set can be divided into small targets, medium targets, and large targets. The image proportions corresponding to different targets are shown in Table 1. The threshold values ​​between small, medium, and large targets in Table 1 can be set based on experience, and this embodiment of the invention does not impose any limitations on this.

[0093] Table 1 Proportion Standards for Different Objectives

[0094] The image proportions of small targets, medium targets, and large targets are β≤0.03, 0.03<β≤0.1, and β>0.1. surface

[0095] Optionally, the initial foreign object detection teacher model described above can be constructed based on the YOLO series of deep neural networks for object detection, using a single neural network to directly regress and predict the category probability and location coordinates of the target in the entire image. Wherein:

[0096] YOLOv1 networks are characterized by their fast detection speed, but their detection accuracy is relatively low. YOLOv2 and YOLOv3 networks addressed this issue with various improvement strategies, including introducing anchor box mechanisms, designing deeper feature extraction networks, and constructing FPN structures, thus improving the model's detection performance. YOLOv4 and YOLOv5 networks further improved model accuracy by introducing mosaic data augmentation, Focus modules, and PAN structures. YOLOv7 networks made improvements in network structure, label allocation, and training methods, significantly surpassing its predecessors in both accuracy and speed.

[0097] To address different performance requirements, the YOLOv7 network offers networks of varying sizes, such as the YOLOv7-tiny deep neural network, the YOLOv7 deep neural network, and the YOLOv7x deep neural network. The YOLOv7 and YOLOv7x deep neural networks are suitable for larger datasets, while the YOLOv7-tiny deep neural network is suitable for smaller datasets. This embodiment of the invention does not impose any limitations on this. Given the relatively small size of the foreign object dataset provided in this embodiment, the initial foreign object detection teacher model is constructed using the YOLOv7-tiny deep neural network. The YOLOv7-tiny deep neural network introduces an anchor box mechanism to output the offset of the predicted target box relative to a preset anchor box. The YOLOv7-tiny deep neural network has prediction branches at three scales. During the training phase, it is necessary to assign the ground truth bounding box (Ground Truth) to different branches and corresponding anchor boxes based on the degree of matching between the ground truth bounding box and the preset anchor boxes at each scale. Then, labels with the same dimensions as the network output are generated. Finally, the loss function is calculated based on the difference between the network's actual output and the labels, and the backpropagation algorithm is used to update the network. This iterative training continues until the network converges.

[0098] Optionally, Figure 2 is a schematic diagram of one example network structure of the initial foreign object detection teacher model provided by the present invention. As shown in Figure 2, the YOLOv7-tiny deep neural network includes: a feature extraction network Backbone, a feature fusion module Neck, and a detection head module Head. The feature extraction network Backbone extracts features from the input image, generates a feature map, and inputs this feature map into the feature fusion module Neck. The feature fusion module Neck fuses feature maps of different scales. The detection head module Head predicts the final detection result at the end of the network.

[0099] 1) The feature extraction network includes 22 convolutional layers and 3 max-pooling layers. Figure 3 is a schematic diagram of the second example network structure of the initial foreign object detection teacher model provided by this invention. As shown in Figure 3, the convolutional unit CBL includes: 1 convolutional layer, a normalization layer, and a ReLU (Rectified Linear Unit) activation layer. Figure 4 is a schematic diagram of the third example network structure of the initial foreign object detection teacher model provided by this invention. As shown in Figure 4, the C5 module includes: 5 convolutional units CBL and one feature connection unit Concat. The 5 convolutional units CBL and the feature connection unit Concat are connected to enhance feature reuse and feature propagation in the network. The max-pooling layer MP is used to downsample the feature map while increasing the number of channels in the feature map. Figure 5 is a schematic diagram of the fourth example network structure of the initial foreign object detection teacher model provided by this invention. The CBC module in the feature extraction network Backbone is shown in Figure 5. The max-pooling layer in the CBC module increases the number of channels in the feature map while downsampling it.

[0100] 2) Figure 6 is the fifth example network structure diagram of the initial foreign object detection teacher model provided by this invention. At the input end, the YOLOv7-tiny deep neural network adopts the same Mosaic data augmentation strategy as the YOLOv5 network. In the feature fusion module Neck structure, FPN and PAN structures are used to enhance the fusion of features at different scales. The FPN and PAN structures are shown in Figure 6. Starting from YOLOv2, the YOLO series introduced the anchor box mechanism, that is, instead of directly predicting the position of the target box, it predicts its offset relative to the pre-set anchor box. A large number of pre-set anchor boxes can densely cover various positions of the image, which greatly improves the recall rate of the YOLO series. As shown in Figure 6, the YOLOv7-tiny deep neural network has three prediction branches at three scales. During the training phase, it is necessary to assign the ground truth boxes to different branches and corresponding anchor boxes according to the matching degree between the ground truth boxes and the pre-set anchor boxes at each scale, and further generate labels with the same dimension as the network output. Finally, based on the difference between the network's true output and the labels, the value of the loss function is calculated, and the backpropagation algorithm is used to update the network. This iterative training continues until the network converges.

[0101] Optionally, the network compression of the foreign object detection teacher model based on sparse coefficients includes:

[0102] Determine the previous iteration sparse foreign object detection model corresponding to the previous training cycle;

[0103] Based on the sparse coefficients corresponding to the current iteration training cycle, the previous iteration sparse foreign object detection model is sparsely trained to obtain the current iteration sparse foreign object detection model.

[0104] Traverse at least two convolutional layers in the current iterative sparse foreign object detection model, prune each convolutional layer, and obtain the updated current iterative sparse foreign object detection model.

[0105] The updated sparse foreign object detection model is subjected to the next round of network compression, and the process stops when the total number of iterations is reached.

[0106] Mainstream lightweight methods for deep neural networks include model pruning, low-rank decomposition, weight quantization, and knowledge distillation. Model pruning primarily compresses the model by removing less important parameters. Low-rank decomposition reduces the number of model parameters by decomposing the weight matrix into multiple low-rank matrices. Weight quantization mainly uses low-bit quantization to compress 32-bit or 64-bit floating-point numbers into 8-bit floating-point numbers, thus achieving model compression. Knowledge distillation uses the output of a trained complex model as a soft objective to guide simplified model training, enabling it to reach or exceed the performance of complex models. Since low-rank decomposition and weight quantization mainly optimize the storage space occupied by the model and have little effect on improving the runtime speed, this patent mainly combines model pruning and knowledge distillation methods for network lightweighting. Sparse training and network pruning are commonly used network compression techniques that reduce the number of network parameters by removing less important nodes, thereby reducing the resource consumption and time overhead of network inference. Figure 8 is a flowchart illustrating the sparse training and pruning process provided by existing technologies. As shown in Figure 8, the key to network pruning is defining a suitable importance evaluation method, which includes the following:

[0107] 1) Inspired by the normalization layer, this method directly uses the γ parameter in the normalization layer as the "importance measure" for each channel, incorporates γ into the loss function for sparse training, and finally prunes based on the γ value corresponding to each channel. The advantage of this method is its clever design, as it does not introduce additional parameters. The disadvantage is that it ignores the parameter values ​​of each channel within the convolutional kernel itself; using only the γ parameter of the normalization layer is insufficient to accurately characterize the importance of channels. Therefore, pruning using this method will cause a significant decrease in model accuracy.

[0108] 2) Using the L2 norm of the convolution kernel as a measure of "importance". Compared with the previous method, the disadvantage of this method is that the L2 norm has a large computational cost, which will lead to a larger computational cost during training, but it does not affect the computational cost during model inference. The advantage is that it takes into account the importance of the convolution kernel weights themselves more fully, and the accuracy of the model is less affected after pruning.

[0109] However, after determining the importance assessment method, sparse training and pruning are performed on the network. Common pruning methods primarily involve hard pruning, where in each training iteration, the importance of weights is calculated and ranked according to a certain criterion, and weights with importance below a threshold are pruned. These pruned weights do not participate in the next training iteration. The advantage of hard pruning is that it prunes low-importance nodes in each training iteration, thus gradually reducing the model size and computational cost as training progresses. However, its disadvantage is that pruned nodes cannot participate in subsequent training, leading to a significant decrease in model accuracy. To address the significant decrease in model accuracy caused by hard pruning, soft pruning strategies for neural networks have been proposed. The core idea is to replace pruning with a strategy of resetting the weights of low-importance nodes to zero. In the next training iteration, nodes whose weights were reset to zero in the previous iteration can still participate in training. After training is complete, all low-importance nodes are then completely pruned. Figure 9 is a flowchart illustrating the soft pruning process provided by existing technology. As shown in Figure 9, the algorithm consists of two loops. The outer loop iterates through the training rounds. In the inner loop, each convolutional layer is traversed, and the L2 norm of each convolutional kernel is calculated as an importance index. The convolutional kernels are then sorted according to their importance index, and the weights of kernels with importance below a threshold are reset to zero. The next loop is then initiated, and training is iterated continuously until a sparse model is obtained. Finally, the sparse model is pruned to obtain a lightweight model. Compared to hard pruning methods, soft pruning has a higher computational cost during training, but the computational cost during model inference is the same as that of hard pruning. Furthermore, because the pruning strategy is relatively "gentle," the model accuracy is generally better than that of hard pruning.

[0110] Specifically, in order to balance the contradiction between compression scale and model detection accuracy and solve the problem of large computational overhead when compressing the foreign object detection teacher model, this embodiment of the invention sets different sparse coefficients for each iteration training cycle. By dynamically determining the sparse coefficients corresponding to each iteration training cycle, and performing sparse training on the previous iteration sparse foreign object detection model obtained from the previous iteration training cycle based on the sparse coefficients, the convolutional layers in the current iteration sparse foreign object detection model obtained from the sparse training are pruned, thereby further improving the compression degree while reducing the impact on detection accuracy.

[0111] Optionally, the sparse coefficients corresponding to the current iteration training cycle are determined based on the following steps:

[0112] Determine the total number of iterations, the stage boost factor, and the upper limit of the sparsity coefficient;

[0113] When the current iteration training cycle is less than the stage threshold of the total number of iterations, the sparse coefficient corresponding to the current iteration training cycle is determined based on the current iteration training cycle, the total number of iterations, the stage boosting factor, and the upper limit of the sparse coefficient.

[0114] If the current iteration training cycle is greater than or equal to the stage threshold of the total number of iterations, the upper limit of the sparse coefficient is determined as the sparse coefficient corresponding to the current iteration training cycle.

[0115] Specifically, in existing technologies, when sparsely training a model, a fixed sparsity coefficient is used to determine the proportion of model parameters that are compressed. However, if a large sparsity coefficient is used, a large proportion of model parameters will be set to zero during training, resulting in a highly sparsity model for foreign object detection, but with significantly reduced detection accuracy, failing to meet detection requirements. Conversely, if a small sparsity coefficient is used, a smaller proportion of model parameters will be set to zero during training, resulting in higher detection accuracy, but insufficient sparsity (i.e., insufficient model compression), leading to a still large model size that cannot meet practical application requirements. Therefore, to balance the contradiction between detection accuracy and model compression, in this embodiment of the invention, the initial sparsity coefficient is set to 0 during training. In the first stage of the total number of iterations, the sparsity coefficient is increased in stages from 0 with a step size until it reaches the upper limit, facilitating observation of the improvement effect. In the second stage of the total number of iterations, the iterative sparse foreign object detection model is sparsely trained with a constant sparsity coefficient until the model converges. By gradually increasing the sparsity coefficients in the first stage, the compression of the foreign object detection teacher model gradually increases, while the impact on detection accuracy is relatively small. This achieves a good balance between model compression and detection accuracy, resulting in a lightweight model. The sparsity coefficient P corresponding to the current iteration training cycle is... i The formula is shown in equation (1), which is:

[0116]

[0117] Where P represents the upper limit of the sparsity coefficient, m represents the improvement ratio factor, and 1 / m represents the improvement ratio when the sparsity coefficient is improved in stages. In this embodiment of the invention, the lower limit of 1 / m is 20%, that is, the upper limit of m is 5, and the range of the improvement ratio is [0.2, 1]. p The current iteration represents the number of training cycles, e represents the total number of iterations, n represents the stage boost factor, 1 / n represents the stage step size for improving the sparse coefficients, and k represents the number of iterations. p e / k is an integer greater than 1. p Represents the stage separation threshold, [0, e / k]p ) represents the first stage of the total number of iterations, [e / k p [e] represents the second stage of the total number of iterations, and / represents integer division, that is, both the divisor and the dividend are integers, and the division result is rounded.

[0118] Optionally, pruning the current convolutional layer based on the sparse coefficients corresponding to the current iteration training cycle includes:

[0119] Determine the L2 norm of the weights corresponding to each convolution kernel in the current convolutional layer, and based on each L2 norm, determine the importance evaluation value corresponding to each convolution kernel;

[0120] Based on the importance evaluation values, the convolutional kernels are sorted in ascending order;

[0121] Based on the sparse coefficients corresponding to the current iteration training cycle and the importance evaluation value, a convolution kernel retention vector is determined. The convolution kernel retention vector is obtained by pruning the convolution kernels that are ranked lower by the importance evaluation value based on the sparse coefficients.

[0122] Traverse the convolution kernel retention vectors, prune the number of channels of each retained convolution kernel, and determine the number of channel updates corresponding to each retained convolution kernel. The number of channel updates for each retained convolution kernel is determined based on the convolution kernel retention vector corresponding to the previous convolutional layer.

[0123] Specifically, in the case of training a foreign object detection teacher model using a YOLOv7-tiny deep neural network, to adapt to the structure of the foreign object detection teacher model and reduce the impact of pruning on the detection accuracy of the sparse foreign object detection model, this embodiment of the invention fully considers three factors: the weight of the convolutional kernel, the number of convolutional kernels, and the number of convolutional channels in the model. The L2 norm of the weights corresponding to each convolutional kernel in the current convolutional layer is determined as the importance evaluation value. Although the computational cost during training increases, it does not affect the cost during model prediction, thus minimizing the impact on the detection accuracy of the foreign object detection teacher model after pruning. After determining the importance evaluation value, the convolutional kernels can be sorted in ascending order based on the importance evaluation value. Then, using the sparsity coefficient corresponding to the current iteration training cycle, convolutional kernels with lower importance evaluation values ​​are pruned, reducing the number of convolutional kernels in the current convolutional layer. This reduces the number of network parameters of the current iteration sparse foreign object detection model corresponding to the current iteration training cycle, achieving a lightweight foreign object detection teacher model. Meanwhile, convolutional kernels with high importance evaluation values ​​are retained, and a convolutional kernel retention vector is constructed. The number of channels of the retained convolutional kernels in the convolutional kernel retention vector is pruned to ensure that the number of channels updated for the retained convolutional kernels in the current convolutional layer is consistent with the number of channels in the output feature map of the previous convolutional layer, thus avoiding pruning the model structure and affecting detection accuracy.

[0124] It should be noted that the above-mentioned ascending order of convolutional kernels involves sorting the kernels according to their importance evaluation values ​​from largest to smallest. Then, using the sparse coefficients corresponding to the current training iteration, convolutional kernels with lower importance evaluation values ​​are pruned, retaining those with higher importance evaluation values, and a convolutional kernel retention vector is constructed. Alternatively, the above-mentioned convolutional kernels can also be sorted according to their importance evaluation values ​​from smallest to largest. Using the sparse coefficients corresponding to the current training iteration, convolutional kernels with higher importance evaluation values ​​are pruned, retaining those with lower importance evaluation values, and a convolutional kernel retention vector is constructed.

[0125] Optionally, determining the number of channel updates corresponding to each of the retained convolutional kernels includes:

[0126] Determine the kernel-preserving vector of at least one previous convolutional layer connected to the current convolutional layer;

[0127] When the number of the previous convolutional layers is 1, the number of the previous convolutional kernels in the previous convolutional kernel retention vector is determined as the number of channel updates corresponding to each of the retained convolutional kernels in the current convolutional layer;

[0128] If the number of previous convolutional layers is greater than 1, determine the union vector of the vectors retained by each previous convolutional kernel, and determine the number of previous convolutional kernels in the union vector as the number of channel updates corresponding to each retained convolutional kernel in the current convolutional layer.

[0129] Specifically, because the current iterative sparse foreign object detection model contains multiple convolutional layers with different connection relationships, the input and output layers directly associated with the pruned convolutional layer also need to be pruned when pruning the network. For relatively simple convolutional neural networks such as LeNet and VGG, the input of a convolutional layer is only related to the previous layer. Therefore, when performing pruning, it is only necessary to ensure that the number of channels of the convolutional kernel of the current convolutional layer matches that of the output feature map of the previous layer after pruning. However, the YOLOv7-tiny network has a concat cross-layer connection structure, and ordinary pruning methods cannot be directly applied to the concat structure. Therefore, to ensure that the number of channels updated for the convolutional kernels in the current convolutional layer is consistent with the number of channels in the output feature map of the previous convolutional layer, and to avoid pruning the model structure and affecting detection accuracy, in this embodiment of the invention, the previous convolutional layer connected to the current convolutional layer is determined. If the current convolutional layer is only connected to one previous convolutional layer, then to ensure the detection accuracy of the pruned model, the number of channels for the convolutional kernels in the current convolutional layer must be equal to the number of previous convolutional kernels in the reserved vector of the previous convolutional layer. If the current convolutional layer is connected to multiple previous convolutional layers, the number of channels updated for each reserved convolutional kernel in the current convolutional layer needs to be determined based on the number of previous convolutional kernels in the union vector of the reserved vectors of each previous convolutional layer. For example, for a simply connected convolutional layer, its channel reserved vector is only related to the previous layer, that is, the convolutional kernel channel reserved vector mask_channel = mask_pre of the current convolutional layer. For the feature connection layer Concat in the YOLOv7-tiny deep neural network in this embodiment of the invention, the feature connection layer Concat is a cross-connection structure, that is, the feature connection layer Concat connects to multiple previous convolutional layers. Taking the connection of 4 previous convolutional layers as an example, the previous convolutional kernel retention vectors corresponding to the 4 previous convolutional layers are mask1, mask2, mask3 and mask4 respectively. Then the number of channels updated by the convolutional kernel in the current convolutional layer, mask_channel, is: mask_channel = mask1∪mask2∪mask3∪mask4.

[0130] Optionally, the distillation loss function corresponding to the foreign object detection model is determined based on the following steps:

[0131] Determine the confidence scaling factor in the first output result, the confidence scaling factor being used as the weight of the negative sample in the distillation loss function;

[0132] Based on the mean square and error functions corresponding to the first and second output results, and the second confidence loss function in the teacher loss function, the first confidence loss function is determined;

[0133] Based on the confidence scaling factor, the KL discrepancy function corresponding to the first output result and the second output result, and the second category loss function in the teacher loss function, the first category loss function is determined;

[0134] Based on the confidence scaling factor, the CIoU loss function corresponding to the first output result and the second output result, and the second position coordinate loss function in the teacher loss function, the first position coordinate loss function is determined;

[0135] The distillation loss function is determined based on the first confidence loss function, the first category loss function, and the first location coordinate loss function.

[0136] Specifically, in conventional knowledge distillation methods in the prior art, the deviation between the output of the teacher model and the output of the student model is directly included in the loss function when calculating the model loss function. However, the foreign object detection teacher model in this embodiment of the invention is built based on a YOLOv7-tiny deep neural network. Since the YOLOv7-tiny deep neural network has a large number of pre-set anchor boxes, in actual training, a large number of anchor boxes do not contain foreign objects and are considered negative samples. Only a few anchor boxes containing objects are considered positive samples. If conventional knowledge distillation methods are used to calculate the loss function, it will lead to a significant increase in computational overhead during training, and over-optimization of negative samples that do not contain the target will also lead to difficulties in model convergence. Therefore, in this embodiment of the invention, the distillation loss function is determined by the sum of the first confidence loss function, the first category loss function, and the first position coordinate loss function.

[0137] in:

[0138] 1) First confidence loss function As shown in equation (2), equation (2) is:

[0139]

[0140] in, This represents the true label value of the confidence score of the i-th bounding box in the feature map. This represents the predicted confidence value of the i-th bounding box in the second output of the iterative sparse foreign object detection model. f represents the predicted confidence value of the i-th bounding box in the first output of the foreign object detection teacher model, i.e., the confidence scaling factor. obj (·) represents the mean square error function. This represents the second confidence loss function of the foreign object detection teacher model. denoted as the confidence loss between the iterative sparse foreign object detection model and the foreign object detection teacher model, and λ represents the weight coefficient of the distillation loss function, used to balance the influence of the true label and the output of the foreign object detection teacher model on the foreign object detection model.

[0141] Second confidence loss function f obj (p i (k),p i (k) is shown in equation (3), which is:

[0142]

[0143] Among them, S 2 p represents the size of the predicted feature map, B represents the number of bounding boxes of different sizes predicted at each location in the predicted feature map, and p represents the size of the predicted feature map. i (k) and p i (k) represent the predicted probability and the true probability of the bounding box corresponding to category k, respectively, and λclass represents the weight coefficient of the second confidence loss function. This indicates that if the target exists in the bounding box corresponding to the i-th position of the predicted feature map, the value is 1; otherwise, it is 0.

[0144] 2) First-class loss function As shown in equation (4), equation (4) is:

[0145]

[0146] in, This represents the label value of the category k to which the bounding box belongs. This represents the predicted value of the bounding box corresponding to category k in the second output of the iterative sparse foreign object detection model. f represents the predicted value of the bounding box corresponding to category k in the first output of the foreign object detection teacher model. cl (·) denotes the KL dispersion function, used to represent the difference between two distributions. This represents the second-class loss function of the foreign object detection teacher model. This represents the class loss between the iterative sparse foreign object detection model and the foreign object detection teacher model. To address the issue of excessive negative bounding boxes in the foreign object detection teacher model, a confidence scaling factor is introduced into the first-class loss function. When the predicted bounding box is a negative sample, the confidence scaling factor The confidence scaling factor approaches 0 when the predicted bounding box is a positive sample. Approaching 1, through the confidence scaling factor Weighting the class probabilities reduces the weight of negative sample prediction boxes in the first class loss function, allowing the iterative sparse foreign object detection model to focus more on learning positive samples containing foreign objects, avoiding over-optimization of negative sample prediction boxes, and improving model convergence efficiency.

[0147] Second category loss function As shown in equation (5), equation (5) is:

[0148]

[0149]

[0150] in, This indicates that if the target exists in the bounding box corresponding to the i-th position of the predicted feature map, the value is 1; otherwise, it is 0. Conversely, if a target exists in the j-th bounding box corresponding to the i-th position in the predicted feature map, the value is 0; otherwise, it is 1. i and c i λ represents the predicted and true values ​​of the bounding box confidence at the i-th position in the feature map, respectively, and can be 0 or 1. obj , λ noobj These represent the weight coefficients of each part of the second-class loss function.

[0151] 3) First position coordinate loss function As shown in equation (6), equation (6) is:

[0152]

[0153] in, Represents the actual target bounding box. This represents the predicted bounding box in the second output of the iterative sparse foreign object detection model. f represents the truth box in the first output of the foreign object detection teacher model. bb (·) denotes the CIoU loss function, used to alleviate the optimization difficulties when there is no overlap between the predicted and target boxes, taking into account the aspect ratio of the predicted bounding box. This represents the loss function for the second position coordinates of the foreign object detection teacher model. This represents the positional coordinate loss between the iterative sparse foreign object detection model and the foreign object detection teacher model. Similar to the first category loss function, the second positional coordinate loss function also introduces a confidence scaling factor. Using confidence scaling factor Weighting the position coordinates can reduce the weight of the negative sample prediction box in the first position coordinate loss function, so that the iterative sparse foreign object detection model can focus more on learning positive samples containing foreign objects, avoid over-optimizing the negative sample prediction box, and improve the model convergence efficiency.

[0154] Second position coordinate loss f bb As shown in equation (7), equation (7) is:

[0155]

[0156] Among them, pred i,j The ground_truth represents the bounding box predicted by the feature map. i λ represents the true target bounding box. coord This represents the weighting coefficient for the position error term.

[0157] 4) Distillation loss function Loss distill As shown in equation (8), equation (8) is:

[0158]

[0159] Optionally, the CIoU loss function can effectively alleviate the optimization difficulty when the predicted target box and the ground truth target box do not overlap, thus accelerating the convergence speed during model training. Determining the CIoU loss function may include the following steps:

[0160] 1) Figure 7 is a schematic diagram of the determination of the CIoU loss function provided by the present invention. As shown in Figure 7, the true target box is determined based on the preset anchor box and label value, and the predicted target box can be obtained through the model output. k1 represents the center point corresponding to the true target box, and k2 represents the center point corresponding to the predicted target box. Based on the position coordinates of the two center points, the Euclidean distance d between the center point k1 and the center point k2 can be obtained. The minimum bounding box is the smallest rectangle that can simultaneously cover the true target box and the predicted target box. The diagonal length c of the minimum bounding box is further determined. Then the CIoU loss function is as shown in Equation (9), which is:

[0161]

[0162] Where IoU represents the predicted bounding box b and the ground truth bounding box b. gt The intersection-over-union ratio, that is, the predicted target box b and the ground truth target box b. gt The ratio of the intersection area to the area of ​​the minimum bounding box. α is shown in equation (10), which is:

[0163]

[0164] μ is shown in equation (11), which is:

[0165]

[0166] Among them, w gt h represents the width of the actual target bounding box. gt represents the height of the true bounding box, w represents the width of the predicted bounding box, and h represents the height of the predicted bounding box.

[0167] Optionally, after training the foreign object detection model, its performance can be tested based on a test training set. The pre-acquired foreign object dataset includes 8180 airport road data images, containing a total of 12196 foreign object targets, including 3119 small targets, 7800 medium targets, and 1277 large targets. The foreign object dataset is randomly sampled at a ratio of 4:1 to divide it into an image sample set and a test dataset. The image sample set includes 6544 airport road data images, and the test dataset includes 1636 airport road data images. When testing the trained foreign object detection model using the test dataset, in this embodiment of the invention, precision, recall, and mean AP can be used as evaluation metrics for detection accuracy.

[0168] Table 2 Confusion Matrix

[0169] Actual positive sample | Actual negative sample | Predicted as positive sample | True Positive (TP) | False Positive (FP) | Predicted as negative sample | False Negative (FN) | True Negative (TN) surface

[0170] 1) The recall and precision can be determined by identifying the confusion matrix corresponding to the foreign object detection model. The confusion matrix is ​​shown in Table 2. Precision refers to the proportion of targets predicted as positive that are actually positive. Precision is shown in Equation (12), which is:

[0171]

[0172] Recall refers to the proportion of all positive targets that are correctly detected. Recall is shown in equation (13), which is:

[0173]

[0174] 2) For a specific target class, a Precision Recall (PR) curve can be plotted for that class. Based on the PR curve, the Accuracy Percentage (AP) value for that class can be determined. After determining the AP values ​​for all classes, the mean accuracy (mAP) value can be determined based on the average of all AP values. The PR curve for a specific class can be plotted using the following methods:

[0175] Given an appropriate IoU threshold, if the IoU between the predicted bounding box and the ground truth bounding box is greater than or equal to the IoU threshold, it indicates that the predicted target bounding box is predicted as a positive sample; otherwise, it is predicted as a negative sample. The predicted target bounding boxes are arranged in descending order of confidence, and the precision and recall corresponding to each type of target are calculated using different confidence values ​​as sampling points for the PR curve. Connecting the sampling points yields the PR curve.

[0176] If the PR curve is continuous, then the AP value for this category is as shown in equation (14), which is:

[0177]

[0178] If the PR curve is discrete, then the AP value for this category is as shown in equation (15), which is:

[0179]

[0180] Optionally, in this embodiment of the invention, mAP@0.5 and mAP@0.5:0.95 are used as metrics for network detection accuracy. mAP@0.5 represents the average of the AP@0.5 for each target class when IoU is set to 0.5, yielding the mAP for all classes. mAP@0.5:0.95 represents the average mAP of the network at IoU = [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95].

[0181] Furthermore, in this embodiment of the invention, the number of model parameters, the number of floating-point operations (FLOPs), and the number of frames per second (FPS) can also be used as indicators to evaluate the computational scale of the model. Under the same hardware and software environment, the larger the number of parameters and the number of floating-point operations, the smaller the number of frames per second. Wherein:

[0182] 1) Network parameter quantity refers to the number of parameters required for a complete storage model, usually measured in megabytes (MB).

[0183] For convolutional layers, the number of network parameters can be determined by the number of channels in the input feature map, the number of channels in the output feature map, and the size of the convolutional kernel. The number of network parameters in a convolutional layer is... nv As shown in equation (16), equation (16) is:

[0184] Parametersconv=Cin×Cout×Kh×Kw+Cout

[0185] Where Cin represents the number of channels in the input feature map, Cout represents the number of channels in the output feature map, and can also represent the number of parameters in the bias term, Kh represents the height of the convolution kernel, Kw represents the width of the convolution kernel, and Cin×Cout×Kh×Kw represents the number of parameters in the convolution kernel.

[0186] For a fully connected layer, the number of network parameters can be determined by the number of input nodes and the number of output nodes. linear As shown in equation (17), equation (17) is:

[0187] Parameters linear =N in ×N out +N out

[0188] Where, N in N represents the number of input nodes. out This indicates the number of output nodes.

[0189] 2) The number of floating-point operations refers to the sum of the number of multiplication or addition operations required for the network to complete one forward inference operation. Its unit is usually gigabit (G), where:

[0190] For a convolutional layer, if the number of input channels is C in The number of output channels is C out The output feature map has a size of h×w, and the convolution kernel size is K. h ×K w The number of multiplications required to perform one convolution operation is C. in ×K h ×K w The number of additions is C. in ×K h ×K w -1 time, totaling 2×C in ×K h ×K w -1 times. The output feature map has a total of C. out Each channel requires h×w convolution operations; therefore, the number of floating-point operations (FLOPs) for the convolutional layer is [number missing]. convAs shown in equation (18), equation (18) is:

[0191] FLOPs conv =(2C in K h K w -1)hwC out

[0192] For a fully connected layer, the number of floating-point operations can be determined by the number of input nodes I and the number of output nodes O. The number of floating-point operations for a fully connected layer is shown in equation (19), which is:

[0193] FLOPsl inear =2IO

[0194] 3) Inference frames per second (fps) refers to the number of forward inferences that the network can complete per unit of time. The calculation method is usually as follows: set a preset time period, such as 100 seconds, count the number of inferences that the network can complete without interruption during this time, and then divide the number of inferences by the preset time period to obtain the inference frames per second (fps).

[0195] Table 3 Test results of the YOLOv7-tiny model

[0196]

[0197] For example, in this embodiment of the invention, YOLOv7-tiny is used as the baseline model. The YOLOv7-tiny deep neural network is trained using the aforementioned image sample set. The hyperparameters for training are configured as follows: network input resolution is 640×640, training data batch size = 16, initial learning rate lr = 0.02, SGD optimizer is used, and the number of iterations is 100 epochs. The loss scaling factors for each part are set as follows: λ cls =0.5, λ obj =1.0, λ noobj =0.05, λ coord =0.05. The loss curves of the benchmark model YOLOv7-tiny are shown in Figures 10 and 11. Figure 10 shows the training loss of the benchmark model, Figure 11 shows the test loss of the benchmark model, Figure 12 shows the precision curve of the benchmark model, Figure 13 shows the recall curve of the benchmark model, Figure 14 shows the mAP@0.5 curve of the benchmark model, and Figure 15 shows the mAP@0.5:0.95 curve of the benchmark model. The horizontal axis represents the number of training epochs, and the vertical axis represents the loss value or evaluation value. The benchmark model YOLOv7-tiny was tested on the test dataset with the following parameters set: network input size of 640×640, confidence threshold... IoU threshold after NMS processing 5. The test results for AP@0.5 for various targets and mAP@0.5 and mAP@0.5:0.95 of the model were calculated on the test set and are shown in Table 3. Other performance indicators of the YOLOv7-tiny model, including the number of parameters, FPS, and FLOPs, are shown in Table 4. Analysis of the results in Tables 4 and 3 shows that the overall performance of the benchmark model YOLOv7-tiny is good, but its detection speed of 49 frames per second is lower than the minimum requirement of 60 frames per second. The number of network parameters and computational load are relatively large, making deployment difficult for mobile platforms with limited computing resources.

[0198] Table 4 Other performance indicators of the YOLOv7-tiny model

[0199] ParamsInference / FPSFLOPs11.7M4913.2G surface

[0200] For example, sparse training and pruning are performed on the pre-trained benchmark model YOLOv7-tiny. To verify the effectiveness of the proposed training strategy of dynamically improving sparse coefficients, the following two experiments are set up:

[0201] 1) The sparsity coefficient is a constant value, P = 0.5;

[0202] 2) The sparsity coefficient adopts a dynamic boosting strategy. The initial value is 0, and it gradually increases with the number of iterations, with the final value being P = 0.5.

[0203] Table 5 Test results of the two sparse models

[0204] Sparse model mAP@0.5:0.95model a 66.1% 44.3% model b 51.3% 28.4% surface

[0205] The hyperparameters during training were configured as follows: network input resolution was 640×640, batch size was 16, initial learning rate was 0.02, SGD optimizer was used, and sparse training was performed for 100 epochs. The loss scaling factors for each part were set as follows: λ cls =0.5, λ obj =1.0, λ noobj =0.05, λ coord=0.05. Figures 16-18 show the performance curves when the sparsity coefficients are constant, and Figures 19-21 show the performance curves when the sparsity coefficients are dynamically increased. As shown in Figures 16-21, the model using the constant sparsity coefficient training strategy has a large initial sparsity coefficient and a large adjustment range of model parameters. At the beginning of training, the model accuracy immediately drops to zero. In subsequent training, the model accuracy gradually recovers, but the final accuracy is still significantly lower than the original model. The model using the dynamic increase training strategy has a certain degree of decrease after the start of training, but because the sparsity coefficients are smaller, the adjustment range of its parameters is also smaller. The overall accuracy during the training process shows a trend of first decreasing slightly and then increasing, and the final model accuracy is higher than the first training strategy. Then, the performance of the two sparse models was tested on the test set, and the results are shown in Table 5. Among them, model a This refers to a model that employs a dynamic improvement training strategy, i.e., model b The model trained using a constant sparsity coefficient strategy. As shown in Table 5, the accuracy of the sparsely trained model decreased compared to the initial model, with the model using constant sparsity coefficients showing the most significant decrease in accuracy.

[0206] For example, after sparse training, the sparse network is pruned based on the method proposed in this embodiment of the invention, and the baseline model YOLOv7-tiny and the dynamic sparse model model are statistically analyzed. a constant sparse model b The parameter count, number of floating-point operations, and inference speed of the three models are statistically shown in Table 6. As can be seen from Table 6, after network pruning, the number of parameters and floating-point operations of the lightweight network are significantly reduced, and the inference speed is significantly improved to 67 frames per second. However, at the same time, the network mAP also decreases significantly. Therefore, subsequent fine-tuning and training of the pruned network is essential.

[0207] Table 6 Comparison of Indicators for the Three Models

[0208] ModelmAP@0.5mAP@0.5-0.95ParamsInference / FPSFLOPsYOLOv7-tiny83.8%62.6%11.7M4913.2Gmodel a 66.1%44.3%1.51M673.3Gmodel b 51.3% 28.4% 1.51M 673.3G surface

[0209] For example, after sparse training and pruning of the above model, the model can be directly fine-tuned, or fine-tuned using the knowledge distillation method proposed in this embodiment of the invention. In direct fine-tuning, the original loss function of YOLOv7-tiny is used to iterate for 50 epochs on the training set, with an initial learning rate lr = 0.01. The resulting performance curves are shown in Figures 22-25. When using the knowledge distillation method proposed in this embodiment for fine-tuning, the baseline model YOLOv7-tiny is used as the foreign object detection teacher model, and the pruned lightweight model is used as the student model. Knowledge distillation is performed using the aforementioned loss function. The hyperparameters are set as follows: distillation temperature T = 2.0, scaling factor α = 0.05, and other parameters are the same as in direct fine-tuning. Iterative training is conducted for 50 epochs, and the resulting performance curves are shown in Figures 26-29. The direct fine-tuning training and knowledge distillation training are tested on the test dataset, and the statistical results are shown in Table 7. As shown in Table 7, compared with direct fine-tuning training, the model accuracy obtained by fine-tuning using the knowledge distillation method is significantly improved, indicating the effectiveness of the knowledge distillation function for the YOLO target detection model proposed in this embodiment of the invention.

[0210] Table 7 Performance Comparison of Two Fine-Tuning Methods

[0211] Strategy mAP@0.5:0.95 Direct fine-tuning 72.9% 45.2% Distillation fine-tuning 80.6% 53.7% surface

[0212] Table 8 Performance Comparison with Benchmark Network

[0213]

[0214] For example, the baseline model YOLOv7-tiny, the pruned dynamic sparse model, and two distillation fine-tuning models are compared, and the performance indicators are shown in Table 8. Table 8 shows that the lightweight model obtained through sparse training, model pruning, and knowledge distillation reduces the number of parameters by 86% and the computational load by 75% compared to the baseline model. The model inference speed is increased by 18 frames per second, reaching 67 frames per second, which meets the minimum frame rate requirement of 60 frames per second for foreign object detection equipment response time stipulated in the normative documents formulated by the Civil Aviation Administration of China's Airport Department. However, mAP@0.5 only decreases by 3.2% and remains above 80%, demonstrating the effectiveness of the model pruning and knowledge distillation strategies provided in this embodiment of the invention.

[0215] Optionally, each model and its network structure in the embodiments of the present invention can be built using PyTorch, and the specific hardware and software configurations are shown in Table 9.

[0216] Table 9 Hardware and software configuration environment parameters

[0217]

[0218] The image foreign object detection method provided by this invention inputs the acquired target image to be detected into a foreign object detection model. Since the foreign object detection model is based on sparse coefficients that change dynamically with the iterative training cycle, the network compression is performed on the foreign object detection teacher model with high accuracy and large scale obtained through training. The distillation loss function is determined by combining the teacher loss function and the first output result corresponding to the foreign object detection teacher model, as well as the second output result corresponding to the iterative sparse foreign object detection model with different iterative training cycles. This results in a knowledge distillation of the network-compressed sparse foreign object detection model, which is smaller in scale but higher in accuracy. This ensures the accuracy of the image foreign object detection results while improving the detection speed and real-time performance of image foreign object detection by reducing the computational load of the foreign object detection model.

[0219] This invention also provides an image foreign object detection device. Figure 30 is a schematic diagram of the structure of the image foreign object detection device provided by this invention. As shown in Figure 30, the image foreign object detection device 3000 includes: an acquisition module 3001 and an output module 3002, wherein:

[0220] The acquisition module 3001 is used to acquire the image of the target to be detected;

[0221] The output module 3002 is used to input the image of the target to be detected into the foreign object detection model and output the foreign object detection result, which characterizes the predicted target's category probability and location coordinates.

[0222] The foreign object detection model is obtained by compressing the foreign object detection teacher model based on sparse coefficients and then performing knowledge distillation on the compressed sparse foreign object detection model. The distillation loss function corresponding to the foreign object detection model is determined based on the teacher loss function and the first output result of the foreign object detection teacher model, as well as the second output result of the iterative sparse foreign object detection model for different iteration training periods. The computational complexity of the foreign object detection model is different from that of the foreign object detection teacher model, and the sparse coefficients are different for different iteration training periods.

[0223] The image foreign object detection device provided by this invention inputs the acquired target image to a foreign object detection model. Since the foreign object detection model is based on sparse coefficients that change dynamically with the iterative training cycle, the network compression is performed on the foreign object detection teacher model with high accuracy and large scale obtained through training. The distillation loss function is determined by combining the teacher loss function and the first output result corresponding to the foreign object detection teacher model, as well as the second output result corresponding to the iterative sparse foreign object detection model with different iterative training cycles. This results in a knowledge distillation of the network-compressed sparse foreign object detection model, which is smaller in scale but higher in accuracy. This ensures the accuracy of the image foreign object detection results while improving the detection speed and real-time performance of image foreign object detection by reducing the computational load of the foreign object detection model.

[0224] Optionally, the output module 3002 is specifically used for:

[0225] The foreign object detection model is trained based on the following steps:

[0226] Obtain an image sample set;

[0227] Based on the image sample set, the initial foreign object detection teacher model is trained to obtain the foreign object detection teacher model;

[0228] Based on the sparse coefficients, the foreign object detection teacher model is compressed to determine the sparse foreign object detection model, wherein the sparse coefficients are different for different iteration training cycles.

[0229] The image sample set is input into the sparse foreign object detection model for knowledge distillation to obtain the foreign object detection model. The distillation loss function corresponding to the foreign object detection model is determined based on the teacher loss function and the first output result of the foreign object detection teacher model, as well as the second output result of the iterative sparse foreign object detection model for different iterative training cycles.

[0230] Optionally, the output module 3002 is specifically used for:

[0231] Determine the previous iteration sparse foreign object detection model corresponding to the previous training cycle;

[0232] Based on the sparse coefficients corresponding to the current iteration training cycle, the previous iteration sparse foreign object detection model is sparsely trained to obtain the current iteration sparse foreign object detection model.

[0233] Traverse at least two convolutional layers in the current iterative sparse foreign object detection model, prune each convolutional layer, and obtain the updated current iterative sparse foreign object detection model.

[0234] The updated sparse foreign object detection model is subjected to the next round of network compression, and the process stops when the total number of iterations is reached.

[0235] Optionally, the output module 3002 is specifically used for:

[0236] The sparse coefficients corresponding to the current iteration training cycle are determined based on the following steps:

[0237] Determine the total number of iterations, the stage boost factor, and the upper limit of the sparsity coefficient;

[0238] When the current iteration training cycle is less than the stage threshold of the total number of iterations, the sparse coefficient corresponding to the current iteration training cycle is determined based on the current iteration training cycle, the total number of iterations, the stage boosting factor, and the upper limit of the sparse coefficient.

[0239] If the current iteration training cycle is greater than or equal to the stage threshold of the total number of iterations, the upper limit of the sparse coefficient is determined as the sparse coefficient corresponding to the current iteration training cycle.

[0240] Optionally, the output module 3002 is specifically used for:

[0241] Iterate through each of the convolutional layers, determine the L2 norm of the weights corresponding to each convolutional kernel in the current convolutional layer, and determine the importance evaluation value corresponding to each convolutional kernel based on each L2 norm.

[0242] Based on the importance evaluation values, the convolutional kernels are sorted in ascending order;

[0243] Based on the sparse coefficients corresponding to the current iteration training cycle and the importance evaluation value, a convolution kernel retention vector is determined. The convolution kernel retention vector is obtained by pruning the convolution kernels that are ranked lower by the importance evaluation value based on the sparse coefficients.

[0244] Traverse the convolution kernel retention vectors, prune the number of channels of each retained convolution kernel, and determine the number of channel updates corresponding to each retained convolution kernel. The number of channel updates for each retained convolution kernel is determined based on the convolution kernel retention vector corresponding to the previous convolutional layer.

[0245] Optionally, the output module 3002 is specifically used for:

[0246] Determine the kernel-preserving vector of at least one previous convolutional layer connected to the current convolutional layer;

[0247] When the number of the previous convolutional layers is 1, the number of the previous convolutional kernels in the previous convolutional kernel retention vector is determined as the number of channel updates corresponding to each of the retained convolutional kernels in the current convolutional layer;

[0248] If the number of previous convolutional layers is greater than 1, determine the union vector of the vectors retained by each previous convolutional kernel, and determine the number of previous convolutional kernels in the union vector as the number of channel updates corresponding to each retained convolutional kernel in the current convolutional layer.

[0249] Optionally, the output module 3002 is specifically used for:

[0250] The distillation loss function corresponding to the foreign object detection model is determined based on the following steps:

[0251] Determine the confidence scaling factor in the first output result, the confidence scaling factor being used as the weight of the negative sample in the distillation loss function;

[0252] Based on the mean square and error functions corresponding to the first and second output results, and the second confidence loss function in the teacher loss function, the first confidence loss function is determined;

[0253] Based on the confidence scaling factor, the KL discrepancy function corresponding to the first output result and the second output result, and the second category loss function in the teacher loss function, the first category loss function is determined;

[0254] Based on the confidence scaling factor, the CIoU loss function corresponding to the first output result and the second output result, and the second position coordinate loss function in the teacher loss function, the first position coordinate loss function is determined;

[0255] The distillation loss function is determined based on the first confidence loss function, the first category loss function, and the first location coordinate loss function.

[0256] Figure 31 is a schematic diagram of the electronic device provided by the present invention. As shown in Figure 31, the electronic device may include: a processor 3110, a communication interface 3120, a memory 3130, and a communication bus 3140. The processor 3110, the communication interface 3120, and the memory 3130 communicate with each other via the communication bus 3140. The processor 3110 can call logical instructions in the memory 3130 to execute an image foreign object detection method, which includes:

[0257] Acquire the image of the target to be detected;

[0258] The image of the target to be detected is input into the foreign object detection model, and the output is the foreign object detection result, which characterizes the predicted target's category probability and location coordinates.

[0259] The foreign object detection model is obtained by compressing the foreign object detection teacher model based on sparse coefficients and then performing knowledge distillation on the compressed sparse foreign object detection model. The distillation loss function corresponding to the foreign object detection model is determined based on the teacher loss function and the first output result of the foreign object detection teacher model, as well as the second output result of the iterative sparse foreign object detection model for different iteration training periods. The computational complexity of the foreign object detection model is different from that of the foreign object detection teacher model, and the sparse coefficients are different for different iteration training periods.

[0260] Furthermore, the logical instructions in the aforementioned memory 3130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0261] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the image foreign object detection method provided by the above methods, the method comprising:

[0262] Acquire the image of the target to be detected;

[0263] The image of the target to be detected is input into the foreign object detection model, and the output is the foreign object detection result, which characterizes the predicted target's category probability and location coordinates.

[0264] The foreign object detection model is obtained by compressing the foreign object detection teacher model based on sparse coefficients and then performing knowledge distillation on the compressed sparse foreign object detection model. The distillation loss function corresponding to the foreign object detection model is determined based on the teacher loss function and the first output result of the foreign object detection teacher model, as well as the second output result of the iterative sparse foreign object detection model for different iteration training periods. The computational complexity of the foreign object detection model is different from that of the foreign object detection teacher model, and the sparse coefficients are different for different iteration training periods.

[0265] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the image foreign object detection methods provided by the methods described above, the method comprising:

[0266] Acquire the image of the target to be detected;

[0267] The image of the target to be detected is input into the foreign object detection model, and the output is the foreign object detection result, which characterizes the predicted target's category probability and location coordinates.

[0268] The foreign object detection model is obtained by compressing the foreign object detection teacher model based on sparse coefficients and then performing knowledge distillation on the compressed sparse foreign object detection model. The distillation loss function corresponding to the foreign object detection model is determined based on the teacher loss function and the first output result of the foreign object detection teacher model, as well as the second output result of the iterative sparse foreign object detection model for different iteration training periods. The computational complexity of the foreign object detection model is different from that of the foreign object detection teacher model, and the sparse coefficients are different for different iteration training periods.

[0269] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0270] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0271] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting foreign objects in an image, characterized in that, include: Acquire an image of the target to be detected; input the image of the target to be detected into a foreign object detection model, and output foreign object detection results representing the category probability and location coordinates of the predicted target; wherein, the foreign object detection model is obtained by compressing the network of the foreign object detection teacher model based on sparse coefficients, and then performing knowledge distillation on the compressed sparse foreign object detection model; the distillation loss function corresponding to the foreign object detection model is determined based on the teacher loss function and the first output result of the foreign object detection teacher model, and the second output result of the iterative sparse foreign object detection model corresponding to different iteration training cycles; the computational complexity of the foreign object detection model is different from that of the foreign object detection teacher model, and different iteration training cycles are also different. The sparse coefficients corresponding to different training periods are different; the foreign object detection model is trained based on the following steps: acquiring an image sample set; training an initial foreign object detection teacher model based on the image sample set to obtain a foreign object detection teacher model; performing network compression on the foreign object detection teacher model based on the sparse coefficients to determine a sparse foreign object detection model, wherein the sparse coefficients corresponding to different training periods are different; inputting the image sample set into the sparse foreign object detection model for knowledge distillation to obtain a foreign object detection model, wherein the distillation loss function corresponding to the foreign object detection model is based on the teacher loss function corresponding to the foreign object detection teacher model and the first output result, as well as the different training periods. The second output result corresponding to the iterative sparse foreign object detection model in the training cycle is determined; the network compression of the foreign object detection teacher model based on the sparse coefficients includes: determining the previous iteration sparse foreign object detection model corresponding to the previous training cycle; performing sparse training on the previous iteration sparse foreign object detection model based on the sparse coefficients corresponding to the current training cycle to obtain the current iteration sparse foreign object detection model; traversing at least two convolutional layers in the current iteration sparse foreign object detection model and pruning each convolutional layer to obtain the updated current iteration sparse foreign object detection model; and performing the next round of network compression on the updated current iteration sparse foreign object detection model until the total iteration is reached. The training stops when the number of iterations is reached. The sparse coefficient corresponding to the current iteration training cycle is determined based on the following steps: determining the total number of iterations, the stage boosting factor, and the upper limit of the sparse coefficient; the reciprocal of the stage boosting factor represents the stage step size for sparse coefficient boosting; when the current iteration training cycle is less than the stage threshold of the total number of iterations, the sparse coefficient corresponding to the current iteration training cycle is determined based on the current iteration training cycle, the total number of iterations, the stage boosting factor, and the upper limit of the sparse coefficient; when the current iteration training cycle is greater than or equal to the stage threshold of the total number of iterations, the upper limit of the sparse coefficient is determined as the sparse coefficient corresponding to the current iteration training cycle.

2. The image foreign object detection method according to claim 1, characterized in that, The pruning of each convolutional layer includes: traversing each convolutional layer, determining the L2 norm of the weights corresponding to each convolutional kernel in the current convolutional layer, and determining the importance evaluation value corresponding to each convolutional kernel based on each L2 norm; sorting each convolutional kernel in ascending order based on the importance evaluation value; determining the convolutional kernel retention vector based on the sparse coefficients corresponding to the current iteration training cycle and the importance evaluation value, wherein the convolutional kernel retention vector is obtained by pruning each convolutional kernel ranked lower in importance evaluation value based on the sparse coefficients; traversing the convolutional kernel retention vector, pruning the number of channels of each retained convolutional kernel, and determining the number of channel updates corresponding to each retained convolutional kernel, wherein the number of channel updates of each retained convolutional kernel is determined based on the convolutional kernel retention vector corresponding to the previous convolutional layer.

3. The image foreign object detection method according to claim 2, characterized in that, Determining the number of channel updates corresponding to each of the retained convolutional kernels includes: determining the previous convolutional kernel retention vector corresponding to at least one previous convolutional layer connected to the current convolutional layer; when the number of previous convolutional layers is 1, determining the number of previous convolutional kernels in the previous convolutional kernel retention vector as the number of channel updates corresponding to each of the retained convolutional kernels in the current convolutional layer; when the number of previous convolutional layers is greater than 1, determining the union vector of each of the previous convolutional kernel retention vectors, and determining the number of previous convolutional kernels in the union vector as the number of channel updates corresponding to each of the retained convolutional kernels in the current convolutional layer.

4. The image foreign object detection method according to claim 1, characterized in that, The distillation loss function corresponding to the foreign object detection model is determined based on the following steps: determining the confidence scaling factor in the first output result, the confidence scaling factor being used to reduce the weight of negative samples in the distillation loss function; determining the first confidence loss function based on the mean square and error functions corresponding to the first and second output results, and the second confidence loss function in the teacher loss function; Based on the confidence scaling factor, the KL discretization function corresponding to the first output result and the second output result, and the second category loss function in the teacher loss function, a first category loss function is determined; based on the confidence scaling factor, the CIoU loss function corresponding to the first output result and the second output result, and the second position coordinate loss function in the teacher loss function, a first position coordinate loss function is determined; based on the first confidence loss function, the first category loss function, and the first position coordinate loss function, the distillation loss function is determined.

5. An apparatus for implementing the image foreign object detection method according to any one of claims 1-4, characterized in that, include: The acquisition module is used to acquire the image of the target to be detected; The output module is used to input the target image to be detected into the foreign object detection model and output the foreign object detection result, which represents the category probability and location coordinates of the predicted target. The foreign object detection model is obtained by compressing the foreign object detection teacher model based on sparse coefficients and then performing knowledge distillation on the compressed sparse foreign object detection model. The distillation loss function corresponding to the foreign object detection model is determined based on the teacher loss function corresponding to the foreign object detection teacher model, the first output result, and the second output result corresponding to the iterative sparse foreign object detection model for different iteration training periods. The computational complexity of the foreign object detection model differs from that of the foreign object detection teacher model, and the sparse coefficients corresponding to different iteration training periods are also different.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image foreign object detection method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image foreign object detection method as described in any one of claims 1 to 4.

Images