A real-time apron target detection and recognition method based on improved SSD model

Abstract

The present invention provides a real-time apron target detection and recognition method based on an improved SSD model. The preprocessed data set is input into the improved SSD model for training to obtain a trained model. The improved SSD model adopts RFB-Net300 as the backbone network, adds a batch-normalization layer, uses a sub-pixel convolution layer instead of upsampling, and fuses low-level and high-level feature information through the MFPN model. The original loss function is improved by introducing the CIoU loss function. In the process of generating the prediction frame, the size of the prior frame is optimized by the K-means clustering algorithm, and the distance between target objects is calculated. The MFPN model realizes multi-scale feature fusion by constructing a feature pyramid, thereby enhancing the detection capability of targets of different sizes. The CIoU loss function comprehensively evaluates the similarity between the predicted frame and the actual frame, taking into account the overlapping area, center point distance, aspect ratio and scale. Through the above improvements, the present invention significantly improves the real-time performance and accuracy of apron target detection and recognition.

Description

Technical field:

[0001] The present invention relates to a method for multi-target detection and recognition during apron operation, and in particular to a real-time apron target detection and recognition method based on an improved SSD model. Background technology:

[0002] The apron is one of the areas with the most frequent activities of aircraft, vehicles, and personnel. Apron target detection technology is becoming increasingly important in ensuring the safety and efficiency of airport operations. Airport surface monitoring systems need to accurately detect and identify targets such as aircraft, vehicles, and ground personnel in real time to support the airport's daily operations and safety management. Currently, computer vision-based apron target detection has been applied to actual airport management systems. It has the advantages of strong real-time performance, data integration, high degree of automation, and rich information content. However, in a multi-target environment, occlusion between targets often leads to reduced detection accuracy, and small targets are frequently missed and misclassified. To address these issues, improving the speed and accuracy of detection and meeting the actual needs of apron safety management are urgent issues that need to be addressed.

[0003] Traditional airport scene target detection algorithms can be roughly divided into four categories, such as target detection based on set features, target detection based on texture features, and target detection based on color features. These algorithms require expert knowledge and manual intervention to select and design features, which is time-consuming, inflexible, and has limited generalization capabilities; such as target detection based on feature extraction and filter-based feature extraction algorithms. The feature extraction methods of these algorithms are usually tightly coupled with specific algorithms, with low versatility and adaptability, and high sensitivity to noise and occlusion; such as algorithms based on model construction, target detection based on statistics, and target detection based on decision tree models. These algorithms usually require a large amount of labeled data for training, which may be difficult to obtain in some application scenarios.

[0004] Although mainstream deep learning target detection algorithms have been studied and improved, most of them only detect target objects without marking the distance between multiple targets. In addition, the recognition effect of small targets at airports and targets that are easily obscured by multiple targets is poor, which cannot meet the management requirements for safe operation of airport aprons. Summary of the invention:

[0005] Aiming at the current target detection of objects in airport operation, key parts in multi-target situations are easily blocked by multiple targets, resulting in missed detection of key parts and classification errors. A real-time apron target detection and recognition method based on an improved SSD model is proposed.

[0006] A real-time apron target detection and recognition method based on an improved SSD model includes:

[0007] Acquire an airport surface operation dataset, the airport surface operation dataset including video data of movement behaviors of aircraft, ground support vehicles, and surface personnel, as well as surveillance video data of key aircraft parts during interactions. The ground support vehicles include boarding bridges, aircraft tractors, passenger elevators, water supply trucks, and fuel trucks. The surface personnel include maintenance personnel, traffic control personnel, cargo handling personnel, and passenger service personnel. The movement behaviors include single-target movement behaviors and multi-target interactive behaviors. The single-target movement behaviors include aircraft arrival and departure. The multi-target interactive behaviors include docking and separation between ground support vehicles, surface personnel, and aircraft. The key aircraft parts include the nose, tail, wingtips, and engine air intakes.

[0008] Preprocessing the airport surface operation data set, calibrating each frame of the scene image, and performing standardization and normalization on the image, wherein the preprocessed image includes the position of each target frame and target category information during the airport surface operation and maintenance process;

[0009] Inputting the preprocessed airport surface operation dataset into an improved target recognition algorithm SSD model for training to obtain a trained improved SSD model; the improved SSD model uses RFB-Net300 as the backbone network of the SSD algorithm, adds batch-normalization to the additional layer, uses a sub-pixel convolution layer instead of upsampling, and adds an MFPN model to fuse low-level and high-level feature information; and introduces a CIoU loss function to improve the original loss function;

[0010] The prediction results are input into the trained improved SSD model to obtain the target detection results.

[0011] Furthermore, the steps of generating the prediction box include:

[0012] Considering that RFB-Net sets the prior box size based on human experience, which is overly dependent on experience, we use the K-means clustering algorithm to set the optimal prior box.

[0013] Determine the distance calculation formula between target objects and initialize the required number of cluster centers;

[0014] Compare the distance d between each actual location box and each cluster center in turn, and assign the actual location box to the cluster with the closest distance to the cluster center;

[0015] After the allocation is completed, the cluster center point of each cluster is recalculated, that is, the average length and width of all actual position boxes in the cluster are calculated;

[0016] Repeat the above steps until the cluster center changes little;

[0017] The final calculated cluster center point is set to the size of the prior box on the feature map.

[0018] Furthermore, the distance calculation formula between target objects is expressed as follows:

[0019] d(box),centroid=1-IOU(box,centroid)#(1)

[0020]

[0021] Where box represents the actual location of the vehicle, centroid represents the cluster center, d(box, centroid) represents the distance from the box to the centroid, and IoU is the intersection over union (IoU) between the box and the centroid. The smaller the distance d, the better, that is, the larger the IoU value, the better.

[0022] Furthermore, the improved SSD models RFB-Net, Sub-pixelconvolutionlayer and MFPN model, the RFB-Net300 backbone network replaces the two newly added layers of SSD with two RFB structures, and conv43 and conv7_fc are respectively connected to RFB-s and RFB structures before connecting to the prediction layer, the BN layer is located between the convolution layer Conv43 and Relu, and the feature map is normalized. The Sub-pixelconvolutionlayer replaces upsampling, and the low-resolution feature map is obtained by convolution and multi-channel reorganization to obtain a high-resolution feature map. The MFPN model integrates a bottom-up and a top-down line horizontally connected feature pyramid network (FPN) structure;

[0023] The stride of the 3×3 convolution kernel of the original Conv43 layer of SSD is modified to 1, and the preprocessed 300×300×3 image is input into it. After the feature layer passes through a series of residual structures corresponding to the Conv43 layer, a 38×38×256 feature matrix is ​​obtained; the remaining convolution layers are generated by convolution, batchnormalization, and RELU. The RFB layer is generated by using three convolution kernels of different sizes of 1×1, 3×3, and 5×5 to form a multi-branch structure, and three dilated convolution layers (such as 3*3conv, rate=1) are introduced.

[0024] Furthermore, the layer performs sub-pixel convolution and multi-channel recombination on the lower-resolution feature map N×(C×r×r)×W×H to output a high-resolution feature map N×C(H×r)×(W×r), where the sub-pixel convolution layer contains multiple convolution kernels, each of which corresponds to a sub-pixel position in the upsampling process, where N is the image size, C is the number of channels, W is the width, H is the height, and r is the upsampling multiple.

[0025] Furthermore, the MFPN model captures multi-scale features by fusing feature maps of different levels to detect targets of different sizes. The model uses a top-down path to transmit high-level semantic information, while using a bottom-up path to retain low-level detail information, and also uses lateral connections to enhance communication between features. MFPN constructs a feature pyramid structure, and exchanges information between feature maps of different resolutions through upsampling and downsampling operations. In the bottom-up process, the size of the feature map becomes smaller and smaller after convolution, while in the top-down process, the high-level feature map is upsampled, and the feature map is sampled and fused with the bottom-up feature map via horizontal connections.

[0026] Furthermore, the CIoU loss function not only considers the overlapping area (IoU) of the bounding box, but also considers the center point distance, aspect ratio and scale of the bounding box to achieve a more comprehensive bounding box evaluation;

[0027] The expression of CIoU loss function βc is

[0028]

[0029]

[0030] where β I It represents the result of the overlap between the output prediction box A and the actual box B compared to the set of two prediction boxes, ρ(A ct , B ct ) represents the distance between the center points of the predicted box A and the measured box B in the image, c represents the slant distance length of the minimum bounding box represented by c, v represents the similarity of the aspect ratio of the predicted box and the actual box, w A 、w B Indicates the length of the predicted box A and the actual box B, h A 、h B Represents the height of the predicted box A and the actual box B, and α represents the weight coefficient.

[0031] Furthermore, for the trained network model, the test sample is used as input to obtain the output prediction result, which is compared with the true value of the sample to calculate the mAP and FPS.

[0032] Step 1: Read the device surveillance camera video and create a dataset containing 10,000 airport scene recognition targets. The training set and test set are randomly divided into 7:3 ratios, with 7,000 images as the training set and the other 3,000 images as the test set to verify the effectiveness of the improved SSD algorithm.

[0033] Step 2: calibrate each scene image and generate a corresponding text file; wherein the text file contains target frame position information and target category information of each target in the corresponding scene image;

[0034] Step 3: The calibrated image is cropped to a size of 300×300×3 and passed into the improved SSD network. The two newly added layers of SSD are replaced with two RFB structures on the RFB-Net300 backbone network, and conv43 and conv7_fc are connected to RFB-s and RFB structures respectively before connecting to the prediction layer. The 3×3 convolution kernel stride of the original Conv43 layer of SSD is modified to 1, and the feature layer obtains a 38×38×256 feature matrix after a series of residual structures corresponding to the Conv4_3 layer; the remaining convolution layers are generated by convolution, batchnormalization, and RELU. The RFB layer is generated by using three convolution kernels of different sizes: 1×1, 3×3, and 5×5 to form a multi-branch structure, and three dilated convolution layers (such as 3*3conv, rate=1) are introduced for generation.

[0035] Step 4: Optimize and generate prediction boxes, use the K-means clustering algorithm to determine the optimal size of the prior box, and achieve this by analyzing the actual target boxes in the training data set. Each actual position box is assigned according to the distance from each cluster center and belongs to the nearest cluster center. Subsequently, the member boxes in each cluster are analyzed, and the cluster center points are recalculated and updated, that is, the new center position is determined by taking the average of the length and width of all boxes in the cluster. This process is iterative, and the clustering and center update steps are repeated until the change in the cluster center reaches a minimum, indicating that the model has converged. Ultimately, these stabilized cluster center points define the size of the prediction box, providing the model with a set of more scientific and data-driven prior boxes, thereby optimizing the prediction box generation process;

[0036] d(box,centroid)=1-IoU(boX,centroid)#(1)

[0037]

[0038] Step 5: Introduce the BN layer to perform feature normalization between convolution and ReLU activation to stabilize the training process. The sub-pixel convolution layer replaces traditional upsampling, transforming the lower-resolution feature map N×(C×r×r)×W×H through sub-pixel convolution and multi-channel reorganization to output a high-resolution feature map N×C(H×r)×(W×r). The sub-pixel convolution layer contains multiple convolution kernels, each of which corresponds to a sub-pixel position in the upsampling process, where N is the image size, C is the number of channels, W is the width, H is the height, and r is the upsampling multiple. Through sub-pixel convolution and channel reorganization, the feature map resolution is effectively improved and key details are retained.

[0039] Step 6: Using the MFPN model, by constructing a feature pyramid, we implement top-down and bottom-up paths and lateral connections to promote the fusion of multi-scale features. Upsampling and downsampling operations enable the effective exchange of information between feature maps of different resolutions, enhancing the model's ability to detect objects of multiple sizes.

[0040] Step 7: Calculate the CIoU loss to comprehensively evaluate the similarity between the predicted box and the actual box, including the overlap area, center point distance, aspect ratio and scale; it can be expressed as:

[0041]

[0042]

[0043] where β I It represents the result of the overlap between the output prediction box A and the actual box B compared to the set of two prediction boxes, ρ(A ct ,B ct ) represents the distance between the center points of the predicted box A and the measured box B in the image, c represents the slant distance length of the minimum bounding box represented by c, v represents the similarity of the aspect ratio of the predicted box and the actual box, w A 、w B Indicates the length of the predicted box A and the actual box B, h A 、h B Represents the height of the predicted box A and the actual box B, and α represents the weight coefficient.

[0044] Step 8: Train the improved SSD algorithm by setting the following parameters: the input image is 300×300, the batch_size is set to 32, the learning rate learning_rate is set to 0.0005, and a total of 200 iterations of training are performed.

[0045] Evaluate the trained model using test samples and calculate parameters such as mAP and FPS to assess the model's accuracy and real-time performance. In object detection, FPS refers to the number of images processed per second. The total number of correctly detected positive samples is TP, the total number of falsely detected positive samples is FP, the total number of correctly detected negative samples is TN, and the total number of falsely detected negative samples is FN.

[0046]

[0047] The AP value represents the comprehensive evaluation of a certain category. The higher the AP value, the higher the accuracy of a certain category. The mAP value is an evaluation of the entire network, where C represents the number of categories contained in the dataset and c represents a single category.

[0048] In summary, the improved SSD model proposed in this paper differs from traditional convolutional neural network models in that it introduces RFB-Net, a sub-pixel convolution layer, and an MFPN model, and uses the CIoU loss function and K-means clustering algorithm to optimize the prior bounding box setting. This significantly improves the real-time performance and accuracy of airport site target detection and recognition. These improvements enhance the detection performance of various targets in complex scenes while maintaining model efficiency. Description of the drawings:

[0049] Figure 1 Improved SSD networking;

[0050] Figure 2 RFB and RFB-S network structures;

[0051] Figure 3 Feature map adjustment process;

[0052] Figure 4 MFPN network model; Specific implementation method:

[0053] A real-time apron target detection and recognition method based on an improved SSD model includes the following steps:

[0054] Step 1: Read the video from the equipment surveillance camera and create a dataset containing 10,000 airport scene recognition targets. The dataset includes video data of the movement of aircraft, ground support vehicles, and ground personnel, as well as surveillance video data of key aircraft parts during interaction. The training set and test set are randomly divided into 7:3, with 7,000 images as the training set and the other 3,000 images as the test set to verify the effectiveness of the improved SSD algorithm.

[0055] Step 2: Calibrate each scene image and generate a corresponding text file; the text file contains the target frame position information and target category information for each target in the corresponding scene image. Specifically, the targets such as aircraft, motor vehicles, and staff in the scene image are enclosed in target frames and their categories are selected. The aircraft is enclosed in a target frame and the target category is marked as "Airplane". At the same time, a text file containing the target frame position and target category information for each target is generated. For example, each line of text in this text file is set to correspond to a target, and each line of text includes five parameters. The first parameter represents the target category code, and the next four parameters represent the target frame center coordinates and the ratio of the target frame length and width to the entire image.

[0056] Step 3: The calibrated image is cropped to a size of 300×300×3 and passed into the improved SSD network. The two newly added layers of SSD are replaced with two RFB structures on the RFB-Net300 backbone network, and conv4_3 and conv7_fc are connected to RFB-s and RFB structures respectively before connecting to the prediction layer. The 3×3 convolution kernel stride of the original Conv4_3 layer of SSD is modified to 1. After the feature layer passes through a series of residual structures corresponding to the Conv4_3 layer, a 38×38×256 feature matrix is ​​obtained; the remaining convolution layers are generated by convolution, batch normalization, and RELU. The RFB layer is generated by using three convolution kernels of different sizes: 1×1, 3×3, and 5×5 to form a multi-branch structure, and three dilated convolution layers (such as 3*3conv, rate=1) are introduced for generation.

[0057] Step 4: Optimize and generate prediction boxes. First, extract the width and height of all target boxes from the preprocessed airport scene dataset, and combine the width and height of each target box into a data point. Then, determine the number of clusters k and initialize the cluster center. Next, for each target box, calculate its distance d to each cluster center, assign each target box to the cluster closest to the cluster center, and recalculate the cluster center point of each cluster, that is, the average of the width and height of all target boxes. Repeat the steps of calculating distance, assigning clusters, and updating cluster centers until the cluster center changes very little. Finally, the obtained cluster center point is used as the size of the prior box to improve the target detection process of the SSD model. In this way, the size of the prior box can be optimized and the detection performance of the SSD model can be improved.

[0058] d(box,centroid)=1-IoU(box,centroid)#(1)

[0059]

[0060] Step 5: Introduce a batch normalization layer to perform feature normalization between convolution and ReLU activation to stabilize the training process. The sub-pixel convolution layer replaces traditional upsampling. The lower-resolution feature map N×(C×r×r)×W×H is transformed through sub-pixel convolution and multi-channel reorganization to output a high-resolution feature map N×C(H×r)×(W×r). The sub-pixel convolution layer contains multiple convolution kernels, each corresponding to a sub-pixel position in the upsampling process, where N is the image size, C is the number of channels, W is the width, H is the height, and r is the upsampling multiple. Through sub-pixel convolution and channel reorganization, the feature map resolution is effectively improved and key details are preserved.

[0061] Step 6: Using the MFPN model, a feature pyramid is constructed. First, a bottom-up path is constructed to extract low-level features layer by layer from the preprocessed image, generating a series of feature maps with decreasing resolution. A top-down path is constructed to upsample the highest-level feature map from the bottom-up layer, increasing its resolution to the same as the lower-level feature maps. The upsampled high-level feature map is then horizontally connected to the corresponding bottom-up feature map to fuse the high-level semantic information with the low-level detail information. In each layer, the bottom-up feature map is connected to the top-down feature map via horizontal connections, and feature fusion is performed through convolution operations, preserving the low-level spatial information and the high-level semantic information. Then, through upsampling and downsampling operations, information exchange is achieved between feature maps of different resolutions in the feature pyramid. Finally, the fused feature map is input into the object detection network to generate the final object detection result.

[0062] Step 7: Calculate the CIoU loss to comprehensively evaluate the similarity between the predicted box and the actual box, including the overlap area, center point distance, aspect ratio and scale; it can be expressed as:

[0063]

[0064]

[0065] where β I It represents the result of the overlap between the output prediction box A and the actual box B compared to the set of two prediction boxes, ρ(A ct ,B ct ) represents the distance between the center points of the predicted box A and the measured box B in the image, c represents the slant distance length of the minimum bounding box represented by c, v represents the similarity of the aspect ratio of the predicted box and the actual box, w A 、w B Indicates the length of the predicted box A and the actual box B, h A 、h BRepresents the height of the predicted box A and the actual box B, and α represents the weight coefficient. The CIoU loss is calculated by combining IoU, center point distance and aspect ratio difference. β c The smaller the value, the higher the similarity between the predicted box and the actual box.

[0066] Step 8: Train the improved SSD algorithm by setting the following parameters: the input image is 300×300, the batch_size is set to 32, the learning rate learning_rate is set to 0.0005, and a total of 200 iterations of training are performed.

[0067] Evaluate the trained model using test samples and calculate parameters such as mAP and FPS to assess the model's accuracy and real-time performance. In object detection, FPS refers to the number of images processed per second. The total number of correctly detected positive samples is TP, the total number of falsely detected positive samples is FP, the total number of correctly detected negative samples is TN, and the total number of falsely detected negative samples is FN.

[0068]

[0069] The AP value represents the comprehensive evaluation of a certain category. The higher the AP value, the higher the accuracy of a certain category. The mAP value is an evaluation of the entire network, where C represents the number of categories contained in the dataset and c represents a single category.

Claims

1. A real-time apron target detection and recognition method based on an improved SSD model, characterized in that: include: Acquire an airport surface operation dataset, the airport surface operation dataset including video data of movement behaviors of aircraft, ground support vehicles, and surface personnel, as well as surveillance video data of key aircraft parts during interactions. The ground support vehicles include boarding bridges, aircraft tractors, passenger elevators, water supply trucks, and fuel trucks. The surface personnel include maintenance personnel, traffic control personnel, cargo handling personnel, and passenger service personnel. The movement behaviors include single-target movement behaviors and multi-target interactive behaviors. The single-target movement behaviors include aircraft arrival and departure. The multi-target interactive behaviors include docking and separation between ground support vehicles, surface personnel, and aircraft. The key aircraft parts include the nose, tail, wingtips, and engine air intakes. Preprocessing the airport surface operation data set, calibrating each frame of the scene image, and performing standardization and normalization on the image, wherein the preprocessed image includes the position of each target frame and target category information during the airport surface operation and maintenance process; Inputting the pre-processed airport surface operation data set into an improved target recognition algorithm SSD model for training to obtain a trained improved SSD model; The improved SSD model uses RFB-Net300 as the backbone network of the SSD algorithm, adds batch-normalization in the additional layer, uses sub-pixel convolution layer instead of upsampling, and adds MFPN model to fuse low-level and high-level feature information; Loss function to improve the original loss function; Input the prediction results into the trained improved SSD model to obtain the target detection results; The step of generating a prediction frame includes: Considering that RFB-Net sets the size of the prior box based on human experience and is overly dependent on experience, the K-means clustering algorithm is used to set the optimal prior box; Determine the distance calculation formula between target objects and initialize the required number of cluster centers; Compare the distance d between each actual location box and each cluster center in turn, and assign the actual location box to the cluster with the closest distance to the cluster center; After the allocation is completed, the cluster center point of each cluster is recalculated, that is, the average length and width of all actual position boxes in the cluster are calculated; Repeat the above steps until the cluster center changes little; Set the final calculated cluster center point to the size of the prior box on the feature map; The improved SSD model RFB-Net, Sub-pixel convolution layer and MFPN model, the RFB-Net300 backbone network replaces the two newly added layers of SSD with two RFB structures, and conv4_3 and conv7_fc are connected to RFB-s and RFB structures respectively before connecting to the prediction layer. The BN layer is located between the convolution layer Conv4_3 and Relu to normalize the feature map. The Sub-pixel convolution layer replaces upsampling, and the low-resolution feature map is obtained by convolution and multi-channel reorganization to obtain a high-resolution feature map. The MFPN model integrates a feature pyramid network FPN structure with a bottom-up and a top-down line horizontal connection; The stride of the original 3×3 convolution kernel of the SSD Conv4_3 layer is modified to 1. The preprocessed 300×300×3 image is input into it. The feature layer passes through a series of residual structures corresponding to the Conv4_3 layer to obtain a 38×38×256 feature matrix. The remaining convolution layers are generated through convolution, batch normalization, and RELU. The RFB layer is generated by using three different sizes of convolution kernels to form a multi-branch structure, and introducing three dilated convolution layers. described The loss function not only considers the bounding box overlap area IoU, but also the center point distance, aspect ratio and scale of the bounding box to achieve a more comprehensive bounding box evaluation; Loss Function The expression is （1） （2） （3） （4） in It indicates that the overlapped part of the output prediction box A and the actual box B is greater than the result of the two prediction box sets. Indicates the distance between the center points of the predicted box A and the measured box B in the image, Indicates that both the predicted box A and the actual box B are contained in The minimum slope distance length in the bounding box represented by Indicates the similarity of the aspect ratio between the predicted box and the actual box, 、 Indicates the length of the predicted box A and the actual box B, 、 Indicates the height of the predicted box A and the actual box B, Represents the weight coefficient.

2. The real-time apron target detection and recognition method based on the improved SSD model according to claim 1 is characterized in that: The expression for calculating the distance between target objects is: （5） （6） Among them, box represents the actual location box of the vehicle, and centroid represents the cluster center. Indicates the distance from the box to the centroid, It is the intersection of box and centroid. The smaller the distance d, the better. The larger the value, the better.

3. The real-time apron target detection and recognition method based on the improved SSD model according to claim 1 is characterized in that: The layer converts the lower resolution feature map Output high-resolution feature maps through sub-pixel convolution and multi-channel reorganization , where the sub-pixel convolution layer contains multiple convolution kernels, each of which corresponds to a sub-pixel position in the upsampling process, where N is the image size, C is the number of channels, W is the width, H is the height, and r is the upsampling multiple.

4. The real-time apron target detection and recognition method based on the improved SSD model according to claim 1 is characterized in that: The MFPN model captures multi-scale features by fusing feature maps of different levels to detect targets of different sizes. The model uses a top-down path to transmit high-level semantic information, while using a bottom-up path to retain low-level detail information. It also uses lateral connections to enhance communication between features. MFPN constructs a feature pyramid structure, and exchanges information between feature maps of different resolutions through upsampling and downsampling operations. In the bottom-up process, the size of the feature map becomes smaller after convolution, while in the top-down process, the high-level feature map is upsampled, and the feature map is horizontally connected and fused with the bottom-up feature map after sampling.

5. The real-time apron target detection and recognition method based on the improved SSD model according to claim 1 is characterized in that: For the trained network model, the test sample is used as input to obtain the output prediction result, which is then compared with the true value of the sample to calculate mAP and FPS.

Images