Airport platform vehicle identification method and system based on image multitask

Abstract

The application relates to an airport platform identification method and system based on image multitasking. The method can realize simultaneous processing of two tasks of rough positioning and fine segmentation, can more accurately position and segment an airport platform vehicle related area in an airport platform, output airport vehicle orientation information and information of a side surface, a docking surface and a top surface of the airport platform vehicle, thereby realizing efficient and accurate airport platform vehicle detection and segmentation, can effectively solve problems existing in a traditional method, improve the accuracy of detection and segmentation, reduce the complexity of an algorithm, improve the generalization ability of a model, realize efficient and accurate airport platform vehicle detection and segmentation, realize monocular image airport platform vehicle target detection, key point detection and semantic segmentation, and the prediction time can meet real-time requirements of a vehicle end.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an airport platform vehicle recognition method and system based on image multitasking. Background Technology

[0002] Currently, airport platform vehicles play an indispensable role in baggage transfer, primarily responsible for loading passengers' luggage onto aircraft. Traditional baggage transfer methods require significant manpower and time to operate these vehicles, making them prone to errors and delays. With advancements in camera sensing and computing technologies, the use of monocular vision, stereo vision, and vision-based multi-sensor fusion has become a very active research direction in the field of intelligent driving. Some robots or automated devices are also being used for baggage transfer. The first step in achieving automated transfer is the detection, localization, and fine segmentation of airport platform vehicles, enabling unmanned transfer vehicles to identify, track, and connect with them. However, traditional airport platform vehicle detection and segmentation methods typically use separate algorithms. These automated detection and segmentation technologies still suffer from several problems, such as low localization accuracy, poor segmentation results, high algorithm complexity, poor robustness, and slow inference speed. Furthermore, these methods cannot identify the front, side, rear, and orientation information of the airport platform vehicle within a 2D bounding box, nor can they identify docking area information. Simultaneously, the lack of connection between multiple models prevents them from mutually reinforcing each other and increases hardware resource consumption. Current multi-task learning methods for detection typically employ an anchor-based approach, which increases the complexity of the detection head and the number of generated results, thereby reducing the model's inference speed. Traditional image processing methods can only complete a single task and struggle to simultaneously handle both long-range coarse localization and close-range fine segmentation of the docking surface, resulting in low accuracy and efficiency in airport platform vehicle detection and segmentation. Summary of the Invention

[0003] Therefore, it is necessary to provide an image-based multi-task airport platform vehicle recognition method and system to address the aforementioned technical problems.

[0004] An airport platform recognition method based on image multi-task, the method comprising: Acquire the image to be tested and preprocess the image to be tested.

[0005] The preprocessed image to be tested is input into a trained object detection and segmentation network to obtain the detection prediction results of the 2D bounding box of the airport platform vehicle, the four corner points, and the segmentation prediction results of the three planes. The object detection and segmentation network includes a backbone network, a Neck feature fusion network, a detection head, and a segmentation head. The backbone network is used to extract features at different levels of the preprocessed image. The Neck feature fusion network is used to fuse features at different levels to obtain fused features at different feature scales. The detection head is used to detect the 2D bounding box and the four corner points of the airport platform vehicle using multiple different detection layers based on the fused features at each feature scale. The segmentation head is used to segment the three planes of the airport platform vehicle based on features at different levels and fused features at different feature scales. The three planes include the top surface, side surface, and rear docking surface of the airport platform vehicle.

[0006] Based on the prediction results of the detection head, the position and orientation of the airport platform vehicle in the 2D bounding box and vehicle coordinate system of the image to be tested are determined. Based on the segmentation prediction results of the segmentation head, the boundary coordinates of the three planar segmentation regions are determined.

[0007] An airport platform vehicle recognition system based on image multitasking, comprising: an unmanned transfer vehicle, an image acquisition device, a processor, a memory, and a control system.

[0008] The unmanned transfer vehicle is used to tow or connect with the airport platform vehicle.

[0009] The image acquisition device is installed on the unmanned transport vehicle and is used to acquire images and transmit the images to the processor.

[0010] The processor is configured to execute any of the above-described image-based multi-task airport platform recognition methods to perform target detection and segmentation in the image, obtain detection and segmentation results, and transmit the detection and segmentation results to the control system.

[0011] The memory is connected to the processor and is used to store the multi-task model and related data.

[0012] The control system is connected to the unmanned transport vehicle and is used to control the movement of the unmanned transport vehicle based on the detection and segmentation results.

[0013] The aforementioned image-based multi-task airport platform vehicle recognition method and system can simultaneously process two tasks: coarse localization and fine segmentation. This allows for more accurate localization and segmentation of the relevant areas of the airport platform vehicle within the airport platform, outputting the vehicle's orientation information, as well as information on its sides, docking surfaces, and top surface. This achieves efficient and accurate airport platform vehicle detection and segmentation, effectively addressing the problems of traditional methods, improving detection and segmentation accuracy, reducing algorithm complexity, enhancing model generalization ability, and enabling monocular image-based target detection, key point detection, and semantic segmentation of the airport platform vehicle. The prediction time can meet the real-time requirements of the vehicle-side. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating an airport platform recognition method based on image multitasking in one embodiment; Figure 2 Here is a diagram of the object detection and segmentation network structure in another embodiment; Figure 3 This is a schematic diagram of the algorithm's identification points in another embodiment; Figure 4 This is a block diagram of an airport platform recognition system based on image multitasking in one embodiment. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0016] The image-based multi-task airport platform recognition method provided in this application can be used to detect the 2D bounding box of an airport platform vehicle, determine its orientation, and segment its three planes. It can simultaneously process two tasks: coarse long-range localization of the airport platform vehicle and fine segmentation of the docking surface at close range, enabling more accurate localization and segmentation of the relevant regions of the airport platform vehicle within the airport platform. This method employs a deep learning single-stage image multi-task network to achieve target detection, keypoint detection, and semantic segmentation of the airport platform vehicle in monocular images, and the prediction time meets the real-time requirements of the vehicle.

[0017] In one embodiment, such as Figure 1 As shown, an airport platform recognition method based on image multi-task is provided, which includes the following steps: Step 100: Acquire the image to be tested and preprocess it.

[0018] Specifically, preprocessing is performed on the image to be tested, including noise removal, image resizing, and formatting. These preprocessing operations can be implemented using image processing software, corresponding libraries in programming languages, or other relevant tools.

[0019] Step 102: Input the preprocessed image to be tested into the trained object detection and segmentation network to obtain the detection prediction results of the 2D bounding box of the airport platform vehicle, the four corner points, and the segmentation prediction results of the three planes. The object detection and segmentation network includes: a backbone network, a Neck feature fusion network, a detection head, and a segmentation head. The backbone network is used to extract features at different levels of the preprocessed image. The Neck feature fusion network is used to fuse features at different levels to obtain fused features at different feature scales. The detection head is used to predict the 2D bounding box and the four corner points of the airport platform vehicle using multiple different detection layers based on the fused features at each feature scale. The segmentation head is used to segment the three planes of the airport platform vehicle based on features at different levels and fused features at different feature scales. The three planes include: the top surface, the side surface, and the rear docking surface of the airport platform vehicle.

[0020] Specifically, the detection head uses multiple detection layers to predict at each feature scale. Our detection branch is based on the anchor-free paradigm. Instead of using preset anchors to generate target boxes, it uses a heatmap to calculate the true prediction value by introducing the Gaussian distribution region of the prediction points and directly regressing the target box size. Finally, the predicted box can be obtained based on the target box size and the center point position of the target box. Therefore, there is no deviation relative to the anchor box in this algorithm.

[0021] In this embodiment, the segmentation head predicts multiple layers at each feature scale and fuses the features extracted by the backbone network, resulting in two path branches: a global feature branch and a detail feature branch. This structure establishes connections between low-level and high-level features, thereby improving the accuracy and efficiency of segmentation. The entire network operates at multiple scales to adapt to airport platform vehicles of different sizes and shapes.

[0022] The structure of an object detection and segmentation network is as follows: Figure 2 As shown.

[0023] In the prediction results, the prediction output is:

[0024] in: To determine which category's Bayesian posterior probability will be output, .

[0025] It realizes multi-task detection and segmentation based on images, which can distinguish the various faces of the airport platform vehicle in the image, determine the orientation of the airport platform vehicle, and identify the three faces of the airport platform vehicle.

[0026] Step 104: Based on the prediction results of the detection head, determine the position and orientation of the airport platform vehicle in the 2D bounding box and vehicle coordinate system of the image to be tested. Based on the segmentation prediction results of the segmentation head, determine the boundary coordinates of the three planar segmentation regions.

[0027] In the aforementioned image-based multi-task airport platform recognition method, the method can simultaneously process two tasks: coarse localization and fine segmentation. This enables more accurate localization and segmentation of the relevant areas of the airport platform vehicle within the airport platform, outputting the vehicle's orientation information, as well as information about its sides, docking surfaces, and top surface. This achieves efficient and accurate airport platform vehicle detection and segmentation, effectively addressing the problems of traditional methods, improving detection and segmentation accuracy, reducing algorithm complexity, enhancing model generalization ability, and realizing efficient and accurate airport platform vehicle detection and segmentation. It also enables monocular image-based airport platform vehicle target detection, key point detection, and semantic segmentation, with prediction time meeting the real-time requirements of the vehicle.

[0028] In one embodiment, the following specific steps are included before step 100: Step 200: Obtain the original image and preprocess it to obtain training samples.

[0029] Step 202: Construct the object detection and segmentation network.

[0030] Step 204: Train the object detection and segmentation network using training samples to obtain a trained object detection and segmentation network.

[0031] Specifically, training the network parameters of the target detection and segmentation network includes two branches: detection and segmentation. The main process is as follows: (1) Read a batch of airport platform vehicle data. For any frame of data within the batch, the training data includes image samples and the label corresponding to each task on the image sample. The detection branch encodes the target and the preset anchor, and the segmentation branch performs one-hot encoding of the data. This indicates the construction of a structure with a length equal to The vector of the range, and the first Set one element to 1 and all other elements to 0; (2) Input data from the read batch data is fed into the multi-task network. The detection branch obtains multi-level multi-task prediction values, and the segmentation branch obtains the Onehot segmentation label ground truth value. (3) Calculate the overall evaluation loss function based on the reference multi-task ground truth and multi-task prediction, including two branches: target and key point detection and semantic segmentation; (4) The Adam optimization method is used to optimize the network parameters of the target detection and segmentation network. Training is stopped when the number of training rounds exceeds the threshold, and the multi-task network that minimizes the overall evaluation loss function is output.

[0032] In one embodiment, the backbone network is a ResNeXt network; step 204 specifically includes the following steps: Step 300: Input the training samples into the ResNeXt network to obtain features at different levels.

[0033] Step 302: Input features at different levels into the Neck feature fusion network to obtain fused features at different feature scales.

[0034] Step 304: Input the fused features at different feature scales into the detection head to obtain the detection prediction results of the 2D bounding box of the airport platform vehicle and the four corner points.

[0035] Step 306: Input the features at different levels and the fused features at different feature scales into the segmentation head to obtain the segmentation prediction results.

[0036] Step 308: Based on the training samples, detection prediction results, and segmentation prediction results, the Adam optimization method is used to optimize and train the object detection and segmentation network to obtain the trained object detection and segmentation network.

[0037] Specifically, the backbone network is composed of ResNeXt, which enables the multi-task network to balance feature extraction capability and prediction speed.

[0038] In one embodiment, the Neck feature fusion network is a weighted bidirectional feature pyramid network.

[0039] Specifically, the Neck feature fusion network consists of a weighted bidirectional feature pyramid network (BiFPN), which achieves simple and fast multi-scale feature fusion. Its output has L feature scales. Compared to the output of the i-th feature scale, the feature map at the (i+1)-th feature scale has half the length and width of the feature map at the i-th feature scale. For a detailed implementation of BiFPN, please refer to "EfficientDet: Scalable and Efficient Object Detection CVPR 2020", therefore, it will not be elaborated upon in this embodiment.

[0040] In one embodiment, the detection head includes a target detection layer, a category detection layer, a bounding box deviation detection layer, a key point visibility detection layer, and a key point deviation detection layer; step 304 specifically includes the following steps: Step 400: Input the fused features at different feature scales into the target detection layer to obtain the target detection prediction results for the airport platform vehicle.

[0041] Step 402: Input the fused features at different feature scales into the category detection layer to obtain the airport platform vehicle category prediction results.

[0042] Step 404: Input the fused features at different feature scales into the target box deviation detection layer to obtain the target box deviation prediction result.

[0043] Step 406: Input the fused features at different feature scales into the key point visibility detection layer to obtain four key point prediction results.

[0044] Step 408: Input the fused features at different feature scales into the key point deviation detection layer to obtain the key point deviation prediction results.

[0045] Step 410: After passing through a transformation layer, the prediction results of the airport platform vehicle target detection, the prediction results of the airport platform vehicle type, the prediction results of the target bounding box deviation, the prediction results of the four key points, and the prediction results of the key point deviation are stitched together to obtain the prediction and detection results of the 2D bounding box of the airport platform vehicle and the four corner points.

[0046] In one embodiment, the object detection layer includes a convolutional layer and an activation layer. The category detection layer includes a convolutional layer and an activation layer.

[0047] Specifically, the target detection layer has an output channel of 1, which represents the confidence level of the airport platform vehicle and the offset prediction of the center point, thereby reducing the accuracy error of the target center point.

[0048] The category detection layer has N output channels, which represent the confidence scores of each airport platform vehicle type label, where N is the total number of defined airport platform vehicle types.

[0049] In one embodiment, the target box deviation detection layer employs an improved anchor-free algorithm. The improved anchor-free algorithm generates target boxes using heatmaps and introduces Gaussian distribution regions of prediction points to calculate the true prediction values. Step 404 includes: inputting fused features of different feature scales into the target box deviation detection layer, processing them using the improved anchor-free algorithm, and obtaining the deviation of the x-coordinate of the target box center point, the deviation of the y-coordinate of the target box center point, the deviation of the target box width, and the deviation of the target box height.

[0050] Specifically, the target bounding box deviation detection layer has four output channels, representing the deviations relative to the target bounding box center point's x-coordinate, y-coordinate, width, and height. In the anchor-free algorithm, target bounding boxes are not generated using pre-defined anchors; instead, the center point and dimensions are directly predicted from the feature map. Therefore, there is no deviation relative to the anchor boxes in this algorithm. A heatmap is used to implement this, introducing a Gaussian distribution region of the predicted points to calculate the true predicted values. Thus, the detection layer includes channels representing the deviations relative to the target bounding box center point's x-coordinate, y-coordinate, width, and height.

[0051] In one embodiment, the key point visibility detection layer includes: a convolutional layer and multiple depthwise separable convolutional layers; step 406 includes: inputting fused features of different feature scales into the key point visibility detection layer, and after passing through a convolutional layer and multiple depthwise separable convolutional layers, obtaining four key point prediction results; the four key point prediction results are the confidence scores of the visibility of the four key points of the airport platform vehicle target: the left rear, right rear, left front, and right front.

[0052] Specifically, the key point visibility detection layer has 4 output channels, which represent the confidence level of whether the four key points of the airport platform vehicle target (left rear, right rear, left front, and right front) are visible, with a value range of [0,1].

[0053] The main function of convolutional layers is to extract features. The method of feature extraction is to perform a dot product between the original image pixels and the weight parameters in the convolution kernel. To obtain more feature information, convolutional blocks of different sizes are needed.

[0054] Depthwise separable convolution mainly consists of two processes: depthwise convolution and pointwise convolution. In depthwise convolution, the number of kernels is the same as the number of channels in the previous layer. One kernel is responsible for one channel, and each channel is convolved by only one kernel. The number of channels in the resulting feature map is exactly the same as the number of input channels. Pointwise convolution is very similar to regular convolution. Its kernel size is 1×1×M, where M is the number of channels in the previous layer. Therefore, this convolution operation weights and combines the feature maps output from depthwise convolution in the depth direction to generate new feature maps. The number of output feature maps equals the number of convolution kernels.

[0055] In one embodiment, the key point deviation detection layer includes a convolutional layer; step 408 includes: inputting fused features of different feature scales into the key point deviation detection layer, and after processing by the convolutional layer, obtaining the key point deviation prediction result, which is the deviation of the xy coordinates of the four key points of the airport platform vehicle target (left rear, right rear, left front, and right front) relative to the xy coordinates of the target box center point.

[0056] Specifically, the key point deviation detection layer has 8 output channels, which respectively represent the deviations of the xy coordinates of the four key points of the airport platform vehicle target (left rear, right rear, left front, and right front) from the target center point xy coordinates.

[0057] In one embodiment, step 410 includes: concatenating the airport platform vehicle target detection prediction result, airport platform vehicle type prediction result, target bounding box deviation prediction result, four key point prediction results, and key point deviation prediction result after passing them through a transformation layer to obtain the predicted detection result of the airport platform vehicle 2D bounding box and four corner points: (1) in, The output is the concatenated data; the input is... It is a four-dimensional vector, with each dimension denoted as . ,in, The number of data in the batch. This indicates the number of predicted feature maps. Represents the height of the feature map in the 2D feature space. The width of the feature map represents the 2D feature space; The function represents the input Ordered by spatial dimensions Rearrange; The function represents the input Transformed into a three-dimensional vector, with a length of in each dimension. ,in, The asterisk (*) indicates a multiplication operation. This represents the total number of feature scales. The function represents the output at each feature scale. Perform splicing according to the first dimension.

[0058] In one embodiment, step 306 includes: passing the output features of the last convolutional layer of the backbone network through a global pooling layer to obtain global high-level features, and then using the output features of the Neck feature fusion network as detailed low-level features; using a segmentation head to predict the features output by multiple layers at each feature scale, and fusing the obtained prediction results with the global high-level features to obtain a segmentation prediction result.

[0059] Specifically, the feature aggregation module combines low-level detailed features obtained from the Neck with high-level features extracted from the backbone network, resulting in two branching paths: a global feature branch and a detailed feature branch. The segmentation head predicts multiple layers at each feature scale, establishing connections between low-level and high-level features, thereby improving segmentation accuracy and efficiency. The entire network operates at multiple scales to adapt to airport platform vehicles of different sizes and shapes.

[0060] The semantic segmentation branch's features are fused from multiple different levels of features, including low-level and high-level features. These features have undergone multiple downsampling steps, so the results need to pass through a feature aggregation module and undergo multiple upsampling steps to finally restore the original image size. The segmentation result includes pixel prediction probability values ​​for three categories: top surface, side surface, and tail surface, plus the background. These features include global features extracted by the backbone network and multi-scale features obtained from networks such as the pyramid network. After the final convolutional layer in feature extraction, a global pooling layer is used to capture the global information of the entire image. This global pooling layer reduces the dimensionality of the feature map of each channel into a single value, allowing a single vector to represent the global information of the entire image.

[0061] To preserve image detail and suppress noise, this invention uses a bilateral filter to process both global and local features. A bilateral filter is a non-linear filter that smooths the image while preserving edge information. By using a bilateral filter, image detail can be preserved, thereby improving the accuracy of segmentation branches.

[0062] To fuse multi-scale features, an attention module feature aggregation is used to combine global high-level features and local low-level features to generate the final semantic segmentation result. This fusion module includes an adaptive average pooling layer, two 1*1 convolutional layers, and an upsampling layer to fuse features from global and local branches. It can learn channel attention weights, helping the network to focus more on important features and improve feature representation capabilities. The main implementation is as follows: First, the output features of high-level and low-level features are concatenated; then, the feature scale is balanced through batch normalization. Next, the concatenated features are pooled into a feature vector, and a weight vector is calculated. This weight vector can reweight the features, playing a role in feature selection and combination, and feature upsampling is performed to produce the final segmentation result.

[0063] In one embodiment, step 104 includes: traversing the prediction outputs of all detection layers of the detection head, storing the prediction outputs that meet the classification score threshold into a first output target set; traversing the first output target set, post-processing the data in the set to obtain valid targets; calculating the angle between each valid target in the first output target set and the y-axis in the vehicle body BEV coordinate system, obtaining the predicted category, angle, and center coordinates, width, and height of the 2D bounding box for each valid target; outputting the position and orientation of the airport vehicle target in the 2D bounding box and vehicle body coordinate system in the image to be tested, and outputting it according to the segmentation prediction results of the segmentation head.

[0064] In a specific embodiment, the specific steps for training the network parameters of the object detection and segmentation network include: Step S1: Read a batch of data. For the detection branch, encode the target in any frame of data within the batch to obtain the reference multi-task ground truth. For the segmentation branch, obtain all ground truth values ​​of the data within the batch. The number of data read is consistent with that of the detection branch. Preprocess the image. The specific implementation process is as follows: Step S1.1: First, the original image needs to be preprocessed, including noise removal, image resizing, and formatting. These preprocessing operations can be implemented using image processing software, corresponding libraries in programming languages, or other relevant tools.

[0065] Step S1.2: Let This is a series of feature points. Since anchors are no longer used, there's no need to define anchor boxes and offsets. We can treat each feature point as the center of the object and use a heatmap to generate a Gaussian distribution to predict the object's center point. The center coordinates (x, y) of each feature point will have the highest response in the heatmap, indicating that this point is most likely the center of the object. A fixed-size rectangle is generated centered on this feature point. When generating the rectangle, a Gaussian distribution can be used to weight the pixel values ​​near the feature point, thus calculating the feature point's position more accurately. x, y represent the feature point's coordinates on the image axes, and w, h represent the width and height of the center point relative to the rectangle.

[0066] Step S1.3: For the selection of positive and negative samples, we adopt a similar idea to Focal Loss. In actual training, the loss of other points around the center point (negative center pointer) is a decayed loss, while the length and width of the target are obtained by regression of w and h corresponding to the current center point. By treating the other points around the center point as negative samples and regressing the length and width of the target, we can obtain the initial positive samples. Because the difficult-to-classify positive and negative samples are usually in the minority, and this method uses other surrounding points for loss calculation, the model pays more attention to the difficult-to-classify samples, thus effectively obtaining the initial positive and negative samples.

[0067] Step S1.4: Since the positive and negative samples after the initial screening are not accurate enough, we combine the Optimal Transport Assignment (OTA) sample selection strategy. This strategy is existing technology, so it will not be described in detail in this embodiment. First, calculate the Intersection over Union (IOU) of the initial screening positive samples and the confidence scores, and sort the IOUs from largest to smallest. Take the sum of the top ten and round it down, denoted as b. Then, calculate the cost function of the initial screening positive samples. Its components include position loss and class loss. The position loss is obtained by calculating the IOU of the previous initial screening positive samples using -torch.log, while the class loss is calculated by calculating the cross-entropy loss of the target classification. The cost function is the weighted sum of the two losses. Then, sort the cost functions from smallest to largest and take the top b samples as positive samples. At the same time, consider the case where the same grid prediction box is associated with two confidence scores. Take the smaller cosine value, and the prediction box is the positive sample of the corresponding confidence score.

[0068] Step S2: Input the data from the read batch data into the multi-task object detection network to obtain the multi-task prediction values:

[0069] in, The target confidence prediction value, This is the predicted score for the airport vehicle category. This is the predicted confidence level of key points for airport vehicles. This is the predicted value of the 2D box deviation. This represents the predicted value of the key point deviation. This represents the predicted value for the segmentation result.

[0070] Step S3: Calculate the overall evaluation loss function based on the true value and predicted value of each multi-task.

[0071] Step S4: The Adam optimization method is used to optimize and train the network parameters of the object detection and segmentation network. Training is stopped when the number of training rounds exceeds a threshold, and the object detection and segmentation network that minimizes the overall evaluation loss function is output.

[0072] After the detection head performs multi-feature scale prediction, for each detection head, a transformation layer is used to concatenate the multi-feature scale predictions, and the concatenated output is shown in Equation (1).

[0073] This method can employ traditional image processing techniques to complete a single task, such as localizing or segmenting an airport platform vehicle. However, the accuracy and efficiency of such methods are inferior to the multi-task object detection and segmentation method of this invention. Performing only a single task can lead to a decrease in the algorithm's accuracy and real-time performance. Other deep learning networks can be used to implement multi-task image processing, such as two-stage networks or other types of single-stage networks. However, these methods may require more computational resources and time, and may not be able to complete the task in real-time on the vehicle. Other image preprocessing methods can be used, such as using different denoising algorithms or not adjusting the image format. However, these alternatives may reduce image quality and the algorithm's robustness. Certain parts of the backbone network, Neck feature fusion network, detection head, and segmentation head can be replaced; for example, using a standard ResNet backbone network and an FPN fusion network can achieve the same functionality. However, these alternatives may affect the algorithm's accuracy and real-time performance; ResNet's feature extraction capability is weaker than ResNeXt, and the model using FPN is less accurate than BiFPN. The most basic implementation method is to use traditional image processing methods for a single task, such as coarse localization of an airport platform vehicle. This method can achieve basic localization, but it cannot perform fine segmentation and multi-task processing. Its accuracy and efficiency are inferior to the multi-task object detection and segmentation method of this invention.

[0074] The multi-task object detection and segmentation method of this invention has excellent accuracy and real-time performance. Implementing alternative solutions may affect the algorithm's performance to varying degrees. The most basic implementation scheme cannot achieve the high accuracy and high efficiency requirements of this invention. The beneficial effects of this method are as follows: (1) Improved accuracy of localization and segmentation of airport platform vehicles: Traditional image processing methods can only complete a single task and are difficult to handle both long-distance coarse localization and close-range fine segmentation of docking surfaces at the same time. However, the image-based multi-task target detection and segmentation method proposed in this invention can handle both tasks at the same time, thereby more accurately locating and segmenting the relevant areas of airport platform vehicles in the airport platform.

[0075] (2) Improved real-time performance of the algorithm: The present invention adopts a deep learning single-stage image multi-task network, which can complete multiple tasks such as target detection, key point detection and semantic segmentation of airport platform vehicles in real time on the vehicle side, thereby meeting the real-time requirements of the vehicle side.

[0076] (3) Improved robustness of the algorithm: The target detection and segmentation network of the present invention consists of four parts: backbone network, Neck feature fusion network, detection head and segmentation head. It makes full use of the characteristics and advantages of each part, thereby improving the robustness and stability of the algorithm.

[0077] (4) Improved accuracy in judging the orientation of the airport vehicle and the three sides of the airport platform vehicle: The present invention provides an effective method to judge the orientation of the airport vehicle and the three sides of the airport platform vehicle, thereby more accurately grasping the status and position of the airport vehicle and the airport platform vehicle, which helps to improve the safety and efficiency of airport operation.

[0078] In summary, the image-based multi-task target detection and segmentation method of the present invention has the advantages of high localization and segmentation accuracy, good real-time performance, and strong robustness, and can be widely applied in the field of automated management and control of airport platform vehicles.

[0079] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0080] In one embodiment, such as Figure 4 As shown, an airport platform vehicle recognition system based on image multitasking is provided. The system includes: an unmanned transfer vehicle 10, an image acquisition device 20, a processor 30, a memory 40, and a control system 50.

[0081] The driverless transfer vehicle 10 is used to tow or connect with airport platform vehicles.

[0082] An image acquisition device 20, installed on the unmanned transport vehicle 10, is used to acquire images and transmit them to the processor 30. Specifically, the image acquisition device may be, but is not limited to, a camera or a digital camera.

[0083] The processor 30 is used to execute any of the above-mentioned image-based multi-task airport platform recognition methods to perform target detection and segmentation in the image, obtain the detection and segmentation results, and transmit the detection and segmentation results to the control system 50.

[0084] The memory 40 is connected to the processor 30 and is used to store the multi-task model and related data.

[0085] The control system 50 is connected to the unmanned transport vehicle 10 and is used to control the movement of the unmanned transport vehicle 10 based on the detection and segmentation results.

[0086] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An airport platform vehicle recognition method based on image multi-task, characterized in that, The method includes: Acquire the image to be tested and preprocess the image to be tested; The preprocessed image to be tested is input into a trained object detection and segmentation network to obtain the detection prediction results of the 2D bounding box of the airport platform vehicle, the four corner points, and the segmentation prediction results of the three planes. The object detection and segmentation network includes a backbone network, a Neck feature fusion network, a detection head, and a segmentation head. The backbone network is used to extract features at different levels of the preprocessed image. The Neck feature fusion network is used to fuse features at different levels to obtain fused features at different feature scales. The detection head is used to predict the 2D bounding box and the four corner points of the airport platform vehicle using multiple different detection layers based on the fused features at each feature scale. The segmentation head is used to segment the three planes of the airport platform vehicle based on features at different levels and fused features at different feature scales. The three planes include the top surface, side surface, and rear docking surface of the airport platform vehicle. Based on the prediction results of the detection head, the position and orientation of the airport platform vehicle in the 2D bounding box and vehicle coordinate system of the image under test are determined. Based on the segmentation prediction results of the segmentation head, the boundary coordinates of the three planar segmentation regions are determined. Specifically, this includes: traversing the prediction outputs of all detection layers of the detection head and storing the prediction outputs that meet the classification score threshold into a first output target set; traversing the first output target set and post-processing the data in the set to obtain valid targets; calculating the angle between each valid target in the first output target set and the y-axis in the vehicle BEV coordinate system, obtaining the predicted category, angle, center coordinates, width, and height of the 2D bounding box of each valid target; outputting the position and orientation of the airport vehicle target in the 2D bounding box and vehicle coordinate system of the image under test, and outputting the boundary coordinates of the three segmentation planes based on the segmentation prediction results of the segmentation head. The detection head includes a target detection layer, a category detection layer, a target bounding box deviation detection layer, a key point visibility detection layer, and a key point deviation detection layer. The fused features at different feature scales are input into the detection head to obtain the detection prediction results of the airport platform vehicle's 2D bounding box and four corner points, including: The fused features at different feature scales are respectively input into the target detection layer, the category detection layer, the target bounding box deviation detection layer, the key point visibility detection layer, and the key point deviation detection layer to obtain the airport platform vehicle target detection prediction result, the airport platform vehicle category prediction result, the target bounding box deviation prediction result, the four key point prediction results, and the key point deviation prediction result; the four key points are the left rear, right rear, left front, and right front corners of the airport platform vehicle target; The prediction results of the airport platform vehicle target detection, the prediction results of the airport platform vehicle type, the prediction results of the target bounding box deviation, the prediction results of the four key points, and the prediction results of the key point deviation are spliced ​​together after passing through a transformation layer to obtain the prediction and detection results of the 2D bounding box of the airport platform vehicle and the four corner points; the transformation layer is a dimensional transformation operation.

2. The method according to claim 1, characterized in that, Before acquiring the image to be tested and preprocessing the image to be tested, the method further includes: The original image is acquired and preprocessed to obtain training samples; Construct an object detection and segmentation network; The target detection and segmentation network is trained using the training samples to obtain a trained target detection and segmentation network.

3. The method according to claim 2, characterized in that, The backbone network is a ResNeXt network; The target detection and segmentation network is trained using the training samples to obtain a trained target detection and segmentation network, including: The training samples are input into the ResNeXt network to obtain features at different levels; The features at different levels are input into the Neck feature fusion network to obtain fused features at different feature scales; The fused features at different feature scales are input into the detection head to obtain the detection prediction results of the 2D bounding box and four corner points of the airport platform vehicle; The fused features at different levels and at different feature scales are input into the segmentation head to obtain the segmentation prediction result; Based on the training samples, the detection prediction results, and the segmentation prediction results, the target detection and segmentation network is optimized and trained using the Adam optimization method to obtain the trained target detection and segmentation network.

4. The method according to claim 3, characterized in that, The Neck feature fusion network is a weighted bidirectional feature pyramid network.

5. The method according to claim 1, characterized in that, The target detection layer includes a convolutional layer and an activation layer; the category detection layer includes a convolutional layer and an activation layer.

6. The method according to claim 1, characterized in that, The target box deviation detection layer uses an improved Anchor-free algorithm, which generates target boxes using heatmaps and introduces the Gaussian distribution region of the prediction points to calculate the true prediction value. The fused features at different feature scales are input into the target box deviation detection layer to obtain the target box deviation prediction result, including: The fused features at different feature scales are input into the target box deviation detection layer and processed using the improved Anchor-free algorithm to obtain the deviations in the x-coordinate of the target box center point, the y-coordinate of the target box center point, the width of the target box, and the height of the target box.

7. The method according to claim 1, characterized in that, The keypoint visibility detection layer includes a convolutional layer and multiple depth-separable convolutional layers; The fused features at different feature scales are input into the keypoint visibility detection layer to obtain four keypoint prediction results, including: The fused features at different feature scales are input into the key point visibility detection layer. After passing through a convolutional layer and multiple depthwise separable convolutional layers, four key point prediction results are obtained. The four key point prediction results are the confidence scores of the visibility of the four key points of the airport platform vehicle target: the left rear, right rear, left front, and right front.

8. The method according to claim 1, characterized in that, The key point deviation detection layer includes a convolutional layer; The fused features at different feature scales are input into the keypoint deviation detection layer to obtain keypoint deviation prediction results, including: The fused features at different feature scales are input into the key point deviation detection layer. After processing by the convolutional layer, the key point deviation prediction result is obtained. The key point deviation prediction result is the deviation of the xy coordinates of the four key points of the airport platform vehicle target (left rear, right rear, left front, and right front) relative to the xy coordinates of the target box center point.

9. The method according to claim 3, characterized in that, The different levels of features and the fused features at different feature scales are input into the segmentation head to obtain segmentation prediction results, including: The output features of the last convolutional layer of the backbone network are passed through a global pooling layer to obtain global high-level features; The fused features at different feature scales output by the Neck feature fusion network are used as low-level detailed features. The segmentation head is used to predict the features output by multiple layers at each feature scale, and the prediction results are fused with the global high-level features to obtain the segmentation prediction result.

10. An airport platform vehicle recognition system based on image multi-tasking, characterized in that, The system includes: an unmanned transport vehicle, an image acquisition device, a processor, a memory, and a control system; The unmanned transfer vehicle is used to tow or connect with the airport platform vehicle; The image acquisition device is installed on the unmanned transport vehicle and is used to acquire images and transmit the images to the processor; The processor is configured to execute the airport platform recognition method based on image multitasking as described in any one of claims 1-9, perform target detection and segmentation in the image, obtain detection and segmentation results, and transmit the detection and segmentation results to the control system; The memory is connected to the processor and is used to store the multi-task model and related data; The control system is connected to the unmanned transport vehicle and is used to control the movement of the unmanned transport vehicle based on the detection and segmentation results.

Images