Airport platform car identification method and system based on image multitask
Through the image multi-task recognition method, combined with the backbone and segmentation head of the target detection segmentation network, the problem of low positioning and segmentation accuracy in the traditional method is solved, and efficient and accurate vehicle detection and segmentation of the airport platform is achieved, meeting the real-time requirements of the vehicle end.
Patent Information
- Application Number
- CN202310589185.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-05-23
AI Technical Summary
The traditional airport platform vehicle detection and segmentation methods have problems such as low positioning accuracy, poor segmentation effect, high algorithm complexity, poor robustness and slow inference speed, and cannot deal with the two long distance rough positioning and close-distance fine segmentation. task, resulting in less accuracy and efficiency of detection and segmentation.
Using an image multi-task recognition method, the target detection segmentation network is achieved through object detection, including backbone network, Neck feature fusion network, detection head and segmentation head, to realize the prediction and segmentation of the 2D frame, four corner points and three planes of the airport platform car. , using the Anchor-free paradigm and heat map for prediction, combining global features and detailed features for fusion, to improve the accuracy and efficiency of detection and segmentation.
It realizes the processing of two tasks of coarse positioning and fine segmentation at the same time, improves the accuracy of vehicle detection and segmentation on the airport platform, reduces the complexity of the algorithm, improves the generalization ability of the model, meets the real-time requirements of the vehicle end, and outputs the airport vehicle. Direction and multi-faceted information.
Smart Images

Figure CN119027943A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an airport platform vehicle recognition method and system based on image multi-tasking. Background Art
[0002] At present, airport platform vehicles play an indispensable role in baggage transfer. They are mainly responsible for delivering passengers' luggage to the plane. The traditional baggage transfer method requires a lot of manpower and time to drive the transfer vehicle, which is prone to errors and delays. With the advancement of camera sensing and computing technology, the use of monocular vision, stereo vision and vision-based multi-sensor fusion has become a very active research direction in the field of intelligent driving. Some robots or automated devices are also used for baggage transfer. The first step to achieve automated transfer is to detect, locate and finely segment the airport platform vehicle so that the unmanned transfer vehicle can identify, track and connect the airport platform vehicle. However, traditional airport platform vehicle detection and segmentation methods are usually processed using independent algorithms. The detection and segmentation technologies of these automated devices still have some problems, such as low positioning accuracy of airport vehicles, poor segmentation effect of airport vehicles, high algorithm complexity, poor robustness, slow reasoning speed, etc. In addition, these methods cannot identify the front, side, rear and orientation information of the airport vehicle in a 2D rectangular box, nor can they identify the docking area information. At the same time, there is a lack of connection between multiple models, which cannot promote each other and will lead to increased hardware resource consumption. The current multi-task learning method detection branch usually adopts the anchor-base solution, which increases the complexity of the detection head and the number of generated results, thereby reducing the reasoning speed of the model. Traditional image processing methods can only complete a single task and it is difficult to handle both long-distance rough positioning and close-range docking surface fine segmentation tasks at the same time, resulting in low accuracy and efficiency of airport platform vehicle detection and segmentation. Summary of the invention
[0003] Based on this, it is necessary to provide an airport platform vehicle recognition method and system based on image multi-tasking to address the above technical problems.
[0004] An airport platform recognition method based on image multi-task, the method comprising:
[0005] Acquire an image to be tested and preprocess the image to be tested.
[0006] The preprocessed image to be tested is input into the trained target detection and segmentation network to obtain the prediction results of the 2D frame, four corner points and three planes of the airport platform vehicle; the target detection and segmentation network includes: a backbone network, a Neck feature fusion network, a detection head and a segmentation head; the backbone network is used to extract features of different levels of the preprocessed image; the Neck feature fusion network is used to fuse features of different levels to obtain fusion features of different feature scales; the detection head is used to predict the 2D frame, four corner points and three planes of the airport platform vehicle using multiple different detection layers according to the fusion features of each feature scale; the segmentation head is used to segment the airport platform vehicle according to features of different levels and fusion features of different feature scales.
[0007] According to the prediction results, the 2D frame of the airport platform vehicle in the image to be tested, the boundary coordinates of the three plane segmentation regions, and the position and orientation in the vehicle body coordinate system are determined.
[0008] An airport platform vehicle recognition system based on image multi-tasking includes an unmanned transfer vehicle, an image acquisition device, a processor, a memory and a control system.
[0009] The unmanned transfer vehicle is used for towing or connecting with the airport platform vehicle.
[0010] The image acquisition device is installed on the unmanned transport vehicle, and is used to acquire images and transmit the images to the processor.
[0011] The processor is used to execute any of the above-mentioned airport platform recognition methods based on image multi-task to perform target detection and segmentation in the image, obtain detection and segmentation results, and transmit the detection and segmentation results to the control system.
[0012] The memory is connected to the processor and is used for storing the multi-task model and related data.
[0013] The control system is connected to the unmanned transport vehicle and is used to control the actions of the unmanned transport vehicle according to the detection and segmentation results.
[0014] The above-mentioned airport platform vehicle recognition method and system based on image multi-task can realize the simultaneous processing of two tasks: rough positioning and fine segmentation. It can more accurately locate and segment the airport platform vehicle-related area in the airport platform, output the airport vehicle orientation information and the side, docking surface and top surface information of the airport platform vehicle, thereby realizing efficient and accurate airport platform vehicle detection and segmentation. It can effectively solve the problems existing in traditional methods, improve the accuracy of detection and segmentation, reduce the complexity of the algorithm, improve the generalization ability of the model, realize efficient and accurate airport platform vehicle detection and segmentation, realize monocular image airport platform vehicle target detection, key point detection and semantic segmentation, and the prediction time can meet the real-time requirements of the vehicle side. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 It is a flowchart of an airport platform identification method based on image multi-task in one embodiment;
[0016] Figure 2 A diagram showing the structure of a target detection and segmentation network in another embodiment;
[0017] Figure 3 A schematic diagram of algorithm identification points in another embodiment;
[0018] Figure 4 It is a structural block diagram of an airport platform identification system based on image multi-tasking in one embodiment. DETAILED DESCRIPTION
[0019] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0020] The airport platform recognition method based on image multi-task provided in this application can be used to detect the 2D frame of the airport platform vehicle, determine the direction of the airport vehicle and segment the three planes of the airport platform vehicle. It can realize the simultaneous processing of the two tasks of long-distance rough positioning of the airport platform vehicle and close-range docking surface fine segmentation, and can more accurately locate and segment the airport platform vehicle related areas in the airport platform. This method adopts a deep learning single-stage image multi-task network to realize monocular image airport platform vehicle target detection, key point detection and semantic segmentation, and the prediction time can meet the real-time requirements of the vehicle end.
[0021] In one embodiment, Figure 1 As shown, a method for airport platform recognition based on image multi-task is provided, and the method comprises the following steps:
[0022] Step 100: Acquire an image to be tested and pre-process the image to be tested.
[0023] Specifically, the image to be tested is preprocessed, including operations such as removing noise, adjusting image size and format, etc. These preprocessing operations can be implemented using image processing software, corresponding libraries in programming languages, or other related tools.
[0024] Step 102: Input the preprocessed image to be tested into the trained target detection and segmentation network to obtain the prediction results of the 2D frame, four corner points and three planes of the airport platform vehicle; the target detection and segmentation network includes: a backbone network, a Neck feature fusion network, a detection head and a segmentation head; the backbone network is used to extract features at different levels of the preprocessed image; the Neck feature fusion network is used to fuse features at different levels to obtain fusion features of different feature scales; the detection head is used to predict the 2D frame, four corner points and three planes of the airport platform vehicle using multiple different detection layers according to the fusion features of each feature scale; the segmentation head is used to segment the airport platform vehicle according to features at different levels and fusion features of different feature scales.
[0025] Specifically, the detection head uses multiple detection layers for prediction at each feature scale, and our detection branch is based on the Anchor-free paradigm. It does not use the preset Anchor to generate the target box, but instead uses a heatmap to implement it. The Gaussian distribution area of the prediction point is introduced to calculate the true prediction value, and the target box size is directly regressed. Finally, the prediction box can be obtained based on the target box size and the center point position of the target box. Therefore, there is no deviation relative to the Anchor box in this algorithm.
[0026] In this embodiment, the segmentation head predicts multiple layers at each feature scale, combines the features extracted by the backbone network, and is divided into two path branches, namely the global feature branch and the detail feature branch. This structure establishes a connection between low-level and high-level features, thereby improving the accuracy and efficiency of segmentation. The entire network operates at multiple scales to accommodate airport platform vehicles of different sizes and shapes.
[0027] The target detection and segmentation network structure is as follows Figure 2 shown.
[0028] In the prediction results, the prediction output is:
[0029]
[0030] in: To output the Bayesian posterior probability of which category,
[0031] Image-based multi-task detection and segmentation are implemented, which can distinguish the various faces of the airport platform vehicle in the image, determine the direction of the airport vehicle and the three faces of the airport platform vehicle.
[0032] Step 104: According to the prediction results, determine the 2D frame of the airport platform vehicle in the image to be tested, the boundary coordinates of the three plane segmentation regions, and the position and orientation in the vehicle body coordinate system.
[0033] In the above-mentioned airport platform recognition method based on image multi-task, the method can realize the simultaneous processing of two tasks, namely, rough positioning and fine segmentation, and can more accurately locate and segment the airport platform vehicle-related areas in the airport platform, output the airport vehicle orientation information and the side, docking surface and top surface information of the airport platform vehicle, thereby realizing efficient and accurate airport platform vehicle detection and segmentation, and can effectively solve the problems existing in traditional methods, improve the accuracy of detection and segmentation, reduce the complexity of the algorithm, improve the generalization ability of the model, realize efficient and accurate airport platform vehicle detection and segmentation, realize monocular image airport platform vehicle target detection, key point detection and semantic segmentation, and the prediction time can meet the real-time requirements of the vehicle side.
[0034] In one embodiment, the following specific steps are also included before step 100:
[0035] Step 200: Acquire an original image and preprocess the original image to obtain a training sample.
[0036] Step 202: Construct a target detection and segmentation network.
[0037] Step 204: Use the training samples to train the target detection and segmentation network to obtain a trained target detection and segmentation network.
[0038] Specifically, the network parameters of the target detection and segmentation network are trained, including two branches: detection and segmentation. The main process is:
[0039] (1) Read a batch of airport platform vehicle data. For any frame of data in the batch, the training data includes image samples and labels corresponding to each task on the image samples. The detection branch encodes the target and the preset anchor, and the segmentation branch performs one-hot encoding (Onehot encoding) of the data. Onehot (c) means constructing a vector with a length equal to the range of c, and setting the cth element to 1 and the other elements to 0;
[0040] (2) The input data in the read batch data is passed into the multi-task network, the detection branch obtains multi-level multi-task prediction values, and the segmentation branch obtains the true value of the Onehot segmentation label;
[0041] (3) Calculate the overall evaluation loss function based on the reference multi-task true value and multi-task predicted value, including two branches: target and key point detection and semantic segmentation;
[0042] (4) The Adam optimization method is used to optimize the network parameters of the target detection and segmentation network, and the training is stopped when the training round exceeds a threshold, and the multi-task network that minimizes the overall evaluation loss function is output.
[0043] In one embodiment, the backbone network is a ResNeXt network; step 204 specifically includes the following steps:
[0044] Step 300: Input the training samples into the ResNeXt network to obtain features at different levels.
[0045] Step 302: Input features at different levels into the Neck feature fusion network to obtain fused features at different feature scales.
[0046] Step 304: Input the fused features of different feature scales into the detection head to obtain the detection prediction results of the 2D frame, four corner points and three planes of the airport platform vehicle.
[0047] Step 306: Input the fusion features of different levels and different feature scales into the segmentation head to obtain the segmentation prediction result.
[0048] Step 308: According to the training samples, the detection prediction results and the segmentation prediction results, the target detection and segmentation network is optimized and trained using the Adam optimization method to obtain a trained target detection and segmentation network.
[0049] Specifically, the backbone network is composed of ResNeXt, which enables the multi-task network to take into account both feature extraction capability and prediction speed.
[0050] In one embodiment, the Neck feature fusion network is a weighted bidirectional feature pyramid network.
[0051] Specifically, the Neck feature fusion network is composed of a weighted bidirectional feature pyramid network (Bidirectional Feature Pyramid Network, referred to as BiFPN), which realizes simple and fast multi-scale feature fusion, and its output is L feature scales. Compared with the i-th feature scale output, the length and width of the feature map of the i+1th feature scale output are 1 / 2 of the length and width of the feature map of the i-th feature scale output. Among them, the specific implementation method of BiFPN can be referred to "EfficientDet: Scalable andEfficient Object Detection CVPR 2020", so it will not be repeated in this embodiment.
[0052] In one embodiment, the detection head includes a target detection layer, a category detection layer, a target frame deviation detection layer, a key point visibility detection layer, and a key point deviation detection layer; step 304 specifically includes the following steps:
[0053] Step 400: Input the fused features of different feature scales into the target detection layer to obtain the airport platform vehicle target detection prediction result.
[0054] Step 402: Input the fused features of different feature scales into the category detection layer to obtain the airport platform vehicle category prediction result.
[0055] Step 404: Input the fused features of different feature scales into the target frame deviation detection layer to obtain the target frame deviation prediction result.
[0056] Step 406: Input the fused features of different feature scales into the key point visibility detection layer to obtain four key point prediction results.
[0057] Step 408: Input the fused features of different feature scales into the key point deviation detection layer to obtain the key point deviation prediction result.
[0058] Step 410: The airport platform vehicle target detection prediction results, the airport platform vehicle type prediction results, the target frame deviation prediction results, the four key point prediction results and the key point deviation prediction results are spliced after the transformation layer to obtain the prediction detection results of the airport platform vehicle 2D frame, four corner points and three planes.
[0059] In one embodiment, the object detection layer includes a convolution layer and an activation layer. The category detection layer includes a convolution layer and an activation layer.
[0060] Specifically, in the target detection layer, the output channel number is 1, which indicates the confidence of the airport platform vehicle, and the offset prediction of the center point, which reduces the accuracy error of the target center point.
[0061] The category detection layer has N output channels, which represent the confidence of each airport platform vehicle type label, where N is the total number of defined airport platform vehicle types.
[0062] In one of the embodiments, the target frame deviation detection layer adopts an improved Anchor-free algorithm, the improved Anchor-free algorithm uses a heat map to generate a target frame, and introduces a Gaussian distribution area of the prediction point to calculate a true prediction value; step 404 includes: inputting fused features of different feature scales into the target frame deviation detection layer, and processing them using the improved Anchor-free algorithm to obtain the deviation of the x-coordinate of the center point of the target frame, the deviation of the y-coordinate of the center point of the target frame, the deviation of the width of the target frame, and the deviation of the height of the target frame.
[0063] Specifically, the target frame deviation detection layer has 4 output channels, which represent the deviation of the x-coordinate of the center point of the target frame, the deviation of the y-coordinate of the center point of the target frame, the deviation of the width of the target frame, and the deviation of the height of the target frame. In the Anchor-free algorithm, the preset anchor is not used to generate the target frame. Instead, the center point and width and height of the target frame are directly predicted based on the feature map. Therefore, there is no deviation relative to the anchor frame in this algorithm. A heatmap is used to implement it, and the Gaussian distribution area of the prediction point is introduced to calculate the true prediction value. Therefore, the detection layer includes the deviation of the x-coordinate of the center point of the target frame, the deviation of the y-coordinate of the center point of the target frame, the deviation of the width of the target frame, and the deviation of the height of the target frame.
[0064] In one of the embodiments, the key point visibility detection layer includes: a convolution layer and multiple depth-separable convolution layers; step 406 includes: inputting fused features of different feature scales into the key point visibility detection layer, and after passing through a convolution layer and multiple depth-separable convolution layers, obtaining four key point prediction results; the four key point prediction results are the confidence levels of visibility of the four key points of the left rear, right rear, left front and right front of the airport platform vehicle target.
[0065] Specifically, the key point visibility detection layer has 4 output channels, which represent the confidence of whether the four key points of the left rear, right rear, left front, and right front of the airport platform vehicle target are visible, and the value range is [0,1].
[0066] The main function of the convolution layer is to extract features by performing dot multiplication between the original image pixels and the weight parameters in the convolution kernel. In order to obtain more feature information, convolution blocks of different sizes are required.
[0067] Depthwise convolution is mainly divided into two processes, namely channel-by-channel convolution (DepthwiseConvolution) and pointwise convolution (PointwiseConvolution). The number of convolution kernels in channel-by-channel convolution is the same as the number of channels in the previous layer. One convolution kernel is responsible for one channel, and one channel is convolved by only one convolution kernel. The number of feature map channels generated by this process is exactly the same as the number of input channels; the pointwise convolution operation is very similar to the conventional convolution operation. The size of its convolution kernel is 1×1×M, where M is the number of channels in the previous layer. Therefore, the convolution operation here will perform a weighted combination of the feature maps output by the channel-by-channel convolution in the depth direction to generate a new feature map. There are as many output feature maps as there are convolution kernels.
[0068] In one embodiment, the key point deviation detection layer includes a convolution layer; step 408 includes: inputting the fused features of different feature scales into the key point deviation detection layer, and obtaining the key point deviation prediction result after processing by the convolution layer, and the key point deviation prediction result is the deviation of the xy coordinates of the four key points of the left rear, right rear, left front and right front of the airport platform vehicle target relative to the xy coordinates of the center point of the target frame.
[0069] Specifically, the key point deviation detection layer has an output channel of 8, which respectively represents the deviation of the xy coordinates of the four key points of the left rear, right rear, left front, and right front of the airport platform vehicle target relative to the xy coordinates of the target center point.
[0070] In one embodiment, step 410 includes: splicing the airport platform vehicle target detection prediction result, the airport platform vehicle type prediction result, the target frame deviation prediction result, the four key point prediction results and the key point deviation prediction result after the transformation layer, and obtaining the predicted detection results of the airport platform vehicle 2D frame, four corner points and three planes as follows:
[0071]
[0072] Among them, Y is the concatenated output; the input X is a four-dimensional vector, and each dimension is recorded as (B, C, H, W), where B is the number of batch data, C is the number of predicted feature maps, H is the height of the feature map in the 2D feature space, and W is the width of the feature map in the 2D feature space; the transpose(X,(0,3,4,1,2)) function indicates that the input x is rearranged in the order of spatial dimensions (0,3,4,1,2); the reshape(transpose(X,(0,3,4,1,2)),(B,A,C)) function indicates that the input transpose(X,(0,3,4,1,2)) is transformed into a three-dimensional vector with a length of (B,A,C) in each dimension, where A=H*W, and the operation relationship represented by * is multiplication; L is the total number of feature scales.
[0073] The function represents the concatenation of the output reshape(transpose(X,(0,3,4,1,2)),(B,A,C)) at each feature scale according to the first dimension.
[0074] In one embodiment, step 306 includes: passing the output features of the last convolutional layer of the backbone network through the global pooling layer to obtain global high-level features, and then using the output features of the Neck feature fusion network as detailed low-level features; using a segmentation head to predict multiple layers at each feature scale, and fusing the obtained prediction results with the global features to obtain a segmentation prediction result.
[0075] Specifically, the detailed low-level features obtained by Neck and the high-level features extracted by the backbone network are fused and divided into two path branches, namely the global feature branch and the detailed feature branch. The segmentation head predicts multiple layers at each feature scale, and the feature aggregation module establishes a connection between low-level and high-level features, thereby improving the accuracy and efficiency of segmentation. The entire network operates at multiple scales to adapt to airport platform vehicles of different sizes and shapes.
[0076] The features of the semantic segmentation branch are fused from features of multiple different levels, including low-level features and high-level features. These features have been downsampled multiple times, so the results need to pass through the feature aggregation module and be upsampled multiple times before being restored to the original image size. The segmentation results include pixel prediction probability values for the top, side, and tail docking surfaces plus the background. The multiple features include global features extracted by the backbone network and multi-scale features obtained by the pyramid network. After the last convolution layer of feature extraction, a global pooling layer is used to capture the global information of the entire image. This global pooling layer reduces the feature map of each channel to a single value, so that a vector can be used to represent the global information of the entire image.
[0077] In order to retain the detail information in the image and suppress noise, the present invention uses a bilateral filter to process global features and local features. The bilateral filter is a nonlinear filter that can smooth the image while retaining edge information. By using a bilateral filter, the detail information in the image can be retained, thereby improving the accuracy of the segmentation branch.
[0078] In order to fuse multi-scale features, an attention module feature aggregation is used to combine global high-level features and local low-level features to generate the final semantic segmentation result. This fusion module contains an adaptive average pooling layer, two 1*1 convolutional layers and an upsampling layer to fuse the features of global and local branches, which can realize the learning of channel attention weights, help the network focus more on important features and improve the representation ability of features. The main implementation method is: first connect the high-level and low-level output features; then, balance the scale of features through batch normalization. Next, pool the connected features into a feature vector and calculate a weight vector. This weight vector can re-weight the features, play a role in feature selection and combination, and perform feature upsampling to produce the final segmentation result.
[0079] In one embodiment, step 104 includes: traversing the predicted outputs of all layers, and storing the predicted outputs that meet the classification score threshold into the first output target set; traversing the first output target set, post-processing the data in the set, and obtaining valid targets; calculating the angle between each valid target in the first output target set and the y-axis in the vehicle body BEV coordinate system, and obtaining the predicted category, angle, and center coordinates, width, and height of the 2D box of each valid target; outputting the position and orientation of the airport vehicle target in the 2D box and vehicle body coordinate system in the image to be tested, as well as the area boundary coordinates of the three segmentation surfaces.
[0080] In a specific embodiment, the specific steps of training the network parameters of the target detection and segmentation network include:
[0081] Step S1: read a batch of data, encode the target in any frame of data in the batch for the detection branch, obtain the reference multi-task true value, and the segmentation branch obtains all the true values of the data in the batch. The number of read data is consistent with the detection branch, and the image is preprocessed. The specific implementation process is as follows:
[0082] Step S1.1: First, the original image needs to be preprocessed, including removing noise, adjusting image size and format, etc. These preprocessing operations can be implemented using image processing software, corresponding libraries in programming languages, or other related tools.
[0083] Step S1.2: Let A j =(A x ,A y ,A i ,A j) is a series of feature points. Since Anchor is no longer used, there is no need to define anchor box and offset. We can regard each feature point as the center of the object and use the heat map to generate a Gaussian distribution to predict the center point of the object. The center coordinates (x, y) of each feature point can find the highest response in the heat map, indicating that the point is most likely the center of the object. Generate a fixed-size rectangular box with the feature point as the center. When generating the rectangular box, Gaussian distribution can be used to weight the pixel values near the feature point, so as to more accurately calculate the position of the feature point. x, y represent the coordinates of the feature point on the image axis, and w, h represent the width and height of the center point from the rectangular box.
[0084] Step S1.3: For the selection of positive and negative samples, we use a similar idea to Focal Loss for screening. In actual training, the loss of other points around the center point (negative center pointer) is the attenuated loss, and the length and width of the target are obtained by regressing w and h corresponding to the current center point. Other points around the center point are used as negative samples, and the length and width of the target are regressed to obtain the initial screening positive samples. Because the positive and negative samples that are difficult to classify are usually a minority, and this method uses other surrounding points for loss calculation, so that the model pays more attention to the samples that are difficult to classify, so that the positive and negative samples after initial screening can be effectively obtained.
[0085] Step S1.4: Since the positive and negative samples after the initial screening are not accurate enough, we combine the optimal transport assignment (OTA) sample screening strategy. This strategy is a prior art, so it will not be repeated in this embodiment. First calculate the intersection over union (IOU) of the initial screening positive sample and the confidence level, and sort the IOU from large to small, take the sum of the top ten and round it, recorded as b. Then calculate the cost loss function (cost function) of the initial screening positive sample, which consists of position loss and category loss. The position loss is obtained by using -torch.log to calculate the previous initial screening positive sample iou, and the category loss is the cross entropy loss of the target classification. The cost loss function is the weighted sum of the two losses, and then the cost function is arranged from small to large, and the samples before the arrangement of b are taken as positive samples. At the same time, consider the case where the same grid prediction box is associated with two confidence levels, take the smaller value of cos, and the prediction box is the positive sample of the corresponding confidence level.
[0086] Step S2: Pass the input data in the read batch data into the multi-task target detection network, and obtain the multi-task prediction value as follows:
[0087]
[0088] in, is the target confidence prediction value, is the predicted value of the airport car category score, is the confidence prediction value of the airport car key point, is the 2D box deviation prediction value, is the predicted value of the key point deviation, Predict the value for the segmentation result.
[0089] Step S3: Calculate the overall evaluation loss function based on each multi-task true value and multi-task predicted value.
[0090] Step S4: Adopting the Adam optimization method to optimize the network parameters of the target detection segmentation network, and stopping the training when the training round exceeds the threshold, and outputting the target detection segmentation network that minimizes the overall evaluation loss function.
[0091] After the detection head performs multi-feature scale prediction, for each detection head, the prediction of multi-feature scale is spliced using the transformation layer to obtain the spliced output as shown in formula (1).
[0092] In this method, traditional image processing methods can be used to complete a single task, such as only positioning or segmenting the airport platform vehicle. However, the accuracy and efficiency of this method are not as good as the multi-task target detection and segmentation method of the present invention. If only a single task is performed, the accuracy and real-time performance of the algorithm will be reduced. Other deep learning networks can be used to implement image multi-task processing, such as using a two-stage network or other types of single-stage networks. However, these methods may require more computing resources and time, and may not be able to complete the task in real time on the vehicle side. Other image preprocessing methods can be used, such as using other denoising algorithms or not adjusting the image format. However, these alternatives may reduce image quality and the robustness of the algorithm. Some parts of the backbone network, Neck feature fusion network, detection head, and segmentation head can be replaced. For example, the backbone network uses ordinary ResNet, and the fusion network uses FPN, which can also achieve the same function. However, these alternatives may affect the accuracy and real-time performance of the algorithm. The feature extraction capability of ResNet is weaker than that of ResNeXt, and the model accuracy of FPN is not as good as BiFPN. The most basic implementation scheme is to use traditional image processing methods to perform a single task, such as only roughly positioning the airport platform vehicle. This method can achieve basic positioning functions, but cannot complete fine segmentation and multi-task processing, and its accuracy and efficiency are not as good as the multi-task target detection and segmentation method of the present invention.
[0093] The multi-task target detection and segmentation method of the present invention has excellent accuracy and real-time performance. The implementation of alternative solutions may have different degrees of impact on the effect of the algorithm. The most basic implementation scheme cannot meet the high accuracy and high efficiency requirements of the present invention. The beneficial effects of this method are as follows:
[0094] (1) Improved the accuracy of airport platform vehicle positioning and segmentation: Traditional image processing methods can only complete a single task and are difficult to handle both long-distance rough positioning and close-range docking surface fine segmentation tasks at the same time. The image-based multi-task target detection and segmentation method proposed in the present invention can handle these two tasks at the same time, thereby more accurately positioning and segmenting the airport platform vehicle-related area on the airport platform.
[0095] (2) Improved real-time performance of the algorithm: The present invention adopts a deep learning single-stage image multi-task network, which can complete multiple tasks such as airport platform vehicle target detection, key point detection and semantic segmentation in real time on the vehicle side, thereby meeting the real-time requirements of the vehicle side.
[0096] (3) Improved algorithm robustness: The target detection and segmentation network of the present invention consists of four parts: backbone network, Neck feature fusion network, detection head and segmentation head, which fully utilizes the characteristics and advantages of each part, thereby improving the robustness and stability of the algorithm.
[0097] (4) Improved accuracy in judging the direction of the airport vehicle and the three sides of the airport platform vehicle: The present invention provides an effective method for judging the direction of the airport vehicle and the three sides of the airport platform vehicle, thereby more accurately grasping the status and position of the airport vehicle and the airport platform vehicle, and helping to improve the safety and efficiency of airport operations.
[0098] In summary, the image-based multi-task target detection and segmentation method of the present invention has the beneficial effects of high positioning and segmentation accuracy, good real-time performance, strong robustness, etc., and can be widely used in the field of automated management and control of airport platform vehicles.
[0099] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0100] In one embodiment, Figure 4 As shown, an airport platform vehicle recognition system based on image multi-tasking is provided, which includes: an unmanned transfer vehicle 10, an image acquisition device 20, a processor 30, a memory 40 and a control system 50.
[0101] The unmanned transfer vehicle 10 is used for towing or connecting with the airport platform vehicle.
[0102] The image acquisition device 20 is installed on the unmanned transport vehicle 10, and is used to acquire images and transmit the images to the processor 30. Specifically, the image acquisition device can be, but is not limited to, a camera or a digital camera.
[0103] The processor 30 is used to execute any of the above-mentioned airport platform recognition methods based on image multi-task to perform target detection and segmentation in the image, obtain detection and segmentation results, and transmit the detection and segmentation results to the control system 50.
[0104] The memory 40 is connected to the processor 30 and is used to store the multi-task model and related data.
[0105] The control system 50 is connected to the unmanned transport vehicle 10 and is used to control the actions of the unmanned transport vehicle 10 according to the detection and segmentation results.
[0106] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0107] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. An airport platform vehicle recognition method based on image multi-task, characterized in that: The method comprises: Acquire an image to be tested and preprocess the image to be tested; The preprocessed image to be tested is input into the trained target detection and segmentation network to obtain the prediction results of the 2D frame, four corner points and three planes of the airport platform vehicle; the target detection and segmentation network includes: a backbone network, a Neck feature fusion network, a detection head and a segmentation head; the backbone network is used to extract features of different levels of the preprocessed image; the Neck feature fusion network is used to fuse features of different levels to obtain fusion features of different feature scales; the detection head is used to use multiple different detection layers according to the fusion features of each feature scale to predict the 2D frame, four corner points and three planes of the airport platform vehicle; the segmentation head is used to segment the airport platform vehicle according to features of different levels and fusion features of different feature scales; According to the prediction results, the 2D frame of the airport platform vehicle in the image to be tested, the boundary coordinates of the three plane segmentation regions, and the position and orientation in the vehicle body coordinate system are determined.
2. The method according to claim 1, characterized in that Before acquiring the image to be tested and preprocessing the image to be tested, the method further includes: Acquire an original image and preprocess the original image to obtain a training sample; Build a target detection and segmentation network; The object detection and segmentation network is trained using the training samples to obtain a trained object detection and segmentation network.
3. The method according to claim 2, characterized in that The backbone network is a ResNeXt network; The target detection segmentation network is trained using the training samples to obtain a trained target detection segmentation network, including: Inputting the training samples into the ResNeXt network to obtain features at different levels; Inputting the features at different levels into the Neck feature fusion network to obtain fusion features at different feature scales; The fused features of different feature scales are input into the detection head to obtain the detection prediction results of the 2D frame, four corner points and three planes of the airport platform vehicle; Inputting the features at different levels and the fused features at different feature scales into the segmentation head to obtain a segmentation prediction result; According to the training samples, the detection prediction results and the segmentation prediction results, the target detection segmentation network is optimized and trained using the Adam optimization method to obtain the trained target detection segmentation network.
4. The method according to claim 3, characterized in that The Neck feature fusion network is a weighted bidirectional feature pyramid network.
5. The method according to claim 3, characterized in that: The detection head includes a target detection layer, a category detection layer, a target frame deviation detection layer, a key point visibility detection layer and a key point deviation detection layer; The fused features of different feature scales are input into the detection head to obtain the detection prediction results of the 2D frame, four corner points and three planes of the airport platform vehicle, including: Input the fused features of different feature scales into the target detection layer to obtain the airport platform vehicle target detection prediction result; Inputting the fused features of different feature scales into the category detection layer to obtain the category prediction result of the airport platform vehicle; Inputting the fused features of different feature scales into the target frame deviation detection layer to obtain a target frame deviation prediction result; Inputting the fused features of different feature scales into the key point visibility detection layer to obtain four key point prediction results; Inputting the fused features of different feature scales into the key point deviation detection layer to obtain a key point deviation prediction result; The airport platform vehicle target detection prediction results, the airport platform vehicle type prediction results, the target frame deviation prediction results, the four key point prediction results and the key point deviation prediction results are spliced after the transformation layer to obtain the prediction detection results of the airport platform vehicle 2D frame, four corner points and three planes.
6. The method according to claim 5, characterized in that The target detection layer includes a convolution layer and an activation layer; the category detection layer includes a convolution layer and an activation layer.
7. The method according to claim 5, characterized in that The target frame deviation detection layer adopts an improved Anchor-free algorithm, which uses a heat map to generate a target frame and introduces a Gaussian distribution area of a prediction point to calculate a true prediction value; Inputting the fused features of different feature scales into the target frame deviation detection layer to obtain a target frame deviation prediction result, including: The fused features of different feature scales are input into the target frame deviation detection layer, and processed using the improved Anchor-free algorithm to obtain the deviation of the x-coordinate of the center point of the target frame, the deviation of the y-coordinate of the center point of the target frame, the deviation of the width of the target frame, and the deviation of the height of the target frame.
8. The method according to claim 5, characterized in that The key point visibility detection layer includes a convolutional layer and a plurality of depth-separable convolutional layers; The fused features of different feature scales are input into the key point visibility detection layer to obtain four key point prediction results, including: The fused features of different feature scales are input into the key point visibility detection layer, and after passing through a convolution layer and multiple depth-separable convolution layers, four key point prediction results are obtained; the four key point prediction results are the visibility confidence of the left rear, right rear, left front and right front four key points of the airport platform vehicle target.
9. The method according to claim 5, characterized in that The key point deviation detection layer includes a convolutional layer; Inputting the fused features of different feature scales into the key point deviation detection layer to obtain key point deviation prediction results, including: The fused features of different feature scales are input into the key point deviation detection layer, and after being processed by the convolution layer, the key point deviation prediction result is obtained; the key point deviation prediction result is the deviation of the xy coordinates of the four key points of the left rear, right rear, left front and right front of the airport platform vehicle target relative to the xy coordinates of the center point of the target frame.
10. The method according to claim 5, characterized in that The airport platform vehicle target detection prediction result, the airport platform vehicle type prediction result, the target frame deviation prediction result, the four key point prediction results and the key point deviation prediction result are spliced after the transformation layer, and the prediction detection results of the airport platform vehicle 2D frame, four corner points and three planes are obtained as follows: Among them, Y is the concatenated output; the input X is a four-dimensional vector, and each dimension is recorded as (B, C, H, W), where B is the number of batch data, C is the number of predicted feature maps, H is the height of the feature map in the 2D feature space, and W is the width of the feature map in the 2D feature space; the transpose(X,(0,3,4,1,2)) function indicates that the input X is rearranged in the order of spatial dimensions (0,3,4,1,2); the reshape(transpose(X,(0,3,4,1,2)),(B,A,C)) function indicates that the input transpose(X,(0,3,4,1,2)) is transformed into a three-dimensional vector with a length of (B,A,C) in each dimension, where A=H*W, * represents the operation relationship of multiplication, and L is the total number of feature scales.
11. The method according to claim 3, characterized in that Inputting the fused features of the features at different levels and different feature scales into the segmentation head to obtain a segmentation prediction result, including: The output features of the last convolutional layer of the backbone network are passed through a global pooling layer to obtain global high-level features; The fused features of different feature scales output by the Neck feature fusion network are used as detailed low-level features, the segmentation head is used to predict multiple layers at each feature scale respectively, and the obtained prediction results are fused with the global features to obtain the segmentation prediction results.
12. The method according to claim 1, characterized in that According to the detection results, the 2D frame, three plane boundary coordinates and the position and orientation of the airport platform vehicle in the image in the vehicle body coordinate system are determined, including: Traverse the prediction outputs of all layers and store the prediction outputs that meet the classification score threshold into the first output target set; Traverse the first output target set, post-process the data in the set, and obtain valid targets; Calculate the angle between each valid target in the first output target set and the y-axis in the vehicle body BEV coordinate system, and obtain the predicted category, angle, and center coordinates, width, and height of the 2D box of each valid target; Output the position and orientation of the airport vehicle target in the 2D frame and vehicle body coordinate system in the image to be tested, as well as the regional boundary coordinates of the three segmentation surfaces.
13. An airport platform vehicle recognition system based on image multi-task, characterized in that: The system includes: an unmanned transport vehicle, an image acquisition device, a processor, a memory and a control system; The unmanned transfer vehicle is used to tow or connect with the airport platform vehicle; The image acquisition device is installed on the unmanned transport vehicle and is used to acquire images and transmit the images to the processor; The processor is used to execute the airport platform recognition method based on image multi-task according to any one of claims 1 to 12 to perform target detection and segmentation in the image, obtain detection and segmentation results, and transmit the detection and segmentation results to the control system; The memory is connected to the processor and is used to store the multi-task model and related data; The control system is connected to the unmanned transport vehicle and is used to control the actions of the unmanned transport vehicle according to the detection and segmentation results.
Citation Information
Patent Citations
Vehicle-mounted visual real-time multi-target multi-task joint sensing method and device
CN111310574A
Anchor-frame-free 3D target detection method based on multi-sensor fusion
CN114118247A
Target detection method based on key point positioning in remote sensing image
CN115631426A
Airport special vehicle identification method and system
CN116071707A
Image segmentation model training method and apparatus, image segmentation method and apparatus, and device and medium
WO2020199593A1
