Airport semantic segmentation method based on three-dimensional digital model and image enhancement technology

Abstract

The application discloses an airport semantic segmentation method based on a three-dimensional digital model and an image enhancement technology, and comprises the following steps: step 1, constructing a three-dimensional digital model of an airplane; step 2, defining a region of interest: using a target detection algorithm to automatically detect and locate the airplane in an image, and generating a defined ROI region; step 3, 3D model projection and matching, outputting a two-dimensional projection of the three-dimensional model of the airplane; step 4, airplane feature enhancement: enhancing airplane information at the image end and the feature end; step 5, semantic segmentation: using the weighted fused image to perform semantic segmentation, so as to identify and extract the airplane region in the airport image. Through the introduction of the three-dimensional digital model and the image enhancement technology, the application provides more abundant geometric information and context information, thereby significantly improving the segmentation precision of the airplane and other objects in the airport scene, and reducing the false segmentation and missed segmentation.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and image processing, and in particular relates to an airport semantic segmentation method based on three-dimensional digital model and image enhancement technology. Background Art

[0002] Airport semantic segmentation algorithms have important applications in computer vision and image processing. Semantic segmentation involves classifying each pixel in an image into distinct semantic categories. For example, in an airport scene, pixels can be classified as aircraft, runways, buildings, and vehicles. High-precision semantic segmentation algorithms have a wide range of applications in airport security monitoring, aircraft navigation, and ground handling management.

[0003] Traditional image segmentation methods mainly include techniques based on edge detection, region growing, cluster analysis and graph cutting. These methods usually rely on low-level features of the image, such as color, texture and edge information. However, due to the complexity of airport scenes, these methods have significant limitations when processing airport images. Edge detection methods: such as Canny, Sobel, etc., are prone to false detection and missed detection when dealing with complex backgrounds and lighting changes. Region growing methods: such as seed-based region growing, rely on the selection of initial seeds and are easily affected by noise and occlusion. Cluster analysis methods: such as K-means clustering, have difficulty processing high-dimensional feature spaces and are sensitive to the initial cluster centers. Graph cutting methods: such as GrabCut, have high computational complexity and are difficult to apply in real time.

[0004] In recent years, deep learning methods have made significant progress in the field of image segmentation. Convolutional neural networks (CNN) and its variants (such as U-Net, SegNet, Mask R-CNN) have achieved excellent performance on multiple benchmark datasets. However, these methods still face some challenges in airport semantic segmentation. Strong data dependence: The performance of deep learning models is highly dependent on the quality and quantity of a large amount of labeled data, and it is costly to obtain and annotate high-quality airport image datasets. Background complexity: In airport scenes, the color and shape similarity between aircraft and runways, buildings, vehicles and other objects is high, resulting in a decrease in segmentation accuracy. Lighting changes: The lighting conditions in the airport environment are complex and changeable. Traditional deep learning models find it difficult to maintain stable segmentation performance under different lighting conditions.

[0005] In summary, existing airport semantic segmentation technologies have significant deficiencies in segmentation accuracy, robustness, and computational efficiency. The main deficiencies are as follows:

[0006] ① Low segmentation accuracy: Existing 2D image-based segmentation algorithms have low segmentation accuracy when dealing with complex backgrounds and changing lighting conditions. This is especially true in airport scenarios, where the color and shape of aircraft, runways, buildings, vehicles, and other objects are highly similar, leading to inaccurate segmentation results.

[0007] Poor robustness: The lighting conditions in airport scenes are complex and changeable, and existing algorithms perform unstably under different lighting conditions. Factors such as shadows, reflections, and weather changes can affect segmentation results and reduce the robustness of the algorithm.

[0008] ③ High computational complexity: Some high-precision segmentation algorithms have high computational complexity, making them difficult to apply in real time. Airport scenes typically require real-time processing of large amounts of image data, which places high demands on the algorithm's computational efficiency.

[0009] ④ Strong data dependency: The performance of deep learning models is highly dependent on the quality and quantity of large amounts of annotated data. Acquiring and annotating high-quality airport image datasets is expensive, and the data lacks diversity and generalization capabilities across different airport scenarios.

[0010] ⑤ Low segmentation efficiency: Semantic segmentation in airport images requires efficient processing while ensuring accuracy. Existing algorithms perform poorly in efficient segmentation tasks and are unable to meet the needs of real-time processing. Summary of the Invention

[0011] This invention aims to overcome the shortcomings of existing technologies by providing an airport semantic segmentation method based on a 3D digital model and image enhancement technology. By incorporating these technologies, the method provides richer geometric and contextual information, significantly improving the segmentation accuracy of aircraft and other objects in airport scenes and reducing the occurrence of mis-segmentation and missed segmentation.

[0012] The object of the present invention is achieved through the following technical solution: an airport semantic segmentation method based on a three-dimensional digital model and image enhancement technology, comprising the following steps:

[0013] Step 1: Build a three-dimensional digital model of the aircraft;

[0014] Step 2: Define the region of interest: Preprocess the collected airport images, use the target detection algorithm to automatically detect and locate the aircraft in the image, generate the bounding box of each aircraft, and define the area where the aircraft bounding box is located as the region of interest, recorded as the ROI area;

[0015] Step 3: 3D model projection and matching: First, the aircraft's pose is initially estimated based on the ROI image to obtain the aircraft's pitch, roll, and yaw angles. These three angles are then used as optimization parameters, and the degree of match between the ROI image and the aircraft's 3D model projection image is used as the objective function. A heuristic optimization algorithm is used to optimize the objective function. Iterations are terminated when the degree of match meets the requirement or the maximum number of iterations is reached. A 2D projection of the aircraft's 3D model is output for subsequent feature enhancement.

[0016] Step 4: Aircraft feature enhancement: Aircraft information is enhanced on both the image and feature sides. On the image side, a weighted fusion method is used to fuse the 2D projection image of the 3D model with the ROI region in the airport image. The fused image is enhanced on the feature side by extracting features from both the airport image and the aircraft model projection image, and enhancing the shallow features of the airport image. Based on the ROI region, aircraft features are extracted from the airport image features and interacted with the features of the corresponding aircraft model projection image.

[0017] Step 5: Semantic Segmentation: Use the weighted fusion image after feature enhancement to perform semantic segmentation to identify and extract the aircraft area in the airport image. The specific steps are as follows:

[0018] Step 5-1, data preprocessing: normalize the fused image and normalize the pixel values ​​to the range of [0,1];

[0019] Step 5-2: Perform data enhancement;

[0020] Step 5-3: Perform feature extraction and segmentation: Use convolutional neural networks for feature extraction and perform semantic segmentation based on the U-Net model.

[0021] By combining 3D digital models with image enhancement technology, this paper provides a high-precision, robust, low-complexity, and efficient airport semantic segmentation algorithm, addressing key issues in existing technologies and meeting the needs of practical applications. The advantages of this paper are specifically reflected in the following aspects:

[0022] ① Improved segmentation accuracy: By introducing three-dimensional digital models and image enhancement technology, richer geometric and contextual information is provided, thereby significantly improving the segmentation accuracy of aircraft and other objects in airport scenes and reducing mis-segmentation and missed segmentation.

[0023] ② Enhanced algorithm robustness: Utilizing multi-view projection and image enhancement technology of three-dimensional models, the robustness of the algorithm in complex backgrounds and changing lighting conditions is improved, enabling it to operate stably in different environments and conditions, and not be affected by shadows, reflections, and weather changes.

[0024] ③ Reduced computational complexity: Although the introduction of three-dimensional models and 2D projections increases the initial computational complexity, by optimizing the algorithm and simplifying the feature extraction process, the overall computational complexity is reduced, the real-time performance and efficiency of the segmentation algorithm are improved, enabling it to process large amounts of image data in real time.

[0025] ④ Reduced data dependency: By utilizing the geometric information and multi-view images provided by the 3D model, the reliance on large amounts of labeled data is reduced, the algorithm’s generalization capability is improved, enabling it to perform well in different airport scenarios, and reducing the cost of acquiring and labeling high-quality data sets.

[0026] ⑤ Achieve efficient segmentation: Combining 3D models and image enhancement technology not only improves segmentation accuracy and robustness, but also enables efficient processing, meeting the needs of real-time processing of large amounts of image data in airport scenarios, thereby improving the system's response speed and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of the airport semantic segmentation method based on three-dimensional digital model and image enhancement technology of the present invention;

[0028] Figure 2 Flowchart for 3D model projection and matching;

[0029] Figure 3 Schematic diagram of how cross-attention features interact;

[0030] Figure 4 This is the semantic segmentation map of this embodiment. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0032] like Figure 1 As shown, the airport semantic segmentation method based on three-dimensional digital model and image enhancement technology of the present invention includes the following steps:

[0033] Step 1: Build a 3D digital aircraft model: This process uses technologies such as laser scanning, structured light, or stereo vision to create a 3D digital model of the aircraft. This model contains detailed geometric information about the aircraft, such as its outline and surface features. The primary objective of this stage is to generate a high-precision 3D point cloud or mesh model of the aircraft, ensuring its geometric accuracy and integrity.

[0034] Step 2: Define the region of interest (ROI): Preprocess the collected airport images and use an object detection algorithm (such as YOLO or Faster R-CNN) to automatically detect and locate aircraft in the images. Generate a bounding box for each aircraft and define the area within the aircraft's bounding box as the region of interest (ROI). These bounding boxes will be used for subsequent 3D model projection and matching to provide accurate initial position information.

[0035] Step 3, 3D model projection and matching: First, make an initial estimate of the aircraft's posture based on the ROI area image to obtain the aircraft's pitch angle, roll angle, and yaw angle; then use the three angles as optimization parameters and the matching degree between the ROI area image and the aircraft's 3D model projection image as the objective function, and use a heuristic optimization algorithm to optimize the objective function; when the matching degree reaches the requirement or the maximum number of iterations is reached, the iteration is stopped and the 2D projection of the aircraft's 3D model is output for subsequent aircraft feature enhancement. The specific process is as follows: Figure 2 shown.

[0036] The specific implementation method of step 3 is:

[0037] Step 3-1: Preliminary view angle estimation, including the following steps:

[0038] Step 3-1-1, edge detection: perform edge detection on the aircraft image in the ROI area and use the Canny edge detection algorithm to extract the contour edge of the aircraft;

[0039] Step 3-1-2, Keypoint Detection: Use the SIFT or ORB algorithm on the edge image to detect the aircraft's key points; these key points can be obvious features such as the aircraft's nose, tail, and wings. You can use algorithms such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF) to detect key points.

[0040] Step 3-1-3, Preliminary View Angle Estimation: Preliminary estimate of the aircraft's attitude based on the detected key point distribution and edge shape; the specific method is to infer the aircraft's pitch angle, roll angle, and yaw angle by analyzing the relative positions of the key points and the aspect ratio of the edges:

[0041] Pitch Angle: The angle of the aircraft's rotation around the lateral axis (the axis from wingtip to wingtip), which indicates the degree to which the aircraft's nose tilts up and down relative to the horizontal plane; Roll Angle: The angle of the aircraft's rotation around the longitudinal axis (the axis from nose to tail), which indicates the degree to which the aircraft's wings tilt; Yaw Angle: The angle of the aircraft's rotation around the vertical axis (the axis running vertically downward from the center of the aircraft), which indicates the degree to which the aircraft's nose deflects left or right relative to the north direction;

[0042]

[0043] (x1, y1), (x2, y2), (x3, y3), and (x4, y4) are the image coordinates of the nose, tail, left wing, and right wing, respectively, obtained through key point detection;

[0044] Step 3-2, viewing angle optimization, includes the following sub-steps:

[0045] Step 3-2-1. Initial matching: Perform preliminary alignment of the generated 2D projection image with the ROI area in the actual image. Initial alignment can be achieved through simple geometric transformations (such as scaling and translation). Use the initially estimated angles to project the 3D model onto the 2D plane. Use optimization algorithms (such as iterative closest point algorithm (ICP) and gradient descent) to continuously adjust the three angles to achieve the best match between the projection image and the actual image.

[0046] Step 3-2-2, optimize the objective function: define a matching optimization objective function that measures the degree of match between the projected image and the actual image; the objective function includes the following components: (1) shape difference: use edge detection algorithms (such as Canny or Sobel) to extract the edges of the projected image and the actual image, and calculate the Hausdorff distance or Chamfer distance between the edges; (2) pixel difference: calculate the difference in pixel intensity between the projected image and the actual image in the overlapping area, for example, using the mean square error (MSE) or structural similarity (SSIM) indicator.

[0047] Step 3-2-3, Optimization Algorithm: Use an optimization algorithm to adjust projection parameters (such as scaling, rotation, and translation) to minimize the objective function. Specific optimization algorithms include: Iterative Closest Point (ICP) algorithm: Minimizes the distance between the projected image and the actual image edge points through iterative optimization. Gradient Descent: Calculates the gradient of the objective function and iteratively updates the projection parameters, gradually approaching the optimal solution. Particle Swarm Optimization (PSO): Simulates the search behavior of a swarm of particles to find the global optimal solution to the objective function. You can choose any optimization algorithm based on your needs.

[0048] Step 3-3: Projection Parameter Adjustment and Best Matching. Parameter Adjustment: Based on the output of the optimization algorithm, adjust the projection parameters of the 3D model, including viewing angle, scaling, rotation, and translation. Best Match: Generate an adjusted 2D projection image and compare it with the ROI in the actual image to check the matching effect. If the matching effect is not ideal, continue iterative optimization until the preset matching accuracy is achieved or the maximum number of iterations is reached.

[0049] Step 4, aircraft feature enhancement: In this invention, in order to enhance the image features and facilitate subsequent semantic segmentation processing, the aircraft information is enhanced at the image end and the feature end. At the image end, a weighted fusion method is used to fuse the two-dimensional projection image of the three-dimensional model with the ROI region in the airport image. The weighted fusion method is simple and intuitive, and can effectively combine the geometric information of the projection image and the texture information of the actual image. The specific method is as follows:

[0050] The weight parameters a and b are selected so that a+b=1; the selection of the weight parameters can be adjusted according to the experimental results to achieve the best fusion effect.

[0051] Pixel weighting calculation: weight calculation is performed on each pixel to fuse the information of the projection image and the actual image; the specific calculation formula is as follows:

[0052] I fused (x, y)=aI projection (x, y)+bI actual (x, y)

[0053] Where I fused is the fused image, I projection is the projection image, I actual is the actual image, and (x, y) is the pixel coordinate. Through the above steps, the fused image can be obtained, which combines the geometric information of the projection image and the texture information of the actual image, enhances the image features, and provides a high-quality input image for subsequent semantic segmentation.

[0054] The fused image is enhanced at the feature end, and the specific method is: the features of the airport image and the projection image of the aircraft model are extracted at the same time, and the shallow features of the airport image are enhanced; then, according to the ROI region, the aircraft features are taken out from the airport image features, and interacted with the corresponding aircraft model projection image features. The specific method is shown in Figure 3 , which respectively inputs the airport ROI image and the matching three-dimensional model projection image into the same image encoder to obtain the airport features and the projection features. In order to enhance the airport features using the projection features, the attention operation is used to fuse the two features. Specifically, the airport features are taken as Query, and the projection features are taken as Key and Value. Query and Key perform matrix multiplication to obtain an attention matrix (indicating the region that needs to be focused on). The attention matrix is multiplied by Value to obtain an attention output feature, and the residual link with the airport feature is completed to enhance the airport feature.

[0055] Step 5, semantic segmentation: using the weighted fused image after feature enhancement for semantic segmentation to identify and extract the aircraft region in the airport image; the specific steps are as follows:

[0056] Step 5-1, data preprocessing: normalize the fused image and normalize the pixel values ​​to the range of [0,1];

[0057] Step 5-2: Perform data enhancement operations such as rotation, scaling, translation, flipping, etc.

[0058] Step 5-3, perform feature extraction and segmentation: Use convolutional neural networks (such as ResNet, VGG) for feature extraction, and perform semantic segmentation based on the U-Net model. U-Net has an encoder-decoder structure, which can effectively fuse feature information at different levels to achieve accurate segmentation. Model training: Use the cross entropy loss function or the Dice loss function to measure the difference between the predicted results and the true labels. Use the Adam optimization algorithm to iteratively update the model parameters to minimize the loss function. Model prediction and post-processing: Input the preprocessed image into the trained segmentation model to obtain the predicted segmentation result. Post-process the predicted results, such as threshold segmentation and morphological operations, to obtain the final binary segmentation map. Result evaluation: Use indicators such as intersection over union (IoU) and Dice coefficient to evaluate the performance of the segmentation results. Visualize the segmentation results and compare them with the actual image to show the segmentation effect, such as Figure 4 As shown, Figure 4 (a) is the original image, Figure 4 (b) is the semantic segmentation image.

[0059] The present invention enhances image features by combining the two-dimensional projection image of the three-dimensional digital model with the actual image for weighted fusion, making the outline and details of the aircraft clearer, thereby significantly improving segmentation accuracy.

[0060] Enhanced robustness: By adopting the iterative closest point (ICP) algorithm and gradient descent optimization algorithm, the projection parameters of the 3D model are automatically adjusted to achieve the best match with the actual image, improving the robustness of the system under different perspectives and complex backgrounds.

[0061] Flexibility and adaptability: Using the weighted fusion method, the weight parameters can be flexibly adjusted to optimize the fusion effect according to actual needs, ensuring that both the geometric information of the projected image and the texture information of the actual image are retained to adapt to different application scenarios.

[0062] In summary, by combining multiple innovative technologies, the present invention outperforms existing technologies in terms of accuracy, robustness, and flexibility, significantly improving the effect of aircraft recognition and segmentation in airport images, and providing reliable technical support for airport management and flight safety.

[0063] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. An airport semantic segmentation method based on three-dimensional digital model and image enhancement technology is characterized by: The following steps are involved: Step 1: Build a three-dimensional digital model of the aircraft; Step 2: Define the region of interest: Preprocess the collected airport images, use the target detection algorithm to automatically detect and locate the aircraft in the image, generate the bounding box of each aircraft, and define the area where the aircraft bounding box is located as the region of interest, recorded as the ROI area; Step 3: 3D model projection and matching: First, the aircraft's pose is initially estimated based on the ROI image to obtain the aircraft's pitch, roll, and yaw angles. These three angles are then used as optimization parameters, and the degree of match between the ROI image and the aircraft's 3D model projection image is used as the objective function. A heuristic optimization algorithm is used to optimize the objective function. Iterations are terminated when the degree of match meets the requirement or the maximum number of iterations is reached. A 2D projection of the aircraft's 3D model is output for subsequent feature enhancement. Step 4: Aircraft feature enhancement: Enhance aircraft information on both the image and feature sides. On the image side, use a weighted fusion method to fuse the 2D projection image of the 3D model with the ROI region in the airport image. The fused image is enhanced at the feature end by extracting features of the airport image and the aircraft model projection image simultaneously, and enhancing the shallow features of the airport image. Then, according to the ROI area, the aircraft features are extracted from the airport image features and interacted with the features of the corresponding aircraft model projection image; Using the cross-attention method, the airport ROI image and the matching 3D model projection image are fed into the same image encoder to obtain airport features and projection features, which are then fused using the attention operation. The airport features are used as the query, and the projection features are used as the key and value. The query and key are matrix multiplied to obtain the attention matrix, which is multiplied by the value to obtain the attention output feature. This is then linked with the airport feature residual to enhance the airport feature. Step 5: Semantic Segmentation: Use the weighted fusion image after feature enhancement to perform semantic segmentation to identify and extract the aircraft area in the airport image. The specific steps are as follows: Step 5-1, data preprocessing: normalize the fused image and normalize the pixel values ​​to the range of [0,1]; Step 5-2: Perform data enhancement; Step 5-3: Perform feature extraction and segmentation: Use convolutional neural networks for feature extraction and perform semantic segmentation based on the U-Net model.

2. The airport semantic segmentation method based on three-dimensional digital model and image enhancement technology according to claim 1 is characterized in that: The specific implementation method of step 3 is: Step 3-1: Preliminary view angle estimation, including the following steps: Step 3-1-1, edge detection: perform edge detection on the aircraft image in the ROI area and use the Canny edge detection algorithm to extract the contour edge of the aircraft; Step 3-1-2, key point detection: Use SIFT or ORB algorithm to detect the key points of the aircraft on the edge image; Step 3-1-3, Preliminary View Angle Estimation: Preliminary estimate of the aircraft's attitude based on the detected key point distribution and edge shape; the specific method is to infer the aircraft's pitch angle, roll angle, and yaw angle by analyzing the relative positions of the key points and the aspect ratio of the edges: Pitch Angle: The angle of the aircraft's rotation around the horizontal axis, which indicates the degree of tilt of the aircraft's nose relative to the horizontal plane; Roll Angle: The angle of the aircraft's rotation around the vertical axis, which indicates the degree of tilt of the aircraft's wings; Yaw Angle: The angle of the aircraft's rotation around the vertical axis, which indicates the degree of left and right deflection of the aircraft's nose relative to the north direction; (x1, y1), (x2, y2), (x3, y3), and (x4, y4) are the image coordinates of the nose, tail, left wing, and right wing, respectively, obtained through key point detection; Step 3-2, viewing angle optimization, includes the following sub-steps: Step 3-2-1, Initial Matching: Preliminarily align the generated 2D projection image with the ROI area in the actual image; project the 3D model onto the 2D plane using the preliminarily estimated angles, and continuously adjust the three angles to achieve the best match between the projection image and the actual image; Step 3-2-2, Optimize the objective function: Define a matching optimization objective function that measures the degree of match between the projected image and the actual image. The objective function includes the following components: (1) Shape difference: Use edge detection algorithm to extract the edges of the projected image and the actual image, and calculate the Hausdorff distance or Chamfer distance between the edges; (2) Pixel difference: Calculate the difference in pixel intensity between the projected image and the actual image in the overlapping area; Step 3-2-3, optimization algorithm: Use the optimization algorithm to adjust the projection parameters and minimize the objective function value; Step 3-3, projection parameter adjustment and optimal matching: According to the output of the optimization algorithm, adjust the projection parameters of the 3D model, generate the adjusted 2D projection image, and compare it with the ROI area in the actual image to check the matching effect; if the matching effect is not ideal, continue iterative optimization until the preset matching accuracy is achieved or the maximum number of iterations is reached.

3. The airport semantic segmentation method based on three-dimensional digital model and image enhancement technology according to claim 1 is characterized in that: In step 4, the specific method of performing fusion at the image end is as follows: Select weight parameters α and β so that α + β = 1; Pixel weighted calculation: Perform weighted calculation on each pixel to fuse the information of the projected image and the actual image. The specific calculation formula is as follows: I fused (x,y)=αI projection (x,y)+βI actual (x,y) Among them, I fused is the fused image, I projection is the projected image, I actual is the actual image, (x,y) is the pixel coordinate.

Images