Airport semantic segmentation method based on three-dimensional digital model and image enhancement technology

Abstract

The invention discloses an airport semantic segmentation method based on a three-dimensional digital model and an image enhancement technology. The airport semantic segmentation method comprises the following steps: step 1, constructing an aircraft three-dimensional digital model; 2, defining a region of interest: automatically detecting and positioning an aircraft in the image by using a target detection algorithm, and generating a defined ROI region; 3, projecting and matching the 3D model, and outputting a two-dimensional projection of the aircraft three-dimensional model; step 4, aircraft feature enhancement: aircraft information enhancement is carried out at the image end and the feature end; and 5, semantic segmentation: carrying out semantic segmentation on the image after weight fusion so as to identify and extract an aircraft area in the airport image. By introducing the three-dimensional digital model and the image enhancement technology, richer geometric information and context information are provided, so that the segmentation precision of airplanes and other objects in an airport scene is remarkably improved, and the conditions of wrong segmentation and missed segmentation are reduced.

Description

Technical Field

[0001] The invention belongs to the field of computer vision and image processing, and in particular relates to an airport semantic segmentation method based on three-dimensional digital model and image enhancement technology. Background Art

[0002] Airport semantic segmentation algorithms have important application value in the fields of computer vision and image processing. Semantic segmentation refers to the classification of each pixel in an image into different semantic categories. For example, in an airport scene, pixels are classified into aircraft, runways, buildings, vehicles, etc. High-precision semantic segmentation algorithms are widely used in airport security monitoring, aircraft navigation, ground service management, etc.

[0003] Traditional image segmentation methods mainly include techniques based on edge detection, region growing, cluster analysis and graph cutting. These methods usually rely on low-level features of the image, such as color, texture and edge information. However, due to the complexity of airport scenes, these methods have significant limitations when processing airport images. Edge detection methods: such as Canny, Sobel, etc., are prone to false detection and missed detection when dealing with complex backgrounds and lighting changes. Region growing methods: such as seed-based region growing, rely on the selection of initial seeds and are easily affected by noise and occlusion. Cluster analysis methods: such as K-means clustering, have difficulty in processing high-dimensional feature spaces and are sensitive to the initial cluster centers. Graph cutting methods: such as GrabCut, have high computational complexity and are difficult to apply in real time.

[0004] In recent years, deep learning methods have made significant progress in the field of image segmentation. Convolutional neural networks (CNN) and their variants (such as U-Net, SegNet, Mask R-CNN) have achieved excellent performance on multiple benchmark datasets. However, these methods still face some challenges in airport semantic segmentation. Strong data dependence: The performance of deep learning models is highly dependent on the quality and quantity of a large amount of labeled data, and it is costly to obtain and annotate high-quality airport image datasets. Background complexity: In airport scenes, the color and shape similarity between aircraft and runways, buildings, vehicles and other objects is high, resulting in a decrease in segmentation accuracy. Lighting changes: The lighting conditions in the airport environment are complex and changeable, and traditional deep learning models find it difficult to maintain stable segmentation performance under different lighting conditions.

[0005] In summary, the existing airport semantic segmentation technology has significant deficiencies in segmentation accuracy, robustness, and computational efficiency. The main deficiencies are as follows:

[0006] ① Low segmentation accuracy: Existing segmentation algorithms based on 2D images have low segmentation accuracy when dealing with complex backgrounds and changing lighting conditions. Especially in airport scenes, the color and shape similarity between aircraft and runways, buildings, vehicles and other objects is high, resulting in inaccurate segmentation results.

[0007] ② Poor robustness: The lighting conditions in airport scenes are complex and changeable, and existing algorithms are unstable under different lighting conditions. Factors such as shadows, reflections, and weather changes will affect the segmentation effect and reduce the robustness of the algorithm.

[0008] ③ High computational complexity: Some high-precision segmentation algorithms have high computational complexity and are difficult to apply in real time. Airport scenes usually require real-time processing of large amounts of image data, which places high demands on the computational efficiency of the algorithm.

[0009] ④ Strong data dependence: The performance of deep learning models is highly dependent on the quality and quantity of a large amount of labeled data. It is expensive to obtain and annotate high-quality airport image datasets, and the diversity and generalization ability of the data are insufficient in different airport scenarios.

[0010] ⑤ Low segmentation efficiency: Semantic segmentation in airport images needs to achieve efficient processing while ensuring accuracy. Existing algorithms perform poorly in efficient segmentation tasks and are difficult to meet the needs of real-time processing. Summary of the invention

[0011] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an airport semantic segmentation method based on three-dimensional digital model and image enhancement technology. By introducing three-dimensional digital model and image enhancement technology, richer geometric information and context information are provided, thereby significantly improving the segmentation accuracy of aircraft and other objects in the airport scene and reducing the occurrence of mis-segmentation and missed segmentation.

[0012] The object of the present invention is achieved through the following technical solution: an airport semantic segmentation method based on a three-dimensional digital model and image enhancement technology, comprising the following steps:

[0013] Step 1: construct a three-dimensional digital model of the aircraft;

[0014] Step 2: Define the region of interest: Preprocess the collected airport images, use the target detection algorithm to automatically detect and locate the aircraft in the image, generate the bounding box of each aircraft, and define the area where the aircraft bounding box is located as the region of interest, recorded as the ROI area;

[0015] Step 3, 3D model projection and matching: First, the aircraft's posture is initially estimated based on the ROI area image to obtain the aircraft's pitch angle, roll angle, and yaw angle; then the three angles are used as optimization parameters, and the matching degree between the ROI area image and the aircraft's 3D model projection image is used as the objective function, and the heuristic optimization algorithm is used to optimize the objective function; when the matching degree reaches the requirement or the maximum number of iterations is reached, the iteration is stopped, and the 2D projection of the aircraft's 3D model is output for subsequent aircraft feature enhancement;

[0016] Step 4, aircraft feature enhancement: aircraft information is enhanced at the image end and feature end; at the image end, the two-dimensional projection image of the three-dimensional model is fused with the ROI area in the airport image using a weighted fusion method; the fused image is enhanced at the feature end, and the specific method is: the features of the airport image and the aircraft model projection image are extracted simultaneously, and the shallow features of the airport image are enhanced; then, according to the ROI area, the aircraft features are extracted from the airport image features, and interact with the features of the corresponding aircraft model projection image;

[0017] Step 5: Semantic segmentation: Use the weighted fused image after feature enhancement to perform semantic segmentation to identify and extract the aircraft area in the airport image; the specific steps are as follows:

[0018] Step 5-1, data preprocessing: normalize the fused image and normalize the pixel values ​​to the range of [0,1];

[0019] Step 5-2: Perform data enhancement;

[0020] Step 5-3: Perform feature extraction and segmentation: Use convolutional neural network for feature extraction and perform semantic segmentation based on U-Net model.

[0021] The present invention combines three-dimensional digital models and image enhancement technology to provide a high-precision, high-robustness, low-computational complexity and efficient airport semantic segmentation algorithm to solve key problems in the prior art and meet the needs of practical applications. The advantages of the present invention are specifically reflected in the following aspects:

[0022] ① Improved segmentation accuracy: By introducing three-dimensional digital models and image enhancement technology, richer geometric information and contextual information are provided, thereby significantly improving the segmentation accuracy of aircraft and other objects in airport scenes and reducing mis-segmentation and missed segmentation.

[0023] ② Enhanced algorithm robustness: The use of multi-view projection and image enhancement technology of three-dimensional models has improved the robustness of the algorithm under complex backgrounds and changing lighting conditions, enabling it to run stably in different environments and conditions and not be affected by shadows, reflections and weather changes.

[0024] ③ Reduced computational complexity: Although the introduction of three-dimensional models and 2D projections increases the initial amount of calculation, the overall computational complexity is reduced by optimizing the algorithm and simplifying the feature extraction process, which improves the real-time performance and efficiency of the segmentation algorithm, enabling it to process large amounts of image data in real time.

[0025] ④ Reduced data dependence: By utilizing the geometric information and multi-view images provided by the 3D model, the dependence on large amounts of labeled data is reduced, the generalization ability of the algorithm is improved, enabling it to perform well in different airport scenarios, and reducing the cost of acquiring and annotating high-quality data sets.

[0026] ⑤ Achieve efficient segmentation: Combining three-dimensional models and image enhancement technology can not only improve segmentation accuracy and robustness, but also achieve efficient processing to meet the needs of real-time processing of large amounts of image data in airport scenarios, thereby improving the system's response speed and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the airport semantic segmentation method based on three-dimensional digital model and image enhancement technology of the present invention;

[0028] Figure 2 Flowchart for 3D model projection and matching;

[0029] Figure 3 Schematic diagram of how cross-attention features interact;

[0030] Figure 4 It is the semantic segmentation map of this embodiment. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0032] like Figure 1 As shown, a method for semantic segmentation of an airport based on a three-dimensional digital model and image enhancement technology of the present invention comprises the following steps:

[0033] Step 1: Build a 3D digital model of the aircraft: Obtain a 3D digital model of the aircraft through laser scanning, structured light or stereo vision technology. The model contains detailed geometric information of the aircraft, such as contours, surface features, etc. The main task of this stage is to generate a high-precision 3D point cloud or mesh model of the aircraft to ensure the geometric accuracy and integrity of the model.

[0034] Step 2: Define the region of interest (ROI): Preprocess the collected airport images, use target detection algorithms (such as YOLO or Faster R-CNN) to automatically detect and locate the aircraft in the image, generate a bounding box for each aircraft, and define the area where the aircraft's bounding box is located as the region of interest, recorded as the ROI area; these bounding boxes will be used for subsequent 3D model projection and matching to provide accurate initial position information.

[0035] Step 3, 3D model projection and matching: First, make an initial estimate of the aircraft's posture based on the ROI area image to obtain the aircraft's pitch angle, roll angle, and yaw angle; then use the three angles as optimization parameters, and the matching degree between the ROI area image and the aircraft's 3D model projection image as the objective function, and use a heuristic optimization algorithm to optimize the objective function; when the matching degree reaches the requirement or the maximum number of iterations is reached, the iteration is stopped, and the 2D projection of the aircraft's 3D model is output for subsequent aircraft feature enhancement. The specific process is as follows: Figure 2 shown.

[0036] The specific implementation method of step 3 is:

[0037] Step 3-1: Preliminary viewing angle estimation, including the following steps:

[0038] Step 3-1-1, edge detection: perform edge detection on the aircraft image in the ROI area and use the Canny edge detection algorithm to extract the contour edge of the aircraft;

[0039] Step 3-1-2, key point detection: Use SIFT or ORB algorithm on the edge image to detect the key points of the aircraft; these key points can be obvious feature points such as the nose, tail, and wings of the aircraft. You can use algorithms such as SIFT (Scale Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF) to detect key points.

[0040] Step 3-1-3, preliminary view angle estimation: Preliminary estimate of the aircraft's attitude based on the detected key point distribution and edge shape; the specific method is: infer the aircraft's pitch angle, roll angle and yaw angle by analyzing the relative position of the key points and the aspect ratio of the edge:

[0041] Pitch Angle: The angle of rotation of the aircraft around the horizontal axis (the axis from wingtip to wingtip), which indicates the degree of up and down tilt of the aircraft's nose relative to the horizontal plane; Roll Angle: The angle of rotation of the aircraft around the longitudinal axis (the axis from the nose to the tail), which indicates the degree of tilt of the aircraft's wings; Yaw Angle: The angle of rotation of the aircraft around the vertical axis (the axis vertically downward from the center of the aircraft), which indicates the degree of left and right deflection of the nose relative to the north direction;

[0042]

[0043] (x1,y1), (x2,y2), (x3,y3), (x4,y4) are the image coordinates of the nose, tail, left wing, and right wing, respectively, obtained through key point detection;

[0044] Step 3-2, viewing angle optimization, includes the following sub-steps:

[0045] Step 3-2-1, initial matching: perform preliminary alignment of the generated 2D projection image with the ROI area in the actual image; initial alignment can be achieved through simple geometric transformations (such as scaling and translation). Use the initially estimated angles to project the 3D model onto the 2D plane, and continuously adjust the three angles through optimization algorithms (such as iterative closest point algorithm ICP, gradient descent, etc.) to achieve the best match between the projection image and the actual image;

[0046] Step 3-2-2, optimize the objective function: define a matching optimization objective function, which measures the matching degree between the projected image and the actual image; the objective function includes the following components: (1) shape difference: use edge detection algorithm (such as Canny or Sobel) to extract the edges of the projected image and the actual image, and calculate the Hausdorff distance or Chamfer distance between the edges; (2) pixel difference: calculate the difference in pixel intensity between the projected image and the actual image in the overlapping area, for example, using mean square error (MSE) or structural similarity (SSIM) indicators.

[0047] Step 3-2-3, Optimization algorithm: Use the optimization algorithm to adjust the projection parameters (such as scaling, rotation, translation) to minimize the objective function value. The specific methods of the optimization algorithm include: Iterative Closest Point (ICP) algorithm: Minimize the distance between the projected image and the edge point of the actual image through iterative optimization. Gradient descent: By calculating the gradient of the objective function, iteratively update the projection parameters and gradually approach the optimal solution. Particle swarm optimization (PSO): Find the global optimal solution of the objective function by simulating the search behavior of the particle group. You can choose any optimization algorithm according to your actual needs.

[0048] Step 3-3, projection parameter adjustment and best match; Parameter adjustment: According to the output of the optimization algorithm, adjust the projection parameters of the 3D model, including viewing angle, scaling, rotation and translation, etc. Best match: Generate the adjusted 2D projection image, compare it with the ROI area in the actual image, and check the matching effect; if the matching effect is not ideal, continue to iterate and optimize until the preset matching accuracy is reached or the maximum number of iterations is reached.

[0049] Step 4, aircraft feature enhancement: In the present invention, in order to enhance the image features and facilitate the subsequent semantic segmentation processing, the aircraft information is enhanced at the image end and the feature end; at the image end, the weighted fusion method is used to fuse the two-dimensional projection image of the three-dimensional model with the ROI area in the airport image. The weighted fusion method is simple and intuitive, and can effectively combine the geometric information of the projection image and the texture information of the actual image. The specific method is as follows:

[0050] The weight parameters α and β are selected so that α+β=1; the selection of the weight parameters can be adjusted according to the experimental results to achieve the best fusion effect.

[0051] Pixel weighted calculation: Perform weighted calculation on each pixel to fuse the information of the projected image and the actual image. The specific calculation formula is as follows:

[0052] I fused (x, y) = αI projection (x,y)+βI actual (x, y)

[0053] Among them, I fused is the fused image, I projection is the projected image, I actual is the actual image, (x, y) is the pixel coordinate. Through the above steps, a fused image can be obtained, which combines the geometric information of the projected image and the texture information of the actual image, enhances the image features, and provides a high-quality input image for subsequent semantic segmentation.

[0054] The fused image is enhanced at the feature end. The specific method is: extract the features of the airport image and the aircraft model projection image at the same time, and enhance the shallow features of the airport image; then, according to the ROI area, extract the aircraft features from the airport image features and interact with the features of the corresponding aircraft model projection image. The specific method used is the cross attention method, such as Figure 3 As shown, the airport ROI image and the matching 3D model projection image are respectively fed into the same image encoder to obtain the airport features and projection features. In order to use the projection features to enhance the airport features, the two features are fused using the attention operation. Specifically, the airport features are used as the query, and the projection features are used as the key and value. The attention matrix (indicating the area that needs to be focused on) is obtained by matrix multiplication of the query and the key. The attention matrix is ​​multiplied by the value to obtain the attention output feature, which is linked to the airport feature residual to complete the enhancement of the airport feature.

[0055] Step 5: Semantic segmentation: Use the weighted fused image after feature enhancement to perform semantic segmentation to identify and extract the aircraft area in the airport image; the specific steps are as follows:

[0056] Step 5-1, data preprocessing: normalize the fused image and normalize the pixel values ​​to the range of [0,1];

[0057] Step 5-2: Perform data enhancement operations such as rotation, scaling, translation, flipping, etc.

[0058] Step 5-3, perform feature extraction and segmentation: Use convolutional neural networks (such as ResNet, VGG) for feature extraction, and perform semantic segmentation based on the U-Net model. U-Net has an encoder-decoder structure, which can effectively integrate feature information at different levels to achieve accurate segmentation. Model training: Use the cross entropy loss function or the Dice loss function to measure the difference between the predicted result and the true label. Use the Adam optimization algorithm to iteratively update the model parameters and minimize the loss function. Model prediction and post-processing: Input the preprocessed image into the trained segmentation model to obtain the predicted segmentation result. Perform post-processing on the predicted results, such as threshold segmentation and morphological operations, to obtain the final binary segmentation map. Result evaluation: Use indicators such as intersection over union (IoU) and Dice coefficient to evaluate the performance of the segmentation results. Visualize the segmentation results and compare them with the actual image to show the segmentation effect, such as Figure 4 As shown, Figure 4 (a) is the original image, Figure 4 (b) is the semantic segmentation image.

[0059] The present invention combines the two-dimensional projection image of the three-dimensional digital model with the actual image for weighted fusion, thereby enhancing the image features, making the outline and details of the aircraft clearer, and thus significantly improving the segmentation accuracy.

[0060] Enhanced robustness: By adopting the iterative closest point (ICP) algorithm and gradient descent optimization algorithm, the projection parameters of the 3D model are automatically adjusted to achieve the best match with the actual image, thereby improving the robustness of the system under different perspectives and complex backgrounds.

[0061] Flexibility and adaptability: Using the weighted fusion method, the weight parameters can be flexibly adjusted to optimize the fusion effect according to actual needs, ensuring that both the geometric information of the projected image and the texture information of the actual image are retained to adapt to different application scenarios.

[0062] In summary, the present invention is superior to the existing technologies in terms of accuracy, robustness, flexibility, etc. through the combination of multiple innovative technologies, which significantly improves the effect of aircraft recognition and segmentation in airport images and provides reliable technical support for airport management and flight safety.

[0063] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. An airport semantic segmentation method based on three-dimensional digital model and image enhancement technology is characterized by: The following steps are involved: Step 1: construct a three-dimensional digital model of the aircraft; Step 2: Define the region of interest: Preprocess the collected airport images, use the target detection algorithm to automatically detect and locate the aircraft in the image, generate the bounding box of each aircraft, and define the area where the aircraft bounding box is located as the region of interest, recorded as the ROI area; Step 3, 3D model projection and matching: First, the aircraft's posture is initially estimated based on the ROI area image to obtain the aircraft's pitch angle, roll angle, and yaw angle; then the three angles are used as optimization parameters, and the matching degree between the ROI area image and the aircraft's 3D model projection image is used as the objective function, and the heuristic optimization algorithm is used to optimize the objective function; when the matching degree reaches the requirement or the maximum number of iterations is reached, the iteration is stopped, and the 2D projection of the aircraft's 3D model is output for subsequent aircraft feature enhancement; Step 4, aircraft feature enhancement: aircraft information is enhanced at the image end and feature end; at the image end, the two-dimensional projection image of the three-dimensional model is fused with the ROI area in the airport image using a weighted fusion method; the fused image is enhanced at the feature end, specifically by extracting features of the airport image and the aircraft model projection image at the same time, and enhancing the shallow features of the airport image; Then, according to the ROI area, the aircraft features are extracted from the airport image features and interacted with the features of the corresponding aircraft model projection image; Step 5: Semantic segmentation: Use the weighted fused image after feature enhancement to perform semantic segmentation to identify and extract the aircraft area in the airport image; the specific steps are as follows: Step 5-1, data preprocessing: normalize the fused image and normalize the pixel values ​​to the range of [0,1]; Step 5-2: Perform data enhancement; Step 5-3: Perform feature extraction and segmentation: Use convolutional neural network for feature extraction and perform semantic segmentation based on U-Net model.

2. The airport semantic segmentation method based on three-dimensional digital model and image enhancement technology according to claim 1 is characterized in that: The specific implementation method of step 3 is: Step 3-1: Preliminary viewing angle estimation, including the following steps: Step 3-1-1, edge detection: perform edge detection on the aircraft image in the ROI area and use the Canny edge detection algorithm to extract the contour edge of the aircraft; Step 3-1-2, key point detection: Use SIFT or ORB algorithm to detect the key points of the aircraft on the edge image; Step 3-1-3, preliminary view angle estimation: Preliminary estimate of the aircraft's attitude based on the detected key point distribution and edge shape; the specific method is: infer the aircraft's pitch angle, roll angle and yaw angle by analyzing the relative position of the key points and the aspect ratio of the edge: Pitch Angle: The angle at which the aircraft rotates around the horizontal axis, which indicates the degree of up and down tilt of the aircraft's nose relative to the horizontal plane; Roll Angle: The angle at which the aircraft rotates around the longitudinal axis, which indicates the degree of tilt of the aircraft's wings; Yaw Angle: The angle at which the aircraft rotates around the vertical axis, which indicates the degree of left and right deflection of the aircraft's nose relative to the north direction; (x1,y1), (x2,y2), (x3,y3), (x4,y4) are the image coordinates of the nose, tail, left wing, and right wing, respectively, obtained through key point detection; Step 3-2, viewing angle optimization, includes the following sub-steps: Step 3-2-1, initial matching: preliminarily align the generated 2D projection image with the ROI area in the actual image; project the 3D model onto the 2D plane using the preliminarily estimated angles, and continuously adjust the three angles to achieve the best match between the projection image and the actual image; Step 3-2-2, optimize the objective function: define a matching optimization objective function, which measures the matching degree between the projected image and the actual image; the objective function includes the following components: (1) shape difference: use the edge detection algorithm to extract the edges of the projected image and the actual image, and calculate the Hausdorff distance or Chamfer distance between the edges; (2) pixel difference: calculate the difference in pixel intensity between the projected image and the actual image in the overlapping area; Step 3-2-3, optimization algorithm: Use the optimization algorithm to adjust the projection parameters and minimize the objective function value; Step 3-3, projection parameter adjustment and optimal matching: According to the output of the optimization algorithm, adjust the projection parameters of the 3D model, generate an adjusted 2D projection image, compare it with the ROI area in the actual image, and check the matching effect; if the matching effect is not ideal, continue to iterate and optimize until the preset matching accuracy is reached or the maximum number of iterations is reached.

3. The airport semantic segmentation method based on three-dimensional digital model and image enhancement technology according to claim 1 is characterized in that: In step 4, the specific method of performing fusion at the image end is as follows: Select weight parameters α and β so that α+β=1; Pixel weighted calculation: Perform weighted calculation on each pixel to fuse the information of the projected image and the actual image. The specific calculation formula is as follows: l fused (x,y)=al projection (x, y)+βl actual (x, y) Among them, I fused is the fused image, I projection is the projected image, I actual is the actual image, (x,y) is the pixel coordinate.

Images