A FOD detection method and system based on semantic segmentation

Abstract

The present disclosure relates to a technical field related to FOD detection, and proposes a FOD detection method and system based on semantic segmentation, the method comprising the following steps: obtaining an image to be detected of an airport runway, the image to be detected comprising a depth image and a grayscale image, extracting depth information and grayscale information in the image; preprocessing the depth information, and using the depth image obtained after preprocessing to preliminarily crop the grayscale image based on the grayscale information; fusing the information data in the processed depth image and the preliminarily cropped grayscale image to enhance the information of the foreign body area; inputting the fused information into an improved semantic segmentation network model for image segmentation, and determining the type and location of the foreign body in real time. The present disclosure improves the accuracy and precision of detection based on the fusion processing of depth information and grayscale information.

Description

Technical Field

[0001] The present disclosure relates to the technical field related to FOD detection, and specifically, to a FOD detection method and system based on semantic segmentation. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Foreign Object Debris (FOD) on the runway refers to some foreign substances, debris or objects that may damage aircraft, such as metal parts, gravel, etc. FOD will cause huge safety hazards during the takeoff and landing of the aircraft, which will not only cause huge economic losses, but also threaten the lives of passengers.

[0004] The inventors found that the current FOD detection mainly relies on manual inspection and capture, which is time-consuming, inefficient and costly. In recent years, some FOD detection methods have been proposed, but these methods mainly rely on sensors such as laser radar and FMCW radar, and require manually designed feature extractors when processing sensor information. These detection methods are also sensitive to changes in air and light, resulting in high detection costs and low detection accuracy. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a FOD detection method and system based on semantic segmentation, which improves the accuracy and precision of detection based on the fusion processing of depth information and grayscale information.

[0006] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0007] One or more embodiments provide a FOD detection method based on semantic segmentation, comprising the following steps:

[0008] Obtain an image to be detected of the airport runway, the image to be detected includes a depth image and a grayscale image H, and extract depth information and grayscale information in the image;

[0009] Preprocessing the depth information, and using the depth image obtained after the preprocessing to preliminarily crop the grayscale image based on the grayscale information;

[0010] The processed depth image and the information data in the preliminarily cropped grayscale image are fused to enhance the information of the foreign body area;

[0011] The fused information is input into the improved semantic segmentation network model for image segmentation to determine the type and location of foreign objects in real time.

[0012] One or more embodiments provide a FOD detection system based on semantic segmentation, including:

[0013] A data acquisition module is configured to acquire an image to be detected of an airport runway, wherein the image to be detected includes a depth image and a grayscale image H, and extract depth information and grayscale information from the image;

[0014] A preliminary cropping module is configured to pre-process the depth information and use the depth image obtained after the pre-processing to perform preliminary cropping on the grayscale image H based on the grayscale information;

[0015] A fusion module is configured to fuse the information data in the processed depth image and the grayscale image after preliminary cropping to enhance the information of the foreign body area;

[0016] The image segmentation module is configured to input the fused information into the improved semantic segmentation network model for image segmentation, so as to determine the type and location of the foreign body in real time.

[0017] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps described in the above method are completed.

[0018] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps described in the above method are completed.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] The detection method disclosed in the present invention, by cropping the grayscale image in combination with the depth information and enhancing the grayscale information through the fusion processing with the depth information, can better highlight the area where the foreign matter is located, and the segmentation can improve the accuracy of the segmentation, so as to extract the foreign matter in the image. Compared with the current FOD detection method, the FOD detection based on semantic segmentation has high efficiency, is less affected by the environment, and is more reliable.

[0021] The advantages of the present disclosure and the advantages of additional aspects will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and the description thereof are used to explain the present disclosure but do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flow chart of the detection method of Example 1 of the present disclosure;

[0024] Figure 2is a schematic diagram of preprocessing depth information in Embodiment 1 of the present disclosure;

[0025] Figure 3 is a preliminary cropping effect diagram of the grayscale image in Example 1 of the present disclosure;

[0026] Figure 4 is a schematic diagram of the segmentation effect of Embodiment 1 of the present disclosure;

[0027] Figure 5 It is a schematic diagram of the structure of the improved semantic segmentation network model CBAM-Deeplab V3+ of Example 1 of the present disclosure;

[0028] Figure 6 It is a schematic diagram of the structure of the ASPP-CBAM module in the improved semantic segmentation network model of Example 1 of the present disclosure. DETAILED DESCRIPTION

[0029] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0031] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments in the present disclosure and the features in the embodiments can be combined with each other. The embodiments will be described in detail below in conjunction with the accompanying drawings.

[0032] Example 1

[0033] In the technical solutions disclosed in one or more embodiments, Figure 1-Figure 6 As shown, a FOD detection method based on semantic segmentation includes the following steps:

[0034] Step 1: Obtain an image to be detected of the airport runway, the image to be detected includes a depth image and a grayscale image H, and extract depth information and grayscale information from the image respectively;

[0035] Step 2: preprocess the depth information, and use the depth image obtained after preprocessing to perform preliminary cropping on the grayscale image H based on the grayscale information;

[0036] Step 3: Fuse the information data in the processed depth image and the grayscale image after preliminary cropping to enhance the information of the foreign body area;

[0037] Step 4: Input the fused information into the improved semantic segmentation network model for image segmentation to determine the type and location of foreign matter in real time.

[0038] The detection method of this embodiment, by cropping the grayscale image in combination with the depth information and enhancing the grayscale information through the fusion processing with the depth information, can better highlight the area where the foreign matter is located, and the segmentation can improve the accuracy of the segmentation, so that the foreign matter in the image can be extracted. Compared with the current FOD detection method, the FOD detection based on semantic segmentation has high efficiency, is less affected by the environment, and is more reliable.

[0039] In some embodiments, the image to be detected can be collected by a vehicle-mounted linear scanning structured light camera, and the grayscale information DC0 (Data Channel 0) and depth information DC2 (Data Channel 2) of the foreign object in the area to be detected are obtained at the same time. The image to be detected taken by the camera includes two types of images. One is a depth image. The other is a grayscale image. These two images are obtained at the same time. The grayscale image is the carrier of the grayscale information, and the depth image contains the depth information and is the carrier of the depth information.

[0040] Optionally, the acquisition equipment includes a carrier vehicle and a vehicle-mounted linear scanning structured light camera arranged on the carrier vehicle. A specific setting method can be set at the rear of the carrier vehicle. The lens direction of the vehicle-mounted linear scanning structured light camera is toward both sides of the carrier vehicle, so that the road can be detected during the movement of the carrier vehicle.

[0041] The high-precision line scanning structured light camera used in this solution has millimeter-level resolution and is less affected by the environment. It can be applied to the FOD detection process and play a huge role.

[0042] Optionally, the depth information includes the distance from the real object to the reference plane; and the grayscale information includes pixel values ​​in the image.

[0043] In step 2, the depth information is preprocessed. For example, Figure 2 As shown, the details are as follows:

[0044] Step 21: set a virtual reference plane, determine the distance between each point and the reference plane according to the depth information, and construct a depth image using the distance as the pixel value of each point.

[0045] The depth information contains the height characteristics of the foreign object. Specifically, a virtual plane at a set distance from the camera is set as the baseline. The virtual plane can be a horizontal plane, that is, parallel to the ground. The pixel of each point in the depth image is the distance between the real object corresponding to the pixel and the baseline. The set distance can be 2 meters.

[0046] Step 22: Apply the Sobel operator to the obtained depth image to extract edge features in the image;

[0047] Step 23, using Gaussian filtering to reduce noise on the depth image after edge extraction;

[0048] Step 24: binarize the Gaussian filtered image to obtain a processed depth image.

[0049] Because most FODs are small in size, setting a suitable threshold can make the pixel values ​​corresponding to most foreign objects be 1, while the proportion of pixel values ​​corresponding to non-foreign objects being 1 is small.

[0050] In step 2, the grayscale image H is initially cropped using the depth information. Specifically, a moving window method may be used to initially crop the grayscale image. Specifically, the steps are as follows:

[0051] Step 2-1 sets the size of the moving window, and moves according to the step size to intercept the processed depth image;

[0052] Step 2-2 calculates the sum of pixel values ​​in the window intercepted image, and obtains the window with the largest sum of pixel values ​​as the cropping window;

[0053] Step 2-3 intercepts the corresponding area in the grayscale image H through the cropping window to obtain the cropped grayscale image.

[0054] A specific cropping method, first, use a moving window of size W*H to intercept the processed depth image, with the upper and lower step lengths s. Then, count the number of 1s in each window and record them. The pixels with a value of 1 are more likely to be the pixels corresponding to the real objects of foreign objects. Finally, select all windows with a large number of 1s, and use the grayscale image corresponding to the window to replace the original photo. The cropped image is as follows Figure 3 As shown, the left side is the original image before preliminary cropping, and the right side is the image after cropping.

[0055] FOD occupies a small area in the image, so it is difficult to achieve good results by directly using the image for image segmentation. Therefore, we use the moving window method to perform preliminary cropping of the grayscale image, which can improve the effect of image segmentation.

[0056] Step 3 is a data fusion step to achieve data enhancement in the foreign body area. Specifically, it includes the following steps:

[0057] Step 31, according to the cropping window for initially cropping the grayscale image, cropping the processed depth image so that the cropped depth image and the cropped grayscale image are in the same area;

[0058] Step 32: multiply the cropped depth image by the magnification factor w, and then add the bias to obtain a weight matrix; the bias can be set, and the set value can preferably be 1.

[0059] Step 33: multiply the weight matrix by the corresponding position of the cropped grayscale image to obtain the fused data after the foreign body area data is enhanced, which is used as the input data of the image segmentation model.

[0060] The processed depth image is also cropped so that the cropped depth image and the cropped grayscale image are in the same area. The part with a pixel value of 1 in the depth image is more likely to be a foreign body than the part with a pixel value of 0, and is what the image segmentation model needs to pay more attention to. Therefore, the cropped depth image is multiplied by the magnification factor w, and then the bias 1 is added to obtain the weight matrix. Finally, the weight matrix is ​​multiplied by the corresponding position of the cropped grayscale image to obtain the input data of the image segmentation model. In this way, through data fusion, more weights are given to the parts that are more likely to be foreign bodies, thereby enhancing the data of the foreign body area, so that the subsequent image segmentation can more accurately obtain the location of the foreign body.

[0061] Optionally, the improved semantic segmentation network model specifically includes a Deeplab V3+ network, a pyramid pooling module of an encoder of the Deeplab V3+ network is connected to an attention module, wherein the Deeplab V3+ network includes an encoder and a decoder connected in sequence, and the encoder includes a pyramid pooling module ASPP.

[0062] In this embodiment, semantic segmentation adopts a semantic segmentation model improved on the basis of Deeplab V3+, which consists of an encoder and a decoder. The backbone network of the encoder is a Resnet101 network, which is used to extract the semantic features of the image. The decoder uses the extracted semantic features to generate the final image. In order to make the features extracted by the encoder more prominent, a CBAM attention module is added after the pyramid pooling module of the encoder to form CBAM-Deeplab V3+. The model structure is as follows: Figure 5 As shown, the ASPP-CBAM module of the pyramid pooling module ASPP cascade CBAM is as follows Figure 6 The image segmentation effect is shown in Figure 4 shown.

[0063] Optionally, the method for performing semantic segmentation by using an improved semantic segmentation network model comprises the following steps:

[0064] Step 41: extract features from the fused data after the foreign body area data is enhanced.

[0065] The fused data after the foreign body area data enhancement is input into the CBAM-Deeplab V3+ backbone network Resnet34. The backbone network performs convolution, pooling, full connection and other operations on the input data to extract the features of the input data as the output data of the encoder stage;

[0066] The encoder is the first part of the segmentation model and can be based on any classification convolutional network model to extract features.

[0067] Step 42 inputs the extracted features into the attention module, increases the attention weight, and obtains scaled new features.

[0068] The output data of the encoder stage is input to the CBAM attention module as the input data of the CBAM attention module, and the scaled new features are obtained according to the attention coefficient.

[0069] The CBAM attention module includes a channel attention module and a spatial attention module. The channel attention module passes the input data through an average pooling module and a maximum pooling module at the same time to obtain the average pooling feature and the maximum pooling feature. Then the two features are sent to the same multi-layer perceptron respectively, the two values ​​are added and the sum is sent to the sigmoid function for activation to obtain the channel attention coefficient. The spatial attention module performs average pooling and maximum pooling operations on the input data on the spatial axis at the same time to obtain a maximum pooling feature description and an average pooling feature description, and then the two descriptions are spliced ​​together according to the channel features, and after a 7x7 convolution and sigmoid function, the spatial attention weight coefficient is obtained. Finally, the input data of the attention module is multiplied by the channel attention coefficient and the spatial attention coefficient in turn to obtain the scaled new features as the output data of the CBAM attention module.

[0070] Step 43: fuse and connect the extracted features with the scaled new features, and obtain the segmented data after decoding.

[0071] Specifically, the features extracted in the first two stages are sent to the decoder stage. The decoder part can fuse the low-level features and high-level features extracted by the encoder part at the same time. It consists of two bilinear difference functions, one of which performs bilinear interpolation on the output of the CBAM attention module, and the other interpolates the low-level features obtained in the second layer of the decoder backbone network. Because the low-level features are of higher dimension and the semantic information is of lower order, a 1x1 convolution is used for dimensionality reduction before bilinear interpolation.

[0072] In this embodiment, by encoding and decoding the input data, features can be fully extracted to achieve a better image segmentation effect.

[0073] Example 2

[0074] Based on Example 1, this embodiment provides a FOD detection system based on semantic segmentation, including:

[0075] A data acquisition module is configured to acquire an image to be detected of an airport runway, wherein the image to be detected includes a depth image and a grayscale image H, and extract depth information and grayscale information from the image;

[0076] A preliminary cropping module is configured to pre-process the depth information and use the depth image obtained after the pre-processing to perform preliminary cropping on the grayscale image based on the grayscale information;

[0077] A fusion module is configured to fuse the information data in the processed depth image and the grayscale image after preliminary cropping to enhance the information of the foreign body area;

[0078] The image segmentation module is configured to input the fused information into the improved semantic segmentation network model for image segmentation, so as to determine the type and location of the foreign body in real time.

[0079] Example 3

[0080] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps described in the method of Embodiment 1 are completed.

[0081] Example 4

[0082] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps described in the method of Embodiment 1 are completed.

[0083] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

[0084] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A FOD detection method based on semantic segmentation, characterized in that: The steps include: Obtain an image to be detected of the airport runway, the image to be detected includes a depth image and a grayscale image, and extract depth information and grayscale information from the image; Preprocessing the depth information, and using the depth image obtained after the preprocessing to perform preliminary cropping of the grayscale image based on the depth information; The information data in the pre-processed depth image and the preliminarily cropped grayscale image are fused to enhance the information of the foreign body area; The fused information is input into the improved semantic segmentation network model for image segmentation, and the type and location of foreign matter are determined in real time; The depth information is preprocessed, specifically including the following steps: A virtual reference plane is set, and the distance between each point and the reference plane is determined according to the depth information, and a depth image is constructed using the distance as the pixel value of each point; Apply the Sobel operator to the obtained depth image to extract edge features in the image; The depth image after edge extraction is subjected to denoising by Gaussian filtering; The Gaussian filtered image is binarized to obtain a processed depth image.

2. The FOD detection method based on semantic segmentation according to claim 1, characterized in that: The image to be detected is collected by a vehicle-mounted linear scanning structured light camera.

3. The FOD detection method based on semantic segmentation according to claim 1, characterized in that: The grayscale image is initially cropped using the depth information. Specifically, the grayscale image is initially cropped using a moving window method.

4. The FOD detection method based on semantic segmentation as claimed in claim 3, characterized in that: Using depth information to perform preliminary cropping of grayscale images includes the following steps: Set the size of the moving window and move it according to the step size to capture the preprocessed depth image; Calculate the sum of pixel values ​​in the window intercepted image, and the window with the largest sum of pixel values ​​is the cropping window; The corresponding area in the grayscale image is captured through the cropping window to obtain the cropped grayscale image.

5. The FOD detection method based on semantic segmentation as claimed in claim 1, characterized in that: The method for enhancing and fusing the information of the foreign body area comprises the following steps: According to the cropping window for initially cropping the grayscale image, the processed depth image is cropped so that the cropped depth image and the cropped grayscale image are in the same area; Multiply the cropped depth image by the magnification factor and then add the bias to get the weight matrix; The weight matrix is ​​multiplied by the corresponding position of the cropped grayscale image to obtain the fused data after the foreign body area data is enhanced.

6. The FOD detection method based on semantic segmentation according to claim 1, characterized in that: The improved semantic segmentation network model includes a Deeplab V3+ network. The Deeplab V3+ network includes an encoder and a decoder connected in sequence. The encoder includes a pyramid pooling module. The pyramid pooling module of the encoder of the Deeplab V3+ network is connected to an attention module, and the output of the attention module is connected to the decoder.

7. A FOD detection method based on semantic segmentation as claimed in claim 6, characterized in that: The method for performing semantic segmentation by using an improved semantic segmentation network model includes the following steps: Extract features from the fused data after the foreign body area data is enhanced; The extracted features are input into the attention module, the attention weight is increased, and the scaled new features are obtained; The features extracted are fused and connected with the scaled new features, and the segmented data is obtained after decoding.

8. A FOD detection system based on semantic segmentation, characterized in that: include: A data acquisition module is configured to acquire an image to be detected of an airport runway, wherein the image to be detected includes a depth image and a grayscale image, and extract depth information and grayscale information from the image; A preliminary cropping module is configured to pre-process the depth information and use the depth image obtained after the pre-processing to perform preliminary cropping on the grayscale image based on the depth information; A fusion module is configured to fuse the information data in the preprocessed depth image and the preliminarily cropped grayscale image to enhance the information of the foreign body area; An image segmentation module is configured to input the fused information into an improved semantic segmentation network model for image segmentation, and determine the type and location of foreign matter in real time; The depth information is preprocessed, specifically including the following steps: A virtual reference plane is set, and the distance between each point and the reference plane is determined according to the depth information, and a depth image is constructed using the distance as the pixel value of each point; Apply the Sobel operator to the obtained depth image to extract edge features in the image; The depth image after edge extraction is subjected to denoising by Gaussian filtering; The Gaussian filtered image is binarized to obtain a processed depth image.

9. An electronic device, characterized in that: It includes a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps of the FOD detection method based on semantic segmentation as described in any one of claims 1 to 7 are completed.

10. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the FOD detection method based on semantic segmentation as described in any one of claims 1 to 7.

Images