A multi-channel visual information fusion FOD detection method and system

Abstract

The present disclosure relates to the technical field of foreign matter detection, and provides a multi-channel visual information fusion FOD detection method and system. The method comprises: acquiring a to-be-detected image, the to-be-detected image comprising a grayscale image, a laser scattering brightness image and a depth image; performing foreign matter edge extraction on the laser scattering brightness image to obtain a processed laser scattering brightness image; performing edge extraction processing on the depth image to obtain an edge map of the depth image; fusing the grayscale image, the processed laser scattering brightness image and the edge map of the depth image as three color channels respectively to obtain a multi-channel fused foreign matter image; and performing foreign matter detection on the multi-channel fused foreign matter image to obtain the type and position of the foreign matter. Through multi-channel fusion, the fused image has three channel features at the same time, the foreign matter edge is more obvious, the difficulty of visual detection by the human eye is overcome, and the detection efficiency and reliability are improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of foreign object detection, and in particular, to a multi-channel visual information fusion FOD detection method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute the prior art.

[0003] Foreign object debris (FOD) refers to foreign substances that can damage an aircraft, such as screws, nuts, metal fragments, stones, plastic debris, etc., which have a huge safety hazard during the takeoff and landing of an aircraft. FOD not only threatens the safety of passengers, but also causes huge economic losses to airlines and airports. FOD hazards are a problem that needs to be solved urgently.

[0004] Currently, the FOD detection method of most airports in the world is still manual visual detection, which is prone to cause personnel fatigue, and is not only low in efficiency, but also poor in reliability. Foreign countries started early in the research of FOD detection, and four typical FOD detection systems have appeared: Tarsier system, FOD Detect system, FODFinder system and IFerret system. The first three systems use ground millimeter wave radar to detect and locate runway foreign objects, and the millimeter wave radar system has a high cost and high requirements for the size of foreign objects; the IFerret system is composed of a high-definition camera and a data processing system, and the system relies on the camera for data acquisition, and has poor performance in bad weather. It can be seen that the current detection method has high detection cost, is limited by foreign objects and detection environment, and has low detection effect. SUMMARY

[0005] In order to solve the above problems, the present disclosure provides a multi-channel visual information fusion FOD detection method and system, which has three channel characteristics after fusion through multi-channel fusion, and the foreign object edge is more obvious, overcoming the difficulty of human visual detection and improving the detection efficiency and reliability.

[0006] In order to achieve the above purpose, the present disclosure adopts the following technical solutions:

[0007] One or more embodiments provide a multi-channel visual information fusion FOD detection method, comprising the following steps:

[0008] An image to be detected is obtained, and the image to be detected includes a gray image, a laser scattering brightness image and a depth image;

[0009] Foreign object edge extraction is performed on the laser scattering brightness image to obtain a processed laser scattering brightness image;

[0010] edge extraction processing is performed on the depth image to obtain an edge map of the depth image;

[0011] The gray-scale image, the processed laser scattering brightness image, and the edge map of the depth image are fused as three color channels respectively to obtain a multi-channel fused foreign object image;

[0012] Foreign object detection is performed on the multi-channel fused foreign object image to obtain the type and position of the foreign object.

[0013] One or more embodiments provide a multi-channel visual information fusion FOD detection system, comprising:

[0014] An acquisition module configured to acquire a to-be-detected image, the to-be-detected image including a gray-scale image, a laser scattering brightness image, and a depth image;

[0015] A laser scattering brightness image processing module configured to perform foreign object edge extraction on the laser scattering brightness image to obtain a processed laser scattering brightness image;

[0016] A depth image processing module configured to perform edge extraction processing on the depth image to obtain an edge map of the depth image;

[0017] A fusion module configured to fuse the gray-scale image, the processed laser scattering brightness image, and the edge map of the depth image as three color channels respectively to obtain a multi-channel fused foreign object image;

[0018] A detection module configured to perform foreign object detection on the multi-channel fused foreign object image to obtain the type and position of the foreign object.

[0019] One or more embodiments provide a multi-channel visual information fusion FOD detection system, comprising: an image acquisition device and a processor;

[0020] The image acquisition device is a vehicle-mounted line scanning structured light camera configured to acquire a to-be-detected image;

[0021] The processor is configured to execute the above-described multi-channel visual information fusion FOD detection method.

[0022] An electronic device comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps of the above-described method are completed.

[0023] Compared with the prior art, the present disclosure has the following beneficial effects:

[0024] The present application collects multiple types of images, and the fused image has three channel characteristics at the same time, the foreign object edge is more obvious, overcomes the difficulty of human visual detection, improves the detection efficiency and reliability; and breaks away from the dependence of the camera on natural light, can be used in the dark environment, and can effectively detect runway foreign objects.

[0025] The advantages of the present disclosure and the advantages of additional aspects will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which form a part of the present disclosure, are intended to provide further understanding of the present disclosure, and the illustrative embodiments of the present disclosure and their description serve the purpose of explaining the present disclosure. They do not constitute limitations on the present disclosure.

[0027] Figure 1 is a detection method flowchart of embodiment 1 of the present disclosure;

[0028] Figure 2 is a gray scale image example collected by the vehicle-mounted line scanning structured light camera of embodiment 1 of the present disclosure;

[0029] Figure 3 is an example laser scattering brightness image processing schematic diagram of embodiment 1 of the present disclosure;

[0030] Figure 4 is a Laplacian operator of example laser scattering brightness image processing of embodiment 1 of the present disclosure;

[0031] Figure 5(a) is a depth image processing method flowchart of embodiment 1 of the present disclosure;

[0032] Figure 5(b) is an example depth image processing schematic diagram of embodiment 1 of the present disclosure;

[0033] Figure 6 is an x-direction Sobel edge detection operator of depth image processing of embodiment 1 of the present disclosure;

[0034] Figure 7 is a y-direction Sobel edge detection operator of depth image processing of embodiment 1 of the present disclosure;

[0035] Figure 8 is an example three-channel fused image of the image to be detected of embodiment 1 of the present disclosure;

[0036] Figure 9(a) is a detection result image of the foreign object being a pliers of embodiment 1 of the present disclosure;

[0037] Figure 9(b) is a detection result image of the foreign object being a screw of embodiment 1 of the present disclosure;

[0038] Figure 9(c) is a detection result image of the foreign object being a stone of embodiment 1 of the present disclosure;

[0039] Fig. 9(d) is a detection result diagram of the foreign matter being grass and wood of the embodiment 1 of the present disclosure;

[0040] Fig. 9(e) is a detection result diagram of the foreign matter being plastic product of the embodiment 1 of the present disclosure. DETAILED DESCRIPTION

[0041] The present disclosure will be further described below in conjunction with the accompanying drawings and embodiments.

[0042] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the present disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0043] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that the terms "comprise" and / or "include" as used in the specification indicate the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that the various embodiments and features in the present disclosure can be combined with each other without conflict, and the embodiments will be described in detail below in conjunction with the accompanying drawings.

[0044] Embodiment 1

[0045] In the technical solutions disclosed in one or more embodiments, as Figure 1 - As shown in Fig. 9, a multi-channel visual information fusion FOD detection method includes the following steps:

[0046] Step 1, acquiring a to-be-detected image, the to-be-detected image including a gray-scale image, a laser scattering brightness image and a depth image;

[0047] Step 2, performing foreign matter edge extraction on the laser scattering brightness image to obtain a processed laser scattering brightness image;

[0048] Step 3, performing edge extraction processing on the depth image to obtain an edge map of the depth image;

[0049] Step 4, fusing the gray-scale image, the processed laser scattering brightness image and the edge map of the depth image as three color channels respectively to obtain a multi-channel fusion foreign matter image;

[0050] Step 5, performing foreign matter detection on the multi-channel fusion foreign matter image to obtain the type and position of the foreign matter.

[0051] The embodiment collects multiple types of images, and the fused image has three channel characteristics at the same time, the foreign object edge is more obvious, the difficulty of visual detection by human eyes is overcome, and the detection efficiency and reliability are improved; and the embodiment is free from the dependence on natural light of the camera, can be used in a dark environment, and can effectively detect runway foreign objects.

[0052] In step 1, the image can be collected by using a vehicle-mounted line scanning structured light camera, and the image to be detected is obtained at the same time.

[0053] Optionally, the embodiment can be arranged on a patrol vehicle and arranged at the tail of the vehicle.

[0054] When the patrol vehicle moves to obtain the image to be detected of the runway in real time, the scanning structured light camera has three shooting channels, and the gray scale information DC0 (Data Channel0) obtained at the same time is a gray scale image, the laser scattering brightness information DC1 (Data Channel1) is a laser scattering brightness image, and the depth information DC2 (Data Channel2) is a depth image.

[0055] In step 2, the foreign object edge of the laser scattering brightness image is extracted, and the processed laser scattering brightness image is obtained. Optionally, the Laplacian operator can be used to extract the foreign object edge.

[0056] Optionally, as shown in Figure 3 , the method for extracting the foreign object edge by using the Laplacian operator is as follows: the laser scattering brightness image is subjected to a convolution operation with a Laplacian operator of a set size.

[0057] Specifically, the Laplacian operator can be set to 3*3 in the embodiment, and the specific operator is as shown in Figure 4 .

[0058] In the embodiment, the extraction of the foreign object edge of the laser scattering brightness image does not need to detect the edges in the x direction and the y direction respectively, and only one edge detection can obtain the image meeting the requirements, thereby greatly improving the detection efficiency.

[0059] The depth image contains the height characteristics of the foreign object, in step 3, the depth image is processed, the edge of the depth image is extracted, and the edge image of the depth image is obtained. Optionally, the Sobel operator can be used to obtain the edge of the depth image, as shown in FIG. 5, and the specific steps are as follows:

[0060] In step 31, the edge in the x direction is extracted, and the original image is subjected to a convolution operation with an x direction Sobel edge detection operator (as shown in Figure 6 ) to extract the edge in the x direction.

[0061] Step 32, extract the edge in the y direction, and perform convolution operation on the original image and the y direction Sobel edge detection operator (as shown in Figure 7

[0062] Step 33, integrate the edge information in the x direction and the y direction to obtain the edge of the whole image, and obtain the edge map of the depth image.

[0063] The method for integrating the edge information in the x direction and the y direction to obtain the edge of the whole image is specifically: summing the absolute values of the pixel values of the corresponding pixels of the two images obtained in steps 1 and 2 to obtain the edge information of the whole image.

[0064] In this embodiment, the edges in the x direction and the y direction are first extracted, and then the edge information in the two directions is integrated to obtain the edge of the whole image, so that the positions where the depth changes in the depth image can be obtained, that is, the contour edges of the foreign matter in the depth image are enhanced, and the purpose of separating the foreign matter from the road surface is achieved.

[0065] In step 4, the edge map of the depth image is fused as three color channels respectively to obtain a multi-channel fused foreign matter image.

[0066] Optionally, the three color channels are G (green) channel, B (blue) channel and R (red) channel. Specifically, the gray-scale image is taken as the G (green) channel, the processed laser scattering brightness image is taken as the B (blue) channel, and the edge map of the depth image is taken as the R (red) channel, and image fusion is performed to form a pseudo-color image. The fused image has three channel characteristics at the same time, and the foreign matter edge is more obvious, as shown in Figure 8 Figure 8

[0067] In this embodiment, a three-channel fused image is obtained by fusion, and finally a detection algorithm is used to detect the fused image to obtain a detection result, which improves the efficiency and reliability of detection to a certain extent.

[0068] In step 5, after obtaining the multi-channel fused foreign matter image, an improved YOLOV5 (You Only Look Once) algorithm is used for runway foreign matter real-time detection. The improved YOLOV5 is specifically that a CBAM attention mechanism module is connected at the output end of the backbone network. After the multi-channel fused foreign matter image is input into the trained YOLOV5 network, the position and type of the foreign matter are obtained.

[0069] ​The Yolov5 network is trained to extract image features and generate an optimal detection model for runway foreign object real-time detection. The specific method for training the Yolov5 network is as follows:

[0070] Step 51, pre-process the images of the constructed training set.

[0071] Specifically, the image is pre-processed at the input end, including Mosaic data enhancement, adaptive anchor frame calculation, adaptive image scaling, etc.

[0072] Step 52, feature extraction on the pre-processed image.

[0073] Optionally, the attention mechanism CBAM is integrated into the Backbone, and Conv, C3, CBAM, and SPPF can be used in the Backbone for image feature extraction.

[0074] Wherein, Backbone is the backbone network, Conv is the combination of convolution, batch normalization and activation function, C3 is a network structure containing three standard convolution layers and multiple Bottleneck modules; Bottleneck is a bottleneck structure; CBAM is a new convolutional attention module; SPPF is a fast spatial pyramid pooling.

[0075] Step 53, feature fusion of the extracted features: high-level features are fused with low-level features through upsampling, and low-level features are fused with high-level features through downsampling, so that different size feature maps contain strong semantic information and strong position information.

[0076] Optionally, FPN+PAN can be used in the Neck of the Yolov5 network to fuse the features extracted by the Backbone network.

[0077] High-level features contain stronger semantic information and weaker position information, while low-level features contain stronger position information and weaker semantic information. FPN transmits high-level semantic information to low-level to enhance semantic expression at multiple scales, and PAN transmits low-level position information to high-level to enhance positioning ability at multiple scales. FPN fuses high-level features with low-level features through upsampling, and PAN fuses low-level features with high-level features through downsampling, so that different size feature maps contain strong semantic information and strong position information.

[0078] The FPN structure performs upsampling from top to bottom, so that the bottom feature map contains strong foreign object semantic information; the PAN structure performs downsampling from bottom to top, so that the top feature contains strong location information of the foreign object, and the two features are finally fused, so that the feature maps of different sizes contain strong foreign object semantic information and strong location information, and the network detection capability is improved.

[0079] wherein FPN is a Feature Pyramid Network, PAN is a Path Aggregation Network, and Neck is a neck network.

[0080] Step 54: The fused features are taken as the input of the Yolov5 network, and the types and positions of the runway foreign objects are taken as the output for training until a set number of training times is reached, and a trained Yolov5 model is obtained.

[0081] The set number of training times can be set as needed, and can be set as 100 rounds of training.

[0082] The multi-channel fused foreign object image to be recognized is input into the trained Yolov5 model for detection, and the types and positions of the runway foreign objects are obtained and marked.

[0083] The types of foreign objects can include tools, sundries, parts, etc., wherein the tools can be aircraft maintenance tools such as pliers, wrenches, etc., the sundries can be stones, plants, etc., and the parts can be parts on vehicles or aircraft, such as bolts, nuts, etc.

[0084] To illustrate the detection effect, the foreign object image is detected, and the output recognition result is shown in FIG. 9. Various types of foreign objects including pliers, screws, stones, plants, and plastic products can achieve good recognition effect, and are not limited by light, improving the reliability and efficiency of foreign object recognition.

[0085] Embodiment 2

[0086] Based on embodiment 1, the present embodiment provides a multi-channel visual information fusion FOD detection system, comprising:

[0087] An acquisition module configured to acquire an image to be detected, wherein the image to be detected includes a grayscale image, a laser scattering brightness image, and a depth image;

[0088] A laser scattering brightness image processing module configured to perform foreign object edge extraction on the laser scattering brightness image to obtain a processed laser scattering brightness image;

[0089] A depth image processing module configured to perform edge extraction processing on the depth image to obtain an edge map of the depth image;

[0090] fusion module: configured to fuse the edge map of the grayscale image, the processed laser scattering brightness image and the depth image as three color channels respectively, to obtain a multi-channel fused foreign object image;

[0091] detection module: configured to detect the foreign object from the multi-channel fused foreign object image, to obtain the type and position of the foreign object.

[0092] It should be noted that each module in the embodiment corresponds to each step in Embodiment 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0093] Embodiment 3

[0094] Based on Embodiment 1, the embodiment provides a multi-channel visual information fusion FOD detection system, comprising: an image acquisition device and a processor;

[0095] The image acquisition device is a vehicle-mounted line scanning structured light camera, configured to acquire a to-be-detected image.

[0096] The processor is configured to execute the multi-channel visual information fusion FOD detection method of Embodiment 1.

[0097] Embodiment 4

[0098] The embodiment provides an electronic device, comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps of the method of Embodiment 1 are completed.

[0099] The above only describes the preferred embodiments of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

[0100] The above describes the specific embodiments of the present disclosure in conjunction with the accompanying drawings, but is not a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications or changes made on the basis of the technical solutions of the present disclosure without creative labor are still within the protection scope of the present disclosure.

Claims

1. A multi-channel visual information fusion FOD detection method, characterized in that, The method comprises the following steps: acquire an image to be detected, the image to be detected comprising a gray image, a laser scattering brightness image and a depth image; perform foreign object edge extraction on the laser scattering brightness image to obtain a processed laser scattering brightness image; perform edge extraction processing on the depth image to obtain an edge map of the depth image; fuse the gray image, the processed laser scattering brightness image and the edge map of the depth image as three color channels respectively to obtain a multi-channel fused foreign object image; in the three color channel fusion, the three color channels are G channel, B channel and R channel respectively; the gray image is taken as the G channel, the processed laser scattering brightness image is taken as the B channel, and the edge map of the depth image is taken as the R channel to perform image fusion to form a pseudo-color image; perform foreign object detection on the multi-channel fused foreign object image to obtain the type and position of the foreign object.

2. The FOD detection method of claim 1, wherein: An image acquisition device and a processor are included.

3. The FOD detection method of claim 1, wherein: The image acquisition device is a vehicle-mounted line scanning structured light camera for acquiring an image to be detected.

4. The multi-channel visual information fusion FOD detection method of claim 1, wherein: The processor is configured to execute a multi-channel visual information fusion FOD detection method according to any one of claims 1-6.

5. The multi-channel visual information fusion FOD detection method of claim 1, wherein: The method comprises the following steps: extracting an x-direction edge, performing convolution operation on the original image and an x-direction Sobel edge detection operator to extract the x-direction edge; extracting a y-direction edge, performing convolution operation on the original image and a y-direction Sobel edge detection operator to extract the y-direction edge; integrating the edge information of the x-direction and the y-direction to obtain the edge of the entire image to obtain the edge map in the depth direction.

6. The FOD detection method of claim 1, wherein: The method for foreign object detection is specifically real-time detection of runway foreign objects through a Yolov5 algorithm.

7. A multi-channel visual information fusion FOD detection system characterized by, It comprises: an acquisition module configured to acquire an image to be detected, the image to be detected comprising a gray image, a laser scattering brightness image and a depth image; a laser scattering brightness image processing module configured to perform foreign object edge extraction on the laser scattering brightness image to obtain a processed laser scattering brightness image; a depth image processing module configured to perform edge extraction processing on the depth image to obtain an edge map of the depth image; a fusion module configured to fuse the gray image, the processed laser scattering brightness image and the edge map of the depth image as three color channels respectively to obtain a multi-channel fused foreign object image; in the three color channel fusion, the three color channels are G channel, B channel and R channel respectively; the gray image is taken as the G channel, the processed laser scattering brightness image is taken as the B channel, and the edge map of the depth image is taken as the R channel to perform image fusion to form a pseudo-color image; a detection module configured to perform foreign object detection on the multi-channel fused foreign object image to obtain the type and position of the foreign object.

8. A multi-channel visual information fusion FOD detection system, characterized in that, It comprises: an image acquisition device and a processor; the image acquisition device is a vehicle-mounted line scanning structured light camera for acquiring an image to be detected; the processor is configured to execute a multi-channel visual information fusion FOD detection method according to any one of claims 1-6.

9. An electronic device, comprising: A computer program product comprising a memory and a processor, and computer instructions stored on the memory and running on the processor, which, when run by the processor, complete the steps of the method of any one of claims 1-6.

Images