FOD detection system and method based on dual-channel dynamic variable structure neural network

Abstract

A FOD detection system and method based on a dual-channel dynamic variable structure neural network. The system includes a housing and an image acquisition device mounted within the housing, two image preprocessing modules, a comparator, an image feature inversion module, an image background removal module, a foreign object location module, a foreign object presence identification module, and a foreign object type identification module. Advantages and benefits of the present invention include rapid location and identification of foreign objects at airports, enabling airport personnel to quickly and accurately remove them, thereby ensuring safe flight takeoff and landing. Furthermore, the system offers excellent detection results and high accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of airport foreign object detection, and in particular relates to a FOD detection system and method based on a dual-channel dynamic variable structure neural network. Background Art

[0002] Because foreign object debris (FOD) at airports poses a threat to aircraft safety, it is crucial to quickly and accurately locate and identify its type. Traditional manual inspections are inefficient and prone to missed detections, failing to fully guarantee safe flight takeoffs and landings. Advanced FOD detection methods primarily include computer vision technologies based on infrared, millimeter-wave radar, and deep learning. In recent years, FOD detection has primarily employed deep learning methods, as neural network technology is relatively mature and widely used in fields such as image recognition and object detection. However, this approach also presents challenges. Neural networks are prone to becoming trapped in local minima during training, preventing them from achieving a global optimal solution, thus impacting network performance. Furthermore, traditional techniques are slow and time-consuming for neural networks to converge to a global optimal solution, resulting in large oscillations and low robustness to varying inputs. In the future, neural networks must be both efficient and stable. Therefore, improving the speed and probability of neural networks converging to a global optimal solution, resulting in better test performance and a more stable network structure, is of great practical significance for the future of deep learning. Summary of the Invention

[0003] In order to solve the above problems, the purpose of the present invention is to provide a FOD detection system and method based on a dual-channel dynamic variable structure neural network.

[0004] In order to achieve the above-mentioned purpose, the FOD detection system based on the dual-channel dynamic variable structure network provided by the present invention includes a shell and an image acquisition device installed inside the shell, two image preprocessing modules, a comparator, an image feature inversion module, an image background removal module, a foreign object positioning module, a foreign object presence recognition module and a foreign object type recognition module; wherein the high-definition camera on the image acquisition device is located at the through hole of the shell, and the image acquisition device is respectively connected to the two image preprocessing modules; the two image preprocessing modules are simultaneously connected to the foreign object presence recognition module through the comparator, the image feature inversion module, and the image background removal module in sequence; the foreign object presence recognition module is simultaneously connected to the foreign object positioning module and the foreign object type recognition module; the foreign object positioning module, the foreign object presence recognition module and the foreign object type recognition module are respectively connected to the airport control center in a wireless manner.

[0005] The image preprocessing module, comparator, image feature inversion module, image background removal module, foreign object positioning module, foreign object presence recognition module and foreign object type recognition module are integrated on an electronic chip, and the electronic chip adopts a Raspberry Pi embedded development board.

[0006] The detection method using the FOD detection system based on the dual-channel dynamic variable structure network includes the following steps in order:

[0007] 1) Using an image acquisition device, a number of n×n pixel-sized images of an airport without foreign objects are captured as reference images. Then, at regular intervals, real airport images of the same size are captured as images to be detected, and the images are transmitted to two image preprocessing modules respectively.

[0008] 2) The two image preprocessing modules extract the RGB value of each pixel in the reference image and the image to be detected respectively, where the RGB value of the reference image is {P′ R ,P′ G ,P′ B}, the RGB value of the image to be detected is , and then transmitted to the comparator;

[0009] 3) The comparator takes the RGB value {P′ R ,P′ G ,P′ B}Subtract the RGB value of each image to be detected , thereby eliminating the same pixels in the two types of images, obtaining several difference images with a pixel size of n×n, which are then transmitted to the image feature inversion module; if there is a foreign object, its RGB value will appear in the difference image;

[0010] 4) Use the image feature inversion module to invert the colors of all RGB values ​​in the difference image and turn the background into white. The formula used is:

[0011]

[0012] Then, the interference information and background in the inverted difference image are removed, and only the obvious pixels are saved. The image feature value with a pixel size of n×n for detection is obtained and transmitted to the foreign object presence recognition module;

[0013] 5) The foreign object presence recognition module first sets an m×m universal foreign object convolution kernel based on the characteristics of the foreign object, including its shape and size, where m<n. It then performs a convolution calculation on this convolution kernel and the image feature value to determine whether a foreign object exists. A threshold β is also set: if all the convolution calculation results are lower than the threshold β, it is determined that there is no foreign object, and the image feature value of the foreign object is wirelessly transmitted to the airport control center for detection of the next image to be detected. If the convolution calculation result exceeds the threshold β, it is determined that there is a foreign object, and the image feature value of the foreign object is transmitted to the foreign object positioning module and the foreign object type recognition module respectively.

[0014] 6) The foreign object positioning module uses the DBSCAN clustering algorithm to perform cluster calculations on the image feature value areas where foreign objects exist, thereby determining the location of the foreign object and transmitting it wirelessly to the airport control center; the foreign object type recognition module uses a dual-channel dynamic variable structure neural network to use the network to identify the image feature values ​​of the foreign object, and finally output the type of the foreign object and transmit it wirelessly to the airport control center.

[0015] In step 5), the threshold β is set as follows:

[0016] Find the maximum value G of the pixel in the original image max And the minimum pixel value G min , the average pixel of the entire image is:

[0017] G 0 =(G sum -G max -G min ) / n*n

[0018] Where G sum is the sum of all pixels in the original image; the size is m×m, and the elements are all G 0 The identity matrix of is convolved with the universal foreign object convolution kernel to obtain the threshold β.

[0019] In step 6), the dual-channel dynamic variable structure neural network includes n input neurons, channel ① connection weight ω 11 ,ω 12 ,L,ω 1m ;ω 21 ,ω 22 ,L,ω 2m ;ω n1 ,ω n2 ,L,ω nm , and channel ② connection weight ω′ 11 ,ω′ 12 ,L,ω′ 1m ;ω′ 21 ,ω′22 ,L,ω′ 2m ;LL;ω′ n1 ,ω′ n2 ,L,ω′ nm ; Connect the output of the input neuron to the hidden layer of network ① and the hidden layer of network ② respectively; The neuron outputs of the two hidden layers of the network are connected through the weight u 11 ,u 12 ,L,u 1z ;u 21 ,u 22 ,Lu 2z ;LL;u m1 ,u m2 ,L,u mz and u′ 11 ,u′ 12 ,L,u′ 1z ;u′ 21 ,u′ 22 ,Lu′ 2z LL u′ m1 ,u′ m2 ,L,u′ mz It is connected to z output neurons, and finally the output of the neural network is obtained through the activation function; at the same time, the m neurons in the hidden layer of network ① and the m neurons in the hidden layer of network ② are connected to each other through the association coefficient weight matrix A, and the association coefficient weight matrix A is a real symmetric matrix.

[0020] Advantages and beneficial effects of the present invention:

[0021] It can quickly locate and identify foreign objects at airports, helping airport staff to quickly and accurately remove them, thereby ensuring the safe takeoff and landing of flights. In addition, the detection effect is good and the accuracy is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the training steps of a dual-channel dynamic variable structure neural network.

[0023] Figure 2 It is a schematic diagram of the dual-channel dynamic variable structure network structure.

[0024] Figure 3 It is a schematic diagram of the connections between the neurons in the hidden layers of the two channel networks.

[0025] Figure 4 This is a block diagram of the FOD detection system based on dual-channel dynamic variable structure provided by the present invention.

[0026] Figure 5 This is a flow chart of the FOD detection method based on a dual-channel dynamic variable structure neural network provided by the present invention. DETAILED DESCRIPTION

[0027] The present invention will be described in further detail below with reference to the accompanying drawings.

[0028] like Figure 4 As shown, the FOD detection system based on the dual-channel dynamic variable structure network provided by the present invention includes a shell 9 and an image acquisition device 1 installed inside the shell 9, two image preprocessing modules 2, a comparator 3, an image feature inversion module 4, an image background removal module 5, a foreign object positioning module 6, a foreign object presence recognition module 7 and a foreign object type recognition module 8; wherein the high-definition camera on the image acquisition device 1 is located at the through hole of the shell 9, and the image acquisition device 1 is respectively connected to the two image preprocessing modules 2; the two image preprocessing modules 2 are simultaneously connected to the foreign object presence recognition module 7 through the comparator 3, the image feature inversion module 4, and the image background removal module 5 in sequence; the foreign object presence recognition module 7 is simultaneously connected to the foreign object positioning module 6 and the foreign object type recognition module 8; the foreign object positioning module 6, the foreign object presence recognition module 7 and the foreign object type recognition module 8 are respectively connected to the airport control center in a wireless manner.

[0029] The image preprocessing module 2, comparator 3, image feature inversion module 4, image background removal module 5, foreign object positioning module 6, foreign object presence recognition module 7 and foreign object type recognition module 8 are integrated on an electronic chip, and the electronic chip adopts an embedded development board of Raspberry Pi.

[0030] like Figure 5 As shown, the detection method using the above-mentioned FOD detection system based on the dual-channel dynamic variable structure network includes the following steps in sequence:

[0031] 1) Using the image acquisition device 1, a number of images of the airport without foreign objects with a pixel size of n×n are captured as reference images. Then, at regular intervals, real images of the airport with the same size are captured as images to be detected, and then transmitted to two image preprocessing modules 2 respectively. In the present invention, the pixel size is 300*300.

[0032] 2) The two image preprocessing modules 2 extract the RGB value of each pixel in the reference image and the image to be detected respectively, where the RGB value of the reference image is {P′ R ,P′ G ,P′ B}, the RGB value of the image to be detected is , and then transmitted to comparator 3;

[0033] 3) Comparator 3 compares the RGB value {P′ R ,P′ G ,P′B}Subtract the RGB value of each image to be detected , thereby eliminating the same pixels in the two types of images, obtaining several difference images with a pixel size of n×n, which are then transmitted to the image feature inversion module 4; if there is a foreign object, its RGB value will appear in the difference image; but the problem is that most of the pixels in the difference image are 0, indicating that the color is black;

[0034] 4) Use the image feature inversion module 4 to invert the colors of all RGB values ​​in the difference image, turning the background white so that the image information of the foreign object becomes more obvious and prominent, which is more conducive to feature extraction. The formula used is:

[0035]

[0036] Then, the interference information and background in the inverted difference image are removed, and only obvious pixels are saved, and the image feature value with a pixel size of n×n for detection is obtained and transmitted to the foreign object presence recognition module 7;

[0037] 5) The foreign object presence identification module 7 first sets an m×m universal foreign object convolution kernel based on the characteristics of the foreign object, including its shape and size, where m<n, and then performs a convolution calculation on this convolution kernel and the image feature value to determine whether a foreign object exists. Considering that not all retained pixels are foreign object pixels, a threshold β needs to be set: if all convolution calculation results are lower than the threshold β, it is determined that there is no foreign object, and the image feature value of no foreign object is wirelessly transmitted to the airport control center for detection of the next image to be detected; and when the convolution calculation result exceeds the threshold β, it is determined that there is a foreign object, and the image feature value of the foreign object is transmitted to the foreign object positioning module 6 and the foreign object type identification module 8 respectively. In the present invention, m=10; the method for setting the threshold β is as follows:

[0038] Find the maximum value G of the pixel in the original image max And the minimum pixel value G min , the average pixel of the entire image is:

[0039] G 0 =(G sum -G max -G min ) / n*n

[0040] Where G sum is the sum of all pixels in the original image; the size is m×m, and the elements are all G 0 The identity matrix of is convolved with the universal foreign object convolution kernel to obtain the threshold β.

[0041] 6) The foreign object positioning module 6 uses the DBSCAN clustering algorithm to perform cluster calculations on the image feature value areas where foreign objects exist, thereby determining the location of the foreign object and transmitting it wirelessly to the airport control center; the foreign object type identification module 8 uses a dual-channel dynamic variable structure neural network to use the network to identify the image feature values ​​where foreign objects exist, and finally outputs the type of foreign object and transmits it wirelessly to the airport control center.

[0042] The following introduces the dual-channel dynamic variable structure neural network:

[0043] like Figure 2 、 Figure 3 As shown, the dual-channel dynamic variable structure neural network includes n input neurons, channel ① connection weight ω 11 ,ω 12 ,L,ω 1m ;ω 21 ,ω 22 ,L,ω 2m ;ω n1 ,ω n2 ,L,ω nm , and channel ② connection weight ω′ 11 ,ω′ 12 ,L,ω′ 1m ;ω′ 21 ,ω′ 22 ,L,ω′ 2m ;LL;ω′ n1 ,ω′ n2 ,L,ω′ nm ; Connect the output of the input neuron to the hidden layer of network ① and the hidden layer of network ② respectively. The output of the neurons in the two hidden layers of the network are connected by weights u 11 ,u 12 ,L,u 1z ;u 21 ,u 22 ,Lu 2z ;LL;u m1 ,u m2 ,L,u mz and u′ 11 ,u′ 12 ,L,u′ 1z ;u′ 21 ,u′ 22 ,Lu′ 2z LL u′ m1 ,u′ m2 ,L,u′ mzIt is connected to z output neurons, and finally the output of the neural network is obtained through the activation function; at the same time, the m neurons in the hidden layer of network ① and the m neurons in the hidden layer of network ② are connected to each other through the association coefficient weight matrix A, and the association coefficient weight matrix A is a real symmetric matrix.

[0044] The dual-channel dynamic variable structure neural network works by alternately training one network while approximating the parameters of the other network through the correlation matrix. After repeated training, the two networks continuously approach the global optimal solution, and then continuously approach each other's optimal solution, ultimately allowing both networks to achieve the global optimal solution. Ultimately, the dual-channel dynamic neural network can achieve the global optimal solution with a higher probability.

[0045] The working principle of the foreign object type identification module 8 is as follows:

[0046] First, input the image size and determine the number of n, m, and z. Correspondingly, the number of connection weights of channel ① and channel ② is also determined. Figure 2 The connection method shown is used to build a dual-channel dynamic variable structure neural network model;

[0047] The initial values ​​of the channel connection weight and the correlation coefficient weight symmetric matrix are all random decimals in the range of (0,1); set K = e -4 ,α=0.99,η=0.001; set the maximum number of iterations Γ according to the training data; the number of batches per iteration is set to 128;

[0048] Solve the multi-output classification problem and choose the cross entropy function as the loss function E:

[0049]

[0050] Where y′ i Indicates the i-th label value, y i Represents the i-th output of this dynamic neural network after the activation function. This formula expresses the i-th output y i Express the i-th label value y′ i The difficulty of the loss function is small, so the smaller the value of the loss function is, the closer the output value is to the true value.

[0051] according to Figure 1The process shown in the figure trains this network. When the number of batches of network training is t = 1, 3, 5, K127, the connection weight of channel ② is closed, and the data enters network ① through channel ①. After one training cycle, the parameters are updated and saved. When the number of batches is t = 2, 4, 6, K, 128, the connection weight of channel ① is closed, and the data enters network ② through channel ②. After one training cycle, the parameters are updated and saved. The hidden layer neurons of the two networks are connected to the hidden layer neurons of the other network through the association coefficient weight matrix A. When the maximum number of batch training cycles is reached, the output of the i-th neuron of this dynamic neural network is y i The expression is:

[0052]

[0053] Take the connection weights ω, ω′, u, u′ as the target to be optimized and use the gradient descent method to update the optimization parameters (taking ω as an example):

[0054] ω t+1 =ω t +Δω

[0055]

[0056]

[0057]

[0058]

[0059] When all the parameters to be optimized have been updated once, that is, an iteration is completed, the correlation coefficient weight matrix has completed a decay, and the learning rate η has completed a decay. The decay expressions are:

[0060]

[0061] By continuously iteratively updating the parameters to be optimized, the loss function's value approaches the global minimum, the network's parameters are updated to the optimal solution, and the network model is saved. The network uses the feature values ​​of the image containing the foreign object as input, and the network outputs the type of foreign object and transmits this information to the airport control center, allowing the entire system to quickly locate and identify the foreign object.

Claims

1. A detection method for a FOD detection system based on a dual-channel dynamic variable structure network, the FOD detection system based on the dual-channel dynamic variable structure network comprising a housing (9) and an image acquisition device (1) installed inside the housing (9), two image pre-processing modules (2), a comparator (3), an image feature inversion module (4), an image background removal module (5), a foreign object positioning module (6), a foreign object presence recognition module (7), and a foreign object type recognition module (8); wherein a high-definition camera on the image acquisition device (1) is located at a through hole of the housing (9), and The image acquisition device (1) is respectively connected to two image pre-processing modules (2); the two image pre-processing modules (2) are simultaneously connected to a foreign object presence recognition module (7) through a comparator (3), an image feature inversion module (4), and an image background removal module (5); the foreign object presence recognition module (7) is simultaneously connected to a foreign object positioning module (6) and a foreign object type recognition module (8); the foreign object positioning module (6), the foreign object presence recognition module (7), and the foreign object type recognition module (8) are respectively connected to an airport control center in a wireless manner; the invention is characterized in that: The detection method includes the following steps in sequence: 1) Use an image acquisition device (1) to capture several airport images without foreign objects with a pixel size of n×n as reference images, and then capture airport real images of the same size at regular intervals as the images to be detected, and then transmit them to two image preprocessing modules (2) respectively; 2) The two image preprocessing modules (2) extract the RGB value of each pixel in the reference image and the image to be detected, respectively. The RGB value of the reference image is {P' R ,P' G P' B }, the RGB value of the image to be detected is Then it is transmitted to the comparator (3); 3) Comparator (3) converts the RGB value {P' R ,P' G P' B }Subtract the RGB value of each image to be detected The same pixels in the two types of images are thus eliminated, and a number of difference images with a pixel size of n×n are obtained, which are then transmitted to the image feature inversion module (4); if there is a foreign object, its RGB value will appear in the difference image; 4) Use an image feature inversion module (4) to perform color inversion on all RGB values in the difference image, turning the background white. The formula used is: Then, remove the interference information and background in the inverted difference image, only save the obvious pixel points, obtain the image feature values with a pixel size of n×n for detection, and transmit them to the foreign object presence recognition module (7); 5) The foreign object presence recognition module (7) first sets a general foreign object convolution kernel of m×m according to the characteristics including the shape and size of the foreign object, where m < n, and then performs convolution calculation on this convolution kernel and the image feature values to determine whether there is a foreign object; And set a threshold β: If all the convolution calculation results are lower than the threshold β, it is determined that there is no foreign object, and the image feature values without foreign objects are transmitted to the airport control center wirelessly to detect the next image to be detected; when the convolution calculation result exceeds the threshold β, it is determined that there is a foreign object, and the image feature values with foreign objects are transmitted to the foreign object positioning module (6) and the foreign object type recognition module (8) respectively; 6) The foreign object positioning module (6) uses the DBSCAN clustering algorithm to perform clustering calculation on the area of the image feature values with foreign objects, thereby determining the position of the foreign object and transmitting it to the airport control center wirelessly; the foreign object type recognition module (8) uses a two-channel dynamic variable structure neural network to identify the image feature values with foreign objects, and finally outputs the type of the foreign object and transmits it to the airport control center wirelessly; In step 6), the dual-channel dynamic variable structure neural network includes n input neurons, channel ① connection weight ω 11 ,ω 12 ,…,ω 1m ;ω 21 ,ω 22 ,…,ω 2m ;……;ω n1 ,ω n2 ,…,ω nm , and channel ② connection weight ω' 11 ,ω' 12 ,…,ω' 1m ;ω' 21 ,ω' 22 ,…,ω' 2m ;……;ω' n1 ,ω' n2 ,…,ω' nm The output of the input neuron is connected to the hidden layer of network ① and the hidden layer of network ② respectively; the neuron outputs of the two hidden layers of the network are connected through the weight u 11 ,u 12 ,…,u 1z ;u 21 ,u 22 ,…,u 2z ;……;u m1 ,u m2 ,…u mz and u' 11 ,u' 12 ,…,u' 1z ;u' 21 ,u' 22 ,…u' 2z ;……;u' m1 ,u' m2 ,…,u' mz It is connected to z output neurons, and finally the output of the neural network is obtained through the activation function; at the same time, the m neurons in the hidden layer of network ① and the m neurons in the hidden layer of network ② are connected to each other through the association coefficient weight matrix A, and the association coefficient weight matrix A is a real symmetric matrix.

2. The detection method of the FOD detection system based on the dual-channel dynamic variable structure network according to claim 1 is characterized in that: The image preprocessing module (2), comparator (3), image feature inversion module (4), image background removal module (5), foreign object positioning module (6), foreign object presence recognition module (7) and foreign object type recognition module (8) are integrated on an electronic chip, and the electronic chip uses an embedded development board of Raspberry Pi.

3. The detection method of the FOD detection system based on the dual-channel dynamic variable structure network according to claim 1 is characterized in that: In step 5), the setting method of the threshold β is as follows: Find the maximum value G of the pixel in the original image max And the minimum pixel value G min , the average pixel of the entire image is: G 0＝ (G sum -G max -G min ) / n*n Where G sum is the sum of all pixels in the original image; the size is m×m, and the elements are all G 0 The identity matrix of is convolved with the universal foreign object convolution kernel to obtain the threshold β.

Images