Airport pavement multispectral analysis method, collection device, server and storage medium

Abstract

The application discloses an airport pavement multispectral analysis method, a collection device, a server and a storage medium, and belongs to the field of data processing, and comprises the following steps: step 1: using an RGB camera to collect RGB response data sets of different light intensities and different frequencies of light, thereby training a frequency response feature expansion neural network, and using the frequency response feature expansion neural network to prepare a multispectral data set of different material foreign matters; step 2: using the multispectral data set to train a material classification network; step 3: using an RGB camera to acquire multispectral data of a pavement and inputting the multispectral data into the material classification network, and classifying foreign matters through an algorithm and outputting. The application mainly uses the characteristics that different material objects have different characteristics under spectral conditions, detects and classifies different material objects in the field of view of an airport pavement detection device, and thus realizes classification of pavement foreign matters and detection of pollutants.

Description

Technical Field

[0001] This application belongs to the field of data processing, and specifically relates to a method for multispectral analysis of airport pavement, a server, and a storage medium. Background Technology

[0002] Foreign objects (FOD) on airport runways are crucial for safe airport operations. In addition, among the global aviation accidents reported by ICAO, there have been numerous cases of aircraft running off the runway, resulting in high rates of fatalities and aircraft damage. The indirect losses caused by aircraft grounding and delays are also particularly severe. To prevent such accidents from happening again, the country attaches great importance to the maintenance of airport runways and has revised the "Regulations on the Operation and Management of Wet and Contaminated Runways by Air Carriers".

[0003] Foreign object debris (FOD) at airports is diverse and originates from various sources. Even those with similar shapes can pose varying degrees of danger to aircraft operation safety depending on their material composition. Classifying FOD by detecting the material composition of objects on the runway facilitates the subsequent identification of FOD sources and the establishment of an airport FOD database. This lays the foundation for effective FOD management, improving both FOD detection efficiency and reducing airport operating costs.

[0004] The detection of pollutants on airport pavements facilitates the airport logistics department's subsequent ground cleaning arrangements, thereby improving the cleaning efficiency of airport pavements, laying the foundation for ensuring the safety of air traffic, benefiting the overall operation of the airport, and reducing the airport's labor costs.

[0005] Currently, airport runway surface condition detection mainly relies on radar-based detection technology, optical image-based detection technology, and multi-sensor hybrid detection technology. However, conventional radar equipment is severely affected by weather, requires high precision in manufacturing and integration, and is costly, making automatic control difficult. Conventional visible light imaging systems also struggle to achieve all-weather, all-time monitoring. Therefore, choosing high-resolution, environmentally adaptable, high-performance optical imaging systems is the future trend for FOD (Frequency of Destructive) detection. Furthermore, current airport runway detection technology primarily detects the presence of foreign objects (FOD). Due to the diverse types of foreign objects detected, effective classification is difficult, and it's challenging to distinguish between oil and rainwater contaminants, or to determine the nature of pollutants according to the "Regulations on the Operation and Management of Wet and Contaminated Runways by Air Carriers." While multispectral cameras can identify foreign object types, current multispectral cameras are expensive, bulky, and require significant transportation and maintenance costs, making continuous detection at airports difficult.

[0006] Therefore, a novel multispectral analysis method for airport pavement is needed to solve the above problems. Summary of the Invention

[0007] To address the shortcomings of the existing technology, this application provides a multispectral analysis method for airport pavement based on the frequency response feature extension of an RGB color camera. This method mainly utilizes the different characteristics of objects of different materials under spectral conditions to detect and classify objects of different materials within the field of view of the airport pavement detection device, thereby classifying foreign objects on the pavement and detecting pollutants.

[0008] The technical effect to be achieved in this application is accomplished through the following solution:

[0009] According to a first aspect of the present invention, a method for multispectral analysis of airport pavement is provided, comprising the following steps:

[0010] Step 1: Use an RGB camera to collect RGB response datasets of light at different intensities and frequencies, and use this dataset to train a frequency response feature extension neural network. Then, use this frequency response feature extension neural network to create a multispectral dataset of different materials and foreign objects.

[0011] Step 2: Train a material classification network using the multispectral dataset;

[0012] Step 3: Use an RGB camera to acquire multispectral data of the road surface and input it into the material classification network. The algorithm then classifies the foreign objects and outputs the classification results.

[0013] Preferably, in step 1, light of different intensities and frequencies is refracted onto a screen through a convex lens and a prism. Then, an RGB camera is used to collect light bands of different wavelengths dispersed on the screen and the wavelengths of the light bands are calibrated. Finally, the wavelengths of the light bands are input into a frequency response feature extended neural network for training.

[0014] Preferably, after the RGB camera acquires the RGB image, it converts the response values ​​of the R, G, and B channels of each pixel in the RGB image into multispectral response values ​​of n channels through a frequency response feature extended neural network, where n is greater than 3.

[0015] Preferably, the specific method for training the frequency response feature extended neural network is as follows: obtain the QE response curves of n channels, input the spectral band data of each light intensity and wavelength in the QE response curves as training data into the frequency response feature extended neural network, and obtain the spectral response curve by corresponding the RGB image with the spectrum of each band, and store the spectral response curve for subsequent use.

[0016] Preferably, the frequency response feature extended neural network processes RGB images using the following formula:

[0017] qn=R*an+G*bn+B*cn

[0018] The autoencoder method is used to train an, bn and cn, where n is greater than 3. The qn value of each pixel in the RGB image is calculated by fitting an, bn and cn to the network, thus obtaining the multispectral dataset corresponding to the entire RGB image.

[0019] Preferably, in step 2, a material classification network is trained using multispectral data of different solid foreign objects and different liquid foreign objects as input, wherein the different liquid foreign objects include at least oil stains and snow in different states.

[0020] According to a second aspect of the present invention, a multispectral data acquisition device for an RGB camera is provided for multispectral data acquisition in step 1. The device includes a light-shielding box in which a light source is installed. The light from the light source is projected onto a screen after being processed by a convex lens and a prism. The camera acquires light bands of different wavelengths dispersed on the screen and calibrates the wavelengths of the light bands. Then, the wavelengths of the light bands are input into a frequency response feature extended neural network for training.

[0021] Preferably, the convex lens is a Fresnel lens; a baffle is provided between the camera and the convex lens and the prism.

[0022] According to a third aspect of the present invention, a server is provided, comprising: a memory and at least one processor;

[0023] The memory stores a computer program, and the at least one processor executes the computer program stored in the memory to implement the airport pavement multispectral analysis method described above.

[0024] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed, the computer program implements the airport pavement multispectral analysis method described in any of the preceding claims.

[0025] According to an embodiment of the present invention, the beneficial effects of using the present invention for airport pavement inspection are as follows:

[0026] This invention analyzes the material properties of objects on the road surface. Conventional multispectral cameras are expensive and difficult to deploy. Therefore, this invention extends the frequency response characteristics of RGB cameras, combining the RGB three channels with different frequency spectra to acquire data, transform it into multispectral data, and then analyze the material properties of the objects.

[0027] For the detection of pollutants such as oil stains, since their characteristics in ordinary RGB images are similar to those in images of rainwater, classification cannot be achieved solely using RGB images. Therefore, this invention relies on spectral feature methods to detect pollutants such as oil stains.

[0028] Regarding the melting state of snow, since the differences in the characteristics of snow in different states are not significant enough in ordinary RGB images, and the water content is different, the characteristics shown in the spectral images at certain frequencies are different. Based on this, the present invention detects the dry and wet state of snow. Attached Figure Description

[0029] To more clearly illustrate the embodiments of this application or the existing technical solutions, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of a multispectral analysis method for airport pavement in one embodiment of this application;

[0031] Figure 2 This is a schematic diagram of the structure of a multispectral data acquisition device for an RGB camera according to one embodiment of this application;

[0032] Figure 3 This is a schematic diagram of the structure of a server according to one embodiment of this application.

[0033] Figure labels: 1. baffle; 2. light source; 3. Fresnel lens; 4. light shield; 5. prism; 6. screen; 7. light shield box; 8. camera. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] like Figure 1 As shown, an embodiment of the present invention provides a multispectral analysis method for airport pavement, comprising the following steps:

[0036] Step 1: Use an RGB camera to collect RGB response datasets of light at different intensities and frequencies, and use this dataset to train a frequency response feature extension neural network. Then, use this frequency response feature extension neural network to create a multispectral dataset of different materials and foreign objects.

[0037] In this step, light of different intensities and frequencies is refracted onto a screen through a convex lens and a prism. Under the refraction of the prism, light of different frequencies from a common light source can be dispersed onto the screen. Then, an RGB camera is used to collect light bands of different wavelengths dispersed on the screen and the wavelengths of the light bands are calibrated. Finally, the wavelengths of the light bands are input into a frequency response feature extension neural network for training.

[0038] Specifically, spectral data of different frequencies at various light intensities are collected using an RGB camera to create an RGB-to-multispectral dataset. Using RGB images as input and frequency spectra as output, a supervised neural network is trained. After training, the dataset can be used to reconstruct multi-channel multispectral images.

[0039] After acquiring an RGB image, the RGB camera converts the response values ​​of the R, G, and B channels of each pixel in the RGB image into multispectral response data with n channels through a frequency response feature extended neural network, where n is greater than 3.

[0040] Converting an RGB image to a multispectral image requires transforming the R, G, and B channels into multiple channels. The number of channels can range from several to over a dozen, such as eight, nine, or sixteen channels. This embodiment does not specifically limit the number of channels or the bands of each channel in the multispectral image. To better understand this embodiment, a nine-channel multispectral image is used as an example. It should be understood that this exemplary description should not be construed as a specific limitation of this embodiment. As a non-limiting example, the obtained image is a nine-channel multispectral image. Each pixel in a nine-channel multispectral image can obtain nine response values: x1, x2, x3, x4, x5, x6, x7, x8, and x9. That is, the RGB three-channel response values ​​of each pixel correspond to the nine response values ​​of the nine channels. Among them, x1 represents the response value of the first channel with q1 response curve characteristics; x2 represents the response value of the second channel with q2 response curve characteristics; x3 represents the response value of the third channel with q3 response curve characteristics; ...; x9 represents the response value of the third channel with q9 response curve characteristics. In other words, xi represents the response value of the i-th channel with the qi response curve characteristics, where i is an integer from 1 to 9. The multispectral image is reconstructed from the red-green-blue (RGB) image. Each pixel in the RGB image has three channel response values: the R value of the R channel, the G value of the G channel, and the B value of the B channel. Reconstructing the multispectral image from the RGB image involves calculating the nine multispectral response values ​​(x1, x2, x3, x4, x5, x6, x7, x8, x9) of each pixel based on its RGB response values.

[0041] Obtain the QE response curve matrices for the nine channels of the camera. These matrices can be denoted as q1, q2, q3, q4, q5, q6, q7, q8, and q9. Matrix q1 represents the response curve of the first channel, q2 the response curve of the second channel, and so on, up to q9 the response curve of the ninth channel. In other words, matrix qj represents the response curve of the j-th channel, where j is an integer from 1 to 9. It should be noted that for a fixed camera (or hardware), these response curves can be obtained by collecting spectral data of different wavelengths and intensities, using this data as training data in a neural network, and then mapping the RGB image to the spectra of each wavelength band to obtain the spectral response curves. After obtaining these curves, they can be pre-stored in the memory of the electronic device and retrieved when needed.

[0042] In this embodiment, the frequency response feature extended neural network processes RGB images using the following formula:

[0043] qn=R*an+G*bn+B*cn

[0044] The values ​​an, bn, and cn are obtained by training an autoencoder, where n is greater than 3. The qn value of each pixel in the RGB image is calculated by fitting an, bn, and cn to the network, thus obtaining the multispectral dataset corresponding to the entire RGB image.

[0045] Specifically, an RGB camera captures light bands of different wavelengths scattered by a prism, the wavelengths of the light bands in the image are calibrated, and the data are input into a neural network for training to obtain fitting parameters for a 9-channel RGB-to-RGB spectral image. The formula is as follows:

[0046] q1=R*a1+G*b1+B*c1

[0047] q2=R*a2+G*b2+B*c2

[0048] q3=R*a3+G*b3+B*c3

[0049] q4=R*a4+G*b4+B*c4

[0050] q5=R*a5+G*b5+B*c5

[0051] q6=R*a6+G*b6+B*c6

[0052] q7=R*a7+G*b7+B*c7

[0053] q8=R*a8+G*b8+B*c8

[0054] q9=R*a9+G*b9+B*c9

[0055] An autoencoder method is used to train a network to obtain the parameter values ​​a1, a2, a3, a4, a5, a6, a7, a8, a9; b1, b2, b3, b4, b5, b6, b7, b8, b9; c1, c2, c3, c4, c5, c6, c7, c8, c9, which are then stored in an electronic device for use in subsequent conversion modules. After network fitting and calculation, the q1, q2, q3, q4, q5, q6, q7, q8, and q9 values ​​for each pixel in the RGB image are obtained, thus deriving the multispectral image corresponding to the entire RGB image; that is, the multispectral image is reconstructed from the RGB image.

[0056] Step 2: Train a material classification network using the multispectral dataset;

[0057] In this step, a material classification network is trained using multispectral data of different solid foreign objects and different liquid foreign objects as input. The different liquid foreign objects include at least oil stains and snow in different states.

[0058] The main steps for collecting and training different solid foreign objects are: using an RGB camera to collect data on different materials and converting it into multispectral data (multispectral images) using the network from the previous step, and recording the material of the collected object as a label for training the network.

[0059] The network is trained by taking multispectral data as input and material classification results as output. A supervised training method is used to train the multispectral classification network based on frequency response feature extension network. After training, it can be put into use to classify the material of objects.

[0060] The training for collecting different liquid foreign objects mainly involves: using a camera to collect data on various oil spills and snow accumulation under different conditions, converting them into multispectral data (multispectral images) using the network in step one, and recording the type of oil spill or the state of snow melting as labels for training the network.

[0061] The network is trained by taking multispectral data as input and pollutant classification results as output. A supervised training method is used to train the multispectral classification network based on frequency response feature extension network. After training, it can be put into use to classify pollutants through the algorithm.

[0062] Step 3: Use an RGB camera to acquire multispectral data of the road surface and input it into the material classification network. The algorithm then classifies the foreign objects and outputs the classification results.

[0063] In this step, the multispectral image output from the network in the previous step is input into a pre-trained material classification network, also known as a multispectral foreign object recognition network. Based on the different spectral characteristics of objects of different materials on the road surface under different spectral conditions, the material of the object can be detected. For a multispectral image with M bands, each pixel has M bands and can be described as an M-dimensional column vector, or abstractly considered as a point in an M-dimensional feature space. In the feature space of the data, the ground objects are first segmented according to a statistical model. The PCA algorithm is used to preserve the main structure of the multispectral data of airport road objects by retaining linear combination components with large variances. The background of the multispectral image is segmented, mainly by calculating the average image of the corrected 400–960 nm spectral image. The Otsu method is used to binarize the average image. A morphological reconstruction algorithm is used to remove noise points generated by reconstructing the multispectral image from RGB. Holes in the airport road multispectral data are filled, a mask image is obtained, and it is compared and analyzed with the corrected reconstructed multispectral image. The road background of the object multispectral image is removed by performing an AND operation between the two. To eliminate the potential impact of scattering, multivariate scattering correction (MSC) was used to preprocess the spectral data before constructing a multispectral foreign object recognition network. A convolutional neural network was used, employing ten convolutional layers to acquire the response features of the object's material across multiple frequency spectra. A pooling layer was added after every two convolutional layers, using max pooling to compress and extract the main features of different materials across multiple spectra output by each convolutional layer. Finally, a stretching layer projected the acquired spectral features into a one-dimensional vector, and a fully connected layer nonlinearly combined the extracted main spectral features of the material to obtain the classification result.

[0064] While identifying foreign material, the system also identifies airport runway contaminants, such as oil stains, engine oil, rainwater, or snow. The different chemical properties of oil stains and snow in different states lead to different spectral responses, thus enabling identification. Similar to the identification of solid foreign objects, the difference lies in the training data. For snow and other contaminants, the focus is on the selection of training data bands. Here, we choose to use changes in the reflectance spectral curves in the visible and near-infrared bands to distinguish contaminants in different states.

[0065] The main application scenario of this invention is to perform qualitative analysis of the material properties of objects, without the need for quantitative analysis. Therefore, the research on the method of expanding RGB images into multispectral images has a significant positive effect on reducing the overall system cost and analyzing natural scene object classification and typical target detection tasks. It solves the problems of high price, inaccessibility, or strict external condition restrictions of general multispectral cameras; moreover, it solves the problem that existing airport pavement detection systems can only detect the presence of foreign objects on the pavement and cannot classify foreign objects according to their material properties (i.e., they cannot distinguish between solid and liquid foreign objects).

[0066] According to a second aspect of the invention, such as Figure 2 As shown, a multispectral data acquisition device for an RGB camera is provided for multispectral data acquisition in step 1. It includes a light-shielding box 7, in which a light source 2 is installed. The light from the light source 2 is projected onto a screen 6 after being processed by a convex lens and a prism 5. The camera 8 acquires light bands of different wavelengths dispersed on the screen 6 and calibrates the wavelengths of the light bands. Then, the wavelengths 5 of the light bands are input into a frequency response feature extended neural network for training.

[0067] The convex lens is a Fresnel lens 3; a baffle 1 is provided between the camera 8 and the convex lens and the prism 5 to prevent unrefractive light from being projected into the camera 8; a light shield 4 is provided between the Fresnel lens and the prism, and the light transmitted through the Fresnel lens enters the prism through the hole in the middle of the light shield 4.

[0068] According to a third aspect of the present invention, a server is provided, such as Figure 3 As shown, it includes: a memory 301 and at least one processor 302;

[0069] The memory 301 stores a computer program, and the at least one processor 302 executes the computer program stored in the memory to implement the airport pavement multispectral analysis method described above.

[0070] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed, the computer program implements the airport pavement multispectral analysis method described in any of the preceding claims.

[0071] This invention primarily involves expanding the frequency response characteristics of data acquired by a conventional RGB camera to convert it into multispectral data. Multispectral analysis techniques and principles are then used to detect pollutants and road surface materials. Multispectral imaging and analysis technologies can acquire information in both spatial image dimensions and...

[0072] We acquire information in the spectral dimension. The principle of multispectral imaging is to split incident light into several narrow bands of light, which are then imaged onto different channels of the image, thus obtaining images of different spectral bands. Here, we input ordinary RGB images into a frequency response feature extension network to extend the frequency response features of a regular camera, forming multispectral three-dimensional data. We then extract material composition and color-related features from the spectral dimension. Finally, we fuse the image dimension information and the spectral dimension information to analyze the target object.

[0073] It should be noted that the above detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0074] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application. As used herein, unless the context clearly indicates otherwise.

[0075] It should be noted that, otherwise the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0076] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0077] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0078] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatial relative descriptions used herein will be interpreted accordingly.

[0079] In the detailed description above, reference has been made to the accompanying drawings, which form part of this document. In the drawings, similar symbols typically identify similar parts unless the context otherwise indicates otherwise. The illustrated embodiments described in the detailed specification, drawings, and claims are not intended to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the subject matter presented herein.

[0080] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multispectral analysis method for airport pavement, characterized in that, Includes the following steps: Step 1: Use an RGB camera multispectral data acquisition device to collect spectral data of different light intensities and frequencies from different foreign objects, and create multispectral datasets of foreign objects of different materials. First, construct a response dataset of the RGB camera multispectral data acquisition device to light of different frequencies, then train a frequency response feature extended neural network, and finally construct a multispectral dataset of foreign objects of different materials. After the RGB camera multispectral data acquisition device acquires RGB images, it converts the response values ​​of the R, G, and B channels of each pixel in the RGB image into multispectral response values ​​of n channels through a frequency response feature extended neural network, where n is greater than 3. Several multispectral response values ​​constitute a QE response curve. The specific method for training the frequency response feature extension neural network is as follows: Obtain the QE response curves of n channels; input the spectral band data of different wavelengths at various light intensities in the QE response curves as training data into the frequency response feature extension neural network; by mapping the RGB image to the spectrum of each band, the spectral response curve can be obtained; store the spectral response curve for later retrieval; the frequency response feature extension neural network processes RGB images using the following formula: qn=R*an+G*bn+B*cn The autoencoder method is used to train an, bn and cn, where n is greater than 3. The qn value of each pixel in the RGB image is calculated by fitting an, bn and cn to the network, and the multispectral dataset corresponding to the entire RGB image is obtained. Step 2: Train a material classification network using the multispectral dataset; Step 3: Use the RGB camera multispectral data acquisition device to acquire multispectral data of the road surface and input it into the material classification network. Use the algorithm to classify foreign objects and output the classification results. The RGB camera multispectral data acquisition device includes a light-shielding box, in which a light source, a convex lens, a prism, a screen, and a camera are installed. The light from the light source is projected onto the screen after being processed by the convex lens and the prism. The camera collects light bands of different wavelengths dispersed on the screen and calibrates the wavelengths of the light bands. Then, the wavelengths of the light bands are input into the frequency response feature extended neural network for training. A baffle is provided between the camera and the convex lens and the prism to prevent unrefractive light from being projected into the camera; the convex lens is a Fresnel lens; a light shield is provided between the Fresnel lens and the prism, and the light transmitted through the Fresnel lens enters the prism through the hole in the middle of the light shield.

2. The airport pavement multispectral analysis method according to claim 1, characterized in that, In step 2, a material classification network is trained using multispectral data of different solid foreign objects and different liquid foreign objects as input. The different liquid foreign objects include at least oil stains and snow in different states.

3. A server, characterized in that, include: Memory and at least one processor; The memory stores a computer program, and the at least one processor executes the computer program stored in the memory to implement the airport pavement multispectral analysis method according to claim 1 or 2.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the airport pavement multispectral analysis method according to claim 1 or 2.

Images