Airfield pavement damage detection method and system

Abstract

The invention provides an airport pavement damage detection method and system, and relates to the technical field of computer vision. The airport pavement monitoring equipment is mounted on the airport inspection vehicle for inspecting the airport to be detected, so that the 2D pavement image and the 3D pavement point cloud data collected by the airport pavement monitoring equipment for the airport to be detected under the driving of the airport inspection vehicle are acquired in real time; extracting a target 2D image and target 3D point cloud data related to a suspected pavement damage area to perform joint low-rank sparse decomposition fusion processing to obtain a target fusion image corresponding to a multi-modal monitoring error correction requirement, and then calling a pavement damage detection model to perform pavement damage identification on the target fusion image. And actual pavement damage condition information corresponding to the suspected pavement damage area is obtained, so that a high-accuracy airport pavement damage condition automatic metering function is quickly realized for the whole to-be-detected airport under the cooperation of the airport inspection vehicle.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more particularly, to a method and system for detecting airport pavement damage. Background Art

[0002] With the rapid development of the air transportation industry, airport pavements, as key infrastructure for aircraft takeoff, landing, taxiing, maintenance, and parking, are of crucial importance in terms of safety and reliability. However, under the long-term erosion of natural factors such as wind, rain, and temperature changes, as well as the frequent impact of aircraft takeoffs and landings, airport pavements are prone to various structural damages such as cracks, potholes, and spalling. If these structural damages are not detected and addressed in a timely manner, they may gradually expand and ultimately affect the bearing capacity of the pavement, or even lead to serious safety accidents. Therefore, regular inspection and maintenance of airport pavements are important steps to ensure flight safety.

[0003] Currently, the damage detection operation for airport pavements mainly relies on manual inspections and experience-based judgments. Inspectors need to use traditional methods such as visual observation and knocking to listen for sounds, and rely on their own experience to manually judge the damage situation of airport pavements. However, it should be noted that this overall pavement damage detection scheme has low efficiency in manual inspections, making it difficult to quickly achieve the detection effect of the entire airport's pavement damage. At the same time, the accuracy and reliability of the corresponding detection results highly depend on the technical level and subjective judgment of the operators, and it is extremely easy to have misjudgments or missed judgments. Often, it is also impossible to accurately measure the specific damage conditions (including information such as damage type, damage size, and damage depth), and thus unable to provide effective data support for subsequent airport pavement maintenance / repair operations. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method and system for detecting airport pavement damage, which can obtain multi-modal pavement monitoring data in real time during the vehicle inspection process of the airport to be detected for multi-modal fusion damage recognition, so as to quickly achieve the automatic measurement function of the damage condition of the airport pavement with high accuracy for the entire airport to be detected, reduce the manual participation in the airport pavement damage detection process, and provide effective data support for subsequent airport pavement maintenance / repair operations.

[0005] To achieve the above purpose, the technical solutions adopted in the embodiments of this application are as follows: In the first aspect, this application provides a method for detecting airport pavement damage, which is applied to a data processing device included in an airport pavement damage detection system. The detection system further includes an airport pavement monitoring device installed on an airport inspection vehicle. The detection method includes: Obtaining in real time the 2D pavement images and 3D pavement point cloud data collected by the airport pavement monitoring device under the drive of the airport inspection vehicle for the airport to be detected; Extract the target 2D image and the target 3D point cloud data corresponding to the suspected pavement damage area from the acquired 2D pavement image and 3D pavement point cloud data; Call the multi-modal low-rank representation fusion model to perform joint low-rank sparse decomposition fusion processing on the target 2D image and the target 3D point cloud data, and obtain a target fusion image that meets the requirements of multi-modal monitoring error correction; Call the pre-trained pavement damage detection model to identify pavement damage in the target fusion image, and obtain the actual pavement damage condition information of the airport to be detected at the corresponding suspected pavement damage area.

[0006] In an alternative embodiment, the airport pavement monitoring device includes a 3D structured light camera and a 2D camera for monitoring the same airport pavement area, where the 3D structured light camera is used to implement the 3D laser point cloud acquisition function, the 2D camera is used to implement the 2D image acquisition function, and the 2D pavement image and the 3D pavement point cloud data belong to the same airport pavement area; at this time, the step of extracting the target 2D image and the target 3D point cloud data corresponding to the suspected pavement damage area from the acquired 2D pavement image and 3D pavement point cloud data includes: Call the pavement damage detection model to identify pavement damage in the 2D pavement image to determine whether there are pavement damage features in the 2D pavement image; In the case where it is determined that the 2D pavement image has pavement damage features, intercept the smallest rectangular image covering all pavement damage features from the 2D pavement image, and use the intercepted smallest rectangular image as the target 2D image corresponding to the suspected pavement damage area; According to the camera coordinate system conversion relationship between the 2D camera and the 3D structured light camera, perform three-dimensional space mapping based on the actual distribution position of the target 2D image in the 2D pavement image, and obtain the target three-dimensional point cloud space range adapted to the target 2D image in the camera coordinate system of the 3D structured light camera; Use the laser point cloud data within the target three-dimensional point cloud space range in the 3D pavement point cloud data as the target 3D point cloud data corresponding to the suspected pavement damage area.

[0007] In an alternative embodiment, the multi-modal low-rank representation fusion model includes a 2D image low-rank sparse decomposition network, a 3D point cloud low-rank sparse decomposition network, four correlation filters, and two feature-level cascade fusers. Then, the step of calling the multi-modal low-rank representation fusion model to perform joint low-rank sparse decomposition fusion processing on the target 2D image and the target 3D point cloud data, and obtaining a target fusion image that meets the requirements of multi-modal monitoring error correction includes: Call the 2D image low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the target 2D image, obtain the first low-rank component and the first sparse component of the target 2D image, and call the 3D point cloud low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the target 3D point cloud data, obtain the second low-rank component and the second sparse component of the target 3D point cloud data; Perform damage feature correlation filtering on the first low-rank component, the first sparse component, the second low-rank component, and the second sparse component respectively through a correlation filter to obtain corresponding first low-rank image features, first sparse image features, second low-rank point cloud features, and second sparse point cloud features; Perform feature cascade fusion on the first low-rank image features and the second low-rank point cloud features through a feature cascade fusion device to obtain corresponding target low-rank fusion features, and perform feature cascade fusion on the first sparse image features and the second sparse point cloud features to obtain corresponding target sparse fusion features; Perform feature superposition fusion on the target low-rank fusion features and the target sparse fusion features to obtain the target fusion image.

[0008] In an alternative embodiment, each low-rank sparse decomposition network in the 2D image low-rank sparse decomposition network and the 3D point cloud low-rank sparse decomposition network includes an initial sparse coding block, a coding data decomposition block, and a plurality of sparse representation convolutional coding blocks connected in cascade. Then, the step of calling the low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the received actual input data to obtain the actual low-rank component and the actual sparse component of the actual input data includes: Perform dictionary splicing on the low-rank dictionary and the sparse dictionary related to pavement damage to obtain a corresponding target coding dictionary, and call the initial sparse coding block to perform sparse representation coding on the actual input data based on the target coding dictionary to obtain corresponding initial coding data; Use the initial coding data as the reference coding data input to the first convolutional coding block in the plurality of sparse representation convolutional coding blocks, and enable each sparse representation convolutional coding block to perform sparse representation convolutional coding based on the actual input data and the input reference coding data to obtain the target sparse representation coding data output by the last convolutional coding block in the plurality of sparse representation convolutional coding blocks, where the sparse representation coding data output by the previous sparse representation convolutional coding block in two adjacent sparse representation convolutional coding blocks is used as the reference coding data input to the subsequent sparse representation convolutional coding block; Call the coding data decomposition block to perform coding coefficient decomposition on the target sparse representation coding data to obtain a corresponding low-rank coefficient matrix and a sparse coefficient matrix; Perform matrix multiplication on the low-rank coefficient matrix and the low-rank dictionary to obtain the actual low-rank component of the actual input data, and perform matrix multiplication on the sparse coefficient matrix and the sparse dictionary to obtain the actual sparse component of the actual input data.

[0009] In an alternative embodiment, the initial encoded data of the actual input data is calculated using the following equation: ; The sparse representation encoded data output by the -th sparse representation convolutional encoding block among the multiple sparse representation convolutional encoding blocks is calculated using the following equation: ; where is used to represent the actual input data, is used to represent the initial encoded data of the actual input data, is used to represent the activation function of the corresponding low-rank sparse decomposition network, is used to represent the target encoding dictionary, is used to represent the first hyperparameter of the corresponding low-rank sparse decomposition network, is used to represent the -th sparse representation encoded data output by the sparse representation convolutional encoding block, is used to represent the first convolutional layer parameter of the -th sparse representation convolutional encoding block, is used to represent the second convolutional layer parameter of the -th sparse representation convolutional encoding block, is used to represent the convolution operator, is used to represent the reference encoded data input to the -th sparse representation convolutional encoding block, is used to represent the second hyperparameter of the corresponding low-rank sparse decomposition network.

[0010] In an alternative embodiment, the step of calling the pre-trained runway damage detection model to perform runway damage identification on the target fusion image to obtain the actual runway damage condition information of the airport to be detected at the corresponding suspected runway damage area includes: Call the pre-trained navigation aid light interference suppression network to remove the navigation aid light interference from the target fusion image to obtain an effective runway image of the corresponding suspected runway damage area; Call the runway damage detection model to perform runway damage identification on the effective runway image to obtain the actual runway damage condition information of the corresponding suspected runway damage area.

[0011] In an alternative embodiment, the detection method further includes: Obtain the actual inspection positions of all suspected pavement damage areas in the airport to be inspected during the inspection process of the airport inspection vehicle; Sort out the inspection reports for the actual inspection positions and actual pavement damage condition information of all the suspected pavement damage areas respectively, and obtain the pavement damage inspection report of the airport to be inspected.

[0012] In an alternative embodiment, the detection method further includes: Obtain a pavement area training sample set, where the pavement area training sample set includes multiple pavement area appearance samples, and each pavement area appearance sample includes a 2D image and 3D laser point cloud data of the corresponding pavement area; Based on the pavement area training sample set, train a low-rank representation fusion model with the aim of minimizing a low-rank representation fusion loss function adapted to the multi-modal monitoring error correction requirement, and obtain the multi-modal low-rank representation fusion model.

[0013] In an alternative embodiment, the low-rank representation fusion loss function adapted to the multi-modal monitoring error correction requirement is represented by the following equation: ; where, is used to represent the low-rank representation fusion loss function, is used to represent the 2D image monitoring correction factor associated with the multi-modal monitoring error correction requirement, is used to represent the 3D point cloud monitoring correction factor associated with the multi-modal monitoring error correction requirement, is used to represent the pixel matrix of the input 2D image of the corresponding multi-modal low-rank representation fusion model, is used to represent the voxel matrix of the input 3D point cloud data of the corresponding multi-modal low-rank representation fusion model, is used to represent the pixel matrix of the output fusion image of the corresponding multi-modal low-rank representation fusion model, is used to represent the Frobenius norm square operator, is used to represent the pixel-level image loss, is used to represent the shallow feature loss detected by a loss-aware network pre-trained based on the VGG-16 network architecture, is used to represent the middle feature loss detected by the loss-aware network, is used to represent the deep feature loss detected by the loss-aware network, is used to represent the shallow features extracted by the first convolutional block of the loss-aware network, is used to represent the deep features extracted by the last convolutional block of the loss-aware network, The total number of cascaded convolutional blocks used to represent the loss perception network, Used to represent the mid-level features extracted by the th convolutional block of the loss perception network, where is a positive integer greater than 2 and less than the total number of cascaded convolutional blocks, Used to represent the 3D point cloud feature loss weight, Used to represent the 2D image feature loss weight, , , and are all used to represent the loss impact weight coefficients.

[0014] In a second aspect, the present application provides an airport pavement damage detection system. The detection system includes a data processing device and an airport pavement monitoring device. The airport pavement monitoring device is installed on an airport inspection vehicle, and the airport inspection vehicle drives the airport pavement monitoring device to conduct inspections within the airport to be detected. The airport pavement monitoring device includes a 3D structured light camera and a 2D camera for monitoring the same airport pavement area. The 3D structured light camera is used to implement the 3D laser point cloud acquisition function, and the 2D camera is used to implement the 2D image acquisition function; The data processing device is communicatively connected to the airport pavement monitoring device and can run a pre-stored computer program to implement the airport pavement damage detection method described in any one of the foregoing embodiments in cooperation with the airport pavement monitoring device.

[0015] In this case, the beneficial effects of the embodiments of the present application may include the following: In this application, the airport pavement monitoring device is installed on the airport inspection vehicle used for inspecting the airport to be detected, so as to obtain in real time the 2D pavement images and 3D pavement point cloud data collected by the airport pavement monitoring device driven by the airport inspection vehicle for the airport to be detected, and extract the target 2D images and target 3D point cloud data related to the suspected pavement damage area from the obtained 2D pavement images and 3D pavement point cloud data. Then, the multi-modal low-rank representation fusion model is called to perform joint low-rank sparse decomposition fusion processing on the target 2D images and target 3D point cloud data to obtain the target fusion image corresponding to the requirements of multi-modal monitoring error correction (that is, eliminating the observation error between the multi-modal pavement monitoring data in the same airport pavement area). Then, the pavement damage detection model is called to identify the pavement damage in the target fusion image, so as to obtain the actual pavement damage condition information at the corresponding suspected pavement damage area (including information such as the true damage type, true damage number, true damage size, true damage depth, and true damage position relationship of the corresponding suspected pavement damage area), so as to quickly realize the automatic measurement function of the airport pavement damage condition with high accuracy under the cooperation of the airport inspection vehicle, reduce the manual participation in the airport pavement damage detection process, and provide effective data support for subsequent airport pavement maintenance / repair operations.

[0016] In order to make the above objects, features, and advantages of the present application more obvious and understandable, the following specifically gives preferred embodiments and, in conjunction with the accompanying drawings, the detailed description is as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0018] Figure 1 It is a schematic diagram of the system composition of the airport pavement damage detection system provided by the embodiment of the present application; Figure 2 It is one of the schematic flowcharts of the airport pavement damage detection method provided by the embodiment of the present application; Figure 3 For Figure 2 the schematic flowchart of the sub-steps included in step S230 in Figure 4 It is a schematic diagram of the model architecture of the multi-modal low-rank representation fusion model provided by the embodiment of the present application; Figure 5 It is a schematic diagram of the network composition of a single low-rank sparse decomposition network provided by the embodiment of the present application; Figure 6The second flowchart diagram of the airport pavement damage detection method provided by the embodiments of the present application; Figure 7 The third flowchart diagram of the airport pavement damage detection method provided by the embodiments of the present application.

[0019] Icon: 10 - Airport pavement damage detection system; 11 - Data processing device; 12 - Airport pavement monitoring device; 121 - 2D camera; 122 - 3D structured light camera. Detailed implementation manners

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Components of the embodiments of the present application generally described and illustrated in the accompanying drawings herein can be arranged and designed in a variety of different configurations.

[0021] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application claimed, but merely represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application.

[0022] It should be noted that like reference numerals and letters denote like items in the following drawings. Therefore, once an item is defined in one drawing, it does not require further definition and explanation in subsequent drawings.

[0023] In the description of the present application, it should be understood that the orientation or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the orientation or positional relationships shown in the drawings, or the orientation or positional relationships in which the products of this application are customarily placed during use, or the orientation or positional relationships commonly understood by those skilled in the art. These are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limiting the present application.

[0024] In the description of the present application, it should also be noted that, unless otherwise clearly defined and limited, the terms "arranged", "installed", "connected", and "coupled" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0025] In addition, in the description of the present application, it can be understood that the relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or device including the said element. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0026] The following will describe in detail some embodiments of the present application with reference to the accompanying drawings. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0027] Please refer to Figure 1 , Figure 1It is a schematic diagram of the system composition of the airport pavement damage detection system 10 provided by the embodiments of the present application. In the embodiments of the present application, the airport pavement damage detection system 10 may include a data processing device 11 and an airport pavement monitoring device 12. The airport pavement monitoring device 12 is installed on an airport patrol vehicle for inspecting the airport to be detected, and the airport pavement monitoring device 12 is driven by the airport patrol vehicle to perform inspections within the airport to be detected. The airport pavement monitoring device 12 includes a 3D structured light camera 122 and a 2D camera 121 for monitoring the same airport pavement area. The 3D structured light camera 122 is used to implement the 3D laser point cloud acquisition function, and the 2D camera 121 is used to implement the 2D image acquisition function. Among them, the relative pose relationship between the 2D camera 121 and the 3D structured light camera 122 on the airport patrol vehicle is fixed; the 3D structured light camera 122 can scan by obliquely emitting multiple laser beams towards the airport pavement and receive the reflected light signals to construct laser point cloud data, obtaining 3D laser point cloud data of the corresponding laser scanning area.

[0028] In this embodiment, the data processing device 11 is respectively communicatively connected to the 2D camera 121 and the 3D structured light camera 122 included in the airport pavement monitoring device 12, and is used to obtain in real time the multi-modal pavement monitoring data (including 2D pavement images and 3D pavement point cloud data) collected by the 2D camera 121 and the 3D structured light camera 122 respectively during the airport patrol process of the airport patrol vehicle for the same airport pavement area. At the same time, the data processing device 11 may pre-store a specific computer program related to the airport pavement damage detection function, and by running the specific computer program, perform high-precision multi-modal fusion damage identification based on the multi-modal pavement monitoring data obtained in real time, so as to quickly realize the automatic measurement function of the airport pavement damage condition with high accuracy for the entire airport to be detected, reduce the manual participation in the airport pavement damage detection process, and provide effective data support for subsequent airport pavement maintenance / repair operations. Among them, the data processing device 11 may be, but is not limited to: a laptop computer, a personal computer, a server, etc.; the data processing device 11 may be deployed on the airport patrol vehicle or at any position within the airport to be detected. In an implementation manner of this embodiment, if the airport patrol vehicle is an intelligent vehicle supporting the airport automatic patrol function, then the airport pavement damage detection system 10 can borrow the airport automatic patrol function of the airport patrol vehicle to automatically and quickly realize the measurement function of the airport pavement damage condition with high accuracy for the entire airport to be detected.

[0029] In addition, during the airport inspection process of the airport inspection vehicle, the data processing device 11 can use a positioning and navigation system (such as the Beidou navigation system, inertial measurement unit, etc.) to obtain vehicle driving information such as the position, speed, and driving direction of the airport inspection vehicle in the airport to be inspected in real time, so as to organically associate the measurement results of the airport pavement damage condition with the vehicle driving information, obtain a pavement damage detection report with geographical information of the damaged area, and then use wireless communication technology to transmit the corresponding pavement damage detection report to the airport runway maintenance management platform, facilitating airport maintenance personnel to uniformly manage, analyze, and make decisions on airport pavement damage data, and improving the intelligent level of airport pavement maintenance.

[0030] It can be understood that Figure 1 The shown composition diagram is only a schematic diagram of one composition of the airport pavement damage detection system 10, and the airport pavement damage detection system 10 may further include more or fewer components than those shown Figure 1 shown, or have a different configuration from that Figure 1 shown. Figure 1 Each component shown can be implemented by hardware, software, or a combination thereof.

[0031] In this application, to ensure that the data processing device 11 in the airport pavement damage detection system 10 can obtain multi-modal pavement monitoring data in real time during the vehicle inspection process of the airport to be inspected for multi-modal fusion damage recognition, so as to quickly realize the automatic measurement function of the airport pavement damage condition with high accuracy for the entire airport to be inspected, the embodiments of this application provide an airport pavement damage detection method applied to the aforementioned data processing device 11 to achieve the aforementioned purpose. The airport pavement damage detection method provided by this application will be described in detail below.

[0032] Please refer to Figure 2 , Figure 2 which is one of the flow schematic diagrams of the airport pavement damage detection method provided by the embodiments of this application. In the embodiments of this application, the airport pavement damage detection method may include step S210 to step S240.

[0033] Step S210, obtain in real time the 2D pavement images and 3D pavement point cloud data collected by the airport pavement monitoring device driven by the airport inspection vehicle for the airport to be inspected.

[0034] In this embodiment, the 2D pavement images and 3D pavement point cloud data obtained by the data processing device 11 at the same moment belong to the same airport pavement area in the airport to be inspected.

[0035] Step S220, extract the target 2D images and target 3D point cloud data corresponding to the suspected pavement damage areas from the obtained 2D pavement images and 3D pavement point cloud data.

[0036] In this embodiment, after the data processing device 11 obtains the 2D pavement image and 3D pavement point cloud data of a certain airport pavement area, it will call the pre-trained pavement damage detection model to preliminarily determine whether there is a suspected pavement damage area with pavement damage characteristics in the airport pavement area based on the 2D pavement image. Then, when it is determined that there is a suspected pavement damage area, the target 2D image corresponding to the suspected pavement damage area is intercepted from the 2D pavement image, and after removing the noise from the 3D pavement point cloud data, the target 3D point cloud data corresponding to the suspected pavement damage area is also extracted from the denoised 3D pavement point cloud data. Among them, the pavement damage detection model can be trained based on a convolutional neural network using a large number of pavement image data marked with different types of pavement damage to ensure that the corresponding pavement damage detection model can have an obvious function of identifying pavement damage characteristics.

[0037] On this basis, step S220 may include sub-steps A to D to accurately extract the effective multi-modal pavement data that may truly involve pavement damage from the real-time multi-modal pavement monitoring data. At this time, the sub-steps A to D are as follows: Sub-step A: Call the pavement damage detection model to identify pavement damage in the 2D pavement image to determine whether there are pavement damage characteristics in the 2D pavement image.

[0038] Sub-step B: When it is determined that the 2D pavement image has pavement damage characteristics, intercept the smallest rectangular image corresponding to covering all pavement damage characteristics from the 2D pavement image, and use the intercepted smallest rectangular image as the target 2D image corresponding to the suspected pavement damage area.

[0039] Sub-step C: According to the camera coordinate system conversion relationship between the 2D camera 121 and the 3D structured light camera 122, perform three-dimensional space mapping based on the actual distribution position of the target 2D image in the 2D pavement image to obtain the target three-dimensional point cloud space range adapted to the target 2D image in the camera coordinate system of the 3D structured light camera 122.

[0040] Sub-step D: Use the laser point cloud data within the target three-dimensional point cloud space range in the 3D pavement point cloud data as the target 3D point cloud data corresponding to the suspected pavement damage area, where the 3D pavement point cloud data and the 2D pavement image belong to the same airport pavement area.

[0041] Thus, the present application can accurately extract the effective multi-modal pavement data that may truly involve pavement damage from the real-time multi-modal pavement monitoring data by executing the above sub-steps A to D.

[0042] In step S230, a multi-modal low-rank representation fusion model is called to perform joint low-rank sparse decomposition fusion processing on the target 2D image and the target 3D point cloud data, so as to obtain a target fusion image that meets the requirements of multi-modal monitoring error correction.

[0043] In this embodiment, the multi-modal monitoring error correction requirements are used to represent the elimination of the observation errors between the multi-modal pavement monitoring data in the same airport pavement area, so that the final fusion image obtained by processing with the multi-modal low-rank representation fusion model can as realistically reflect the actual pavement details of the corresponding airport pavement area. Among them, the multi-modal monitoring error correction requirements can be effectively reflected in the model training process of the multi-modal low-rank representation fusion model. By minimizing the low-rank representation fusion loss function of the multi-modal low-rank representation fusion model, it is ensured that the finally trained multi-modal low-rank representation fusion model itself has the function of multi-modal monitoring error correction. Among them, the low-rank representation fusion loss function adapted to the multi-modal monitoring error correction requirements can be used to comprehensively characterize the feature difference conditions of the multi-modal pavement monitoring data and the final fusion image at different dimensional levels, and it is represented by the following equation: ; Where is used to represent the low-rank representation fusion loss function, is used to represent the 2D image monitoring correction factor associated with the multi-modal monitoring error correction requirements, is used to represent the 3D point cloud monitoring correction factor associated with the multi-modal monitoring error correction requirements, is used to represent the pixel matrix of the input 2D image of the corresponding multi-modal low-rank representation fusion model, is used to represent the voxel matrix of the input 3D point cloud data of the corresponding multi-modal low-rank representation fusion model, is used to represent the pixel matrix of the output fusion image of the corresponding multi-modal low-rank representation fusion model, is used to represent the Frobenius norm square operator, is used to represent the pixel-level image loss, is used to represent the shallow feature loss detected by the loss-aware network pre-trained based on the VGG (Visual Geometry Group)-16 network architecture, is used to represent the middle feature loss detected by the loss-aware network, is used to represent the deep feature loss detected by the loss-aware network, is used to represent the shallow features extracted by the first convolutional block of the loss-aware network, Used to represent the deep features extracted by the last convolutional block of the loss perception network, Used to represent the total number of cascaded convolutional blocks of the loss perception network, Used to represent the mid-level features extracted by the th convolutional block of the loss perception network, where is a positive integer greater than 2 and less than the total number of cascaded convolutional blocks, Used to represent the 3D point cloud feature loss weight, Used to represent the 2D image feature loss weight, 、 、 and are all used to represent the loss influence weight coefficients. Among them, all the convolutional blocks included in the loss perception network are cascaded with each other, and the total number of cascaded convolutional blocks is greater than or equal to 4.

[0044] On this basis, please refer to Figure 3 and Figure 4 , where Figure 3 is Figure 2 the schematic flow diagram of the sub-steps included in step S230 of Figure 4 is the schematic diagram of the model architecture of the multi-modal low-rank representation fusion model provided by the embodiment of the present application. In the embodiment of the present application, the multi-modal low-rank representation fusion model may include a 2D image low-rank sparse decomposition network, a 3D point cloud low-rank sparse decomposition network, four correlation filters, and two feature cascade fusion units. The step S230 may include sub-steps S231 to S234 to perform highly reliable multi-modal fusion on the target 2D image and the target 3D point cloud data related to the same suspected pavement damage area, so that the finally output target fusion image can truly reflect the actual pavement details of the corresponding suspected pavement damage area.

[0045] Sub-step S231, call the 2D image low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the target 2D image to obtain the first low-rank component and the first sparse component of the target 2D image, and call the 3D point cloud low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the target 3D point cloud data to obtain the second low-rank component and the second sparse component of the target 3D point cloud data.

[0046] In this embodiment, please refer to Figure 5, for any one of the 2D image low-rank sparse decomposition network and the 3D point cloud low-rank sparse decomposition network, the low-rank sparse decomposition network may include an initial sparse coding block, a coded data decomposition block, and a plurality of sparse representation convolutional coding blocks connected in cascade, where the initial sparse coding block may be implemented by a convolutional layer, the coded data decomposition block may be implemented by a decoder, each sparse representation convolutional coding block includes two convolutional layers (i.e., the first convolutional layer and the second convolutional layer), the parameters of the first convolutional layer of each of the plurality of sparse representation convolutional coding blocks are kept consistent, and the parameters of the second convolutional layer of each of the plurality of sparse representation convolutional coding blocks are kept consistent. At this time, for any one of the low-rank sparse decomposition networks, the step of calling the low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the actually input data received (for example, the target 2D image input to the 2D image low-rank sparse decomposition network, and the target 3D point cloud data input to the 3D point cloud low-rank sparse decomposition network), and obtaining the actual low-rank component (for example, the first low-rank component associated with the target 2D image, and the second low-rank component associated with the target 3D point cloud data) and the actual sparse component (for example, the first sparse component associated with the target 2D image, and the second sparse component associated with the target 3D point cloud data) of the actually input data may include: Perform dictionary splicing on the low-rank dictionary and the sparse dictionary related to the pavement damage to obtain the corresponding target coding dictionary, and based on the target coding dictionary, call the initial sparse coding block to perform sparse representation coding on the actually input data to obtain the corresponding initial coded data; Use the initial coded data as the reference coded data input to the first convolutional coding block in the plurality of sparse representation convolutional coding blocks, so that each sparse representation convolutional coding block performs sparse representation convolutional coding based on the actually input data and the input reference coded data to obtain the target sparse representation coded data output by the last convolutional coding block in the plurality of sparse representation convolutional coding blocks, where the sparse representation coded data output by the previous sparse representation convolutional coding block in two adjacent sparse representation convolutional coding blocks is used as the reference coded data input to the latter sparse representation convolutional coding block; Call the coded data decomposition block to perform coding coefficient decomposition on the target sparse representation coded data to obtain the corresponding low-rank coefficient matrix and sparse coefficient matrix; Perform matrix multiplication on the low-rank coefficient matrix and the low-rank dictionary to obtain the actual low-rank component of the actually input data, and perform matrix multiplication on the sparse coefficient matrix and the sparse dictionary to obtain the actual sparse component of the actually input data.

[0047] In this process, the low-rank dictionary and the sparse dictionary related to the 2D image low-rank sparse decomposition network are trained using a large amount of pavement image data labeled with different types of pavement damage; the low-rank dictionary and the sparse dictionary related to the 3D point cloud low-rank sparse decomposition network are trained using a large amount of pavement three-dimensional point cloud data labeled with different types of pavement damage. The initial encoded data obtained by encoding the actual input data by the initial sparse coding block in a single low-rank sparse decomposition network is calculated using the following formula: ; where, is used to represent the actual input data, is used to represent the initial encoded data of the actual input data, is used to represent the activation function of the corresponding low-rank sparse decomposition network, is used to represent the target encoding dictionary, is used to represent the first hyperparameter of the corresponding low-rank sparse decomposition network.

[0048] And the sparse representation encoded data output by the th sparse representation convolutional coding block in a single low-rank sparse decomposition network is calculated using the following formula: ; where, is used to represent the actual input data, is used to represent the activation function of the corresponding low-rank sparse decomposition network, is used to represent the sparse representation encoded data output by the th sparse representation convolutional coding block, is used to represent the parameters of the first convolutional layer of the th sparse representation convolutional coding block, is used to represent the parameters of the second convolutional layer of the th sparse representation convolutional coding block, is used to represent the convolutional operator, is used to represent the second hyperparameter of the corresponding low-rank sparse decomposition network; is used to represent the reference encoded data input to the th sparse representation convolutional coding block, and is also used to represent the sparse representation encoded data output by the th sparse representation convolutional coding block.

[0049] Thus, this application can decompose the multi-modal pavement monitoring data into appropriate multi-modal low-rank representation components (i.e., the first low-rank component and the second low-rank component) and multi-modal sparse representation components (i.e., the first sparse component and the second sparse component) according to the pavement damage characteristics by executing the specific step process of the above sub-step S231.

[0050] Sub-step S232: Perform damage feature correlation filtering on the first low-rank component, the first sparse component, the second low-rank component, and the second sparse component respectively through a correlation filter to obtain corresponding first low-rank image features, first sparse image features, second low-rank point cloud features, and second sparse point cloud features.

[0051] In this embodiment, any correlation filter is used to implement the correlation filtering function that focuses on pavement damage features.

[0052] Sub-step S233: Perform feature cascade fusion on the first low-rank image features and the second low-rank point cloud features through a feature cascade fusion device to obtain corresponding target low-rank fusion features, and perform feature cascade fusion on the first sparse image features and the second sparse point cloud features to obtain corresponding target sparse fusion features.

[0053] In this embodiment, the first low-rank image features and the second low-rank point cloud features are cascaded through a Concat operation and then undergo feature fusion through a convolutional layer of the feature cascade fusion device to obtain the target low-rank fusion features; the first sparse image features and the second sparse point cloud features are cascaded through a Concat operation and then undergo feature fusion through a convolutional layer of the feature cascade fusion device to obtain the target sparse fusion features.

[0054] Sub-step S234: Perform feature superposition fusion on the target low-rank fusion features and the target sparse fusion features to obtain a target fusion image.

[0055] Thus, this application can perform highly reliable multi-modal fusion on the target 2D image and the target 3D point cloud data related to the same suspected pavement damage area by executing the above sub-steps S231 to S234, so that the finally output target fusion image can truly reflect the actual pavement details of the corresponding suspected pavement damage area.

[0056] Step S240: Invoke a pre-trained pavement damage detection model to perform pavement damage recognition on the target fusion image to obtain the actual pavement damage condition information of the corresponding suspected pavement damage area of the airport to be detected.

[0057] Optionally, in an implementation manner of this embodiment, after the data processing device 11 obtains the target fusion image of a certain suspected pavement damage area, it can directly invoke the pavement damage detection model to perform pavement damage recognition on the target fusion image to obtain the actual pavement damage condition information of the suspected pavement damage area (including information such as the true damage type, true damage number, true damage size, true damage depth, and true damage position relationship of the corresponding suspected pavement damage area).

[0058] Optionally, in another implementation manner of this embodiment, the data processing device 11 pre-stores a navigation aid light interference suppression network. During the model training stage of the navigation aid light interference suppression network, a large number of airport runway images containing navigation aid light interference and interference-free airport runway images can be used as training data, and the network parameters are continuously adjusted through the backpropagation algorithm to enable it to accurately remove navigation aid light interference. Thus, to further improve the accuracy of pavement damage recognition, the navigation aid light interference suppression network can be used to remove navigation aid light interference from the obtained target fusion image, so as to obtain a clear and accurate pavement image for pavement damage recognition even in complex lighting environments (such as strong light illumination at night, weak light illumination in the early morning, etc.). At this time, step S240 may include: Invoking the pre-trained navigation aid light interference suppression network to remove navigation aid light interference from the target fusion image to obtain an effective pavement image corresponding to the suspected pavement damage area; Invoking the pavement damage detection model to perform pavement damage recognition on the effective pavement image to obtain the actual pavement damage condition information corresponding to the suspected pavement damage area.

[0059] Thus, this application can perform multi-modal fusion damage recognition by executing the above steps S210 to S240 to obtain multi-modal pavement monitoring data in real time during the vehicle inspection process of the airport to be detected, so as to quickly realize the automatic measurement function of the airport pavement damage condition with high accuracy for the entire airport to be detected, reduce the manual participation in the airport pavement damage detection process, and provide effective data support for subsequent airport pavement maintenance / repair operations.

[0060] Optionally, please refer to Figure 6 , Figure 6 which is the second flowchart of the airport pavement damage detection method provided by the embodiment of this application. In the embodiment of this application, compared with the airport pavement damage detection method shown in Figure 2 , the airport pavement damage detection method shown in Figure 6 may further include steps S250 to S260 to ensure that the generated pavement damage detection report can intuitively display the respective distribution positions and damage details of all airport pavement damages in the airport to be detected.

[0061] Step S250, obtaining the actual inspection positions of all suspected pavement damage areas in the airport to be detected during the inspection process of the airport inspection vehicle.

[0062] In this embodiment, the data processing device 11 will obtain the vehicle driving information of the airport inspection vehicle in real time during the airport inspection process of the airport inspection vehicle, so that when any suspected pavement damage area is detected through the above step S220, the vehicle position is extracted from the vehicle driving information when the airport inspection vehicle patrols to the suspected pavement damage area as the corresponding actual inspection position, so as to determine the actual inspection positions (i.e., damage area positions) of all suspected pavement damage areas in the airport to be detected along with the airport inspection operation of the airport inspection vehicle.

[0063] Step S260, organize the detection reports of the actual inspection positions and actual pavement damage condition information of all suspected pavement damage areas to obtain the pavement damage detection report of the airport to be detected.

[0064] In this embodiment, the pavement damage detection report can be presented in the form of an electronic map, and the actual pavement damage condition information of all suspected pavement damage areas is marked on the electronic map of the airport to be detected according to the corresponding actual inspection positions, so as to visually present the distribution positions and damage details of all airport pavement damages in the airport to be detected in the form of a chart display.

[0065] Thus, this application can ensure that the generated pavement damage detection report can visually display the distribution positions and damage details of all airport pavement damages in the airport to be detected by executing the above step S250 to step S260.

[0066] Optionally, please refer to Figure 7 , Figure 7 which is the third flowchart of the airport pavement damage detection method provided by the embodiment of this application. In the embodiment of this application, compared with the airport pavement damage detection methods shown in Figure 2 or Figure 6 , the airport pavement damage detection method shown in Figure 7 may further include steps S270 to S280 to ensure that the trained multi-modal low-rank representation fusion model can, on the basis of realizing the multi-modal pavement monitoring data fusion function, have a multi-modal monitoring error correction function that meets the requirements of multi-modal monitoring error correction, so that the output fusion image can as realistically reflect the actual pavement details of the corresponding airport pavement area.

[0067] Step S270, obtain a pavement area training sample set, where the pavement area training sample set includes multiple pavement area appearance samples, and each pavement area appearance sample includes a 2D image and 3D laser point cloud data of the corresponding pavement area.

[0068] Step S280: Based on the runway area training sample set, train a low-rank representation fusion model with the aim of minimizing the low-rank representation fusion loss function adapted to the multi-modal monitoring error correction requirements, and obtain the multi-modal low-rank representation fusion model.

[0069] In this embodiment, the model architecture adopted during the training process of the low-rank representation fusion model is as Figure 4 and Figure 5 shown. The low-rank representation fusion loss function adapted to the multi-modal monitoring error correction requirements can be referred to the detailed description of step S230 above, and will not be elaborated here one by one.

[0070] Thus, by executing the above step S270 to step S280, the present application can ensure that the trained multi-modal low-rank representation fusion model can, on the basis of realizing the multi-modal runway monitoring data fusion function, have a multi-modal monitoring error correction function that meets the multi-modal monitoring error correction requirements, so that the fused image output can as realistically as possible reflect the actual road surface details of the corresponding airport runway area.

[0071] In the embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of the code, and the module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0072] In addition, each functional module in various embodiments of the present application may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part. If the various functions provided by the present application are implemented in the form of software functional modules and sold or used as an independent product, they may be stored in a readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions for causing the above-mentioned data processing device 11 to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned readable storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0073] The above are only various implementation manners of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An airport pavement damage detection method, characterized in that, The data processing device applied to the airport pavement damage detection system, and the detection system further includes an airport pavement monitoring device installed on an airport inspection vehicle; the detection method includes: Obtaining in real time the 2D pavement images and 3D pavement point cloud data collected by the airport pavement monitoring device driven by the airport inspection vehicle for the airport to be detected; Extracting the target 2D image and the target 3D point cloud data corresponding to the suspected pavement damage area from the obtained 2D pavement images and 3D pavement point cloud data; Invoking a multi-modal low-rank representation fusion model to perform joint low-rank sparse decomposition fusion processing on the target 2D image and the target 3D point cloud data, and obtaining a target fusion image corresponding to the requirements of multi-modal monitoring error correction; Invoking a pre-trained pavement damage detection model to identify pavement damage in the target fusion image, and obtaining the actual pavement damage condition information of the airport to be detected at the corresponding suspected pavement damage area.

2. The detection method according to claim 1, characterized in that, The airport pavement monitoring device includes a 3D structured light camera and a 2D camera for monitoring the same airport pavement area, wherein the 3D structured light camera is used to implement the 3D laser point cloud acquisition function, and the 2D camera is used to implement the 2D image acquisition function, and the 2D pavement image and the 3D pavement point cloud data belong to the same airport pavement area; at this time, the step of extracting the target 2D image and the target 3D point cloud data corresponding to the suspected pavement damage area from the obtained 2D pavement images and 3D pavement point cloud data includes: Invoking the pavement damage detection model to identify pavement damage in the 2D pavement image to determine whether there are pavement damage features in the 2D pavement image; In the case of determining that the 2D pavement image has pavement damage features, intercepting the smallest rectangular image corresponding to covering all pavement damage features from the 2D pavement image, and taking the intercepted smallest rectangular image as the target 2D image corresponding to the suspected pavement damage area; According to the camera coordinate system conversion relationship between the 2D camera and the 3D structured light camera, performing three-dimensional space mapping based on the actual distribution position of the target 2D image in the 2D pavement image, and obtaining the target three-dimensional point cloud space range adapted to the target 2D image in the camera coordinate system of the 3D structured light camera; Taking the laser point cloud data within the target three-dimensional point cloud space range in the 3D pavement point cloud data as the target 3D point cloud data corresponding to the suspected pavement damage area.

3. The detection method according to claim 1, wherein The multi-modal low-rank representation fusion model includes a 2D image low-rank sparse decomposition network, a 3D point cloud low-rank sparse decomposition network, four correlation filters and two feature concatenation fusion devices, then the step of invoking the multi-modal low-rank representation fusion model to perform joint low-rank sparse decomposition fusion processing on the target 2D image and the target 3D point cloud data, and obtaining a target fusion image corresponding to the requirements of multi-modal monitoring error correction includes: Call the 2D image low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the target 2D image, obtain the first low-rank component and the first sparse component of the target 2D image, and call the 3D point cloud low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the target 3D point cloud data, obtain the second low-rank component and the second sparse component of the target 3D point cloud data; Perform damage feature correlation filtering on the first low-rank component, the first sparse component, the second low-rank component, and the second sparse component respectively through a correlation filter to obtain corresponding first low-rank image features, first sparse image features, second low-rank point cloud features, and second sparse point cloud features; Perform feature cascade fusion on the first low-rank image features and the second low-rank point cloud features through a feature cascade fusion device to obtain corresponding target low-rank fusion features, and perform feature cascade fusion on the first sparse image features and the second sparse point cloud features to obtain corresponding target sparse fusion features; Perform feature superposition fusion on the target low-rank fusion features and the target sparse fusion features to obtain the target fusion image.

4. The detection method according to claim 3, wherein Each low-rank sparse decomposition network in the 2D image low-rank sparse decomposition network and the 3D point cloud low-rank sparse decomposition network includes an initial sparse coding block, a coding data decomposition block, and multiple sparse representation convolutional coding blocks connected in cascade. Then, the step of calling the low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the received actual input data to obtain the actual low-rank component and the actual sparse component of the actual input data includes: Perform dictionary splicing on the low-rank dictionary and the sparse dictionary related to pavement damage to obtain a corresponding target coding dictionary, and based on the target coding dictionary, call the initial sparse coding block to perform sparse representation coding on the actual input data to obtain corresponding initial coding data; Use the initial coding data as the reference coding data input to the first convolutional coding block in the multiple sparse representation convolutional coding blocks, so that each sparse representation convolutional coding block performs sparse representation convolutional coding based on the actual input data and the input reference coding data to obtain the target sparse representation coding data output by the last convolutional coding block in the multiple sparse representation convolutional coding blocks, where the sparse representation coding data output by the previous sparse representation convolutional coding block in two adjacent sparse representation convolutional coding blocks is used as the reference coding data input to the latter sparse representation convolutional coding block; Call the coding data decomposition block to perform coding coefficient decomposition on the target sparse representation coding data to obtain corresponding low-rank coefficient matrix and sparse coefficient matrix; Perform matrix multiplication on the low-rank coefficient matrix and the low-rank dictionary to obtain the actual low-rank component of the actual input data, and perform matrix multiplication on the sparse coefficient matrix and the sparse dictionary to obtain the actual sparse component of the actual input data.

5. The detection method according to claim 4, wherein The initial coding data of the actual input data is calculated by the following formula: ； The sparse representation encoded data output by the th sparse representation convolutional coding block among the multiple sparse representation convolutional coding blocks is calculated by the following formula: ； Among them, is used to represent the actual input data, is used to represent the initial coding data of the actual input data, is used to represent the activation function corresponding to the low-rank sparse decomposition network, is used to represent the target coding dictionary, is used to represent the first hyperparameter corresponding to the low-rank sparse decomposition network, is used to represent the sparse representation coding data output by the th sparse representation convolutional coding block, is used to represent the first convolutional layer parameters of the th sparse representation convolutional coding block, is used to represent the second convolutional layer parameters of the th sparse representation convolutional coding block, is used to represent the convolutional operator, is used to represent the reference coding data input to the th sparse representation convolutional coding block, is used to represent the second hyperparameter corresponding to the low-rank sparse decomposition network.

6. The detection method according to claim 1, characterized in that The step of calling the pre-trained pavement damage detection model to identify pavement damage in the target fused image, and obtaining the actual pavement damage condition information of the airport to be detected in the corresponding suspected pavement damage area includes: Calling the pre-trained navigation aid light interference suppression network to remove the navigation aid light interference from the target fused image, and obtaining an effective pavement image of the corresponding suspected pavement damage area; Calling the pavement damage detection model to identify pavement damage in the effective pavement image, and obtaining the actual pavement damage condition information of the corresponding suspected pavement damage area.

7. The detection method according to claim 1, wherein The detection method further includes: Obtaining the actual inspection positions of all suspected pavement damage areas in the airport to be detected during the inspection process of the airport inspection vehicle; Sorting out the detection reports for the actual inspection positions and actual pavement damage condition information of all suspected pavement damage areas respectively, and obtaining the pavement damage detection report of the airport to be detected.

8. The detection method according to any one of claims 1-7, characterized in that, The detection method further includes: Obtaining a pavement area training sample set, where the pavement area training sample set includes multiple pavement area appearance samples, and each pavement area appearance sample includes a 2D image and 3D laser point cloud data of the corresponding pavement area; Based on the pavement area training sample set, training a low-rank representation fusion model with the aim of minimizing the low-rank representation fusion loss function adapted to the multi-modal monitoring error correction requirement, and obtaining the multi-modal low-rank representation fusion model.

9. The detection method according to claim 8, wherein The low-rank representation fusion loss function adapted to the multi-modal monitoring error correction requirement is represented by the following equation: ； Among them, is used to represent the low-rank representation fusion loss function, is used to represent the 2D image monitoring correction factor associated with the multi-modal monitoring error correction requirement, is used to represent the 3D point cloud monitoring correction factor associated with the multi-modal monitoring error correction requirement, is used to represent the pixel matrix of the input 2D image corresponding to the multi-modal low-rank representation fusion model, is used to represent the voxel matrix of the input 3D point cloud data corresponding to the multi-modal low-rank representation fusion model, is used to represent the pixel matrix of the output fusion image corresponding to the multi-modal low-rank representation fusion model, is used to represent the Frobenius norm square operator, is used to represent the pixel-level image loss, is used to represent the shallow feature loss detected by the loss-aware network pre-trained based on the VGG-16 network architecture, is used to represent the middle feature loss detected by the loss-aware network, is used to represent the deep feature loss detected by the loss-aware network, is used to represent the shallow features extracted by the first convolutional block of the loss-aware network, is used to represent the deep features extracted by the last convolutional block of the loss-aware network, is used to represent the total number of cascaded convolutional blocks of the loss-aware network, is used to represent the middle features extracted by the th convolutional block of the loss-aware network, is a positive integer greater than 2 but less than the total number of cascaded convolutional blocks, is used to represent the 3D point cloud feature loss weight, is used to represent the 2D image feature loss weight, 、 、 and are all used to represent the loss influence weight coefficients.

10. An airport pavement damage detection system, characterized in that, The detection system includes a data processing device and an airport pavement monitoring device. The airport pavement monitoring device is installed on the airport inspection vehicle, and the airport inspection vehicle drives the airport pavement monitoring device to conduct inspections in the airport to be detected. The airport pavement monitoring device includes a 3D structured light camera and a 2D camera for monitoring the same airport pavement area. The 3D structured light camera is used to implement the 3D laser point cloud acquisition function, and the 2D camera is used to implement the 2D image acquisition function; The data processing device is communicatively connected to the airport pavement monitoring device and can run a pre-stored computer program to implement the airport pavement damage detection method according to any one of claims 1-9 with the cooperation of the airport pavement monitoring device.

Images