Airport pavement damage detection method and system

Abstract

The present application provides an airport pavement damage detection method and system, which relates to the field of computer vision technology. The present application installs an airport pavement monitoring device on an airport inspection vehicle used to inspect an airport to be inspected, so as to obtain in real time the 2D pavement images and 3D pavement point cloud data collected by the airport pavement monitoring device at the airport to be inspected under the guidance of the airport inspection vehicle, and extracts the target 2D image and target 3D point cloud data of the suspected pavement damage area for joint low-rank sparse decomposition and fusion processing to obtain a target fused image that meets the requirements of multimodal monitoring error correction, and then calls a pavement damage detection model to perform pavement damage identification on the target fused image to obtain the actual pavement damage status information corresponding to the suspected pavement damage area, thereby quickly realizing a high-accuracy automatic measurement function of the airport pavement damage status for the entire airport to be inspected with the cooperation of the airport inspection vehicle.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a method and system for detecting damage to an airport pavement. Background Art

[0002] With the rapid development of the aviation industry, airport pavements, as critical infrastructure for aircraft takeoff, landing, taxiing, maintenance, and parking, are crucial for their safety and reliability. However, under the long-term erosion of natural factors such as wind, rain, and temperature fluctuations, as well as the impact of frequent aircraft takeoffs and landings, airport pavements are prone to various structural damages such as cracks, pits, and spalling. If these structural damages are not discovered and addressed promptly, they may gradually expand and ultimately affect the pavement's load-bearing capacity, even leading to serious safety accidents. Therefore, regular inspection and maintenance of airport pavements are crucial to ensuring flight safety.

[0003] Currently, damage detection for airport pavements relies primarily on manual inspections and empirical judgment. Inspectors rely on their own experience to manually determine the damage to the pavement using traditional methods such as visual observation, tapping, and listening. However, this approach to pavement damage detection suffers from low manual inspection efficiency, making it difficult to quickly detect damage across the entire airport. Furthermore, the accuracy and reliability of the test results are highly dependent on the operator's technical skills and subjective judgment, making misjudgments and missed detections highly likely. Furthermore, it often fails to accurately measure the specific damage condition (including damage type, size, depth, and other information), hindering the ability to provide effective data support for subsequent pavement maintenance and repair operations. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an airport pavement damage detection method and system, which can obtain multimodal pavement monitoring data in real time during the vehicle inspection process of the airport to be inspected to perform multimodal fusion damage identification, so as to quickly realize high-accuracy automatic measurement function of airport pavement damage status for the entire airport to be inspected, reduce the degree of manual participation in the airport pavement damage detection process, and provide effective data support for subsequent airport pavement maintenance / repair operations.

[0005] In order to achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0006] In a first aspect, the present application provides an airport pavement damage detection method, which is applied to a data processing device included in an airport pavement damage detection system, wherein the detection system also includes an airport pavement monitoring device installed on an airport inspection vehicle; the detection method comprises:

[0007] Real-time acquisition of 2D pavement images and 3D pavement point cloud data collected by the airport pavement monitoring equipment driven by the airport inspection vehicle for the airport to be inspected;

[0008] Extracting target 2D images and target 3D point cloud data corresponding to suspected road surface damage areas from the acquired 2D road surface images and 3D road surface point cloud data;

[0009] Calling a multimodal low-rank representation fusion model to perform a joint low-rank sparse decomposition and fusion process on the target 2D image and the target 3D point cloud data to obtain a target fusion image that meets the requirements of multimodal monitoring error correction;

[0010] A pre-trained pavement damage detection model is called to perform pavement damage identification on the target fusion image to obtain actual pavement damage status information of the airport to be detected at the corresponding suspected pavement damage area.

[0011] In an optional embodiment, the airport pavement monitoring equipment includes a 3D structured light camera and a 2D camera for monitoring the same airport pavement area, wherein the 3D structured light camera is used to implement a 3D laser point cloud acquisition function, and the 2D camera is used to implement a 2D image acquisition function, and the 2D pavement image and the 3D pavement point cloud data belong to the same airport pavement area; in this case, the step of extracting a target 2D image and target 3D point cloud data corresponding to the suspected pavement damage area from the acquired 2D pavement image and 3D pavement point cloud data includes:

[0012] calling the pavement damage detection model to perform pavement damage identification on the 2D pavement image to determine whether the 2D pavement image has pavement damage features;

[0013] If it is determined that the 2D road surface image has road surface damage features, extracting a minimum rectangular image corresponding to all road surface damage features from the 2D road surface image, and using the extracted minimum rectangular image as a target 2D image corresponding to the suspected road surface damage area;

[0014] performing three-dimensional spatial mapping based on the actual distribution position of the target 2D image in the 2D road surface image according to a camera coordinate system conversion relationship between the 2D camera and the 3D structured light camera, to obtain a target three-dimensional point cloud spatial range adapted to the target 2D image in the camera coordinate system of the 3D structured light camera;

[0015] The laser point cloud data in the 3D road surface point cloud data that is within the target three-dimensional point cloud space is used as the target 3D point cloud data corresponding to the suspected road surface damage area.

[0016] In an optional embodiment, the multimodal low-rank representation fusion model includes a 2D image low-rank sparse decomposition network, a 3D point cloud low-rank sparse decomposition network, four correlation filters and two feature cascade fusers, and the step of calling the multimodal low-rank representation fusion model to perform joint low-rank sparse decomposition and fusion processing on the target 2D image and the target 3D point cloud data to obtain a target fusion image that meets the multimodal monitoring error correction requirements includes:

[0017] Calling the 2D image low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the target 2D image to obtain a first low-rank component and a first sparse component of the target 2D image, and calling the 3D point cloud low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the target 3D point cloud data to obtain a second low-rank component and a second sparse component of the target 3D point cloud data;

[0018] Performing damage feature correlation filtering on the first low-rank component, the first sparse component, the second low-rank component, and the second sparse component through a correlation filter to obtain corresponding first low-rank image features, first sparse image features, second low-rank point cloud features, and second sparse point cloud features;

[0019] Performing feature cascade fusion on the first low-rank image feature and the second low-rank point cloud feature through a feature cascade fuser to obtain a corresponding target low-rank fused feature, and performing feature cascade fusion on the first sparse image feature and the second sparse point cloud feature to obtain a corresponding target sparse fused feature;

[0020] The target low-rank fusion features and the target sparse fusion features are superimposed and fused to obtain the target fusion image.

[0021] In an optional embodiment, each low-rank sparse decomposition network in the 2D image low-rank sparse decomposition network and the 3D point cloud low-rank sparse decomposition network includes an initial sparse coding block, a coded data decomposition block, and a plurality of sparse representation convolutional coding blocks cascaded with each other, and the step of calling the low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the received actual input data to obtain the actual low-rank component and the actual sparse component of the actual input data includes:

[0022] Perform dictionary concatenation on a low-rank dictionary and a sparse dictionary related to pavement damage to obtain a corresponding target coding dictionary, and call the initial sparse coding block to perform sparse representation coding on the actual input data based on the target coding dictionary to obtain corresponding initial coding data;

[0023] Using the initial coded data as reference coded data input to a leading convolutional coding block among the multiple sparse representation convolutional coding blocks, causing each sparse representation convolutional coding block to perform sparse representation convolutional coding based on the actual input data and the input reference coded data, to obtain target sparse representation coded data output by a last convolutional coding block among the multiple sparse representation convolutional coding blocks, wherein the sparse representation coded data output by a preceding sparse representation convolutional coding block of two adjacent sparse representation convolutional coding blocks serves as reference coded data input to a succeeding sparse representation convolutional coding block;

[0024] Calling the coded data decomposition block to perform coding coefficient decomposition on the target sparse representation coded data to obtain a corresponding low-rank coefficient matrix and a sparse coefficient matrix;

[0025] A matrix multiplication operation is performed on the low-rank coefficient matrix and the low-rank dictionary to obtain an actual low-rank component of the actual input data, and a matrix multiplication operation is performed on the sparse coefficient matrix and the sparse dictionary to obtain an actual sparse component of the actual input data.

[0026] In an optional embodiment, the initial encoding data of the actual input data is calculated using the following formula:

[0027] ;

[0028] The first of the multiple sparse representation convolutional coding blocks The sparse representation coding data output by the sparse representation convolutional coding block is calculated using the following formula:

[0029] ;

[0030] in, is used to represent the actual input data, Initial encoding data for representing the actual input data, Used to represent the activation function corresponding to the low-rank sparse decomposition network, is used to represent the target encoding dictionary, Used to represent the first hyperparameter of the corresponding low-rank sparse decomposition network, Used to indicate the The sparse representation coded data output by the sparse representation convolutional coding block, Used to indicate the The first convolutional layer parameters of the sparse representation convolutional coding block, Used to indicate the The second convolutional layer parameters of the sparse representation convolutional coding block, Used to represent the convolution operator, Used to indicate the The reference coded data of the sparse representation convolutional coding block input, The second hyperparameter used to represent the corresponding low-rank sparse decomposition network.

[0031] In an optional embodiment, the step of calling a pre-trained pavement damage detection model to perform pavement damage identification on the target fused image to obtain actual pavement damage status information at the airport to be inspected corresponding to the suspected pavement damage area includes:

[0032] Calling a pre-trained navigation light interference suppression network to remove the navigation light interference from the target fusion image to obtain a valid road surface image corresponding to the suspected road surface damage area;

[0033] The pavement damage detection model is called to perform pavement damage identification on the valid pavement image to obtain actual pavement damage status information corresponding to the suspected pavement damage area.

[0034] In an optional embodiment, the detection method further comprises:

[0035] Obtaining the actual inspection positions of all suspected pavement damage areas within the airport to be inspected during the inspection process of the airport inspection vehicle;

[0036] The actual inspection positions and actual pavement damage status information of all the suspected pavement damage areas are collated into an inspection report to obtain a pavement damage inspection report for the airport to be inspected.

[0037] In an optional embodiment, the detection method further comprises:

[0038] Acquire a road surface area training sample set, wherein the road surface area training sample set includes a plurality of road surface area appearance samples, each road surface area appearance sample includes a 2D image and 3D laser point cloud data of a corresponding road surface area;

[0039] Based on the road surface area training sample set, a low-rank representation fusion model is trained with the purpose of minimizing a low-rank representation fusion loss function adapted to the multimodal monitoring error correction requirement to obtain the multimodal low-rank representation fusion model.

[0040] In an optional embodiment, the low-rank representation fusion loss function adapted to the multimodal monitoring error correction requirement is expressed by the following equation:

[0041] ;

[0042] in, Used to represent the low-rank representation fusion loss function, used to represent a 2D image monitoring correction factor associated with the multimodal monitoring error correction requirement, used to represent the 3D point cloud monitoring correction factor associated with the multimodal monitoring error correction requirement, The pixel matrix used to represent the input 2D image corresponding to the multimodal low-rank representation fusion model, Used to represent the voxel matrix of the input 3D point cloud data corresponding to the multimodal low-rank representation fusion model, The pixel matrix used to represent the output fusion image of the corresponding multimodal low-rank representation fusion model, Used to represent the Frobenius norm square operator, Used to represent pixel-level image loss, Used to represent the shallow feature loss detected by the loss-aware network pre-trained on the VGG-16 network architecture, is used to represent the mid-level feature loss detected by the loss-aware network, is used to represent the deep feature loss detected by the loss-aware network, Used to represent the shallow features extracted by the first convolutional block of the loss-aware network, Used to represent the deep features extracted by the last convolutional block of the loss-aware network, The total number of cascaded convolutional blocks used to represent the loss-aware network, is used to represent the loss-aware network The middle-level features extracted by the convolution blocks, is a positive integer greater than 2 but less than the total number of cascaded convolution blocks, Used to represent the 3D point cloud feature loss weight, Used to represent the 2D image feature loss weight, Used to represent Gram matrix calculation function, 、 、 and Both are used to represent the loss impact weight coefficient.

[0043] In a second aspect, the present application provides an airport pavement damage detection system, the detection system comprising a data processing device and an airport pavement monitoring device, the airport pavement monitoring device being mounted on an airport inspection vehicle and driven by the airport inspection vehicle to perform inspections within the airport to be inspected, wherein the airport pavement monitoring device comprises a 3D structured light camera and a 2D camera for monitoring the same airport pavement area, the 3D structured light camera being used to implement a 3D laser point cloud acquisition function, and the 2D camera being used to implement a 2D image acquisition function;

[0044] The data processing device is communicatively connected to the airport pavement monitoring device and can run a pre-stored computer program to implement the airport pavement damage detection method described in any one of the aforementioned embodiments with the cooperation of the airport pavement monitoring device.

[0045] In this case, the beneficial effects of the embodiments of the present application may include the following:

[0046] The present application installs the airport pavement monitoring equipment on the airport inspection vehicle used to inspect the airport to be inspected, so as to obtain the 2D pavement images and 3D pavement point cloud data collected by the airport pavement monitoring equipment at the airport to be inspected in real time under the drive of the airport inspection vehicle, and extracts the target 2D image and target 3D point cloud data involving the suspected pavement damage area from the acquired 2D pavement image and 3D pavement point cloud data, and then calls the multimodal low-rank representation fusion model to perform joint low-rank sparse decomposition and fusion processing on the target 2D image and target 3D point cloud data, and obtains the corresponding multimodal monitoring error correction requirements (that is, eliminating the multimodal pavement monitoring of the same airport pavement area). The target fusion image (observation error between data) is obtained, and the pavement damage detection model is then called to perform pavement damage identification on the target fusion image to obtain the actual pavement damage status information corresponding to the suspected pavement damage area (including the real damage type, real damage number, real damage size, real damage depth and real damage position relationship of the suspected pavement damage area). In this way, with the cooperation of the airport inspection vehicle, the high-accuracy automatic measurement function of the airport pavement damage status can be quickly achieved for the entire airport to be inspected, reducing the manual participation in the airport pavement damage detection process and providing effective data support for subsequent airport pavement maintenance / repair operations.

[0047] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A schematic diagram of the system composition of the airport pavement damage detection system provided in an embodiment of the present application;

[0050] Figure 2 This is a flow chart of a method for detecting damage to an airport pavement provided in an embodiment of the present application;

[0051] Figure 3 for Figure 2 A schematic flow chart of the sub-steps included in step S230;

[0052] Figure 4 A schematic diagram of the model architecture of the multimodal low-rank representation fusion model provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of the network composition of a single low-rank sparse decomposition network provided in an embodiment of the present application;

[0054] Figure 6 This is a second flow chart of the airport pavement damage detection method provided in an embodiment of the present application;

[0055] Figure 7 This is the third flow chart of the airport pavement damage detection method provided in the embodiment of the present application.

[0056] Icons: 10-Airport pavement damage detection system; 11-Data processing equipment; 12-Airport pavement monitoring equipment; 121-2D camera; 122-3D structured light camera. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0058] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0059] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0060] In the description of this application, it should be understood that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, or are the orientation or position relationship in which the product of the application is usually placed when in use, or are the orientation or position relationship commonly understood by those skilled in the art. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0061] It should also be noted that, in the description of this application, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0062] In addition, in the description of the present application, it is understood that relational terms such as the terms "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also include elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or equipment comprising the elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0063] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other.

[0064] Please refer to Figure 1 , Figure 1It is a schematic diagram of the system composition of the airport pavement damage detection system 10 provided in an embodiment of the present application. In an embodiment of the present application, the airport pavement damage detection system 10 may include a data processing device 11 and an airport pavement monitoring device 12, wherein the airport pavement monitoring device 12 is installed on an airport inspection vehicle for inspecting the airport to be inspected, and the airport inspection vehicle drives the airport pavement monitoring device 12 to inspect within the airport to be inspected, wherein the airport pavement monitoring device 12 includes a 3D structured light camera 122 and a 2D camera 121 for monitoring the same airport pavement area, the 3D structured light camera 122 is used to realize the 3D laser point cloud acquisition function, and the 2D camera 121 is used to realize the 2D image acquisition function. The relative posture relationship between the 2D camera 121 and the 3D structured light camera 122 on the airport inspection vehicle is fixed; the 3D structured light camera 122 can scan by emitting multiple laser beams obliquely toward the airport pavement, and receive the reflected light signal to construct the laser point cloud data, thereby obtaining the 3D laser point cloud data corresponding to the laser scanning area.

[0065] In this embodiment, the data processing device 11 is communicatively connected to the 2D camera 121 and the 3D structured light camera 122 included in the airport pavement monitoring device 12, respectively, to acquire, in real time, multimodal pavement monitoring data (including 2D pavement images and 3D pavement point cloud data) collected by each of the 2D camera 121 and the 3D structured light camera 122 from the same airport pavement area during the airport inspection process of the airport inspection vehicle. Furthermore, the data processing device 11 may pre-store a specific computer program related to airport pavement damage detection. By running this specific computer program, the data processing device 11 performs highly accurate multimodal fusion damage identification based on the real-time acquired multimodal pavement monitoring data. This allows for rapid and highly accurate automatic measurement of airport pavement damage status for the entire airport being inspected, reducing manual intervention in the airport pavement damage detection process and providing effective data support for subsequent airport pavement maintenance / repair operations. The data processing device 11 may be, but is not limited to, a laptop, a personal computer, a server, etc.; the data processing device 11 may be deployed on the airport inspection vehicle or anywhere within the airport to be inspected. In one implementation of this embodiment, the airport inspection vehicle is an intelligent vehicle that supports automatic airport inspection. The airport pavement damage detection system 10 can leverage the automatic airport inspection function of the airport inspection vehicle to automatically and quickly achieve highly accurate airport pavement damage measurement for the entire airport to be inspected.

[0066] In addition, during the airport inspection process of the airport inspection vehicle, the data processing equipment 11 can use a positioning and navigation system (for example, a Beidou navigation system, an inertial measurement unit, etc.) to obtain in real time vehicle driving information such as the position, speed, and driving direction of the airport inspection vehicle within the airport to be inspected, so as to organically associate the airport pavement damage condition measurement results with the vehicle driving information, obtain a pavement damage detection report with geographical information of the damaged area, and then use wireless communication technology to transmit the corresponding pavement damage detection report to the airport runway maintenance management platform, so as to facilitate airport maintenance personnel to uniformly manage, analyze and make decisions on the airport pavement damage data, thereby improving the intelligent level of airport pavement maintenance.

[0067] It is understandable that Figure 1 The composition diagram shown is only a schematic diagram of the composition of the airport pavement damage detection system 10. The airport pavement damage detection system 10 may also include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0068] In this application, to ensure that the data processing device 11 in the airport pavement damage detection system 10 can acquire multimodal pavement monitoring data in real time during vehicle inspections at the airport to be inspected, and perform multimodal fusion damage identification, thereby rapidly and accurately achieving automatic measurement of airport pavement damage conditions for the entire airport to be inspected, an embodiment of this application provides an airport pavement damage detection method applied to the aforementioned data processing device 11 to achieve the aforementioned purpose. The airport pavement damage detection method provided in this application is described in detail below.

[0069] Please refer to Figure 2 , Figure 2 This is one of the flow charts of the airport pavement damage detection method provided in the embodiment of the present application. In the embodiment of the present application, the airport pavement damage detection method may include steps S210 to S240.

[0070] Step S210 , obtaining in real time the 2D pavement images and 3D pavement point cloud data collected by the airport pavement monitoring equipment driven by the airport inspection vehicle for the airport to be inspected.

[0071] In this embodiment, the 2D pavement image and 3D pavement point cloud data acquired by the data processing device 11 at the same time belong to the same airport pavement area within the airport to be inspected.

[0072] Step S220 : extracting a target 2D image and target 3D point cloud data corresponding to the suspected road surface damage area from the acquired 2D road surface image and 3D road surface point cloud data.

[0073] In this embodiment, after acquiring a 2D pavement image and 3D pavement point cloud data of a particular airport pavement area, the data processing device 11 invokes a pre-trained pavement damage detection model to preliminarily determine whether a suspected pavement damage area with pavement damage characteristics exists within the airport pavement area based on the 2D pavement image. If a suspected pavement damage area is determined to exist, the data processing device 11 extracts a target 2D image corresponding to the suspected pavement damage area from the 2D pavement image. Furthermore, after performing noise removal on the 3D pavement point cloud data, the data processing device 11 also extracts target 3D point cloud data corresponding to the suspected pavement damage area from the denoised 3D pavement point cloud data. The pavement damage detection model can be trained using a large amount of pavement image data labeled with different types of pavement damage based on a convolutional neural network, ensuring that the corresponding pavement damage detection model can clearly identify pavement damage characteristics.

[0074] On this basis, step S220 may include sub-steps A to D to accurately extract valid multi-modal pavement data that may actually involve pavement damage from the multi-modal pavement monitoring data acquired in real time. In this case, sub-steps A to D are as follows:

[0075] Sub-step A: calling the pavement damage detection model to perform pavement damage identification on the 2D pavement image to determine whether the 2D pavement image has pavement damage features.

[0076] Sub-step B: When it is determined that the 2D road surface image has road surface damage features, a minimum rectangular image corresponding to and covering all road surface damage features is cut out from the 2D road surface image, and the cut-out minimum rectangular image is used as a target 2D image corresponding to the suspected road surface damage area.

[0077] Sub-step C, based on the camera coordinate system conversion relationship between the 2D camera 121 and the 3D structured light camera 122, performs three-dimensional spatial mapping based on the actual distribution position of the target 2D image in the 2D road surface image, and obtains the target three-dimensional point cloud spatial range adapted to the target 2D image in the camera coordinate system of the 3D structured light camera 122.

[0078] Sub-step D: Use the laser point cloud data within the target three-dimensional point cloud space in the 3D pavement point cloud data as the target 3D point cloud data corresponding to the suspected pavement damage area, wherein the 3D pavement point cloud data and the 2D pavement image belong to the same airport pavement area.

[0079] Therefore, the present application can accurately extract effective multimodal pavement data that may actually involve pavement damage from the multimodal pavement monitoring data acquired in real time by executing the above sub-steps A to D.

[0080] Step S230 , calling a multimodal low-rank representation fusion model to perform joint low-rank sparse decomposition and fusion processing on the target 2D image and the target 3D point cloud data, to obtain a target fusion image that meets the requirements of multimodal monitoring error correction.

[0081] In this embodiment, the multimodal monitoring error correction requirement is used to indicate the elimination of observation errors between multimodal pavement monitoring data of the same airport pavement area, so that the final fusion image obtained by processing the multimodal low-rank representation fusion model can reflect the actual road surface details of the corresponding airport pavement area as realistically as possible. Among them, the multimodal monitoring error correction requirement can be effectively reflected in the model training process of the multimodal low-rank representation fusion model, which ensures that the multimodal low-rank representation fusion model finally trained itself has the multimodal monitoring error correction function by minimizing the low-rank representation fusion loss function of the multimodal low-rank representation fusion model. Among them, the low-rank representation fusion loss function adapted to the multimodal monitoring error correction requirement can be used to comprehensively characterize the feature differences between the multimodal pavement monitoring data and the final fusion image at different dimensional levels, which is expressed by the following equation:

[0082] ;

[0083] in, Used to represent the low-rank representation fusion loss function, used to represent a 2D image monitoring correction factor associated with the multimodal monitoring error correction requirement, used to represent the 3D point cloud monitoring correction factor associated with the multimodal monitoring error correction requirement, The pixel matrix used to represent the input 2D image corresponding to the multimodal low-rank representation fusion model, Used to represent the voxel matrix of the input 3D point cloud data corresponding to the multimodal low-rank representation fusion model, The pixel matrix used to represent the output fusion image of the corresponding multimodal low-rank representation fusion model, Used to represent the Frobenius norm square operator, Used to represent pixel-level image loss, Used to represent the loss of shallow features detected by a loss-aware network pre-trained on the VGG (Visual Geometry Group)-16 network architecture, is used to represent the mid-level feature loss detected by the loss-aware network, is used to represent the deep feature loss detected by the loss-aware network, Used to represent the shallow features extracted by the first convolutional block of the loss-aware network, Used to represent the deep features extracted by the last convolutional block of the loss-aware network, The total number of cascaded convolutional blocks used to represent the loss-aware network, is used to represent the loss-aware network The middle-level features extracted by the convolution blocks, is a positive integer greater than 2 but less than the total number of cascaded convolution blocks, Used to represent the 3D point cloud feature loss weight, Used to represent the 2D image feature loss weight, Used to represent Gram matrix calculation function, 、 、 and All of them are used to represent the loss impact weight coefficient. Wherein, all convolution blocks included in the loss-aware network are cascaded with each other, and the total number of the cascaded convolution blocks is greater than or equal to 4.

[0084] On this basis, please refer to Figure 3 and Figure 4 ,in Figure 3 yes Figure 2 The flowchart of the sub-steps included in step S230 is as follows: Figure 4 : This is a schematic diagram of the model architecture of the multimodal low-rank representation fusion model provided in an embodiment of the present application. In an embodiment of the present application, the multimodal low-rank representation fusion model may include a 2D image low-rank sparse decomposition network, a 3D point cloud low-rank sparse decomposition network, four correlation filters, and two feature cascade fusers. Step S230 may include sub-steps S231 to S234 to perform highly reliable multimodal fusion of the target 2D image and target 3D point cloud data involved in the same suspected road damage area, so that the target fused image finally outputted can truly reflect the actual road surface details of the corresponding suspected road damage area.

[0085] Sub-step S231, calling the 2D image low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the target 2D image to obtain the first low-rank component and the first sparse component of the target 2D image, and calling the 3D point cloud low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the target 3D point cloud data to obtain the second low-rank component and the second sparse component of the target 3D point cloud data.

[0086] In this embodiment, please refer to Figure 5For any one of the 2D image low-rank sparse decomposition network and the 3D point cloud low-rank sparse decomposition network, the low-rank sparse decomposition network may include an initial sparse coding block, a coding data decomposition block, and multiple sparse representation convolution coding blocks cascaded together, wherein the initial sparse coding block may be implemented using a convolution layer, the coding data decomposition block may be implemented using a decoder, each sparse representation convolution coding block includes two convolution layers (i.e., a first convolution layer and a second convolution layer), the first convolution layer parameters of the multiple sparse representation convolution coding blocks remain consistent, and the second convolution layer parameters of the multiple sparse representation convolution coding blocks remain consistent. At this time, for any low-rank sparse decomposition network, calling the low-rank sparse decomposition network to perform low-rank sparse decomposition processing on the actual input data received (for example, the target 2D image input to the 2D image low-rank sparse decomposition network, and the target 3D point cloud data input to the 3D point cloud low-rank sparse decomposition network) to obtain the actual low-rank component of the actual input data (for example, the first low-rank component associated with the target 2D image, and the second low-rank component associated with the target 3D point cloud data) and the actual sparse component (for example, the first sparse component associated with the target 2D image, and the second sparse component associated with the target 3D point cloud data) step may include:

[0087] Perform dictionary concatenation on a low-rank dictionary and a sparse dictionary related to pavement damage to obtain a corresponding target coding dictionary, and call the initial sparse coding block to perform sparse representation coding on the actual input data based on the target coding dictionary to obtain corresponding initial coding data;

[0088] Using the initial coded data as reference coded data input to a leading convolutional coding block among the multiple sparse representation convolutional coding blocks, causing each sparse representation convolutional coding block to perform sparse representation convolutional coding based on the actual input data and the input reference coded data, to obtain target sparse representation coded data output by a last convolutional coding block among the multiple sparse representation convolutional coding blocks, wherein the sparse representation coded data output by a preceding sparse representation convolutional coding block of two adjacent sparse representation convolutional coding blocks serves as reference coded data input to a succeeding sparse representation convolutional coding block;

[0089] Calling the coded data decomposition block to perform coding coefficient decomposition on the target sparse representation coded data to obtain a corresponding low-rank coefficient matrix and a sparse coefficient matrix;

[0090] A matrix multiplication operation is performed on the low-rank coefficient matrix and the low-rank dictionary to obtain an actual low-rank component of the actual input data, and a matrix multiplication operation is performed on the sparse coefficient matrix and the sparse dictionary to obtain an actual sparse component of the actual input data.

[0091] During this process, the low-rank dictionary and sparse dictionary associated with the 2D image low-rank sparse decomposition network are trained using a large amount of road surface image data labeled with different types of road surface damage; the low-rank dictionary and sparse dictionary associated with the 3D point cloud low-rank sparse decomposition network are trained using a large amount of road surface three-dimensional point cloud data labeled with different types of road surface damage. The initial encoded data obtained by encoding the actual input data in the initial sparse coding block of a single low-rank sparse decomposition network is calculated using the following formula:

[0092] ;

[0093] in, is used to represent the actual input data, Initial encoding data for representing the actual input data, Used to represent the activation function corresponding to the low-rank sparse decomposition network, is used to represent the target encoding dictionary, Used to represent the first hyperparameter of the corresponding low-rank sparse decomposition network.

[0094] In a single low-rank sparse decomposition network, The sparse representation coding data output by the sparse representation convolutional coding block is calculated using the following formula:

[0095] ;

[0096] in, is used to represent the actual input data, Used to represent the activation function corresponding to the low-rank sparse decomposition network, Used to indicate the The sparse representation coded data output by the sparse representation convolutional coding block, Used to indicate the The first convolutional layer parameters of the sparse representation convolutional coding block, Used to indicate the The second convolutional layer parameters of the sparse representation convolutional coding block, Used to represent the convolution operator, A second hyperparameter representing the corresponding low-rank sparse decomposition network; Used to indicate the The reference coded data of the sparse representation convolutional coding block input is also used to represent the The sparse representation coded data output by the sparse representation convolutional coding block.

[0097] Therefore, the present application can decompose the multimodal pavement monitoring data into appropriate multimodal low-rank representation components (i.e., the first low-rank component and the second low-rank component) and multimodal sparse representation components (i.e., the first sparse component and the second sparse component) according to the pavement damage characteristics by executing the specific step process of the above-mentioned sub-step S231.

[0098] In sub-step S232, the first low-rank component, the first sparse component, the second low-rank component and the second sparse component are respectively subjected to damage feature correlation filtering through a correlation filter to obtain corresponding first low-rank image features, first sparse image features, second low-rank point cloud features and second sparse point cloud features.

[0099] In this embodiment, any correlation filter is used to implement a correlation filtering function that focuses on pavement damage characteristics.

[0100] Sub-step S233, performing feature cascade fusion on the first low-rank image feature and the second low-rank point cloud feature through a feature cascade fusion device to obtain the corresponding target low-rank fusion feature, and performing feature cascade fusion on the first sparse image feature and the second sparse point cloud feature to obtain the corresponding target sparse fusion feature.

[0101] In this embodiment, the first low-rank image feature and the second low-rank point cloud feature are cascaded through a Concat operation and then fused through a convolutional layer of the feature cascade fuser to obtain the target low-rank fused feature; the first sparse image feature and the second sparse point cloud feature are cascaded through a Concat operation and then fused through a convolutional layer of the feature cascade fuser to obtain the target sparse fused feature.

[0102] In sub-step S234 , the target low-rank fusion features and the target sparse fusion features are superimposed and fused to obtain a target fusion image.

[0103] Therefore, the present application can perform highly reliable multimodal fusion of the target 2D image and target 3D point cloud data involved in the same suspected road damage area by executing the above-mentioned sub-steps S231 to S234, so that the final output target fusion image can truly reflect the actual road surface details of the corresponding suspected road damage area.

[0104] Step S240: Calling a pre-trained pavement damage detection model to perform pavement damage recognition on the target fusion image, and obtaining actual pavement damage status information of the airport to be detected at the corresponding suspected pavement damage area.

[0105] Optionally, in one implementation of this embodiment, after obtaining a target fused image of a suspected road damage area, the data processing device 11 may directly call the road damage detection model to perform road damage identification on the target fused image to obtain actual road damage status information at the suspected road damage area (including information such as the actual damage type, actual number of damages, actual damage size, actual damage depth, and actual damage position relationship corresponding to the suspected road damage area).

[0106] Optionally, in another implementation of this embodiment, the data processing device 11 pre-stores a navigation light interference suppression network. During the model training phase of the navigation light interference suppression network, a large number of airport runway images containing navigation light interference and images of airport runways without interference can be used as training data. The network parameters are continuously adjusted through a back-propagation algorithm to accurately remove navigation light interference. Therefore, to further improve the accuracy of pavement damage identification, the navigation light interference suppression network can be used to remove navigation light interference from the obtained target fused image. This allows for obtaining clear and accurate pavement images for pavement damage identification even in complex lighting environments (e.g., strong light at night, weak light in the early morning, etc.). In this case, step S240 may include:

[0107] Calling a pre-trained navigation light interference suppression network to remove the navigation light interference from the target fusion image to obtain a valid road surface image corresponding to the suspected road surface damage area;

[0108] The pavement damage detection model is called to perform pavement damage identification on the valid pavement image to obtain actual pavement damage status information corresponding to the suspected pavement damage area.

[0109] Therefore, the present application can obtain multimodal pavement monitoring data in real time during the vehicle inspection process of the airport to be inspected by executing the above steps S210 to S240 to perform multimodal fusion damage identification, so as to quickly realize the high-accuracy automatic measurement function of the airport pavement damage condition for the entire airport to be inspected, reduce the manual participation in the airport pavement damage detection process, and provide effective data support for subsequent airport pavement maintenance / repair operations.

[0110] Alternatively, see Figure 6 , Figure 6 This is the second flow chart of the airport pavement damage detection method provided in the embodiment of the present application. Figure 2 Compared with the airport pavement damage detection method shown, Figure 6The airport pavement damage detection method shown may further include steps S250 to S260 to ensure that the generated pavement damage detection report can intuitively display the distribution locations and damage details of all airport pavement damage within the airport to be detected.

[0111] Step S250: obtaining the actual inspection positions of all suspected pavement damage areas in the airport to be inspected during the inspection process of the airport inspection vehicle.

[0112] In this embodiment, the data processing device 11 will obtain the vehicle driving information of the airport patrol vehicle in real time during the airport patrol process of the airport patrol vehicle, so that when any suspected pavement damage area is detected through the above-mentioned step S220, the vehicle position is extracted from the vehicle driving information when the airport patrol vehicle inspects the suspected pavement damage area as the corresponding actual inspection position, thereby determining the actual inspection position (i.e., the damage area position) of all suspected pavement damage areas in the airport to be inspected along with the airport patrol operation of the airport patrol vehicle.

[0113] Step S260 , compiling a detection report of the actual inspection positions and actual pavement damage status information of all suspected pavement damage areas to obtain a pavement damage detection report for the airport to be inspected.

[0114] In this embodiment, the pavement damage detection report can be presented in the form of an electronic map, and the actual pavement damage status information of all suspected pavement damage areas is marked on the electronic map of the airport to be inspected according to the corresponding actual inspection locations, so as to intuitively present the distribution locations and damage details of all airport pavement damages in the airport to be inspected through graphical display.

[0115] Therefore, the present application can ensure that the generated pavement damage detection report can intuitively display the distribution locations and damage details of all airport pavement damages within the airport to be inspected by executing the above steps S250 to S260.

[0116] Alternatively, see Figure 7 , Figure 7 This is the third flow chart of the airport pavement damage detection method provided in the embodiment of the present application. Figure 2 or Figure 6 Compared with the airport pavement damage detection method shown, Figure 7The airport pavement damage detection method shown may also include steps S270 to S280 to ensure that the trained multimodal low-rank representation fusion model can, on the basis of realizing the multimodal pavement monitoring data fusion function, have a multimodal monitoring error correction function that meets the multimodal monitoring error correction requirements, so that the fusion image it outputs can reflect the actual road surface details of the corresponding airport pavement area as realistically as possible.

[0117] Step S270 : Acquire a road surface area training sample set, wherein the road surface area training sample set includes a plurality of road surface area appearance samples, and each road surface area appearance sample includes a 2D image and 3D laser point cloud data of a corresponding road surface area.

[0118] In step S280 , based on the road surface area training sample set, a low-rank representation fusion model is trained with the purpose of minimizing a low-rank representation fusion loss function adapted to the multimodal monitoring error correction requirement, thereby obtaining a multimodal low-rank representation fusion model.

[0119] In this embodiment, the low-rank representation fusion model is used in the training process of the model architecture as follows Figure 4 and Figure 5 As shown, the low-rank representation fusion loss function adapted to the multimodal monitoring error correction requirement is described in detail above for step S230, which will not be repeated here.

[0120] Therefore, the present application can ensure that the trained multimodal low-rank representation fusion model can, on the basis of realizing the multimodal pavement monitoring data fusion function, have a multimodal monitoring error correction function that meets the multimodal monitoring error correction requirements by executing the above steps S270 to S280, so that the fusion image it outputs can reflect the actual road surface details of the corresponding airport pavement area as realistically as possible.

[0121] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0122] In addition, the functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the various functions provided by the present application are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a readable storage medium and includes several instructions for causing the above-mentioned data processing device 11 to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned readable storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0123] The above are merely various embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for detecting damage to an airport pavement, characterized in that: The data processing equipment used in the airport pavement damage detection system includes an airport pavement monitoring device installed on an airport inspection vehicle; the detection method includes: Real-time acquisition of 2D pavement images and 3D pavement point cloud data collected by the airport pavement monitoring equipment driven by the airport inspection vehicle for the airport to be inspected; Extracting target 2D images and target 3D point cloud data corresponding to suspected road surface damage areas from the acquired 2D road surface images and 3D road surface point cloud data; Calling a multimodal low-rank representation fusion model to perform a joint low-rank sparse decomposition and fusion process on the target 2D image and the target 3D point cloud data to obtain a target fusion image that meets the requirements of multimodal monitoring error correction; wherein, the multimodal low-rank representation fusion model includes a 2D image low-rank sparse decomposition network, a 3D point cloud low-rank sparse decomposition network, four correlation filters and two feature cascade fusers, and the aforementioned steps include: calling the 2D image low-rank sparse decomposition network to perform a low-rank sparse decomposition process on the target 2D image to obtain a first low-rank component and a first sparse component of the target 2D image, and calling the 3D point cloud low-rank sparse decomposition network to perform a low-rank sparse decomposition process on the target 3D point cloud data to obtain the target 3D point cloud data the second low-rank component and the second sparse component of the image processing unit; performing damage feature correlation filtering on the first low-rank component, the first sparse component, the second low-rank component and the second sparse component respectively through a correlation filter to obtain corresponding first low-rank image features, first sparse image features, second low-rank point cloud features and second sparse point cloud features; performing feature cascade fusion on the first low-rank image features and the second low-rank point cloud features through a feature cascade fusion device to obtain corresponding target low-rank fusion features, and performing feature cascade fusion on the first sparse image features and the second sparse point cloud features to obtain corresponding target sparse fusion features; performing feature superposition fusion on the target low-rank fusion features and the target sparse fusion features to obtain the target fused image; A pre-trained pavement damage detection model is called to perform pavement damage identification on the target fusion image to obtain actual pavement damage status information of the airport to be detected at the corresponding suspected pavement damage area.

2. The detection method according to claim 1, wherein The airport pavement monitoring equipment includes a 3D structured light camera and a 2D camera for monitoring the same airport pavement area, wherein the 3D structured light camera is used to implement a 3D laser point cloud acquisition function, and the 2D camera is used to implement a 2D image acquisition function, and the 2D pavement image and the 3D pavement point cloud data belong to the same airport pavement area; in this case, the step of extracting a target 2D image and target 3D point cloud data corresponding to a suspected pavement damage area from the acquired 2D pavement image and 3D pavement point cloud data includes: calling the pavement damage detection model to perform pavement damage identification on the 2D pavement image to determine whether the 2D pavement image has pavement damage features; If it is determined that the 2D road surface image has road surface damage features, extracting a minimum rectangular image corresponding to all road surface damage features from the 2D road surface image, and using the extracted minimum rectangular image as a target 2D image corresponding to the suspected road surface damage area; performing three-dimensional spatial mapping based on the actual distribution position of the target 2D image in the 2D road surface image according to a camera coordinate system conversion relationship between the 2D camera and the 3D structured light camera, to obtain a target three-dimensional point cloud spatial range adapted to the target 2D image in the camera coordinate system of the 3D structured light camera; The laser point cloud data in the 3D road surface point cloud data that is within the target three-dimensional point cloud space is used as the target 3D point cloud data corresponding to the suspected road surface damage area.

3. The detection method according to claim 1, wherein Each low-rank sparse decomposition network in the 2D image low-rank sparse decomposition network and the 3D point cloud low-rank sparse decomposition network includes an initial sparse coding block, a coded data decomposition block, and a plurality of sparse representation convolutional coding blocks cascaded with each other, and the low-rank sparse decomposition network is called to perform low-rank sparse decomposition processing on the received actual input data to obtain the actual low-rank component and the actual sparse component of the actual input data, including: Perform dictionary concatenation on a low-rank dictionary and a sparse dictionary related to pavement damage to obtain a corresponding target coding dictionary, and call the initial sparse coding block to perform sparse representation coding on the actual input data based on the target coding dictionary to obtain corresponding initial coding data; Using the initial coded data as reference coded data input to a leading convolutional coding block among the multiple sparse representation convolutional coding blocks, causing each sparse representation convolutional coding block to perform sparse representation convolutional coding based on the actual input data and the input reference coded data, to obtain target sparse representation coded data output by a last convolutional coding block among the multiple sparse representation convolutional coding blocks, wherein the sparse representation coded data output by a preceding sparse representation convolutional coding block of two adjacent sparse representation convolutional coding blocks serves as reference coded data input to a succeeding sparse representation convolutional coding block; Calling the coded data decomposition block to perform coding coefficient decomposition on the target sparse representation coded data to obtain a corresponding low-rank coefficient matrix and a sparse coefficient matrix; A matrix multiplication operation is performed on the low-rank coefficient matrix and the low-rank dictionary to obtain an actual low-rank component of the actual input data, and a matrix multiplication operation is performed on the sparse coefficient matrix and the sparse dictionary to obtain an actual sparse component of the actual input data.

4. The detection method according to claim 3, characterized in that The initial encoding data of the actual input data is calculated using the following formula: ； The first of the multiple sparse representation convolutional coding blocks The sparse representation coding data output by the sparse representation convolutional coding block is calculated using the following formula: ； in, is used to represent the actual input data, Initial encoding data for representing the actual input data, Used to represent the activation function corresponding to the low-rank sparse decomposition network, is used to represent the target encoding dictionary, Used to represent the first hyperparameter of the corresponding low-rank sparse decomposition network, Used to indicate the The sparse representation coded data output by the sparse representation convolutional coding block, Used to indicate the The first convolutional layer parameters of the sparse representation convolutional coding block, Used to indicate the The second convolutional layer parameters of the sparse representation convolutional coding block, Used to represent the convolution operator, Used to indicate the The reference coded data of the sparse representation convolutional coding block input, The second hyperparameter used to represent the corresponding low-rank sparse decomposition network.

5. The detection method according to claim 1, wherein The step of calling a pre-trained pavement damage detection model to perform pavement damage recognition on the target fusion image to obtain actual pavement damage status information at the airport to be detected corresponding to the suspected pavement damage area includes: Calling a pre-trained navigation light interference suppression network to remove the navigation light interference from the target fusion image to obtain a valid road surface image corresponding to the suspected road surface damage area; The pavement damage detection model is called to perform pavement damage identification on the valid pavement image to obtain actual pavement damage status information corresponding to the suspected pavement damage area.

6. The detection method according to claim 1, characterized in that The detection method further comprises: Obtaining the actual inspection positions of all suspected pavement damage areas within the airport to be inspected during the inspection process of the airport inspection vehicle; The actual inspection positions and actual pavement damage status information of all the suspected pavement damage areas are collated into an inspection report to obtain a pavement damage inspection report for the airport to be inspected.

7. The detection method according to any one of claims 1 to 6, characterized in that The detection method further comprises: Acquire a road surface area training sample set, wherein the road surface area training sample set includes a plurality of road surface area appearance samples, each road surface area appearance sample includes a 2D image and 3D laser point cloud data of a corresponding road surface area; Based on the road surface area training sample set, a low-rank representation fusion model is trained with the purpose of minimizing a low-rank representation fusion loss function adapted to the multimodal monitoring error correction requirement to obtain the multimodal low-rank representation fusion model.

8. The detection method according to claim 7, characterized in that The low-rank representation fusion loss function adapted to the multimodal monitoring error correction requirement is expressed by the following equation: ； in, Used to represent the low-rank representation fusion loss function, used to represent a 2D image monitoring correction factor associated with the multimodal monitoring error correction requirement, used to represent the 3D point cloud monitoring correction factor associated with the multimodal monitoring error correction requirement, The pixel matrix used to represent the input 2D image corresponding to the multimodal low-rank representation fusion model, Used to represent the voxel matrix of the input 3D point cloud data corresponding to the multimodal low-rank representation fusion model, The pixel matrix used to represent the output fusion image of the corresponding multimodal low-rank representation fusion model, Used to represent the Frobenius norm square operator, Used to represent pixel-level image loss, Used to represent the shallow feature loss detected by the loss-aware network pre-trained on the VGG-16 network architecture, is used to represent the mid-level feature loss detected by the loss-aware network, is used to represent the deep feature loss detected by the loss-aware network, Used to represent the shallow features extracted by the first convolutional block of the loss-aware network, Used to represent the deep features extracted by the last convolutional block of the loss-aware network, The total number of cascaded convolutional blocks used to represent the loss-aware network, is used to represent the loss-aware network The middle-level features extracted by the convolution blocks, is a positive integer greater than 2 but less than the total number of cascaded convolution blocks, Used to represent the 3D point cloud feature loss weight, Used to represent the 2D image feature loss weight, Used to represent Gram matrix calculation function, 、 、 and Both are used to represent the loss impact weight coefficient.

9. An airport pavement damage detection system, characterized in that: The detection system includes a data processing device and an airport pavement monitoring device. The airport pavement monitoring device is installed on an airport inspection vehicle and driven by the airport inspection vehicle to perform inspections within the airport to be inspected. The airport pavement monitoring device includes a 3D structured light camera and a 2D camera for monitoring the same airport pavement area. The 3D structured light camera is used to implement a 3D laser point cloud acquisition function, and the 2D camera is used to implement a 2D image acquisition function. The data processing device is communicatively connected to the airport pavement monitoring device and can run a pre-stored computer program to implement the airport pavement damage detection method described in any one of claims 1 to 8 in cooperation with the airport pavement monitoring device.

Images