Road surface defect detection method and device, storage medium and electronic equipment

Abstract

The invention discloses a pavement defect detection method and device, a storage medium and electronic equipment, relates to the technical field of computer vision and intelligent transportation, and aims at improving the fusion efficiency of multi-modal features by aligning multi-scale texture features and depth and reflection intensity features through a cross-modal attention mechanism. Due to the fact that visual-radar contrast learning of a cross-modal attention mechanism aligns semantic information of multi-scale texture features and semantic information of depth and reflection intensity features in a feature space, the similarity of samples in the same defect area is maximized, the similarity of negative samples is minimized, and deviation between modals is reduced. And a contrast learning loss function is introduced into the discriminator to distinguish the cross-modal consistency of the pavement generation defect and the real defect, so that the accuracy of pavement defect detection is improved. A high-precision detection model can be trained through a cross-modal data generation technology without real defect labeling, zero-sample and few-sample defect detection is realized, and the pavement defect detection cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer vision and intelligent transportation technology, and more specifically, to a road surface defect detection method, device, storage medium, and electronic device. Background Art

[0002] Highway pavement defect detection refers to the inspection of highway pavement through a series of technical means to identify and evaluate various defects on the road surface, such as cracks, potholes, ruts, subsidence, etc.

[0003] These defects can reduce vehicle handling, increase the risk of traffic accidents, and affect the lifespan and maintenance costs of roads. Therefore, timely detection and treatment of these defects are crucial to ensuring driving safety, improving road performance, and extending road life.

[0004] Existing road surface defect detection methods typically utilize traditional deep learning models, with generative models providing output. This high cost is due to the high cost of acquiring road defect data, which requires a large number of labeled defect samples. Furthermore, since on-board equipment is subject to interference from lighting fluctuations and turbulence, the output of the generative models is prone to artifacts, leading to increased false positives.

[0005] Therefore, how to reduce the cost of road surface defect detection and improve the accuracy of road surface defect detection is an urgent problem to be solved in this application. Summary of the Invention

[0006] In view of this, the present application discloses a road surface defect detection method, device, storage medium and electronic equipment, aiming to improve the accuracy of road surface defect detection and reduce the cost of road surface defect detection.

[0007] In order to achieve the above purpose, the disclosed technical solutions are as follows:

[0008] In a first aspect, the present application discloses a road surface defect detection method, the method comprising:

[0009] Acquire a visual data stream and target radar point cloud data; wherein the target radar point cloud data is point cloud data related to the road surface;

[0010] Calculating road curvature information using a preset calculation method and the target radar point cloud data;

[0011] Extracting multi-scale texture features from the visual data stream, and extracting depth and reflection intensity features from the target radar point cloud data;

[0012] Aligning the multi-scale texture features and the depth and reflection intensity features through a cross-modal attention mechanism;

[0013] Correlating the road surface curvature information with the aligned multi-scale texture features and the aligned depth and reflection intensity features to obtain a depth information set with curvature attributes;

[0014] Generate a three-dimensional spatial constraint matrix based on the cross-modal data generation technology and the depth information set with curvature attributes;

[0015] The three-dimensional spatial constraint matrix is ​​input into a cross-modal generative adversarial network, so that the discriminator of the cross-modal generative adversarial network that introduces a contrastive learning loss function synthesizes samples of crack morphology that meet preset conditions to complete the process of real-time detection of road defects.

[0016] Preferably, the obtaining of visual data stream and target radar point cloud data includes:

[0017] The visual data stream is collected by the vehicle-mounted acquisition device, and the raw point cloud data is obtained by the millimeter wave radar; wherein the raw point cloud data is the radar point cloud data that has not been filtered;

[0018] Performing filtering on the original point cloud data to obtain a plurality of voxel grids; wherein the original point cloud data at least includes reflection intensity information;

[0019] For a plurality of voxel grids, downsampling the voxel grids to remove noise points and redundant points to obtain evenly distributed initial point cloud data;

[0020] Determining classification information based on the reflection intensity information; wherein the classification information includes at least road surface information, vehicle information, and pedestrian information;

[0021] Filtering point cloud data related to a road surface according to the classification information and the initial point cloud data;

[0022] The point cloud data related to the road surface is determined as target radar point cloud data.

[0023] Preferably, calculating the road curvature information using a preset calculation method and the target radar point cloud data includes:

[0024] Taking each point in the target radar point cloud data as the center, select a field point within a preset range;

[0025] Fitting a local plane or curved surface using the least squares method and the domain points within the preset range;

[0026] Geometric parameters are determined according to the local plane or curved surface, and road curvature information is calculated according to the geometric parameters.

[0027] Preferably, the extracting multi-scale texture features from the visual data stream and extracting depth and reflection intensity features from the target radar point cloud data include:

[0028] Extracting multi-scale texture features from the visual data stream through the visual branch of the deep residual network RestNet structure;

[0029] By introducing the radar branch of 3D sparse convolution, depth and reflection intensity features are extracted from the target radar point cloud data.

[0030] Preferably, aligning the multi-scale texture features and the depth and reflection intensity features through a cross-modal attention mechanism includes:

[0031] generating a weight matrix by guiding spatial attention through radar and the depth and reflection intensity features;

[0032] Enhance the pixel response of the multi-scale texture feature related to the defect area by using the weight matrix;

[0033] In the feature space, the semantic information of the enhanced multi-scale texture features and the semantic information of the depth and reflection intensity features are aligned through vision-radar contrastive learning.

[0034] Preferably, generating a three-dimensional space constraint matrix based on the cross-modal data generation technology and the depth information set with curvature attributes includes:

[0035] Based on a Cartesian coordinate system, the road surface area is divided into a three-dimensional grid according to a preset resolution; the three-dimensional grid includes a plurality of grid cells;

[0036] For each grid cell, calculating constraint parameters of the grid cell based on point cloud information and curvature information in the depth information set with curvature attributes falling within the grid cell; wherein the constraint parameters include at least a height offset and a curvature degree factor;

[0037] The constraint parameters are integrated into a matrix to generate a three-dimensional space constraint matrix; wherein each element of the three-dimensional space constraint matrix corresponds to the constraint information of a grid unit.

[0038] Preferably, the three-dimensional spatial constraint matrix is ​​input into a cross-modal generative adversarial network, so that a discriminator of the cross-modal generative adversarial network that introduces a contrastive learning loss function synthesizes samples of crack morphology that meet preset conditions to complete the process of real-time road surface defect detection, including:

[0039] Inputting the three-dimensional spatial constraint matrix into a cross-modal generative adversarial network, so that a generator of the cross-modal generative adversarial network selects a vertical height according to the height offset when determining a crack starting location, and adjusts the crack direction according to the curvature factor when extending the crack to conform to the curvature change of the road surface;

[0040] Obtaining the crack morphology of the road surface through the vertical height and the adjusted crack direction;

[0041] The crack morphology of the pavement is evaluated using a discriminator of a cross-modal generative adversarial network that introduces a contrastive learning loss function and an adversarial mechanism of the generative adversarial network to determine whether the crack morphology of the pavement conforms to the physical laws prescribed by the three-dimensional spatial constraint matrix and whether it is similar to real pavement cracks.

[0042] The crack morphology of the evaluated pavement is subjected to adversarial training until the generator generates samples of crack morphology that conform to the physical laws specified by the three-dimensional spatial constraint matrix and are similar to real pavement cracks, thereby completing the process of real-time detection of pavement defects.

[0043] A second aspect of the present application discloses a road surface defect detection device, comprising:

[0044] An acquisition unit, configured to acquire a visual data stream and target radar point cloud data; wherein the target radar point cloud data is point cloud data related to a road surface;

[0045] a calculation unit, configured to calculate road curvature information using a preset calculation method and the target radar point cloud data;

[0046] an extraction unit, configured to extract multi-scale texture features from the visual data stream, and extract depth and reflection intensity features from the target radar point cloud data;

[0047] an alignment unit, configured to align the multi-scale texture features and the depth and reflection intensity features through a cross-modal attention mechanism;

[0048] an associating unit, configured to associate the road surface curvature information with the aligned multi-scale texture features and the aligned depth and reflection intensity features, respectively, to obtain a depth information set with curvature attributes;

[0049] A generating unit, configured to generate a three-dimensional spatial constraint matrix based on a cross-modal data generation technique and the depth information set with curvature attributes;

[0050] The defect detection unit is used to input the three-dimensional spatial constraint matrix into a cross-modal generative adversarial network, so that the discriminator of the cross-modal generative adversarial network that introduces a contrastive learning loss function synthesizes samples of crack morphology that meet preset conditions to complete the process of real-time detection of road defects.

[0051] A third aspect of the present application discloses a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the road surface defect detection method as described in any one of the first aspects.

[0052] The fourth aspect of the present application discloses an electronic device comprising a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by one or more processors to perform the road surface defect detection method as described in any one of the first aspects.

[0053] It can be seen from the above technical solution that the present application discloses a road surface defect detection method, device, storage medium and electronic device, which obtain visual data stream and target radar point cloud data, wherein the target radar point cloud data is point cloud data related to the road surface, and the road surface curvature information is calculated through a preset calculation method and target radar point cloud data, multi-scale texture features are extracted from the visual data stream, and depth and reflection intensity features are extracted from the target radar point cloud data. The multi-scale texture features and depth and reflection intensity features are aligned through a cross-modal attention mechanism, and the road surface curvature information is associated with the aligned multi-scale texture features and the aligned depth and reflection intensity features respectively to obtain a depth information set with curvature attributes. According to the cross-modal data generation technology and the depth information set with curvature attributes, a three-dimensional spatial constraint matrix is ​​generated, and the three-dimensional spatial constraint matrix is ​​input into a cross-modal generative adversarial network, so that the discriminator of the cross-modal generative adversarial network that introduces a contrastive learning loss function synthesizes samples of crack morphology that meet preset conditions to complete the process of real-time detection of road surface defects.

[0054] The beneficial effects of the present application are: aligning multi-scale texture features and depth and reflection intensity features, i.e., multi-modal features, through a physical consistency cross-modal attention mechanism composed of radar-guided spatial attention and visual-radar contrast learning, thereby improving the fusion efficiency of multi-modal features. Since visual-radar contrast learning aligns the semantic information of multi-scale texture features and the semantic information of depth and reflection intensity features in the feature space, the deviation between modalities is reduced by maximizing the similarity of positive samples, i.e., samples of the same defect area, and minimizing the similarity of negative samples. In addition, the discriminator in the cross-modal generative adversarial network introduces a contrastive learning loss function to distinguish the cross-modal consistency between pavement-generated defects and real defects, thereby improving the accuracy of pavement defect detection. Through cross-modal data generation technology and a depth information set with curvature attributes, a three-dimensional spatial constraint matrix is ​​generated, and a high-precision detection model can be trained without real defect annotation, achieving zero-sample and few-sample defect detection, and reducing the cost of pavement defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0056] Figure 1 A schematic flow chart of a road surface defect detection method disclosed in an embodiment of the present application;

[0057] Figure 2 This is a schematic diagram of the process of obtaining visual data stream and target radar point cloud data disclosed in the embodiment of this application;

[0058] Figure 3 A schematic diagram of the process of calculating road curvature information disclosed in an embodiment of the present application;

[0059] Figure 4 A schematic diagram of a process for aligning multi-scale texture features and depth and reflection intensity features through a cross-modal attention mechanism disclosed in an embodiment of the present application;

[0060] Figure 5 A schematic diagram of a process for generating a three-dimensional space constraint matrix disclosed in an embodiment of the present application;

[0061] Figure 6 A schematic diagram of a process for completing real-time detection of road surface defects disclosed in an embodiment of the present application;

[0062] Figure 7 This is an architectural diagram of a road surface defect detection system disclosed in an embodiment of the present application;

[0063] Figure 8 This is a schematic structural diagram of a road surface defect detection device disclosed in an embodiment of the present application;

[0064] Figure 9 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0066] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0067] As can be seen from the background technology, the existing road surface defect detection methods are costly and have a high false positive rate. Therefore, how to reduce the cost of road surface defect detection and improve the accuracy of road surface defect detection is an urgent problem to be solved in this application.

[0068] To address the aforementioned issues, this application discloses a pavement defect detection method, apparatus, storage medium, and electronic device. This method uses a physically consistent cross-modal attention mechanism comprised of radar-guided spatial attention and visual-radar contrastive learning to align multi-scale texture features with depth and reflection intensity features, i.e., multimodal features, thereby improving the efficiency of multimodal feature fusion. Because visual-radar contrastive learning aligns the semantic information of multi-scale texture features with that of depth and reflection intensity features in feature space, it reduces inter-modal bias by maximizing the similarity of positive samples (i.e., samples from the same defect area) and minimizing the similarity of negative samples. Furthermore, a contrastive learning loss function is introduced into the discriminator within the cross-modal generative adversarial network to distinguish the cross-modal consistency between generated pavement defects and real defects, thereby improving the accuracy of pavement defect detection. By using cross-modal data generation technology and a depth information set with curvature attributes, a three-dimensional spatial constraint matrix is ​​generated. This allows for training high-precision detection models without the need for real defect annotation, enabling zero- and few-shot defect detection and reducing the cost of pavement defect detection. The specific implementation is illustrated in the following examples.

[0069] It should be noted that the road surface defect detection method, device, storage medium and electronic device provided in this application can be used in the technical field of the intersection of computer vision and intelligent transportation, and is particularly suitable for zero-sample / few-sample scenarios of multi-sensor collaboration such as on-board cameras and millimeter-wave radars. The above is only an example and does not limit the application field of the road surface defect detection method, device, storage medium and electronic device provided in this application.

[0070] refer to Figure 1 FIG. 1 is a flow chart of a road surface defect detection method disclosed in an embodiment of the present application. The road surface defect detection method mainly includes the following steps:

[0071] S101: Acquire a visual data stream and target radar point cloud data; wherein the target radar point cloud data is point cloud data related to the road surface.

[0072] The visual data stream is a color mode (RGB) image, which can be collected by a vehicle-mounted acquisition device or other equipment.

[0073] Target radar point cloud data represents a radar data stream, which includes at least point cloud reflection intensity and depth data. Target radar point cloud data can be obtained using equipment such as millimeter-wave radar.

[0074] The specific process of obtaining visual data stream and target radar point cloud data is as follows: Figure 2 As shown, it mainly includes the following steps:

[0075] S201: Collect visual data streams (RGB images) through the vehicle-mounted acquisition device, and obtain raw point cloud data through the millimeter wave radar; wherein the raw point cloud data is radar point cloud data that has not been filtered.

[0076] Among them, the vehicle-mounted collection equipment includes but is not limited to vehicle-mounted cameras.

[0077] S202: Filtering the original point cloud data to obtain a plurality of voxel grids; wherein the original point cloud data at least includes reflection intensity information.

[0078] The filtering process includes but is not limited to voxel grid filtering. The filtering process of the present application is preferably voxel grid filtering.

[0079] The voxel grid filtering method is used to divide the point cloud data into small voxel grids.

[0080] S203: For the multiple voxel grids, downsample the voxel grids to remove noise points and redundant points, and obtain evenly distributed initial point cloud data.

[0081] In S203 , the points in each grid are downsampled by averaging or other methods to remove noise points and redundant points, thereby obtaining relatively clean and evenly distributed initial point cloud data.

[0082] S204: Determine classification information based on the reflection intensity information; wherein the classification information at least includes road surface information, vehicle information, and pedestrian information.

[0083] In S204 , different materials or objects are preliminarily classified based on the reflection intensity information of the point cloud to obtain classification information, which is used to distinguish, for example, road surfaces, vehicles, pedestrians, etc.

[0084] S205: Filtering out point cloud data related to the road surface based on the classification information and the initial point cloud data.

[0085] Based on the classification information and initial point cloud data, the point cloud data related to the road surface is screened out to prepare for the subsequent construction of constraints related to the road curvature.

[0086] S206: Determine the point cloud data related to the road surface as target radar point cloud data.

[0087] S102: Calculate road curvature information using a preset calculation method and target radar point cloud data.

[0088] In S102 , for the filtered road point cloud data, a method such as local neighborhood fitting may be used to calculate the road curvature.

[0089] The specific process of calculating road curvature information, such as Figure 3As shown, it mainly includes the following steps:

[0090] S301: Taking each point in the target radar point cloud data as the center, select a field point within a preset range.

[0091] The preset range is set according to actual conditions and is not specifically limited in this application.

[0092] For example, with each point in the target radar point cloud data as the center, select neighborhood points within a certain radius.

[0093] S302: Fitting a local plane or curved surface using the least squares method and domain points within a preset range.

[0094] In S302, a local plane or surface is fitted by using the least squares method and the area points within a preset range. Based on the fitting of the local plane or surface, the fitted geometric parameters are determined, including but not limited to the normal vector, curvature radius, etc.

[0095] S303: Determine geometric parameters according to the local plane or curved surface, and calculate road curvature information according to the geometric parameters.

[0096] Taking each point as the center, select neighborhood points within a certain radius, fit a local plane or surface using the least squares method, and calculate the road curvature at that point based on the fitted geometric parameters (such as normal vector, curvature radius, etc.).

[0097] S103: Extract multi-scale texture features from the visual data stream (RGB image), and extract depth and reflection intensity features from the target radar point cloud data.

[0098] The bimodal encoder with a cross-modal feature alignment architecture extracts multi-scale texture features from the visual data stream, and extracts depth and reflection intensity features from the target radar point cloud data, namely the point cloud reflection intensity and depth data.

[0099] Dual-modal encoder: This system uses a dual-branch U-Net architecture to process the target radar point cloud data (voxelized features) from the millimeter-wave radar and the visual image (RGB texture) from the visual data stream. 3D sparse convolution is introduced in the radar branch to extract depth and reflection intensity features; in the visual branch, ResNet-50 is used to extract multi-scale texture features.

[0100] In S103, the visual branch of the deep residual network RestNet-50 structure is used to extract multi-scale texture features from the visual data stream (RGB image); by introducing the radar branch with 3D sparse convolution, the depth and reflection intensity features are extracted from the target radar point cloud data.

[0101] Specifically, multi-scale texture features are extracted from the visual data stream (RGB image). By introducing the radar branch of 3D sparse convolution, the depth and reflection intensity features are extracted from the target radar point cloud data. The operation process includes feature fusion and joint analysis and decision optimization based on consistency judgment. The specific feature fusion and joint analysis and decision optimization based on consistency judgment are explained as follows:

[0102] Feature fusion and joint analysis:

[0103] Fusion Strategy: Adopt an adaptive feature fusion method based on the attention mechanism. Traditional fusion methods often rely on simple concatenation or weighted summation, lacking dynamic consideration of the importance of features from different modalities. This solution designs a Modality-Feature Attention Network (MFAN) to dynamically adjust feature weights based on the effectiveness of features in the current scenario. For example, in complex lighting scenarios, the depth and reflection intensity features of millimeter-wave radar are more reliable, so the MFAN will increase the weight assigned to radar features (i.e., depth and reflection intensity features). Conversely, in scenes rich in structural detail, visual texture features are given a higher weight. Through this adaptive fusion, a more discriminative joint feature representation is generated.

[0104] Joint consistency measurement: Most existing technologies calculate MAE error and SSIM similarity separately, lacking joint consideration. This solution constructs a new joint loss function To measure consistency. The joint loss function of this scheme integrates the two and introduces the modal consistency coefficient To balance the radar reflection intensity distribution error Similarity to visual texture The specific effect is shown in formula (1).

[0105] (1)

[0106] in, is the joint loss function; is the modal consistency coefficient, Dynamic adjustments can be made through reinforcement learning based on different application scenarios (such as high-speed driving and urban road driving), thereby more accurately determining the consistency of generated samples in the two modalities. is the radar reflection intensity distribution error; is the visual texture similarity.

[0107] Decision optimization based on consistency judgment:

[0108] Multimodal-guided decision calibration: Traditional decision models mostly rely on a single modality or simple fusion features, and lack in-depth exploration of inter-modal consistency. This application introduces consistency judgment results as decision calibration factors to correct preliminary decisions based on fusion features (i.e., features that fuse multi-scale texture features with depth and reflection intensity features). For example, in a target detection task, if a preliminary judgment is made based on fusion features that there is a target in a certain area, but after consistency judgment, it is found that there is a large difference between the radar features and the visual modality features (the consistency is lower than the preset threshold, which is set according to the actual situation and is not specifically limited in this application), then the secondary feature extraction and analysis module is started to re-examine the area to avoid misjudgment, thereby improving the accuracy.

[0109] Consistency-driven model adaptation: A model adaptation module based on consistency feedback is designed. When consistency judgment results indicate that the current model has problems processing modal features in certain scenarios, this adaptive module automatically adjusts model parameters or structure. For example, if the radar reflection intensity distribution is found to be inconsistent, that is, below a preset threshold, the convolution kernel parameters of the radar branch are automatically fine-tuned to enhance the ability to extract depth and reflection intensity features. This allows the model to better adapt to the consistency of modal features in different scenarios, improving the overall performance and reliability of the system.

[0110] S104: Aligning multi-scale texture features with depth and reflection intensity features via cross-modal attention mechanism (CMA).

[0111] Among them, the cross-modal attention mechanism is determined by radar-guided spatial attention and visual-radar contrastive learning.

[0112] Radar-guided spatial attention is used to utilize depth and reflection intensity features, i.e., the spatial positions of radar point clouds, to generate a weight matrix to enhance pixel responses associated with defect areas in the visual image (e.g., crack edges, pothole boundaries).

[0113] Vision-radar contrastive learning is used to align the semantic information of the two modalities (i.e., enhanced multi-scale texture features, and depth and reflection intensity features) in the feature space, thereby reducing the inter-modality bias by maximizing the similarity of positive samples (the same defect area) and minimizing the similarity of negative samples.

[0114] A physical consistency alignment mechanism of visual and radar features is established through a cross-modal attention mechanism to improve the efficiency of multimodal feature fusion.

[0115] Specifically, the process of aligning multi-scale texture features and depth and reflection intensity features through the cross-modal attention mechanism is as follows: Figure 4 As shown, it mainly includes the following steps:

[0116] S401: Generate a weight matrix by guiding spatial attention and depth and reflection intensity features through radar.

[0117] Among them, radar-guided spatial attention is used to generate a weight matrix using depth and reflection intensity features, that is, the spatial position of the radar point cloud.

[0118] S402: Enhance the pixel response related to the multi-scale texture feature and the defect area through a weight matrix.

[0119] Based on the pixel responses associated with defect areas in the enhanced visual image (such as crack edges, pothole boundaries).

[0120] S403: In the feature space, the semantic information of the enhanced multi-scale texture features and the semantic information of the depth and reflection intensity features are aligned through vision-radar comparative learning.

[0121] Among them, visual-radar contrastive learning is used to align the semantic information of the two modalities (i.e., enhanced multi-scale texture features, and depth and reflection intensity features) in the feature space, thereby reducing the inter-modal bias by maximizing the similarity of positive samples (the same defect area) and minimizing the similarity of negative samples.

[0122] S105: Correlating the road surface curvature information with the aligned multi-scale texture features and the aligned depth and reflection intensity features, respectively, to obtain a depth information set with curvature attributes.

[0123] S106: Generate a three-dimensional space constraint matrix based on the cross-modal data generation technology and the depth information set with curvature attributes.

[0124] Cross-modal fusion constraint enhancement is achieved through a three-dimensional spatial constraint matrix. In addition to utilizing the depth information of the radar point cloud, modal fusion constraint enhancement also innovatively integrates visual texture information obtained from on-board cameras to further enhance constraints. Through feature alignment and fusion technology, information such as edges and contours in the visual texture is combined with the depth-curvature constraints of the radar point cloud. For example, when a clear boundary is detected on the road surface in the visual texture, the restriction on the extension direction of the crack is strengthened at the corresponding position in the three-dimensional spatial constraint matrix to avoid the boundary area, thereby achieving multi-modal information complementarity and generating a more realistic and reasonable crack morphology.

[0125] Through cross-modal data generation technology, high-precision detection models can be trained without the need for real defect annotation, solving the problem of zero-sample defect detection.

[0126] The specific process of generating a three-dimensional space constraint matrix is ​​as follows: Figure 5 As shown, it mainly includes the following steps:

[0127] S501: Based on a Cartesian coordinate system, a road surface area is divided into a three-dimensional grid according to a preset resolution; the three-dimensional grid includes a plurality of grid cells.

[0128] The preset resolution may be a resolution of 1280×720 pixels, a resolution of 1920×1080 pixels, etc. The preset resolution is set according to actual conditions and is not specifically limited in this application.

[0129] S502: For each grid cell, calculate the constraint parameters of the grid cell according to the point cloud information and curvature information in the depth information set with curvature attributes falling into the grid cell; wherein the constraint parameters include at least a height offset and a curvature degree factor.

[0130] The constraint parameters include but are not limited to a height offset determined based on depth and a curvature factor determined based on curvature information.

[0131] The height offset is used to limit the depth range of the generated cracks in the vertical direction.

[0132] The curvature factor is used to guide the bending shape of the crack in the horizontal direction.

[0133] S503: Integrate the constraint parameters into a matrix to generate a three-dimensional space constraint matrix (3D space constraint matrix); wherein each element of the three-dimensional space constraint matrix corresponds to the constraint information of a grid unit.

[0134] Each element of the three-dimensional spatial constraint matrix corresponds to the constraint information of a grid unit, which fully describes the 3D spatial physical characteristics of the road surface.

[0135] Dynamic adaptive constraint adjustment is performed through a dynamic adaptive mechanism. Because traditional constraint construction is mostly based on static a priori rules, it lacks adaptability to changes in different scenarios. The dynamic adaptive mechanism of this application adjusts the parameters of the three-dimensional spatial constraint matrix in real time based on real-time road condition information (such as vehicle speed, road surface temperature, etc.) and statistical characteristics of point cloud data (such as changes in point cloud density). For example, when a vehicle is traveling at high speed, the accuracy constraints on the horizontal direction of the crack morphology are appropriately relaxed to take into account factors such as dynamic blur; in sections of road with large temperature changes, the depth constraint range in the vertical direction is adjusted according to the principle of thermal expansion and contraction, so that the generated crack morphology is more in line with the actual scene.

[0136] S107: Input the three-dimensional spatial constraint matrix into the cross-modal generative adversarial network, so that the discriminator of the cross-modal generative adversarial network that introduces the contrastive learning loss function can synthesize samples of crack morphology that meet the preset conditions (crack and pothole samples that conform to the actual physical distribution) to complete the process of real-time detection of road defects.

[0137] In S107, a physical constraint generation strategy is used, that is, the constructed three-dimensional space constraint matrix is ​​used to guide the generator to synthesize crack morphologies that meet preset conditions, that is, crack morphologies that meet the curvature of the road surface, the real physical distribution of cracks, and pothole samples.

[0138] The loss function of the generative adversarial network includes at least multimodal contrast loss and physical consistency constraint loss.

[0139] The cross-modal generative adversarial network consists of an optimized generator and a discriminator.

[0140] The generator uses a dual-branch U-Net architecture to process the visual image and target radar point cloud data separately. The spatial attention module's cross-modal attention mechanism aligns multi-scale texture features with depth and reflection intensity features. A three-dimensional spatial constraint matrix based on radar depth information is introduced to standardize the generated morphology, enhance physical consistency, and assist cross-modal fusion.

[0141] The process of generator optimization is as follows:

[0142] Physical Constraint Layer:

[0143] Based on the depth information of the radar point cloud (such as road curvature and reflection intensity), a three-dimensional spatial constraint matrix is ​​constructed to constrain the crack morphology synthesized by the generator to conform to real physical laws (such as the positive correlation between crack width and depth);

[0144] Multi-scale residual module:

[0145] Residual blocks are embedded in U-Net skip connections to fuse shallow texture features (vision) with deep structural features (radar), enhancing the detail fidelity of generated defects (such as the continuity of subtle cracks).

[0146] The discriminator introduces a contrastive learning loss function to distinguish the cross-modal consistency between generated defects and real defects.

[0147] The design of the discriminator is as follows:

[0148] Multimodal Contrastive Discriminator:

[0149] The input is the cross-modal joint features of the generated defect image and the real defect image. The contrast loss is used to judge the consistency of the generated samples in radar reflection intensity distribution (MAE error) and visual texture (SSIM similarity).

[0150] Multimodal contrast loss function: Combines visual texture similarity (SSIM) and radar reflection intensity difference (MAE) to enhance the cross-modal authenticity of generated samples.

[0151] The contrastive learning loss function is introduced, which is the joint loss function.

[0152] The specific structure of the joint loss function is as follows:

[0153] Adversarial Loss: Standard Cross-Modality GAN loss, which optimizes adversarial training between the generator and the discriminator.

[0154] Physics Loss: Calculates the KL divergence between the radar reflection intensity distribution of the generated defect (such as the scattering characteristics of the pothole area) and the real data to ensure that the physical properties match.

[0155] Cross-modal contrastive loss (CMCL): aligns the embedding spaces of visual features and radar features through the InfoNCE loss function to improve modal complementarity.

[0156] In S107, the constructed three-dimensional spatial constraint matrix is ​​input into the generator of the cross-modal generative adversarial network. When the generator synthesizes the crack morphology, the information in the constraint matrix is ​​referenced in each step of the calculation. For example, when determining the starting position of the crack, the appropriate vertical height is selected based on the range of the height offset in the constraint matrix; when extending the crack, the direction of the crack is adjusted according to the bending degree factor to make it conform to the change in road curvature. At the same time, combined with the adversarial mechanism of GAN, the discriminator will evaluate the generated crack morphology to determine whether it meets the preset conditions, namely the physical laws specified by the constraint matrix and the similarity with real road cracks. Through continuous adversarial training, the generator can gradually synthesize crack morphologies that not only meet physical constraints such as road curvature but also have high fidelity.

[0157] Real-time detection of road defects, namely lightweight real-time detection of road defects, is used to support the lightweight deployment of on-board edge computing devices and meet the low-latency requirements in dynamic scenarios.

[0158] Lightweight deployment compresses detection models to 3MB or even smaller, achieving inference speeds of 30 FPS (NVIDIA Jetson Xavier) or even faster. Detection models can also be deployed dynamically.

[0159] The engine for dynamically deploying detection models includes edge device adaptive compression and EnsorRT quantization.

[0160] Adaptive compression on edge devices:

[0161] Knowledge distillation strategy: The output of the pre-trained YOLOv11 model (teacher network) is used as the supervision signal to train a lightweight MobileNetV3 (student network) to retain the discriminative ability of multimodal features.

[0162] 1. Initialization and preparation:

[0163] First, we obtain a pre-trained YOLOv11 model as the teacher network. The YOLOv11 model has been trained on large-scale datasets and possesses strong object detection and multimodal feature discrimination capabilities. We also initialize a lightweight MobileNetV3 as the student network. This student network has a simple structure and low computational overhead, making it suitable for deployment on edge devices.

[0164] 2. Feature Distillation:

[0165] During training, the same multimodal input data (such as fused millimeter-wave radar point cloud data and RGB image data from a vehicle camera) is fed into both the teacher and student networks. The teacher network outputs feature maps containing rich semantic information and discriminative features, while the student network attempts to learn these features.

[0166] Specifically, by calculating the difference between the feature maps of the teacher network and the student network, for example, using the mean square error (MSE) loss function, the calculation formula of the MSE loss function is shown in formula (2).

[0167] (2)

[0168] in, is the characteristic distillation loss; is the total number of elements in the feature map; is the eigenvalue of the teacher network at the i-th position; is the eigenvalue of the student network at the i-th position.

[0169] Through the back-propagation algorithm, the parameters of the student network are continuously adjusted to make its feature map as close as possible to the teacher network.

[0170] 3. Output distillation:

[0171] In addition to feature distillation, the network output is also distilled. The category probability distribution output by the teacher network can be considered a soft label, containing richer information than the true label. The KL divergence (Kullback-Leibler Divergence, KLD) is used to measure the difference between the probability distribution Pstudent output by the student network and the probability distribution Pteacher output by the teacher network. The loss function is shown in Formula (3).

[0172] (3)

[0173] in, is the output distillation loss; j is the category index; is the probability distribution output by the teacher network; is the probability distribution output by the student network.

[0174] By minimizing the loss function of formula (3), the student network can learn the knowledge of the teacher network in category prediction and retain the discriminative ability of multimodal features.

[0175] In order to further improve the efficiency and effect of distillation, an adaptive weight adjustment mechanism is introduced. That is, according to the different characteristics of the input data (such as scene complexity, target diversity, etc.), the feature distillation loss is dynamically adjusted. and output distillation loss For example, in simple scenarios, appropriately increasing the weight of output distillation allows the student network to more quickly learn the classification knowledge of the teacher network; in complex scenarios, increasing the weight of feature distillation allows the student network to better learn the feature extraction capabilities of the teacher network.

[0176] TensorRT Quantization:

[0177] FP16 quantization of the generator and discriminator, combined with dynamic memory optimization, increases inference speed to 30 FPS (NVIDIA Jetson Xavier) or even faster.

[0178] 1. Model import and analysis:

[0179] First, import the trained generator and discriminator models into the TensorRT environment. TensorRT analyzes the model's structure and parameters, identifying the various layers in the model (such as convolutional layers and fully connected layers) and the connections between them.

[0180] 2. FP16 quantization:

[0181] Because traditional models in existing technologies typically use 32-bit floating-point numbers (FP32) to represent weights and activation values, which consumes a lot of memory and computing resources, TensorRT quantization uses 16-bit floating-point numbers (FP16) to quantize models. It analyzes the dynamic range of model parameters and maps FP32 values ​​to the FP16 range. Specifically, TensorRT quantization counts the maximum and minimum values ​​of model parameters, determines scaling factors and offsets based on specific quantization algorithms (such as linear quantization), and converts FP32 parameters to FP16 format. This process minimizes precision loss and ensures model performance through a carefully designed quantization algorithm.

[0182] 3. Dynamic memory optimization:

[0183] In addition to FP16 quantization, TensorRT quantization also optimizes the model's memory usage. It dynamically allocates memory based on the model's execution plan. For example, for convolutional layer calculations, TensorRT rationally arranges the memory locations of input data, weight data, and output data, avoiding unnecessary memory copying and access overhead. Furthermore, memory pooling technology pre-allocates memory blocks of a certain size. When new memory is needed during model execution, it is directly obtained from the memory pool, reducing the time overhead of memory allocation and release.

[0184] 4. Inference Acceleration and Deployment:

[0185] After FP16 quantization and dynamic memory optimization, TensorRT quantization generates a highly optimized inference engine. Deploying this inference engine on edge devices such as the NVIDIA Jetson Xavier significantly improves model inference speed. Compared to existing unoptimized models, inference speeds can reach 30 FPS or even higher. Furthermore, a key innovation lies in the introduction of a dynamic optimization mechanism based on runtime feedback. During model inference, model performance metrics (such as inference time and memory usage) are monitored in real time. Based on this feedback, quantization parameters and memory allocation strategies are dynamically adjusted to further optimize the model's inference speed and resource utilization efficiency under different loads and scenarios.

[0186] Specifically complete the process of real-time detection of road surface defects, such as Figure 6 As shown, it mainly includes the following steps:

[0187] S601: Input the three-dimensional spatial constraint matrix into the cross-modal generative adversarial network, so that the generator of the cross-modal generative adversarial network selects the vertical height according to the height offset when determining the starting position of the crack, and adjusts the direction of the crack according to the curvature factor when extending the crack to make it conform to the change of road curvature.

[0188] S602: Obtain the crack morphology of the road surface through the vertical height and the adjusted crack direction.

[0189] S603: Based on the discriminator of the cross-modal generative adversarial network that introduces the contrastive learning loss function and the adversarial mechanism of the generative adversarial network, the crack morphology of the road surface is evaluated to determine whether the crack morphology of the road surface conforms to the physical laws prescribed by the three-dimensional spatial constraint matrix and its similarity to real road surface cracks.

[0190] S604: Conduct adversarial training on the crack morphology of the evaluated pavement until the generator generates samples of crack morphology that conform to the physical laws specified by the three-dimensional spatial constraint matrix and are similar to real pavement cracks, thereby completing the process of real-time pavement defect detection.

[0191] In order to facilitate the understanding of the process of road surface defect detection, combined with Figure 7 To illustrate, Figure 7 The road surface defect detection architecture diagram is shown. This road surface defect detection architecture diagram includes a multimodal data acquisition module, a cross-modal feature alignment module, a cross-modal generative adversarial module, and a detection deployment module. The multimodal data acquisition module includes an onboard camera and millimeter-wave radar.

[0192] Figure 7 In the,vision data stream (RGB image) is obtained through the on-board camera in the,multimodal data acquisition module, and the radar data stream (point cloud,reflection intensity) is obtained through the millimeter-wave radar,in the multimodal data acquisition module;

[0193] The cross-modal feature alignment module introduces a U-net state encoder with multi-scale RestNet-50 feature extraction to extract multi-scale texture features from the millimeter-wave radar data stream, and the cross-modal feature alignment module introduces a U-net state encoder with 3D sparse convolution voxelization to extract depth and reflection intensity features.

[0194] Align multi-scale texture features with depth and reflection intensity features through a cross-modal attention mechanism;

[0195] Generate a weight matrix by guiding spatial attention and depth and reflection intensity features through radar;

[0196] Enhance the pixel response related to the defect area with multi-scale texture features through weight matrix;

[0197] In the feature space, the semantic information of the enhanced multi-scale texture features is aligned with the semantic information of the depth and reflectance features through visual-radar contrastive learning;

[0198] The road surface curvature information is associated with the aligned multi-scale texture features and the aligned depth and reflection intensity features to obtain a depth information set with curvature attributes;

[0199] Generate a three-dimensional spatial constraint matrix based on cross-modal data generation technology and a depth information set with curvature attributes;

[0200] Input the three-dimensional spatial constraint matrix into the YOLOv11 model in the retrieval deployment module for training;

[0201] The output of the trained YOLOv11 model is used as a supervisory signal to train a lightweight MobileNetV3 model.

[0202] During the training process of the MobileNetV3 model, the parameters of the student network of the MobileNetV3 model are continuously adjusted through the real-time data augmentation module and the backpropagation algorithm (InverseAug reverse enhancement), so that its feature map is as close as possible to the teacher network of the YOLOv11 model. InverseAug reverse enhancement is used to address the bumpy interference in dynamic vehicle scenes. It restores the spatial alignment between the radar point cloud and the image through inverse transformation, eliminating the impact of motion artifacts on the generated quality.

[0203] Perform FP16 quantization and dynamic memory optimization on the generator and discriminator of the detection deployment module. After FP16 quantization and dynamic memory optimization, TensorRT quantization generates a highly optimized inference engine.

[0204] Deploy the inference engine to the edge, such as on edge devices such as NVIDIA Jetson Xavier.

[0205] The solution of this application can improve sample generation efficiency, optimize detection accuracy and achieve lightweight deployment.

[0206] Improved sample generation efficiency: Experiments show that this solution only takes 2 hours (NVIDIA A100 GPU) to generate 100,000 defect images, reducing costs by at least 90%.

[0207] Detection accuracy optimization: Through relevant experiments, this solution achieved a mAP of 92.7% and a false positive rate of less than 3% in both the GTA5 synthetic dataset and real-world road test data.

[0208] Lightweight deployment: Experiments show that the model compressed by this solution can be reduced to at least 3MB, with an inference speed of at least 30FPS (NVIDIA Jetson Xavier).

[0209] This application addresses the issue of insufficient multimodal data samples collected by on-board cameras and millimeter-wave radars. By integrating visual imagery with radar point cloud data, this solution proposes a defect generation scheme that uses radar depth information to constrain the generator's spatial structure and employs an attention mechanism to align visual texture with radar reflection intensity features, generating crack and pothole samples that conform to their physical distribution. Combined with a comparative learning strategy, this solution achieves pixel-level defect localization without labeled data. This solution significantly reduces reliance on labeled data, achieving a defect recall rate exceeding 95% and a false detection rate below 3% in real-world road tests, supporting real-time detection on onboard edge devices.

[0210] The beneficial effects of the embodiments of the present application are as follows: the multi-scale texture features and depth and reflection intensity features, i.e., multi-modal features, are aligned through a physical consistency cross-modal attention mechanism composed of radar-guided spatial attention and visual-radar contrastive learning, thereby improving the fusion efficiency of multi-modal features. Since visual-radar contrastive learning aligns the semantic information of multi-scale texture features and the semantic information of depth and reflection intensity features in the feature space, the deviation between modalities is reduced by maximizing the similarity of positive samples, i.e., samples of the same defect area, and minimizing the similarity of negative samples. In addition, the discriminator in the cross-modal generative adversarial network introduces a contrastive learning loss function to distinguish the cross-modal consistency between pavement-generated defects and real defects, thereby improving the accuracy of pavement defect detection. Through cross-modal data generation technology and a depth information set with curvature attributes, a three-dimensional spatial constraint matrix is ​​generated, and a high-precision detection model can be trained without the need for real defect annotation, thereby achieving zero-sample and few-sample defect detection and reducing the cost of pavement defect detection.

[0211] Based on the road surface defect detection method disclosed in the above embodiment, the present application also discloses a road surface defect detection device. Figure 8 As shown, the road surface defect detection device includes:

[0212] An acquisition unit 801 is configured to acquire a visual data stream and target radar point cloud data; wherein the target radar point cloud data is point cloud data related to a road surface;

[0213] The calculation unit 802 is used to calculate the road curvature information using a preset calculation method and target radar point cloud data;

[0214] Extraction unit 803, used to extract multi-scale texture features from the visual data stream, and extract depth and reflection intensity features from the target radar point cloud data;

[0215] an alignment unit 804 for aligning multi-scale texture features and depth and reflection intensity features through a cross-modal attention mechanism;

[0216] An associating unit 805 is configured to associate the road curvature information with the aligned multi-scale texture features and the aligned depth and reflection intensity features, respectively, to obtain a depth information set with curvature attributes;

[0217] A generating unit 806 is configured to generate a three-dimensional space constraint matrix based on a cross-modal data generation technique and a depth information set with curvature attributes;

[0218] The defect detection unit 807 is used to input the three-dimensional spatial constraint matrix into the cross-modal generative adversarial network, so that the discriminator of the cross-modal generative adversarial network that introduces the contrastive learning loss function can synthesize samples of crack morphology that meet the preset conditions to complete the process of real-time detection of road defects.

[0219] Furthermore, the acquisition unit 801 includes:

[0220] The acquisition module is used to collect visual data streams through the vehicle-mounted acquisition device and obtain raw point cloud data through the millimeter wave radar. The raw point cloud data is the radar point cloud data that has not been filtered;

[0221] A filtering processing module, configured to filter the original point cloud data to obtain a plurality of voxel grids; wherein the original point cloud data at least includes reflection intensity information;

[0222] A downsampling module is used to downsample multiple voxel grids to remove noise points and redundant points and obtain evenly distributed initial point cloud data;

[0223] A first determining module is configured to determine classification information based on the reflection intensity information; wherein the classification information includes at least road surface information, vehicle information, and pedestrian information;

[0224] A screening module is used to screen out point cloud data related to the road surface based on the classification information and the initial point cloud data;

[0225] The second determination module is used to determine the point cloud data related to the road surface as the target radar point cloud data.

[0226] Furthermore, the calculation unit 802 includes:

[0227] The selection module is used to select the area points within a preset range with each point in the target radar point cloud data as the center;

[0228] The fitting module is used to fit a local plane or surface using the least squares method and domain points within a preset range;

[0229] The first calculation module is used to determine geometric parameters according to the local plane or curved surface, and calculate road curvature information according to the geometric parameters.

[0230] Furthermore, the extraction unit 803 includes:

[0231] The first extraction module is used to extract multi-scale texture features from the visual data stream through the visual branch of the deep residual network RestNet structure;

[0232] The second extraction module is used to extract depth and reflection intensity features from the target radar point cloud data by introducing the radar branch of 3D sparse convolution.

[0233] Furthermore, the alignment unit 804 includes:

[0234] A generation module for generating a weight matrix using radar-guided spatial attention and depth and reflection intensity features;

[0235] An enhancement module for enhancing the pixel response associated with the defect area with multi-scale texture features through a weight matrix;

[0236] The alignment module is used to align the semantic information of the enhanced multi-scale texture features and the semantic information of the depth and reflection intensity features in the feature space through visual-radar contrast learning.

[0237] Furthermore, the generating unit 806 includes:

[0238] A partitioning module is used to divide the road surface area into three-dimensional grids according to a preset resolution based on a Cartesian coordinate system; the three-dimensional grid includes a plurality of grid cells;

[0239] A second calculation module is configured to calculate, for each grid cell, constraint parameters of the grid cell based on the point cloud information and curvature information in the depth information set with curvature attributes falling within the grid cell; wherein the constraint parameters include at least a height offset and a curvature degree factor;

[0240] The integration generation module is used to integrate the constraint parameters into a matrix to generate a three-dimensional space constraint matrix; wherein each element of the three-dimensional space constraint matrix corresponds to the constraint information of a grid unit.

[0241] Furthermore, the defect detection unit 807 includes:

[0242] An adjustment module is selected to input the three-dimensional spatial constraint matrix into the cross-modal generative adversarial network so that the generator of the cross-modal generative adversarial network selects a vertical height according to the height offset when determining the starting position of the crack, and adjusts the crack direction according to the curvature factor when extending the crack to make it conform to the curvature change of the road surface;

[0243] An acquisition module is used to obtain the crack morphology of the pavement through the vertical height and the adjusted crack direction;

[0244] An evaluation module is used to evaluate the pavement crack morphology based on the discriminator of a cross-modal generative adversarial network that introduces a contrastive learning loss function and the adversarial mechanism of the generative adversarial network to determine whether the pavement crack morphology conforms to the physical laws prescribed by the three-dimensional spatial constraint matrix and whether it is similar to real pavement cracks;

[0245] The adversarial training module is used to perform adversarial training on the crack morphology of the evaluated pavement until the generator generates samples of crack morphology that conform to the physical laws specified by the three-dimensional spatial constraint matrix and are similar to real pavement cracks, thereby completing the process of real-time detection of pavement defects.

[0246] The beneficial effects of the embodiments of the present application are as follows: the multi-scale texture features and depth and reflection intensity features, i.e., multi-modal features, are aligned through a physical consistency cross-modal attention mechanism composed of radar-guided spatial attention and visual-radar contrastive learning, thereby improving the fusion efficiency of multi-modal features. Since visual-radar contrastive learning aligns the semantic information of multi-scale texture features and the semantic information of depth and reflection intensity features in the feature space, the deviation between modalities is reduced by maximizing the similarity of positive samples, i.e., samples of the same defect area, and minimizing the similarity of negative samples. In addition, the discriminator in the cross-modal generative adversarial network introduces a contrastive learning loss function to distinguish the cross-modal consistency between pavement-generated defects and real defects, thereby improving the accuracy of pavement defect detection. Through cross-modal data generation technology and a depth information set with curvature attributes, a three-dimensional spatial constraint matrix is ​​generated, and a high-precision detection model can be trained without the need for real defect annotation, thereby achieving zero-sample and few-sample defect detection and reducing the cost of pavement defect detection.

[0247] An embodiment of the present application further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned road surface defect detection method.

[0248] The present application also provides an electronic device, the structure of which is shown in FIG. Figure 9 As shown, it specifically includes a memory 901 and one or more instructions 902, wherein the one or more instructions 902 are stored in the memory 901 and are configured to be executed by one or more processors 903 to execute the one or more instructions 902 to perform the above-mentioned road surface defect detection method.

[0249] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0250] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For system-related embodiments, since they are generally similar to method-related embodiments, their description is relatively simple. For relevant details, refer to the description of the method-related embodiments.

[0251] The steps in the methods of the various embodiments of the present application can be adjusted in sequence, combined, and deleted according to actual needs.

[0252] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0253] The above description of the disclosed embodiments will enable those skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

[0254] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A road surface defect detection method, characterized in that: The method comprises: Acquire a visual data stream and target radar point cloud data; wherein the target radar point cloud data is point cloud data related to the road surface; Calculating road curvature information using a preset calculation method and the target radar point cloud data; Extracting multi-scale texture features from the visual data stream, and extracting depth and reflection intensity features from the target radar point cloud data; Aligning the multi-scale texture features and the depth and reflection intensity features through a cross-modal attention mechanism; Correlating the road surface curvature information with the aligned multi-scale texture features and the aligned depth and reflection intensity features to obtain a depth information set with curvature attributes; Generate a three-dimensional spatial constraint matrix based on the cross-modal data generation technology and the depth information set with curvature attributes; The three-dimensional spatial constraint matrix is ​​input into a cross-modal generative adversarial network, so that the discriminator of the cross-modal generative adversarial network that introduces a contrastive learning loss function synthesizes samples of crack morphology that meet preset conditions to complete the process of real-time detection of road defects.

2. The method according to claim 1, characterized in that The obtaining of the visual data stream and the target radar point cloud data includes: The visual data stream is collected by the vehicle-mounted acquisition device, and the raw point cloud data is obtained by the millimeter wave radar; wherein the raw point cloud data is the radar point cloud data that has not been filtered; Performing filtering on the original point cloud data to obtain a plurality of voxel grids; wherein the original point cloud data at least includes reflection intensity information; For a plurality of voxel grids, downsampling the voxel grids to remove noise points and redundant points to obtain evenly distributed initial point cloud data; Determining classification information based on the reflection intensity information; wherein the classification information includes at least road surface information, vehicle information, and pedestrian information; Filtering point cloud data related to a road surface according to the classification information and the initial point cloud data; The point cloud data related to the road surface is determined as target radar point cloud data.

3. The method according to claim 1, characterized in that Calculating the road curvature information using a preset calculation method and the target radar point cloud data includes: Taking each point in the target radar point cloud data as the center, select a field point within a preset range; Fitting a local plane or curved surface using the least squares method and the domain points within the preset range; Geometric parameters are determined according to the local plane or curved surface, and road curvature information is calculated according to the geometric parameters.

4. The method according to claim 1, wherein The extracting multi-scale texture features from the visual data stream and extracting depth and reflection intensity features from the target radar point cloud data includes: Extracting multi-scale texture features from the visual data stream through the visual branch of the deep residual network RestNet structure; By introducing the radar branch of 3D sparse convolution, depth and reflection intensity features are extracted from the target radar point cloud data.

5. The method according to claim 1, wherein The aligning of the multi-scale texture features and the depth and reflection intensity features through a cross-modal attention mechanism includes: generating a weight matrix by guiding spatial attention through radar and the depth and reflection intensity features; Enhance the pixel response of the multi-scale texture feature related to the defect area by using the weight matrix; In the feature space, the semantic information of the enhanced multi-scale texture features and the semantic information of the depth and reflection intensity features are aligned through vision-radar contrastive learning.

6. The method according to claim 1, wherein The generating of a three-dimensional space constraint matrix based on the cross-modal data generation technology and the depth information set with curvature attributes includes: Based on a Cartesian coordinate system, the road surface area is divided into a three-dimensional grid according to a preset resolution; the three-dimensional grid includes a plurality of grid cells; For each grid cell, calculating constraint parameters of the grid cell based on point cloud information and curvature information in the depth information set with curvature attributes falling within the grid cell; wherein the constraint parameters include at least a height offset and a curvature degree factor; The constraint parameters are integrated into a matrix to generate a three-dimensional space constraint matrix; wherein each element of the three-dimensional space constraint matrix corresponds to the constraint information of a grid unit.

7. The method according to claim 6, characterized in that The process of inputting the three-dimensional spatial constraint matrix into a cross-modal generative adversarial network, and enabling a discriminator of the cross-modal generative adversarial network that introduces a contrastive learning loss function to synthesize samples of crack morphology that meet preset conditions, thereby completing real-time detection of road surface defects, includes: Inputting the three-dimensional spatial constraint matrix into a cross-modal generative adversarial network, so that a generator of the cross-modal generative adversarial network selects a vertical height according to the height offset when determining a crack starting location, and adjusts the crack direction according to the curvature factor when extending the crack to conform to the curvature change of the road surface; Obtaining the crack morphology of the road surface through the vertical height and the adjusted crack direction; The crack morphology of the pavement is evaluated using a discriminator of a cross-modal generative adversarial network that introduces a contrastive learning loss function and an adversarial mechanism of the generative adversarial network to determine whether the crack morphology of the pavement conforms to the physical laws prescribed by the three-dimensional spatial constraint matrix and whether it is similar to real pavement cracks. The crack morphology of the evaluated pavement is subjected to adversarial training until the generator generates samples of crack morphology that conform to the physical laws specified by the three-dimensional spatial constraint matrix and are similar to real pavement cracks, thereby completing the process of real-time detection of pavement defects.

8. A road surface defect detection device, characterized in that: The device comprises: An acquisition unit, configured to acquire a visual data stream and target radar point cloud data; wherein the target radar point cloud data is point cloud data related to a road surface; a calculation unit, configured to calculate road curvature information using a preset calculation method and the target radar point cloud data; an extraction unit, configured to extract multi-scale texture features from the visual data stream, and extract depth and reflection intensity features from the target radar point cloud data; an alignment unit, configured to align the multi-scale texture features and the depth and reflection intensity features through a cross-modal attention mechanism; an associating unit, configured to associate the road surface curvature information with the aligned multi-scale texture features and the aligned depth and reflection intensity features, respectively, to obtain a depth information set with curvature attributes; A generating unit, configured to generate a three-dimensional spatial constraint matrix based on a cross-modal data generation technique and the depth information set with curvature attributes; The defect detection unit is used to input the three-dimensional spatial constraint matrix into a cross-modal generative adversarial network, so that the discriminator of the cross-modal generative adversarial network that introduces a contrastive learning loss function synthesizes samples of crack morphology that meet preset conditions to complete the process of real-time detection of road defects.

9. A storage medium, characterized in that: The storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the road surface defect detection method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The system comprises a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by one or more processors to execute the road surface defect detection method according to any one of claims 1 to 7.

Images