Road surface disease identification method and device based on three-dimensional image and electronic equipment

Abstract

The application relates to the technical field of disease identification, and provides a road surface disease identification method and device based on three-dimensional images and electronic equipment. The method comprises the following steps: performing pretreatment correction on road surface precision three-dimensional image data to obtain corrected image data; inputting the corrected image data into a disease target extraction model to obtain disease target information; inputting the disease target information into a disease target classification model to obtain disease category information; the disease target extraction model and the disease target classification model are obtained by jointly training a combination of multiple base models based on an original labeled sample set and an expanded labeled sample set formed based on historical corrected image data; and the road surface precision three-dimensional image data comprises road surface grayscale image data and road surface depth image data. The two-stage artificial intelligence model obtained by introducing the road surface precision three-dimensional image data and the joint training can effectively improve the identification accuracy of road surface diseases.

Description

Technical Field

[0001] This application relates to the field of road surface defect identification technology, specifically to a method, device, and electronic device for identifying road surface defects based on three-dimensional images. Background Technology

[0002] As highway service life increases and transportation pressure intensifies, various road surface defects arise, leading to a significant reduction in performance and service life. This results in a substantial increase in the amount of maintenance work required, and scientific maintenance decisions depend on the accurate identification of road surface defects.

[0003] Existing methods for identifying pavement defects still rely on two-dimensional image recognition technology. On the one hand, this technology cannot obtain comprehensive information about pavement defects. On the other hand, this technology is generally based on relatively simple image processing and feature engineering to identify pavement defects, resulting in low accuracy in identifying pavement defects. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for identifying road surface defects based on three-dimensional images, in order to solve the technical problem that existing road surface defect identification methods still use two-dimensional image recognition technology, resulting in low accuracy in identifying road surface defects.

[0005] In a first aspect, embodiments of this application provide a method for identifying pavement defects based on three-dimensional images, including: Preprocessing and correction are performed on the precise three-dimensional image data of the road surface to obtain the corrected image data; The corrected image data is input into the disease target extraction model to obtain the disease target information output by the disease target extraction model; The disease target information is input into the disease target classification model to obtain the disease category information output by the disease target classification model; The disease target extraction model and the disease target classification model are obtained by jointly training a combination of multiple base models based on the original labeled sample set and the expanded labeled sample set formed by historical corrected image data. The precise three-dimensional image data of the road surface includes grayscale image data of the road surface and depth image data of the road surface.

[0006] In one embodiment, the original labeled sample set and the expanded labeled sample set are obtained based on the following method: Select typical disease category image data from the historical corrected image data; The image data of the typical disease categories are labeled to obtain the original labeled sample set; wherein, the labeling information includes disease area, disease category, pavement category, road grade and road area; The original labeled sample set is classified according to the disease category to obtain multiple category labeled sample sets; If the number of samples in any category-labeled sample set is less than a preset number, the disease area is expanded on the samples in the category-labeled sample set to obtain expanded disease area samples; the expansion includes stretching, rotation, flipping, translation and superposition. Simulate and generate road surface texture background image data for different road surface types, road grades, and road areas; The expanded disease area samples are superimposed on the road surface texture background image data to obtain an expanded labeled sample set.

[0007] In one embodiment, the step of filtering typical disease category image data from the historically corrected image data includes: The historical corrected image data is pre-classified using manual recognition methods or a pre-trained unsupervised model to obtain the pre-classification result of the historical corrected image data; Based on the pre-classification results, typical disease category image data are selected from the historical corrected image data.

[0008] In one embodiment, each base model combination includes a target extraction base model and a target classification base model; the disease target extraction model and the disease target classification model are obtained based on the following method: For each combination of base models, the original labeled sample set and the expanded labeled sample set are used as training sample sets and input together with the disease knowledge base into the combination of base models to obtain the disease target information and disease category information output by the combination of base models; If the joint loss of the disease target information and the disease category information fails to converge, after adjusting the parameters of the target extraction base model and / or the target classification base model, return to the step of using the original labeled sample set and the expanded labeled sample set as training sample sets, and inputting them together with the disease knowledge base into the combination of the base models, until the number of training times is reached or the joint loss converges, and obtain the candidate target extraction model and candidate target classification model corresponding to the combination of the base models; Based on the road surface category, the road grade, and / or the road region, the optimal target extraction model and the optimal target classification model are selected from the candidate target extraction models and candidate target classification models corresponding to the combination of the multiple base models; The optimal target extraction model is determined as the disease target extraction model, and the optimal target classification model is determined as the disease target classification model.

[0009] In one embodiment, the preprocessing and correction of the precise three-dimensional image data of the road surface to obtain corrected image data includes: The grayscale image data of the road surface is subjected to brightness correction to obtain corrected grayscale image data; The road surface depth image data is subjected to attitude correction to obtain corrected three-dimensional image data.

[0010] In one embodiment, performing brightness correction on the road surface grayscale image data to obtain corrected grayscale image data includes: Acquire historical road surface grayscale image data where the road surface grayscale image data is located within a preset range; The grayscale correction coefficient of the road surface grayscale image data is obtained by calculating the ratio of the mean grayscale value of the historical road surface grayscale image data to the preset target grayscale value. Based on the grayscale correction coefficient, the brightness of the road surface grayscale image data is corrected to obtain the corrected grayscale image data.

[0011] In one embodiment, the step of performing attitude correction on the road surface depth image data to obtain corrected three-dimensional image data includes: Based on the measurement attitude compensation method, the attitude of the road surface depth image data is corrected to obtain corrected depth image data; The corrected depth image data is then rendered in three dimensions to obtain corrected three-dimensional image data.

[0012] In one embodiment, the typical disease category image data includes typical disease category grayscale image data and typical disease category three-dimensional image data; The annotation of the image data of the typical disease categories includes: Annotate the grayscale image data and / or three-dimensional image data of the typical disease categories; or The fused image data of the grayscale image data and the three-dimensional image data of the typical disease categories are labeled; Specifically, for either the grayscale image data or the three-dimensional image data of the typical disease category, the annotation information in the image data is synchronously updated to the other image data.

[0013] Secondly, embodiments of this application provide a pavement defect identification device based on three-dimensional images, comprising: The image correction module is used to: preprocess and correct the precise three-dimensional image data of the road surface to obtain corrected image data; The target extraction module is used to: input the corrected image data into the disease target extraction model to obtain the disease target information output by the disease target extraction model; The target classification module is used to: input the disease target information into the disease target classification model to obtain the disease category information output by the disease target classification model; The disease target extraction model and the disease target classification model are obtained by jointly training a combination of multiple base models based on the original labeled sample set and the expanded labeled sample set formed by historical corrected image data. The precise three-dimensional image data of the road surface includes grayscale image data of the road surface and depth image data of the road surface.

[0014] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the road surface defect identification method based on three-dimensional images described in the first aspect.

[0015] The method, apparatus, and electronic equipment for pavement defect identification based on 3D images provided in this application preprocess and correct precise 3D pavement image data to obtain corrected image data. The corrected image data is then input into a defect target extraction model to obtain defect target information output by the defect target extraction model. The defect target information is then input into a defect target classification model to obtain defect category information output by the defect target classification model. The defect target extraction model and the defect target classification model are obtained by jointly training a combination of multiple base models based on an original labeled sample set and an expanded labeled sample set formed from historical corrected image data. The precise 3D pavement image data includes pavement grayscale image data and pavement depth image data. This application uses precise 3D road surface images as the basis for road surface defect identification. After preliminary preprocessing and correction, the images are input into a two-stage defect model. The first stage uses a defect target extraction model to extract defect target information, and the second stage uses a defect target classification model to identify the defect category. On the one hand, the precise 3D road surface image data includes road surface grayscale image data and road surface depth image data, adding important depth information to the traditional 2D image, thus enabling the acquisition of more comprehensive information about road surface defects. On the other hand, the two-stage model is trained based on the original labeled sample set and the expanded labeled sample set formed by historical corrected image data, introducing an artificial intelligence model into the vertical field of road surface defect identification. Through joint training of the two stages, the adaptability between the two stages and the adaptability for road surface defect identification are effectively improved. In summary, this application, by introducing precise 3D road surface image data and a jointly trained two-stage artificial intelligence model, can effectively improve the accuracy of road surface defect identification. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is one of the flowcharts illustrating the method for identifying road surface defects based on three-dimensional images provided in this application embodiment; Figure 2 This is the second flowchart illustrating the method for identifying road surface defects based on three-dimensional images provided in this application embodiment; Figure 3 This is the third flowchart illustrating the method for identifying road surface defects based on three-dimensional images provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of the road surface defect identification device based on three-dimensional images provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that in the description of the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, or integral connections; they can be mechanical connections or electrical connections; they can be direct connections or indirect connections through an intermediate medium; and they can be internal connections between two elements. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0020] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0021] Figure 1 This is one of the flowcharts illustrating the pavement defect identification method based on three-dimensional images provided in this application. (Refer to...) Figure 1 This application provides a method for identifying road surface defects based on three-dimensional images, which may include: Step 101: Preprocess and correct the precise three-dimensional image data of the road surface to obtain corrected image data; Step 102: Input the corrected image data into the disease target extraction model to obtain the disease target information output by the disease target extraction model; Step 103: Input the disease target information into the disease target classification model to obtain the disease category information output by the disease target classification model.

[0022] The disease target extraction model and the disease target classification model are obtained by jointly training a combination of multiple base models based on the original labeled sample set and the expanded labeled sample set formed by historical corrected image data. The precise three-dimensional image data of the road surface includes grayscale image data and depth image data, which can be acquired by a line-scanning three-dimensional device. This line-scanning three-dimensional device can include a laser and a high-frequency three-dimensional camera. The laser is used to transmit line laser along the width of the road surface, and the high-frequency three-dimensional camera is used to acquire grayscale image data and depth image data of the road surface in the area covered by the line laser.

[0023] In step 101, since the precise three-dimensional image data of the road surface includes road surface grayscale image data and road surface depth image data, the road surface grayscale image data and road surface depth image data can be preprocessed and corrected separately to initially smooth some abnormal data and avoid introducing data noise and interference in subsequent processing.

[0024] In steps 102 to 103, since the disease target extraction model and the disease target classification model are obtained by jointly training a combination of multiple base models based on the original labeled sample set and the expanded labeled sample set formed by historically corrected image data, on the one hand, the data of the training samples can be enhanced by expanding the labeled sample set on the basis of the original labeled sample set, thereby improving the joint training effect of the base model combination. On the other hand, based on the joint training results of the combination of multiple base models, the combination with the best disease identification effect can be selected as the final two-stage model, thereby realizing the accurate extraction of disease target information and the accurate identification of disease category information.

[0025] The pavement defect identification method based on three-dimensional images provided in this embodiment preprocesses and corrects precise three-dimensional pavement image data to obtain corrected image data. The corrected image data is then input into a defect target extraction model to obtain defect target information output by the defect target extraction model. The defect target information is then input into a defect target classification model to obtain defect category information output by the defect target classification model. The defect target extraction model and the defect target classification model are obtained by jointly training a combination of multiple base models based on the original labeled sample set and the expanded labeled sample set formed by historical corrected image data. The precise three-dimensional pavement image data includes pavement grayscale image data and pavement depth image data. This embodiment uses precise 3D road surface images as the basis for road surface defect identification. After preliminary preprocessing and correction, the images are input into a two-stage defect model. The first stage uses a defect target extraction model to extract defect target information, and the second stage uses a defect target classification model to identify the defect category. On the one hand, the precise 3D road surface image data includes road surface grayscale image data and road surface depth image data, adding important depth information to the traditional 2D image, thus enabling the acquisition of more comprehensive information about road surface defects. On the other hand, the two-stage model is trained based on the original labeled sample set and the expanded labeled sample set formed from historical corrected image data, introducing an artificial intelligence model into the vertical field of road surface defect identification. Through joint training of the two stages, the adaptability between the two-stage models and their adaptability for road surface defect identification are effectively improved. In summary, this embodiment, by introducing precise 3D road surface image data and a jointly trained two-stage artificial intelligence model, can effectively improve the accuracy of road surface defect identification.

[0026] Figure 2 This is the second schematic flowchart of the pavement defect identification method based on three-dimensional images provided in this application. (Refer to...) Figure 2 In one embodiment, the original labeled sample set and the expanded labeled sample set can be obtained based on the following method: Step 201: Select typical disease category image data from the historical corrected image data; Step 202: Annotate the image data of typical disease categories to obtain the original annotated sample set; The labeling information includes the affected area, the type of affected area, the pavement type, the road grade, and the road area. Step 203: Classify the original labeled sample set according to the disease category to obtain multiple category labeled sample sets; Step 204: If the number of samples in any category of labeled sample set is less than the preset number, expand the disease area of ​​the samples in that category of labeled sample set to obtain expanded disease area samples. Extensions include stretching, rotating, flipping, translating, and stacking; Step 205: Simulate and generate road surface texture background image data for different road surface categories, different road grades, and different road areas; Step 206: Overlay the extended disease area samples onto the road surface texture background image data to obtain an expanded labeled sample set.

[0027] In steps 201 to 202, manual identification methods or pre-trained unsupervised models can be used to pre-classify the historical corrected image data to obtain the pre-classification results of the historical corrected image data. Then, based on the pre-classification results, typical disease category image data can be selected from the historical corrected image data and labeled, which can effectively improve the labeling efficiency and quickly obtain the original labeled sample set.

[0028] In step 203, since the annotation information includes the disease area, disease category, pavement category, road grade, and road area, the original annotation sample set can be classified according to the disease category based on this annotation information.

[0029] In step 204, since the number of disease samples of various types in the original labeled sample set is not balanced, the model will focus on learning the disease sample categories with a large number of samples, reducing the recognition accuracy of disease sample categories with a small number of samples. Therefore, it is necessary to classify the disease samples in the original labeled sample set and expand the samples of some disease categories with a small number of samples. First, the disease regions of these disease samples can be subjected to operations such as stretching, rotating, flipping, translating and superimposing, so that the disease information of the original disease region can be expanded to a larger scale, thereby obtaining the expanded disease region.

[0030] In step 205, furthermore, artificial intelligence algorithms can be used to simulate and generate various road surface texture background image data for different road surface categories, different road grades, and different road areas. Each type of road surface texture background image data includes road surface texture background grayscale image data and road surface texture background three-dimensional image data.

[0031] In step 206, the extended disease area samples obtained in step 204 are superimposed on the various road surface texture background image data generated in step 205 to obtain a smaller number of extended samples of disease sample categories in the various road surface texture background image data, i.e., an expanded labeled sample set.

[0032] This embodiment quickly obtains an original labeled sample set by selecting typical disease category image data from historical corrected image data and labeling them. Then, the samples in the original labeled sample set are classified, and for categories with fewer samples, disease areas are expanded, multiple background image data are generated, and disease areas and background images are superimposed to expand the number of samples in that category. Effective data augmentation is performed on the basis of the original labeled sample set, which helps the model learn various categories of disease samples and improves the model's recognition accuracy.

[0033] Figure 3 This is the third schematic flowchart of the pavement distress identification method based on three-dimensional images provided in this application. (Refer to...) Figure 3 In one embodiment, each base model combination includes a target extraction base model and a target classification base model; the disease target extraction model and the disease target classification model can be obtained based on the following: Step 301: For each combination of base models, the original labeled sample set and the expanded labeled sample set are used as training sample sets and input together with the disease knowledge base into the combination of base models to obtain the disease target information and disease category information output by the combination of base models. Step 302: If the joint loss of disease target information and disease category information has not converged, adjust the parameters of the target extraction base model and / or the target classification base model, and then return to step 301. Step 303: When the number of training iterations or the joint loss converges, obtain the candidate target extraction model and the candidate target classification model corresponding to the combination of the base models; Step 304: Based on road surface category, road grade and / or road area, select the optimal target extraction model and the optimal target classification model from the candidate target extraction model and candidate target classification model corresponding to the combination of multiple base models; Step 305: Determine the optimal target extraction model as the disease target extraction model and the optimal target classification model as the disease target classification model.

[0034] In step 301, the pavement disease knowledge base can include multi-dimensional and multi-modal knowledge data such as text, images, and 3D models of pavement diseases. When constructing it, it is necessary to refer to industry standards and national standards, and combine them with road maintenance needs to semantically define various types of pavement diseases and quantitatively define the parameters for the identification of various types of pavement diseases. For example, pavement subsidence is defined as local subsidence of the pavement, and pavement potholes are defined as local depressions or damages of the pavement with an area greater than 0.1 square meters and a depth greater than 25 millimeters. Inputting the disease knowledge base and the training sample set into the base model combination can enable each base model combination to learn rich disease knowledge.

[0035] In steps 302 to 303, the joint loss of the base model combination output is obtained in each training session to measure the performance of the base model combination. If the joint loss does not converge, the parameters of at least one base model in the combination are adjusted, and training is continued until the number of training sessions or the joint loss converges. The base model combination is considered to have achieved the best training effect, and the candidate target extraction model and candidate target classification model corresponding to the base model combination at this time are obtained.

[0036] In steps 304 and 305, after training each base model combination is completed, the optimal target extraction model and the optimal target classification model can be selected from the candidate target extraction models and candidate target classification models corresponding to all base model combinations based on at least one dimension of road surface category, road grade, and road region. The optimal target extraction model and the optimal target classification model are the model combinations with the highest comprehensive recognition accuracy in at least one dimension. Then, the optimal target extraction model can be determined as the final disease target extraction model, and the optimal target classification model can be determined as the final disease target classification model. The disease target extraction model can accurately output the position of the disease target in its corresponding corrected grayscale image data, the position in the corrected three-dimensional image data, and the coverage area data in these two types of corrected image data. The disease target classification model can accurately output the category of the disease target.

[0037] This embodiment trains each of the multiple base model combinations, compares the training effects of the multiple base model combinations, and selects the base model combination with the best training effect in the road surface area, road grade, and at least one dimension of the road area that needs to be identified for disease identification. Then, the target extraction model corresponding to the base model combination is determined as the final disease target extraction model, and the target classification model corresponding to the base model combination is determined as the final disease target classification model. This achieves the highest comprehensive recognition accuracy of the two-stage model composed of the disease target extraction model and the disease target classification model.

[0038] In one embodiment, step 101 may include: 1. Perform brightness correction on the grayscale image data of the road surface to obtain the corrected grayscale image data; Since grayscale image data of road surfaces is easily affected by brightness errors caused by uneven laser line energy or ambient light, brightness correction is necessary. This correction can include: Step 1a: Obtain historical road surface grayscale image data within a preset range for the location of the road surface grayscale image data; Step 1b: Calculate the ratio of the mean gray value of the historical road surface grayscale image data to the preset target grayscale value to obtain the grayscale correction coefficient of the road surface grayscale image data; Step 1c: Based on the grayscale correction coefficient, perform brightness correction on the road surface grayscale image data to obtain the corrected grayscale image data. In step 1a, since the brightness error is consistent in the same column of the road surface grayscale image data but inconsistent in different columns, brightness correction can be performed column by column. For each column of grayscale image data in the road surface grayscale image data, taking the road surface position of the grayscale image data in that column as a reference, historical grayscale image data of all pixels within a preset road surface interval range are obtained along the column direction. The preset road surface interval range can be set according to actual needs and is not limited here. For example, it can be set to the interval range between a certain position before and after the road surface position of the grayscale image data in that column, the interval range between the road surface position of the grayscale image data in that column and a certain position before it, or the interval range between the road surface position of the grayscale image data in that column and a certain position after it, etc. Here, the column direction refers to the road driving direction.

[0039] In step 1b, the ratio of the mean of all historical grayscale image data of all pixels obtained in step 1a to the preset target grayscale value is calculated to obtain the column grayscale correction coefficient of the column grayscale image data.

[0040] In step 1c, based on the grayscale correction coefficient of that column, the brightness of the grayscale image data of each pixel in the column of grayscale image data is corrected to obtain the corrected grayscale image data corresponding to that column of grayscale image data.

[0041] The above steps are performed on each column of grayscale image data in the road surface grayscale image data to obtain the corrected grayscale image data corresponding to the road surface grayscale image data.

[0042] 2. Perform attitude correction on the road surface depth image data to obtain corrected 3D image data.

[0043] Since road surface depth image data is primarily susceptible to attitude errors of the measuring equipment, attitude correction is necessary, which may include: Step 2a: Based on the measurement attitude compensation method, perform attitude correction on the road surface depth image data to obtain corrected depth image data; Step 2b: Perform 3D rendering on the corrected depth image data to obtain the corrected 3D image data.

[0044] In step 2a, since the attitude error is consistent within the same row of the road surface depth image data but inconsistent across different rows, attitude correction can be performed row by row. For each row of depth image data in the road surface depth image data, the installation parameters of the line-scan 3D device and the measurement attitude corresponding to that row of depth image data are obtained. Based on the installation parameters and the measurement attitude, attitude compensation is performed on that row of depth image data to obtain the corrected depth image data corresponding to that row of depth image data. Here, the row direction refers to the road width direction.

[0045] The above steps are performed on each row of depth image data in the road surface depth image data to obtain the corrected depth image data corresponding to the road surface depth image data.

[0046] In step 2b, preset 3D rendering parameters, such as depth scaling ratio and depth translation parameters, can be used to perform 3D rendering on the corrected depth image data to make the 3D features of the corrected depth image data more prominent. The resulting corrected 3D image data is more conducive to subsequent disease extraction and classification.

[0047] This embodiment addresses different error types that affect road surface grayscale image data and road surface depth image data, employing appropriate error correction methods. For road surface grayscale image data, brightness correction is applied to column data, resulting in more uniform brightness among pixels in the corrected grayscale image data within the same column. Furthermore, since each column of grayscale image data has a column grayscale correction coefficient specific to its own brightness error, it also ensures more uniform brightness among pixels in grayscale image data from different columns after correction. For road surface depth image data, pose correction is applied to row data, eliminating pose errors in the road surface depth image data. This is followed by 3D rendering to highlight its 3D features, making subsequent defect extraction and classification more efficient.

[0048] In one embodiment, the typical disease category image data includes typical disease category grayscale image data and typical disease category three-dimensional image data; step 202 may include: Annotate grayscale image data and / or 3D image data of typical disease categories; or Annotate the fused image data of grayscale image data and three-dimensional image data of typical disease categories; Specifically, for either grayscale image data or 3D image data of typical disease categories, the annotation information in the image data is synchronously updated to the other image data.

[0049] This embodiment features four image data annotation methods: 1. Label the grayscale image data of typical disease categories and synchronize the labeling information to the three-dimensional image data of typical disease categories; 2. Label the 3D image data of typical disease categories and synchronize the labeling information to the grayscale image data of typical disease categories; 3. Label the grayscale image data and three-dimensional image data of typical disease categories respectively; 4. Fuse the grayscale image data and three-dimensional image data of typical disease categories to obtain fused image data, and then label the fused image data.

[0050] This embodiment provides diverse annotation methods for typical disease category image data, which can be flexibly selected according to actual needs. Among them, methods 1 and 2 have higher annotation efficiency, while method 3 can maintain the independence of annotation between image data and avoid the deviation in the annotation of one type of image data being synchronized to the other type of image data. Method 4 is more suitable for annotation of two types of image data in fusion scenarios.

[0051] The pavement defect identification device based on three-dimensional images provided in the embodiments of this application will be described below. The pavement defect identification device based on three-dimensional images described below can be referred to in correspondence with the pavement defect identification method based on three-dimensional images described above.

[0052] Figure 4 This is a schematic diagram of the pavement defect identification device based on three-dimensional images provided in an embodiment of this application. (Refer to...) Figure 4 This application provides a pavement defect identification device based on three-dimensional images, which may include: Image correction module 401 is used to: preprocess and correct the precise three-dimensional image data of the road surface to obtain corrected image data; The target extraction module 402 is used to: input the corrected image data into the disease target extraction model to obtain the disease target information output by the disease target extraction model; The target classification module 403 is used to: input the disease target information into the disease target classification model to obtain the disease category information output by the disease target classification model; The disease target extraction model and the disease target classification model are obtained by jointly training a combination of multiple base models based on the original labeled sample set and the expanded labeled sample set formed by historical corrected image data. The precise three-dimensional image data of the road surface includes grayscale image data of the road surface and depth image data of the road surface.

[0053] The pavement defect identification device based on three-dimensional images provided in this embodiment preprocesses and corrects precise three-dimensional pavement image data to obtain corrected image data. The corrected image data is then input into a defect target extraction model to obtain defect target information output by the defect target extraction model. The defect target information is then input into a defect target classification model to obtain defect category information output by the defect target classification model. The defect target extraction model and the defect target classification model are obtained by jointly training a combination of multiple base models based on the original labeled sample set and the expanded labeled sample set formed by historical corrected image data. The precise three-dimensional pavement image data includes pavement grayscale image data and pavement depth image data. This embodiment uses precise 3D road surface images as the basis for road surface defect identification. After preliminary preprocessing and correction, the images are input into a two-stage defect model. The first stage uses a defect target extraction model to extract defect target information, and the second stage uses a defect target classification model to identify the defect category. On the one hand, the precise 3D road surface image data includes road surface grayscale image data and road surface depth image data, adding important depth information to the traditional 2D image, thus enabling the acquisition of more comprehensive information about road surface defects. On the other hand, the two-stage model is trained based on the original labeled sample set and the expanded labeled sample set formed from historical corrected image data, introducing an artificial intelligence model into the vertical field of road surface defect identification. Through joint training of the two stages, the adaptability between the two-stage models and their adaptability for road surface defect identification are effectively improved. In summary, this embodiment, by introducing precise 3D road surface image data and a jointly trained two-stage artificial intelligence model, can effectively improve the accuracy of road surface defect identification.

[0054] In one embodiment, a sample set acquisition module (not shown in the figure) is further included, for: Select typical disease category image data from the historical corrected image data; The image data of the typical disease categories are labeled to obtain the original labeled sample set; wherein, the labeling information includes disease area, disease category, pavement category, road grade and road area; The original labeled sample set is classified according to the disease category to obtain multiple category labeled sample sets; If the number of samples in any category-labeled sample set is less than a preset number, the disease area is expanded on the samples in the category-labeled sample set to obtain expanded disease area samples; the expansion includes stretching, rotation, flipping, translation and superposition. Simulate and generate road surface texture background image data for different road surface types, road grades, and road areas; The expanded disease area samples are superimposed on the road surface texture background image data to obtain an expanded labeled sample set.

[0055] In one embodiment, the sample set acquisition module is specifically used for: The historical corrected image data is pre-classified using manual recognition methods or a pre-trained unsupervised model to obtain the pre-classification result of the historical corrected image data; Based on the pre-classification results, typical disease category image data are selected from the historical corrected image data.

[0056] In one embodiment, each base model combination includes an object extraction base model and an object classification base model; it also includes a model training module (not shown in the figure) for: For each combination of base models, the original labeled sample set and the expanded labeled sample set are used as training sample sets and input together with the disease knowledge base into the combination of base models to obtain the disease target information and disease category information output by the combination of base models; If the joint loss of the disease target information and the disease category information fails to converge, after adjusting the parameters of the target extraction base model and / or the target classification base model, return to the step of using the original labeled sample set and the expanded labeled sample set as training sample sets, and inputting them together with the disease knowledge base into the combination of the base models, until the number of training times is reached or the joint loss converges, and obtain the candidate target extraction model and candidate target classification model corresponding to the combination of the base models; Based on the road surface category, the road grade, and / or the road region, the optimal target extraction model and the optimal target classification model are selected from the candidate target extraction models and candidate target classification models corresponding to the combination of the multiple base models; The optimal target extraction model is determined as the disease target extraction model, and the optimal target classification model is determined as the disease target classification model.

[0057] In one embodiment, the image correction module 401 is specifically used for: The grayscale image data of the road surface is subjected to brightness correction to obtain corrected grayscale image data; The road surface depth image data is subjected to attitude correction to obtain corrected three-dimensional image data.

[0058] In one embodiment, the image correction module 401 is specifically used for: Acquire historical road surface grayscale image data where the road surface grayscale image data is located within a preset range; The grayscale correction coefficient of the road surface grayscale image data is obtained by calculating the ratio of the mean grayscale value of the historical road surface grayscale image data to the preset target grayscale value. Based on the grayscale correction coefficient, the brightness of the road surface grayscale image data is corrected to obtain the corrected grayscale image data.

[0059] In one embodiment, the image correction module 401 is specifically used for: Based on the measurement attitude compensation method, the attitude of the road surface depth image data is corrected to obtain corrected depth image data; The corrected depth image data is then rendered in three dimensions to obtain corrected three-dimensional image data.

[0060] In one embodiment, the typical disease category image data includes typical disease category grayscale image data and typical disease category three-dimensional image data; the sample set acquisition module is specifically used for: Annotate the grayscale image data and / or three-dimensional image data of the typical disease categories; or The fused image data of the grayscale image data and the three-dimensional image data of the typical disease categories are labeled; Specifically, for either the grayscale image data or the three-dimensional image data of the typical disease category, the annotation information in the image data is synchronously updated to the other image data.

[0061] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call a computer program in the memory 530 to execute the steps of a road surface defect identification method based on three-dimensional images, such as: Preprocessing and correction are performed on the precise three-dimensional image data of the road surface to obtain the corrected image data; The corrected image data is input into the disease target extraction model to obtain the disease target information output by the disease target extraction model; The disease target information is input into the disease target classification model to obtain the disease category information output by the disease target classification model; The disease target extraction model and the disease target classification model are obtained by jointly training a combination of multiple base models based on the original labeled sample set and the expanded labeled sample set formed by historical corrected image data. The precise three-dimensional image data of the road surface includes grayscale image data of the road surface and depth image data of the road surface.

[0062] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0063] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the road surface defect identification method based on three-dimensional images provided in the above embodiments, such as including: Preprocessing and correction are performed on the precise three-dimensional image data of the road surface to obtain the corrected image data; The corrected image data is input into the disease target extraction model to obtain the disease target information output by the disease target extraction model; The disease target information is input into the disease target classification model to obtain the disease category information output by the disease target classification model; The disease target extraction model and the disease target classification model are obtained by jointly training a combination of multiple base models based on the original labeled sample set and the expanded labeled sample set formed by historical corrected image data. The precise three-dimensional image data of the road surface includes grayscale image data of the road surface and depth image data of the road surface.

[0064] On the other hand, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program thereon, the computer program being used to cause a processor to execute the steps of the road surface defect identification method based on three-dimensional images provided in the above embodiments, for example including: Preprocessing and correction are performed on the precise three-dimensional image data of the road surface to obtain the corrected image data; The corrected image data is input into the disease target extraction model to obtain the disease target information output by the disease target extraction model; The disease target information is input into the disease target classification model to obtain the disease category information output by the disease target classification model; The disease target extraction model and the disease target classification model are obtained by jointly training a combination of multiple base models based on the original labeled sample set and the expanded labeled sample set formed by historical corrected image data. The precise three-dimensional image data of the road surface includes grayscale image data of the road surface and depth image data of the road surface.

[0065] The non-transitory computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0066] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0067] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for identifying road surface defects based on three-dimensional images, characterized in that, include: Preprocessing and correction are performed on the precise three-dimensional image data of the road surface to obtain the corrected image data; The corrected image data is input into the disease target extraction model to obtain the disease target information output by the disease target extraction model; The disease target information is input into the disease target classification model to obtain the disease category information output by the disease target classification model; The disease target extraction model and the disease target classification model are obtained by jointly training a combination of multiple base models based on the original labeled sample set and the expanded labeled sample set formed by historical corrected image data. The precise three-dimensional image data of the road surface includes grayscale image data of the road surface and depth image data of the road surface.

2. The method for identifying road surface defects based on three-dimensional images according to claim 1, characterized in that, The original labeled sample set and the expanded labeled sample set were obtained based on the following method: Select typical disease category image data from the historical corrected image data; The image data of the typical disease categories are labeled to obtain the original labeled sample set; wherein, the labeling information includes disease area, disease category, pavement category, road grade and road area; The original labeled sample set is classified according to the disease category to obtain multiple category labeled sample sets; If the number of samples in any category-labeled sample set is less than a preset number, the disease area is expanded on the samples in the category-labeled sample set to obtain expanded disease area samples; the expansion includes stretching, rotation, flipping, translation and superposition. Simulate and generate road surface texture background image data for different road surface types, road grades, and road areas; The expanded disease area samples are superimposed on the road surface texture background image data to obtain an expanded labeled sample set.

3. The method for identifying road surface defects based on three-dimensional images according to claim 2, characterized in that, The step of filtering typical disease category image data from the historical corrected image data includes: The historical corrected image data is pre-classified using manual recognition methods or a pre-trained unsupervised model to obtain the pre-classification result of the historical corrected image data; Based on the pre-classification results, typical disease category image data are selected from the historical corrected image data.

4. The method for identifying road surface defects based on three-dimensional images according to claim 2, characterized in that, Each base model combination includes a target extraction base model and a target classification base model; the disease target extraction model and the disease target classification model are obtained based on the following method: For each combination of base models, the original labeled sample set and the expanded labeled sample set are used as training sample sets and input together with the disease knowledge base into the combination of base models to obtain the disease target information and disease category information output by the combination of base models; If the joint loss of the disease target information and the disease category information fails to converge, after adjusting the parameters of the target extraction base model and / or the target classification base model, return to the step of using the original labeled sample set and the expanded labeled sample set as training sample sets, and inputting them together with the disease knowledge base into the combination of the base models, until the number of training times is reached or the joint loss converges, and obtain the candidate target extraction model and candidate target classification model corresponding to the combination of the base models; Based on the road surface category, the road grade, and / or the road region, the optimal target extraction model and the optimal target classification model are selected from the candidate target extraction models and candidate target classification models corresponding to the combination of the multiple base models; The optimal target extraction model is determined as the disease target extraction model, and the optimal target classification model is determined as the disease target classification model.

5. The method for identifying road surface defects based on three-dimensional images according to claim 1, characterized in that, The preprocessing and correction of the precise three-dimensional image data of the road surface to obtain corrected image data includes: The grayscale image data of the road surface is subjected to brightness correction to obtain corrected grayscale image data; The road surface depth image data is subjected to attitude correction to obtain corrected three-dimensional image data.

6. The method for identifying pavement defects based on three-dimensional images according to claim 5, characterized in that, The step of performing brightness correction on the road surface grayscale image data to obtain corrected grayscale image data includes: Acquire historical road surface grayscale image data where the road surface grayscale image data is located within a preset range; The grayscale correction coefficient of the road surface grayscale image data is obtained by calculating the ratio of the mean grayscale value of the historical road surface grayscale image data to the preset target grayscale value. Based on the grayscale correction coefficient, the brightness of the road surface grayscale image data is corrected to obtain the corrected grayscale image data.

7. The method for identifying road surface defects based on three-dimensional images according to claim 5, characterized in that, The process of performing attitude correction on the road surface depth image data to obtain corrected three-dimensional image data includes: Based on the measurement attitude compensation method, the attitude of the road surface depth image data is corrected to obtain corrected depth image data; The corrected depth image data is then rendered in three dimensions to obtain corrected three-dimensional image data.

8. The method for identifying road surface defects based on three-dimensional images according to claim 2, characterized in that, The image data of typical disease categories includes grayscale image data of typical disease categories and three-dimensional image data of typical disease categories; The annotation of the image data of the typical disease categories includes: Annotate the grayscale image data and / or three-dimensional image data of the typical disease categories; or The fused image data of the grayscale image data and the three-dimensional image data of the typical disease categories are labeled; Specifically, for either the grayscale image data or the three-dimensional image data of the typical disease category, the annotation information in the image data is synchronously updated to the other image data.

9. A road surface defect identification device based on three-dimensional images, characterized in that, include: The image correction module is used to: preprocess and correct the precise three-dimensional image data of the road surface to obtain corrected image data; The target extraction module is used to: input the corrected image data into the disease target extraction model to obtain the disease target information output by the disease target extraction model; The target classification module is used to: input the disease target information into the disease target classification model to obtain the disease category information output by the disease target classification model; The disease target extraction model and the disease target classification model are obtained by jointly training a combination of multiple base models based on the original labeled sample set and the expanded labeled sample set formed by historical corrected image data. The precise three-dimensional image data of the road surface includes grayscale image data of the road surface and depth image data of the road surface.

10. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the road surface defect identification method based on three-dimensional images as described in any one of claims 1 to 8.

Images