Road disease identification method and device, electronic equipment and storage medium

Abstract

The embodiment of the present application provides a road disease identification method, when road ponding is scanned, a scanning image and an infrared thermal image of the road ponding are acquired; the scanning image and the infrared thermal image are subjected to image generation processing through a trained image generation model, and a target image without road ponding is obtained; and road diseases under the road ponding are identified based on the target image. The target image without road ponding is generated through the scanning image and the infrared thermal image of the road ponding, the target image is used to identify road diseases corresponding to the road ponding, because the specific heat capacity of the ponding and the road is different, the heat dissipation modes of the road and the ponding are also different, the infrared thermal images of different roads and different ponding are different, the infrared thermal image can implicitly have the difference between the road and the ponding in terms of heat, therefore, the implicit information of the infrared thermal image and the texture of the scanning image can be used to generate the image without ponding, and the accuracy of road disease identification is further improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a road hazard identification method, device, electronic device and storage medium. Background Art

[0002] Scanning vehicles can scan roads for defects, eliminating the need for manual inspection and saving significant manpower. However, since road scanning is performed outdoors, some road defects may be caused by accumulated water. The road surface beneath the accumulated water may or may not contain the defect. Because the accumulated water obscures the road surface, images of the road surface beneath the water cannot be captured for defect identification. Therefore, the accuracy of defect scanning is low when water is present. Summary of the Invention

[0003] An embodiment of the present invention provides a method for identifying road defects, which aims to solve the problem of low accuracy in defect scanning when water accumulates on the road. A target image without water accumulation is generated by scanning an image of water accumulation and using an infrared thermal image. The target image is used to identify defects on the road surface corresponding to the water accumulation. Since the specific heat capacity of water accumulation and road is different, the heat dissipation methods of the road and water accumulation are also different. The infrared thermal images of different road surfaces and different water accumulation are different. The infrared thermal image can implicitly contain the thermal differences between the road and water accumulation. Therefore, the implicit information of the infrared thermal image and the texture of the scanning image can be used to generate an image without water accumulation, thereby improving the accuracy of road defect identification.

[0004] In a first aspect, an embodiment of the present invention provides a method for identifying road damage, the method comprising:

[0005] When water accumulation on the road is detected, a scanning image and an infrared thermal image of the water accumulation are obtained, wherein the scanning image and the infrared thermal image have the same image parameters;

[0006] Performing image generation processing on the scanned image and the infrared thermal image using a trained image generation model to obtain a target image without the road surface water;

[0007] Defects of the road surface under the water are identified based on the target image.

[0008] Optionally, performing image generation processing on the scanned image and the infrared thermal image using a trained image generation model to obtain a target image without the road surface water includes:

[0009] Aligning the scanned image and the infrared thermal image to obtain an aligned scanned image and infrared thermal image;

[0010] The aligned scanned image and infrared thermal image are processed by image generation using a trained image generation model to obtain a target image without the road surface water.

[0011] Optionally, the aligning the scan image and the infrared thermal image to obtain the aligned scan image and infrared thermal image includes:

[0012] Performing image segmentation processing on the scan image and the infrared thermal image respectively to obtain a first water accumulation area in the scan image and a second water accumulation area in the infrared thermal image;

[0013] The scanning image and the infrared thermal image are aligned using the first water accumulation area and the second water accumulation area to obtain an aligned scanning image and infrared thermal image.

[0014] Optionally, the image generation model includes a convolution structure and a deconvolution structure, and performing image generation processing on the aligned scan image and infrared thermal image using the trained image generation model to obtain a target image without the road surface water includes:

[0015] Perform channel fusion on the aligned scan image and infrared thermal map to obtain an input image, where the input image includes a scan image channel and an infrared thermal map channel;

[0016] Performing feature extraction on the input image through the convolution structure to obtain a feature vector of the input image;

[0017] A deconvolution operation is performed on the feature vector through the deconvolution structure to obtain a target image without the road surface water.

[0018] Optionally, before performing image generation processing on the scanned image and the infrared thermal image using the trained image generation model to obtain a generated image without the road surface water, the method further includes:

[0019] Obtaining a data set and a model to be trained, wherein the data set includes image groups corresponding to different disease types, each image group corresponding to a non-water-logging image, a water-logging image, and an infrared thermal image of a disease, and the model to be trained includes a generator to be trained and a discriminator to be trained;

[0020] Inputting the water accumulation image and the water accumulation infrared thermal image into the generator to be trained for image generation processing to obtain a generated image, and inputting the generated image into the discriminator to be trained for identification with the water-free image to obtain an identification result;

[0021] The parameters of the generator to be trained and the discriminator to be trained are adjusted according to the identification result, and the parameter adjustment process is iterated. After the iteration is completed, a trained generator is obtained, and the trained generator is determined as the image generation model.

[0022] Optionally, adjusting parameters of the generator to be trained and the discriminator to be trained according to the identification result includes:

[0023] When the identification result is true, determining a first error loss of the discriminator to be trained;

[0024] Keeping the parameters of the generator to be trained unchanged, and adjusting the parameters of the discriminator to be trained based on the first error loss;

[0025] When the identification result is false, determining a second error loss of the generator to be trained;

[0026] Keeping the parameters of the discriminator to be trained unchanged, the parameters of the generator to be trained are adjusted based on the second error loss.

[0027] Optionally, the identifying damage to the road surface under water based on the target image includes:

[0028] Performing disease recognition on the target image to obtain a disease recognition result, wherein the disease recognition result includes a disease confidence level;

[0029] If the disease confidence is greater than or equal to the confidence threshold, it is determined that there is a disease on the road surface under the road surface water;

[0030] If the disease confidence is less than the confidence threshold, it is determined that there is no disease on the road surface under the road surface water.

[0031] In a second aspect, an embodiment of the present invention further provides a road damage identification device, the road damage identification device comprising:

[0032] A first acquisition module is configured to acquire a scanning image and an infrared thermal image of the road surface water when water on the road surface is detected, wherein the scanning image and the infrared thermal image have the same image parameters;

[0033] A first processing module is configured to perform image generation processing on the scan image and the infrared thermal image using a trained image generation model to obtain a target image without the road surface water;

[0034] The recognition module is used to identify road surface defects under the road surface water based on the target image.

[0035] In a third aspect, an embodiment of the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the road hazard identification method provided in the embodiment of the present invention are implemented.

[0036] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the road damage identification method provided in the embodiment of the invention are implemented.

[0037] In an embodiment of the present invention, when water accumulation is detected on the road surface, a scanning image and an infrared thermal image of the water accumulation are obtained, and the scanning image and the infrared thermal image have the same image parameters; the scanning image and the infrared thermal image are processed by a trained image generation model to obtain a target image without the water accumulation; and road surface defects are identified based on the target image. A target image without water accumulation is generated using the scanning image and the infrared thermal image of the water accumulation, and the target image is used to identify defects on the road surface corresponding to the water accumulation. Since the specific heat capacity of water accumulation and the road are different, the heat dissipation methods of the road and the water accumulation are also different. The infrared thermal images of different road surfaces and different water accumulation are different. The infrared thermal image can implicitly contain the difference in heat between the road and the water accumulation. Therefore, the implicit information of the infrared thermal image and the texture of the scanning image can be used to generate an image without water accumulation, thereby improving the accuracy of road defect identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 This is a flow chart of a road hazard identification method provided by an embodiment of the present invention;

[0040] Figure 2 1 is a schematic structural diagram of a road hazard identification device provided in an embodiment of the present invention;

[0041] Figure 3 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0043] like Figure 1 As shown, Figure 1 This is a flow chart of a method for identifying road damage provided by an embodiment of the present invention. The method for identifying road damage includes the following steps:

[0044] 101. When water accumulation on the road is detected, a scanning image and an infrared thermal image of the water accumulation on the road are obtained.

[0045] In an embodiment of the present invention, the above-mentioned road defect identification method can be deployed on a server, an edge, or a vehicle-mounted terminal. The above-mentioned server can be a cloud server or a local server, the above-mentioned edge can be an electronic device set on the roadside or on a scanning vehicle, and the above-mentioned vehicle-mounted terminal can be an electronic device set on a scanning vehicle.

[0046] The above-mentioned road surface water can be accumulated water after rain, accumulated water caused by artificial sprinkling, or accumulated water caused by other reasons. When performing a road scanning task, the scanning vehicle can detect road surface water in real time. Specifically, when performing a road scanning task, the scanning vehicle scans the road image in real time and detects water accumulation on the road image. If water accumulation is detected in the road image, it is determined that road surface water has been scanned. If no water accumulation is detected in the road image, the road image can be directly inspected for defects.

[0047] Water accumulation detection can be performed on road images using a water accumulation detection model. The water accumulation detection model can be a model based on a deep convolutional neural network, such as ResNet, YOLOV, Faster R-CNN, SSD, or RetinaNet.

[0048] Furthermore, the water accumulation detection model outputs a water accumulation detection frame (x1, y1, w1, h1, u1), where (x1, y1) is the center position of the water accumulation detection frame, w1 is the width of the water accumulation detection frame, h1 is the height of the water accumulation detection frame, and u1 represents the confidence of the water accumulation detection frame. The area of ​​the water accumulation detection frame can be determined based on the width w1 of the water accumulation detection frame and the height h1 of the water accumulation detection frame. When the area of ​​the water accumulation detection frame is larger than the preset area, it is determined that road water accumulation is detected. When the water accumulation detection frame is not detected or the area of ​​the water accumulation detection frame is smaller than the preset area, it is determined that road water accumulation is not detected. Since water accumulation with a smaller area does not significantly obstruct defects and has little effect on defect identification, the embodiment of the present application determines water accumulation with a larger water accumulation detection frame area as road water accumulation that obstructs more defects.

[0049] If there is water on the road surface in the road image, it can be determined that water on the road surface has been scanned. When water on the road surface is scanned, a scanning image of the water on the road surface can be collected by an image scanning device on the scanning vehicle, and an infrared thermal image of the water on the road surface can be collected by an infrared image collection device. The scanning image of the water on the road surface and the infrared thermal image of the water on the road surface are collected at the same time, the distance between the image scanning device and the infrared image collection device is less than a preset value, or the image scanning device and the infrared image collection device are integrated in the same device, and the scanning image and the infrared thermal image of the water on the road surface can be collected at the same time.

[0050] An infrared thermogram is a temperature distribution map obtained by measuring the infrared radiation energy radiated from an object's surface. In an infrared thermogram, areas of different temperatures are displayed in different colors. In this embodiment of the present invention, the infrared image acquisition device and the image scanning device have the same resolution parameters, so that the scanned image and the corresponding infrared thermogram have the same image parameters. These image parameters may include image resolution, capture time, and capture location.

[0051] 102. The scanned image and infrared thermal image are processed by the trained image generation model to obtain a target image without road water accumulation.

[0052] In the embodiment of the present application, the image generation model can be a model based on a deep convolutional neural network. Specifically, the area of accumulated water on the road is found according to the infrared thermal map. Generally, the temperature of the accumulated water area is lower than the temperature of the road surface in the infrared thermal map. The area of accumulated water on the road can be found according to this feature. Threshold segmentation processing, morphological processing and other methods can be used to extract the area of accumulated water on the infrared thermal map. According to the road surface image and the area of accumulated water, the scanning map is segmented into road surface area and accumulated water area. Image segmentation algorithms can be used to achieve the segmentation of the road surface area and the accumulated water area in the scanning map, such as convolutional neural network-based image segmentation, threshold-based binary algorithm, morphological dilation and erosion algorithm, etc. By segmenting the scanning map, the road surface and the accumulated water can be separated to obtain the road surface area and the accumulated water area image. For the image of the accumulated water area, the threshold-based binary algorithm, the morphological dilation and erosion algorithm, and other methods can be used to remove the accumulated water area from the scanning map according to the accumulated water area in the infrared thermal map to obtain the real road surface image. The real road surface image is determined as the target image.

[0053] The image generation model can be a generator in a generative adversarial network. Specifically, the generator and the discriminator in the generative adversarial network can be used to learn and train the rule of generating a non-accumulated water image from an accumulated water image. The obtained generator can generate a non-accumulated water image from an accumulated water image. Further, the input of the generative adversarial network is an accumulated water image and an accumulated water infrared thermal map. During the learning and training process of the generative adversarial network, the road texture in the accumulated water image and the thermal distribution in the accumulated water infrared thermal map can be extracted. The road texture and the thermal distribution can be used to generate a non-accumulated water image, which can obtain a more realistic non-accumulated water image. After the learning and training are completed, the trained generator is used as the image generation model. The input of the trained image generation model is a scanning map and an infrared thermal map, and the output is a target image.

[0054] It should be noted that the specific heat capacity of the road surface and the accumulated water is different. Generally, the specific heat capacity of the accumulated water is larger, which can absorb more heat, so that the surface temperature is relatively low. The larger the volume of the accumulated water, the more heat it absorbs, and the lower the surface temperature. The specific heat capacity of the road is smaller, which can absorb less heat, so that the surface temperature of the road is relatively high. Due to the accumulated water on the road, different road conditions result in different accumulated water volumes, and different accumulated water volumes reflect different accumulated water thermal distributions, such as different accumulated water thermal distributions corresponding to shallow pits and deep pits. Therefore, the infrared thermal map can be used to assist the scanning map to eliminate accumulated water and obtain a target image without accumulated water on the road.

[0055] The scanning map can be one or more, and the infrared thermal map corresponds to the scanning map Figure 1 Each scanning map corresponds to one infrared thermal map.

[0056] In one possible embodiment, for a waterlogged area, multiple scans and infrared thermal images can be collected, each corresponding to an infrared thermal image. A trained image generation model is then used to generate images of each scan and its corresponding infrared thermal image, yielding a corresponding waterlogged image and image confidence. The image confidence indicates the degree of confidence between the waterlogged image and the scan. The greater the image confidence, the greater the degree of confidence between the waterlogged image and the scan. The waterlogged image with the highest image confidence can be determined as the target image.

[0057] In a possible embodiment, for a waterlogged area, multiple scans and multiple infrared thermal maps can be collected, each scan corresponds to a waterlogging detection frame, and each scan corresponds to an infrared thermal map. Based on the waterlogging detection frame and infrared thermal map corresponding to the scan, a scan and its corresponding infrared thermal map can be selected as input data for an image generation model, and a target image without waterlogging on the road is output, and the target image corresponds to the selected scan. Specifically, the quality score of the scan can be determined based on the waterlogging detection frame and infrared thermal map corresponding to the scan, and the scan with the highest quality score and the corresponding infrared thermal map are determined as input data for the image generation model. More specifically, the quality score of the above scan can be determined by the following formula:

[0058]

[0059] In the above formula, s is the quality score of the scanned image, w1 is the width of the water detection frame, h1 is the height of the water detection frame, u1 is the confidence of the water detection frame, a is the empirical coefficient, which is a positive integer, and t is the confidence level of the water detection frame. max is the maximum thermal value in the infrared thermal map, the above t min is the minimum thermal value in the infrared thermal image. From the above formula, it can be seen that the image quality score is positively correlated with the area and confidence of the water detection frame, and the image quality score is positively correlated with t max -t min Positively correlated, t max -t min The larger the e -a(t min -t max ) is smaller, e (t min -t max ) is smaller, s is larger, t max -t minThe difference between the maximum and minimum thermal values ​​is the same. The larger the difference, the clearer the infrared thermal image, and the smaller the difference, the blurrier the infrared thermal image. After determining the quality scores corresponding to multiple scans using the above formula, the scan with the highest quality score can be used as the input data for the image generation model.

[0060] 103. Identify road surface defects under water based on target images.

[0061] In an embodiment of the present invention, after obtaining the target image, since the target image is equivalent to the water-removed restoration of the scanned image, there is no road water in the target image. Therefore, the trained disease detection model can be directly used to identify the disease of the target image to obtain the disease identification result of the road surface under the road water. The above-mentioned disease identification result may include the disease type, disease degree, and disease location.

[0062] The above-mentioned disease detection model can be trained using a pre-prepared dataset, and a trained disease detection model is obtained after the training is completed. The above-mentioned disease detection model can be a model based on a deep convolutional neural network, and the above-mentioned water accumulation detection model can be a model based on a deep convolutional neural network such as ResNet, YOLOV, Faster R-CNN, SSD, RetinaNet, etc. The above-mentioned trained disease detection model outputs a disease detection frame (x0, y0, w0, h0, u0, r0, v0), where (x0, y0) represents the center position of the disease detection frame, w0 represents the width of the disease detection frame, h0 represents the height of the disease detection frame, u0 represents the confidence level of the disease detection frame, r0 represents the type of disease in the disease detection frame, and v0 represents the degree of disease in the disease detection frame.

[0063] The above-mentioned types of defects may include potholes, road depressions, road subsidence, wavy roads, etc. The above-mentioned defect location may be the location of the defect in the target image, and the defect location may be represented by the center position of the defect detection frame. In a possible embodiment, each target image corresponds to a scan image, and each scan image has a corresponding shooting time and shooting location. After determining the shooting location of the scan image, the defect location in the target image can be converted into the defect location in the real world through the conversion relationship between the camera coordinate system of the image scanning device and the real-world coordinate system. The above-mentioned degree of defect can be determined by v of the defect detection frame, and the degree of defect can be represented in the form of a score or a level. The higher the score of the degree of defect, the more serious the defect, or the higher the degree of defect, the more serious the defect.

[0064] Optionally, in the step of identifying road surface defects under water based on the target image, the target image can be subjected to defect identification to obtain a defect identification result, the defect identification result including a defect confidence level; if the defect confidence level is greater than or equal to a confidence level threshold, it is determined that there is a defect on the road surface under water; if the defect confidence level is less than the confidence level threshold, it is determined that there is no defect on the road surface under water.

[0065] In an embodiment of the present invention, the above-mentioned defect confidence can be the confidence u0 of the defect detection frame. The higher the defect confidence, the higher the confidence that it is a defect. The lower the defect confidence, the lower the confidence that it is a defect. The above-mentioned confidence threshold can be set by relevant personnel based on experience. The fixed threshold can be determined based on the image confidence of the target image. The lower the image confidence of the target image, the lower the generated quality of the target image. In this case, the confidence threshold can be set higher, and can be dynamically set to be higher than the fixed threshold to avoid false detection. The higher the image confidence of the target image, the higher the generated quality of the target image. A fixed threshold can be used. The above-mentioned confidence threshold can also be determined based on the quality score s of the scanned image. The lower the quality score s of the scanned image, the lower the generated quality of the target image. In this case, the confidence threshold can be set higher, and can be dynamically set to be higher than the fixed threshold to avoid false detection. The lower the quality score s of the scanned image, the higher the generated quality of the target image. A fixed threshold can be used.

[0066] In an embodiment of the present invention, when water accumulation is detected on the road surface, a scanning image and an infrared thermal image of the water accumulation are obtained, and the scanning image and the infrared thermal image have the same image parameters; the scanning image and the infrared thermal image are processed by a trained image generation model to obtain a target image without water accumulation; and road surface defects are identified based on the target image. A target image without water accumulation is generated using the scanning image and the infrared thermal image of water accumulation, and the target image is used to identify road defects corresponding to the water accumulation. Since the specific heat capacity of water accumulation and the road are different, the heat dissipation methods of the road and water accumulation are also different. The infrared thermal images of different road surfaces and different water accumulation are different. The infrared thermal image can implicitly contain the difference in heat between the road and water accumulation. Therefore, the implicit information of the infrared thermal image and the texture of the scanning image can be used to generate an image without water accumulation, thereby improving the accuracy of road defect identification.

[0067] It is understandable that in the specific implementation of this application, related data such as scan images, infrared thermal images, and data sets required for training are involved. When the embodiments in this application are applied to specific products or technologies, it is necessary to obtain permission or consent from relevant departments, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0068] Optionally, in the step of performing image generation processing on the scan image and the infrared thermal map through a trained image generation model to obtain a target image without road water accumulation, the scan image and the infrared thermal map can be aligned to obtain an aligned scan image and infrared thermal map; and the aligned scan image and the infrared thermal map can be image generated by a trained image generation model to obtain a target image without road water accumulation.

[0069] In an embodiment of the present invention, the scanned image and the infrared thermal map have the same image parameters, which may be shooting time, shooting location, and image resolution. The scanned image and the infrared thermal map can be aligned by image center alignment, specifically by aligning the image center of the scanned image with the image center of the infrared thermal map. The scanned image and the infrared thermal map can also be aligned by feature point matching, specifically by calculating the feature points of the scanned image and the feature points of the infrared thermal map, matching the feature points of the scanned image with the feature points of the infrared thermal map, and aligning the scanned image and the infrared thermal map through the successfully matched feature point pairs.

[0070] The aligned scanning image and infrared thermal image can more accurately determine the position of the road surface water in the scanning image and the infrared thermal image, and then determine the position in the scanning image where the road surface water needs to be eliminated.

[0071] After obtaining the aligned scan image and infrared thermal map, the aligned scan image and infrared thermal map can be input as input data into an image generation model, and the image generation model outputs a target image without road surface water accumulation.

[0072] Optionally, in the step of aligning the scanning image and the infrared thermal map to obtain the aligned scanning image and infrared thermal map, the scanning image and the infrared thermal map can be separately segmented to obtain a first water accumulation area in the scanning image and a second water accumulation area in the infrared thermal map; the scanning image and the infrared thermal map are aligned through the first water accumulation area and the second water accumulation area to obtain the aligned scanning image and infrared thermal map.

[0073] In an embodiment of the present invention, the scanned image can be subjected to image segmentation processing to obtain the road area and the road surface waterlogging area of ​​the scanned image, and the road surface waterlogging area in the scanned image is determined as the first waterlogging area. The infrared thermal map can be subjected to image segmentation processing to obtain the road area and the road surface waterlogging area of ​​the infrared thermal map, and the road surface waterlogging area in the infrared thermal map is determined as the second waterlogging area.

[0074] The above-mentioned image segmentation processing can be based on an image segmentation model to perform image segmentation processing on the scanned image and the infrared thermal map. The image segmentation model can be based on a model obtained by a deep convolutional neural network, and can adopt a neural network model such as U-Net, FCN, DeepLab, Mask R-CNN, etc.

[0075] After obtaining the first and second water accumulation areas, the scan image and the infrared thermal map can be aligned using their contours to obtain an aligned scan image and infrared thermal map. Alternatively, the scan image and the infrared thermal map can be aligned using their geometric centers to obtain an aligned scan image and infrared thermal map.

[0076] By performing image segmentation processing on the scan image and the infrared thermal map, and using the accumulated water area after the image segmentation processing to align the scan image and the infrared thermal map, the alignment accuracy of the scan image and the infrared thermal map can be improved.

[0077] Optionally, the image generation model includes a convolution structure and a deconvolution structure. In the step of performing image generation processing on the aligned scan image and infrared thermal map through the trained image generation model to obtain a target image without road water accumulation, the aligned scan image and infrared thermal map can be channel-fused to obtain an input image, which includes a scan image channel and an infrared thermal map channel; feature extraction is performed on the input image through the convolution structure to obtain a feature vector of the input image; and deconvolution operation is performed on the feature vector through the deconvolution structure to obtain a target image without road water accumulation.

[0078] In an embodiment of the present invention, the above-mentioned image generation model may include a convolution structure and a deconvolution structure, wherein the above-mentioned convolution structure is used to extract implicit information of the scanned image and the infrared thermal map. The implicit information of the scanned image may be road surface texture information, and the implicit information of the infrared thermal map may be water accumulation and road surface thermal distribution information. The implicit water accumulation and road surface thermal distribution information can be used to guide the elimination of water accumulation, and the implicit road surface texture information and the implicit water accumulation and road surface thermal distribution information can be used to guide the generation of new road surfaces.

[0079] Specifically, the aligned scanned image is used as an image channel, the aligned infrared thermal map is also used as an image channel, the scanned image and the infrared thermal map are channel-fused to obtain a dual-channel fused image, and the fused image is determined as the input image of the image generation model.

[0080] In the image generation model, a convolution operation is performed on the input image through a convolution structure, thereby extracting features from the input image and extracting a feature vector of the input image. The feature vector contains implicit information about the road surface texture and the infrared thermal map. After obtaining the feature vector, a deconvolution operation is performed on the feature vector through a deconvolution structure. In the deconvolution operation, only the road surface texture is restored to obtain a target image with the same resolution as the scanned image. Specifically, in the deconvolution operation, the water accumulation and road surface thermal distribution information are mainly used to assist in determining the size and depth of the water accumulation area, and the road surface texture information is used to generate the road texture of the water accumulation area. The road texture of the road area remains unchanged, thereby obtaining a target image with the same resolution as the scanned image.

[0081] Optionally, before performing image generation processing on the scanned image and the infrared thermal map through the trained image generation model to obtain a generated image without road water accumulation, a data set and a model to be trained can also be obtained, the data set including image groups corresponding to different disease types, each image group corresponding to a disease image without water accumulation, an image with water accumulation, and an infrared thermal map with water accumulation, the model to be trained including a generator to be trained and a discriminator to be trained; the image with water accumulation and the infrared thermal map with water accumulation are input into the generator to be trained for image generation processing to obtain a generated image, the generated image is input into the discriminator to be trained for identification with the image without water accumulation to obtain an identification result; according to the identification result, the parameters of the generator to be trained and the discriminator to be trained are adjusted, and the parameter adjustment process is iterated, the trained generator is obtained after the iteration is completed, and the trained generator is determined as the image generation model.

[0082] In an embodiment of the present invention, the above-mentioned data set includes a preset number of image groups, an image group is an image triplet, and one image group corresponds to a specific disease. The disease actually occurs and is collected into corresponding non-water accumulation images, water accumulation images, and water accumulation infrared thermal images. The shooting positions and image resolutions of the above-mentioned non-water accumulation images, water accumulation images, and water accumulation infrared thermal images are the same.

[0083] The above-mentioned model to be trained can be a generative adversarial network, and the above-mentioned model to be trained includes a generator to be trained and a discriminator to be trained, wherein the above-mentioned generator to be trained is used to generate an image, and the above-mentioned discriminator to be trained is used to identify whether the generated result is a real image.

[0084] During the training process, for a given image set, images with water accumulation and infrared thermal images with water accumulation are fed as input data into the generator to be trained for image generation, generating a generated image. After the generated image is obtained, it is fed into the discriminator to be trained for identification with the image without water accumulation. The discriminator to be trained identifies whether the generated image is a real image, obtaining an identification result. The identification result may include whether the generated image is a real image or not. If the generated image is a real image, the parameters of the discriminator to be trained are adjusted. If the generated image is not a real image, the parameters of the generator to be trained are adjusted.

[0085] The above parameter adjustment process is iterated until the discriminator to be trained cannot identify the generated image as a non-real image after parameter adjustment when the gradient is minimum, or the number of iterations reaches the preset number of iterations, then the learning and training is stopped, and the trained generator is obtained. The trained generator is deployed to obtain a trained image generation model.

[0086] Optionally, in the step of adjusting the parameters of the generator to be trained and the discriminator to be trained according to the identification result, when the identification result is true, the first error loss of the discriminator to be trained can be determined; the parameters of the generator to be trained are kept unchanged, and the parameters of the discriminator to be trained are adjusted based on the first error loss; when the identification result is false, the second error loss of the generator to be trained is determined; the parameters of the discriminator to be trained are kept unchanged, and the parameters of the generator to be trained are adjusted based on the second error loss.

[0087] In this embodiment of the present invention, after obtaining the identification result, if the identification result is true, it means that the identification ability of the current discriminator to be trained cannot accurately identify the generated image of the generator to be trained. Therefore, it is necessary to adjust the parameters of the discriminator to be trained. Specifically, the first error loss of the discriminator to be trained can be calculated by the following formula:

[0088] Loss1=(y-1)log(D(z))-ylogD(x)

[0089] wherein the Loss1 is a first error loss, y is a label value, the label value of the water-free image is 1, the label value of the generated image is 0, z is the generated image, D(z) is a discrimination result of the to-be-trained discriminator for the generated image, x is the water-free image, and D(x) is a discrimination result of the to-be-trained discriminator for the water-free image. As can be seen from the above formula, the training process of the to-be-trained discriminator is an optimization process with the optimization objective of minimizing the error loss. When the input of the to-be-trained discriminator is the generated image, Loss1 = -log(D(z)), at this time, the optimization expectation is that the discrimination result D(z) approaches 0, the closer D(z) is to 0, the smaller Loss1 is. When the input of the to-be-trained discriminator is the water-free image, Loss1 = -logD(x), the optimization expectation is that the discrimination result D(x) approaches 1, the closer D(x) is to 1, the smaller Loss1 is.

[0090] Further, when the discrimination ability of the to-be-trained discriminator cannot accurately discriminate the generated image of the to-be-trained generator, the first error of the to-be-trained discriminator can be determined through the above formula, the optimization objective is to minimize the first error loss, and the parameters of the to-be-trained generator are kept unchanged, and the to-be-trained discriminator is adjusted in parameters through the error back propagation algorithm.

[0091] After obtaining the discrimination result, if the discrimination result is false, it indicates that the generation ability of the current to-be-trained generator cannot generate a generated image that can deceive the to-be-trained discriminator, therefore, the parameters of the to-be-trained generator need to be adjusted. Specifically, the second error loss of the to-be-trained generator can be calculated through the following formula:

[0092]

[0093] wherein the Loss2 is a second error loss, y is a label value, the label value of the water-free image is 1, the label value of the generated image is 0, z is the generated image, D(z) is a discrimination result of the to-be-trained discriminator for the generated image, is rounded to an integer, and the category probability greater than or equal to a preset value is 1, and the category probability less than the preset value is 0. As can be seen from the above formula, the training process of the to-be-trained generator is an optimization process with the optimization objective of minimizing the error loss. In the above formula, the optimization expectation is that the discrimination result D(z) approaches 1, the closer D(z) is to 1, the smaller Loss2 is.

[0094] Furthermore, when the generation capability of the generator to be trained cannot generate a generated image that can deceive the discriminator to be identified, the second error of the generator to be trained can be determined by the above formula, and the optimization goal is to minimize the loss of the second error, while keeping the parameters of the discriminator to be trained unchanged, and adjusting the parameters of the generator to be trained through the error back propagation algorithm.

[0095] It should be noted that the road damage identification method provided in the embodiment of the present invention can be applied to devices such as photographing equipment, smart phones, computers, servers, etc. that can perform road damage identification.

[0096] like Figure 2 As shown, an embodiment of the present invention provides a road damage identification device, which includes:

[0097] The first acquisition module 201 is configured to acquire a scanning image and an infrared thermal image of the road surface water when water is detected on the road surface, wherein the scanning image and the infrared thermal image have the same image parameters;

[0098] A first processing module 202 is configured to perform image generation processing on the scan image and the infrared thermal image using a trained image generation model to obtain a target image without the road surface water;

[0099] The identification module 203 is configured to identify road surface defects under the water based on the target image.

[0100] Optionally, the first processing module 202 includes:

[0101] A first processing submodule is configured to align the scan image and the infrared thermal image to obtain an aligned scan image and infrared thermal image;

[0102] The second processing submodule is used to perform image generation processing on the aligned scan image and infrared thermal image through a trained image generation model to obtain a target image without the road surface water.

[0103] Optionally, the first processing submodule includes:

[0104] a first processing unit, configured to perform image segmentation processing on the scan image and the infrared thermal image, respectively, to obtain a first water accumulation area in the scan image and a second water accumulation area in the infrared thermal image;

[0105] The second processing unit is configured to align the scanning image and the infrared thermal image using the first water accumulation area and the second water accumulation area to obtain an aligned scanning image and infrared thermal image.

[0106] Optionally, the first processing module 202 includes:

[0107] A third processing submodule is configured to perform channel fusion on the aligned scan image and the infrared thermal map to obtain an input image, where the input image includes a scan image channel and an infrared thermal map channel;

[0108] an extraction submodule, configured to perform feature extraction on the input image through the convolution structure to obtain a feature vector of the input image;

[0109] The fourth processing submodule is configured to perform a deconvolution operation on the feature vector using the deconvolution structure to obtain a target image without the road surface water.

[0110] Optionally, the device further includes:

[0111] a second acquisition module, configured to acquire a data set and a model to be trained, wherein the data set includes image groups corresponding to different disease types, each image group corresponding to an image without water accumulation, an image with water accumulation, and an infrared thermal image with water accumulation of a disease, and the model to be trained includes a generator to be trained and a discriminator to be trained;

[0112] a second processing module, configured to input the water accumulation image and the water accumulation infrared thermal image into a generator to be trained for image generation processing to obtain a generated image, and input the generated image into a discriminator to be trained for identification with the water-free image to obtain an identification result;

[0113] The third processing module is used to adjust the parameters of the generator to be trained and the discriminator to be trained according to the identification result, and iterate the parameter adjustment process to obtain a trained generator after the iteration is completed, and determine the trained generator as the image generation model.

[0114] Optionally, the third processing module includes:

[0115] a first determining submodule, configured to determine a first error loss of the discriminator to be trained when the identification result is true;

[0116] a first adjustment submodule, configured to keep the parameters of the generator to be trained unchanged and adjust the parameters of the discriminator to be trained based on the first error loss;

[0117] A second determination submodule, configured to determine a second error loss of the generator to be trained when the identification result is false;

[0118] The second adjustment submodule is configured to keep the parameters of the discriminator to be trained unchanged and adjust the parameters of the generator to be trained based on the second error loss.

[0119] Optionally, the identification module 203 includes:

[0120] an identification submodule, configured to perform disease identification on the target image to obtain a disease identification result, wherein the disease identification result includes a disease confidence level;

[0121] a third determination submodule, configured to determine that a disease exists on the road surface under the road surface water if the disease confidence is greater than or equal to a confidence threshold;

[0122] The fourth determination submodule is configured to determine that no damage exists on the road surface under the road surface water if the damage confidence is less than a confidence threshold.

[0123] It should be noted that the road damage identification device provided in the embodiment of the present invention can be applied to devices such as photographing equipment, smart phones, computers, servers, etc. that can perform road damage identification.

[0124] The road damage identification device provided in the embodiment of the present invention can implement each process implemented by the road damage identification method in the above method embodiment and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0125] See also Figure 3 , Figure 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, it includes: a memory 302, a processor 301, and a computer program of a road damage identification method stored in the memory 302 and executable on the processor 301, wherein:

[0126] The processor 301 is configured to call the computer program stored in the memory 302 and execute the following steps:

[0127] When water accumulation on the road is detected, a scanning image and an infrared thermal image of the water accumulation are obtained, wherein the scanning image and the infrared thermal image have the same image parameters;

[0128] Performing image generation processing on the scanned image and the infrared thermal image using a trained image generation model to obtain a target image without the road surface water;

[0129] Defects of the road surface under the water are identified based on the target image.

[0130] Optionally, the processor 301 performs image generation processing on the scan image and the infrared thermal image using the trained image generation model to obtain a target image without the road surface water, including:

[0131] Aligning the scanned image and the infrared thermal image to obtain an aligned scanned image and infrared thermal image;

[0132] The aligned scanned image and infrared thermal image are processed by image generation using a trained image generation model to obtain a target image without the road surface water.

[0133] Optionally, the processor 301 performs the alignment process on the scan image and the infrared thermal image to obtain the aligned scan image and infrared thermal image, including:

[0134] Performing image segmentation processing on the scan image and the infrared thermal image respectively to obtain a first water accumulation area in the scan image and a second water accumulation area in the infrared thermal image;

[0135] The scanning image and the infrared thermal image are aligned using the first water accumulation area and the second water accumulation area to obtain an aligned scanning image and infrared thermal image.

[0136] Optionally, the image generation model includes a convolution structure and a deconvolution structure, and the processor 301 performs image generation processing on the aligned scan image and infrared thermal image using the trained image generation model to obtain a target image without the road surface water, including:

[0137] Perform channel fusion on the aligned scan image and infrared thermal map to obtain an input image, where the input image includes a scan image channel and an infrared thermal map channel;

[0138] Performing feature extraction on the input image through the convolution structure to obtain a feature vector of the input image;

[0139] A deconvolution operation is performed on the feature vector through the deconvolution structure to obtain a target image without the road surface water.

[0140] Optionally, before performing image generation processing on the scan image and the infrared thermal image using the trained image generation model to obtain a generated image without the road surface water, the method executed by the processor 301 further includes:

[0141] Obtaining a data set and a model to be trained, wherein the data set includes image groups corresponding to different disease types, each image group corresponding to a non-water-logging image, a water-logging image, and an infrared thermal image of a disease, and the model to be trained includes a generator to be trained and a discriminator to be trained;

[0142] Inputting the water accumulation image and the water accumulation infrared thermal image into the generator to be trained for image generation processing to obtain a generated image, and inputting the generated image into the discriminator to be trained for identification with the water-free image to obtain an identification result;

[0143] adjust parameters of the to-be-trained generator and the to-be-trained discriminator according to the discrimination result, and iteratively adjust the parameters until the to-be-trained generator is trained and the trained generator is determined as the image generation model.

[0144] Optionally, the adjusting parameters of the to-be-trained generator and the to-be-trained discriminator according to the discrimination result performed by the processor 301 comprises:

[0145] when the discrimination result is true, determining a first error loss of the to-be-trained discriminator;

[0146] keeping parameters of the to-be-trained generator unchanged, and adjusting parameters of the to-be-trained discriminator based on the first error loss;

[0147] when the discrimination result is false, determining a second error loss of the to-be-trained generator;

[0148] keeping parameters of the to-be-trained discriminator unchanged, and adjusting parameters of the to-be-trained generator based on the second error loss.

[0149] Optionally, the disease identification of the road surface under water based on the target image performed by the processor 301 comprises:

[0150] performing disease identification on the target image to obtain a disease identification result, the disease identification result comprising a disease confidence;

[0151] if the disease confidence is greater than or equal to a confidence threshold, it is determined that the road surface under water has a disease;

[0152] if the disease confidence is less than the confidence threshold, it is determined that the road surface under water does not have a disease.

[0153] It should be noted that the electronic device provided by the embodiment of the present application can be applied to a smart phone, a computer, a server and the like which can perform the road disease identification method.

[0154] The electronic device provided by the embodiment of the present application can realize each process of the road disease identification method in the method embodiment, and can achieve the same beneficial effects. To avoid repetition, details are not repeated here.

[0155] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, each process of the road disease identification method or the application end road disease identification method provided by the embodiment of the present application is realized, and the same technical effects can be achieved. To avoid repetition, details are not repeated here.

[0156] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0157] The above disclosure is merely a preferred embodiment of the present invention and certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A road disease identification method, characterized in that: The method comprises the following steps: When water accumulation on the road is detected, a scanning image and an infrared thermal image of the water accumulation are obtained, wherein the scanning image and the infrared thermal image have the same image parameters; Performing image generation processing on the scanned image and the infrared thermal image using a trained image generation model to obtain a target image without the road surface water; Defects of the road surface under the water are identified based on the target image.

2. The road hazard identification method according to claim 1, wherein: The performing image generation processing on the scan image and the infrared thermal image by using the trained image generation model to obtain a target image without the road surface water includes: Aligning the scanned image and the infrared thermal image to obtain an aligned scanned image and infrared thermal image; The aligned scanned image and infrared thermal image are processed by image generation using a trained image generation model to obtain a target image without the road surface water.

3. The road hazard identification method according to claim 2, wherein: The aligning process of the scan image and the infrared thermal image to obtain the aligned scan image and infrared thermal image includes: Performing image segmentation processing on the scan image and the infrared thermal image respectively to obtain a first water accumulation area in the scan image and a second water accumulation area in the infrared thermal image; The scanning image and the infrared thermal image are aligned using the first water accumulation area and the second water accumulation area to obtain an aligned scanning image and infrared thermal image.

4. The road hazard identification method according to claim 3, wherein: The image generation model includes a convolution structure and a deconvolution structure. The image generation processing is performed on the aligned scan image and the infrared thermal image by the trained image generation model to obtain a target image without the road surface water, including: Perform channel fusion on the aligned scan image and infrared thermal map to obtain an input image, where the input image includes a scan image channel and an infrared thermal map channel; Performing feature extraction on the input image through the convolution structure to obtain a feature vector of the input image; A deconvolution operation is performed on the feature vector through the deconvolution structure to obtain a target image without the road surface water.

5. The road hazard identification method according to claim 1, wherein: Before performing image generation processing on the scanned image and the infrared thermal image using the trained image generation model to obtain a generated image without the road surface water, the method further includes: Obtaining a data set and a model to be trained, wherein the data set includes image groups corresponding to different disease types, each image group corresponding to a non-water-logging image, a water-logging image, and an infrared thermal image of a disease, and the model to be trained includes a generator to be trained and a discriminator to be trained; Inputting the water accumulation image and the water accumulation infrared thermal image into the generator to be trained for image generation processing to obtain a generated image, and inputting the generated image into the discriminator to be trained for identification with the water-free image to obtain an identification result; The parameters of the generator to be trained and the discriminator to be trained are adjusted according to the identification result, and the parameter adjustment process is iterated. After the iteration is completed, a trained generator is obtained, and the trained generator is determined as the image generation model.

6. The road hazard identification method according to claim 5, characterized in that: The step of adjusting parameters of the generator to be trained and the discriminator to be trained according to the identification result includes: When the identification result is true, determining a first error loss of the discriminator to be trained; Keeping the parameters of the generator to be trained unchanged, and adjusting the parameters of the discriminator to be trained based on the first error loss; When the identification result is false, determining a second error loss of the generator to be trained; Keeping the parameters of the discriminator to be trained unchanged, the parameters of the generator to be trained are adjusted based on the second error loss.

7. The road hazard identification method according to any one of claims 1 to 6, characterized in that: The identifying of road surface defects under the water based on the target image includes: Performing disease recognition on the target image to obtain a disease recognition result, wherein the disease recognition result includes a disease confidence level; If the disease confidence is greater than or equal to the confidence threshold, it is determined that there is a disease on the road surface under the road surface water; If the disease confidence is less than the confidence threshold, it is determined that there is no disease on the road surface under the road surface water.

8. A road hazard identification device, characterized in that: The road hazard identification device comprises: A first acquisition module is configured to acquire a scanning image and an infrared thermal image of the road surface water when water on the road surface is detected, wherein the scanning image and the infrared thermal image have the same image parameters; A first processing module is configured to perform image generation processing on the scan image and the infrared thermal image using a trained image generation model to obtain a target image without the road surface water; The recognition module is used to identify road surface defects under the road surface water based on the target image.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the road hazard identification method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the road damage identification method according to any one of claims 1 to 7.

Images