An AI-based road inspection method and device

Abstract

The present application discloses an artificial intelligence-based road surface inspection method and device, which relate to data processing and road surface inspection technologies, and include: pre-acquiring a remote sensing image of a road to be inspected under specified vision conditions; acquiring a first inspection image obtained when an inspection vehicle travels on a first main inspection lane; extracting first image features of the remote sensing image and the first inspection image; fusing the remote sensing image features and the first image features, and using a trained road surface recognition model to perform road surface disease recognition; judging whether there is a region with doubtful recognition results in the region outside the second main inspection lane at the position corresponding to the second inspection image; and performing road surface disease recognition on the remaining lanes based on the second inspection image and the remote sensing image and the road surface recognition model. The present application is used to solve the problem of low accuracy in disease recognition of lanes where the inspection vehicle does not travel, and can greatly improve the inspection efficiency by reasonably planning inspection rules.

Description

Technical Field

[0001] The present application relates to data processing and road surface inspection technologies, and in particular, to a road surface inspection method and device based on artificial intelligence. Background Art

[0002] In the work of maintaining urban roads, it is necessary to frequently inspect the roads in order to timely detect road surface diseases and perform repairs. In the past, manual patrols along the roads were usually carried out to find areas with diseases. In order to efficiently identify road surface diseases, a method of taking pictures along the way by inspection vehicles has been introduced for road surface inspection.

[0003] However, in order to save manpower and material resources, during the daily inspection process, inspection vehicles usually identify diseases at intervals of lanes, for example, alternately inspect lanes 1 and 3, and lanes 2 and 4. Although side cameras can be installed, due to complex environments and long distances, etc., the inspection effect on lanes where the inspection vehicle does not drive is poor, resulting in the overall inspection not meeting the actual requirements. In this case, only by performing higher-density inspections, or waiting until the inspection vehicle is arranged to inspect these lanes during the next inspection, can further results be obtained. However, obviously, this method will cost more manpower and material resources to achieve ideal results. Summary of the Invention

[0004] Embodiments of the present application provide a road surface inspection method and device based on artificial intelligence, which solve the problems of low accuracy and great difficulty in identifying diseases in lanes where the inspection vehicle does not drive, and can greatly improve the inspection efficiency by reasonably planning inspection rules.

[0005] An embodiment of the present application proposes a road surface inspection method based on artificial intelligence, including:

[0006] Pre-obtain a remote sensing image of the road to be inspected under specified vision conditions;

[0007] Obtain a first inspection image obtained when the inspection vehicle is driving in the first main inspection lane;

[0008] Extract the remote sensing image features of the remote sensing image, and extract the first image features of the first inspection image;

[0009] Fuse the remote sensing image features and the first image features, and use the trained road surface recognition model to identify road surface diseases in the first main inspection lane and the remaining recognizable lanes, and mark the areas with doubtful recognition results in the remaining recognizable lanes;

[0010] In the case of obtaining a second inspection image obtained when the inspection vehicle is traveling in the second main inspection lane, it is determined whether there is an area with doubtful recognition results in the area outside the second main inspection lane at the position corresponding to the second inspection image;

[0011] Based on the second inspection image and the remote sensing image, and based on the road surface recognition model, road surface diseases are recognized for the second main inspection lane and the remaining lanes that cannot be recognized in the first inspection image; wherein, if there is an area with doubtful recognition results in the area outside the second main inspection lane, supplementary road surface disease recognition is performed on the area with doubtful recognition results according to the second inspection image.

[0012] Optionally, before obtaining the second inspection image, it further includes:

[0013] When the inspection vehicle is traveling in the second main inspection lane, it is determined whether there is an area with doubtful recognition results at the current position of the inspection vehicle;

[0014] At the position where there is an area with doubtful recognition results, an instruction is sent to the image acquisition device on the inspection vehicle to obtain the second inspection image at a higher sampling frequency.

[0015] Optionally, the supplementary road surface disease recognition for the area with doubtful recognition results according to the second inspection image includes:

[0016] The areas with doubtful recognition results in the first inspection image, the second inspection image, and the remote sensing image are subjected to fusion enhancement processing, wherein the fusion ratio of the remote sensing image is less than that of the first inspection image or the second inspection image;

[0017] Based on the image after fusion enhancement processing, road surface disease recognition is performed to obtain a supplementary recognition result.

[0018] Optionally, the supplementary road surface disease recognition for the area with doubtful recognition results according to the second inspection image includes:

[0019] Obtain a first recognition result obtained by performing road surface disease recognition based on the first inspection image and a second recognition result obtained by performing road surface disease recognition based on the second inspection image;

[0020] Determine the shooting distances between the shooting positions of the first inspection image and the second inspection image and the area with doubtful recognition results;

[0021] Determine the final recognition result according to the change trend of the recognition results in the order from far to near of the shooting distances.

[0022] Optionally, the step of determining the final recognition result according to the change trend of the recognition results in the order from far to near of the shooting distance includes:

[0023] If it is determined that in the order from far to near of the shooting distance, the possibility of no disease in the recognition results of road surface disease recognition increases, then modify the recognition result of the area with doubtful recognition result to no disease;

[0024] Otherwise, modify the recognition result of the area with doubtful recognition result to having disease.

[0025] Optionally, if it is determined that there is no area with doubtful recognition result in the area outside the second main inspection lane at the position corresponding to the second inspection image, then remove the area outside the second main inspection lane in the second inspection image that coincides with the first inspection image.

[0026] Optionally, before obtaining the second inspection image obtained when the inspection vehicle is driving in the second main inspection lane, the road surface inspection method further includes:

[0027] When there are multiple pre-planned main inspection lanes, if there is a lane with a driving direction opposite to that of the first main inspection lane, then select the lane with a driving direction opposite to that of the first main inspection lane as the second main inspection lane.

[0028] Optionally, before obtaining the second inspection image obtained when the inspection vehicle is driving in the second main inspection lane, the road surface inspection method further includes:

[0029] Identify the lane to which the area with doubtful recognition result belongs;

[0030] Dynamically adjust the pre-planned second main inspection lane according to the number of areas with doubtful recognition results in each lane;

[0031] Send a notice of temporarily adjusting the main inspection lane to a predetermined device;

[0032] Among them, dynamically adjusting the pre-planned second main inspection lane includes:

[0033] Assign different weights to each area with doubtful recognition result according to the distance relationship between the lane and the inspection vehicle, where the farther the distance, the greater the weight;

[0034] Determine the total weight of the areas with doubtful recognition results when determining the adjacent lane of the pre-planned second main inspection lane as the main inspection lane;

[0035] In the case where the sum of weights of an adjacent lane is smaller than the sum of weights of the pre-planned second main inspection lane and the difference in the sum of weights exceeds a set threshold, the adjacent lane is taken as the actual second main inspection lane.

[0036] Optionally, the set threshold is determined based on the interval since the pre-planned second main inspection lane was last used as a main inspection lane. Specifically, the longer the interval since the pre-planned second main inspection lane was last used as a main inspection lane, the larger the set threshold.

[0037] An embodiment of the present application also provides a road surface inspection device based on artificial intelligence, including a processor and a memory. A computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the aforementioned road surface inspection method based on artificial intelligence are implemented.

[0038] In the embodiment of the present application, in combination with remote sensing data, road surface inspection and disease identification are realized for the remaining recognizable lanes around the first main inspection lane. When a second inspection image obtained when the inspection vehicle is driving on the second main inspection lane is acquired, it is determined whether there is a region with doubtful identification results in the region outside the second main inspection lane at the position corresponding to the second inspection image; if so, supplementary road surface disease identification is performed on the region with doubtful identification results according to the second inspection image. The method of the present application realizes the inspection and supplementary identification of lanes that the inspection vehicle does not drive on, improves the completion degree of the overall road inspection, and improves the accuracy of road disease identification.

[0039] The above description is only an overview of the technical solution of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the description. And in order to make the above and other purposes, features and advantages of the present application more obvious and understandable, the following specifically illustrates the embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0041] Figure 1 is a schematic diagram of the basic process of the road surface inspection method based on artificial intelligence in this embodiment;

[0042] Figure 2 is a schematic diagram of vehicle road surface inspection in this embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be completely conveyed to those skilled in the art.

[0044] An embodiment of the present application proposes a road inspection method based on artificial intelligence, as Figure 1 shown, including the following steps:

[0045] In step S101, remotely sensed images of the road to be inspected under specified vision conditions are obtained in advance. For example, before the inspection starts, remotely sensed images of the upcoming road surface to be inspected under good vision conditions can be obtained, and vehicle recognition is performed on the obtained remotely sensed images. Remotely sensed images with less vehicle coverage on the road surface are selected or combined and stored locally for use in conjunction with the inspection and recognition.

[0046] In step S102, a first inspection image obtained when the inspection vehicle travels in the first main inspection lane is obtained. In some specific examples, the first inspection image can be an image obtained by an image acquisition device mounted on the inspection vehicle. During the process of road inspection and disease identification, the inspection vehicle is arranged to travel on the road to be inspected, and image data (the first inspection image) of the road is captured along the way. By analyzing this image data, road disease identification can be performed.

[0047] The main inspection lane referred to in the embodiment of the present application is the lane in which the inspection vehicle travels. Since the inspection vehicle travels in the main inspection lane, the main content in the inspection image will relatively completely reflect the road conditions of the main inspection lane. However, even if there is an image acquisition device with a large shooting range on the inspection vehicle, due to reasons such as the complex environment and distance of the actual road, the information quality of other lanes outside the main inspection lane in the inspection image is necessarily lower than that of the main inspection lane. Therefore, the road disease identification of the remaining lanes is often low.

[0048] In step S103, remotely sensed image features of the remotely sensed image are extracted, and first image features of the first inspection image are extracted. Specifically, SIFT (Scale-Invariant Feature Transform) features of the image can be extracted.

[0049] In step S104, the remotely sensed image features and the first image features are fused, and the trained road surface recognition model is used to identify road diseases in the first main inspection lane and the other recognizable lanes, and regions with doubtful recognition results in the other recognizable lanes are marked.

[0050] By fusing the image features, the trained road surface recognition model is used to identify road surface diseases in the first main inspection lane and the remaining recognizable lanes. In some specific examples, the pre-trained road surface recognition model can be, for example, VGG16, YOLOv3, etc., which is not specifically limited here. In specific training, various types of road surface disease images can be collected in advance and labeled to perform the training. Since the trained model is based on relatively clear image data, in the embodiments of the present application, the image features based on remote sensing images are used to supplement the information of other lanes outside the main inspection lane to improve the recognition effect of the road surface recognition model on other lanes.

[0051] If it is possible to determine whether there is a disease area in the remaining recognizable lanes based on the first inspection image and the remote sensing image, the disease identification of this lane can be directly completed. Since the shooting position of the first inspection image is relatively far from these lanes, even in combination with the remote sensing image, the recognition accuracy is lower than that of the main inspection lane. In this embodiment, the areas with doubtful results are marked. For lanes that are not photographed or have a low picture ratio, they are ignored during the process of this road surface disease identification.

[0052] In some examples, based on the probability scores output by models such as VGG16 and YOLOv3, it is determined whether there is a road surface disease. As an example, when the probability score of the model output indicating the existence of a road surface disease is higher than the set value, it is considered that there is a road surface disease in the corresponding area; otherwise, it is considered that there is no road surface disease in the corresponding area. Specifically, the probability score interval for doubtful recognition results can be set. When the probability score output by the model is within this probability score interval for doubtful recognition results, it is considered that the recognition result of the road surface disease in this area is doubtful. For example, if this probability score interval is 50%-70%, it means that when the probability score of the model output indicating the existence of a road surface disease is below 50%, the recognition result is that there is no road surface disease; when the probability score is above 70%, the recognition result is that there is a road surface disease; and if the probability score is within the interval of 50%-70%, the corresponding area is marked as having a doubtful recognition result. In practical applications, the type of the disease can also be recognized at the same time. For example, cracks are 80%, collapses are 15%..., then the recognition result is a disease of the crack type. For the sake of simplicity, the types of diseases are not elaborated in the embodiments of the present application.

[0053] In step S105, when the second inspection image obtained when the inspection vehicle is driving in the second main inspection lane is acquired, it is determined whether there is an area with a doubtful recognition result in the area outside the second main inspection lane at the position corresponding to the second inspection image. In a specific example, the second main inspection lane is a lane different from the first main inspection lane. It can be understood that during the process of road surface disease recognition, the shooting position of the inspection image is known, which can be obtained through positioning means such as GPS on the equipment carried by the inspection vehicle or inspection personnel. And when marking the areas with diseases or suspected diseases, the positions of these disease areas will be recorded.

[0054] In step S106, based on the second inspection image and the remote sensing image, road surface disease recognition is performed on the second main inspection lane and the remaining lanes that cannot be recognized in the first inspection image based on the road surface recognition model; wherein, if there is an area with a doubtful recognition result in the area outside the second main inspection lane, supplementary road surface disease recognition is performed on the area with a doubtful recognition result according to the second inspection image.

[0055] That is, in a specific example, the area with a doubtful recognition result will be recognized again, which can improve the completion and accuracy of disease recognition. The specific method of supplementary road surface disease recognition based on the second inspection image and the remote sensing image can be the same as that when performing disease recognition based on the first inspection image; in some implementation processes, different recognition methods from those when performing disease recognition based on the first inspection image can also be adopted.

[0056] In practical applications, if there is no area with a doubtful recognition result, then during the process of disease recognition based on the second inspection image, the lanes with definite recognition results obtained when the inspection vehicle was driving in the first main inspection lane can be ignored. During the process of the inspection vehicle driving in the second main inspection lane, road surface disease recognition is mainly performed on the second main inspection lane and the lanes that cannot be recognized in the first inspection image, which can save computing power and avoid repeated recognition of the same area. That is, if there is no area with a doubtful recognition result, only road surface disease recognition is performed on the second main inspection lane and the lanes that cannot be recognized in the first inspection image; when there is an area with a doubtful recognition result in the area outside the second main inspection lane in the second inspection image, supplementary road surface disease recognition is also performed on the area with a doubtful recognition result.

[0057] For ease of understanding, as Figure 2 shown, it is described in combination with a specific lane example. In this example, a two-way 6-lane (3 lanes in one direction) is taken as an example, but the road surface inspection method of this embodiment is applicable to any number of lanes and lane directions of two or more. For ease of description, with the direction shown in the figure as a reference, from left to right are lane 1, lane 2... lane 6, and the arrows in the figure indicate the lane directions.

[0058] In the actual inspection process, generally, not all lanes will be inspected in a single inspection. Exemplarily, the main inspection lanes for a single inspection may be any one of the groups of lane 1 and lane 4, lane 2 and lane 5, lane 3 and lane 6; in the next inspection, the main inspection lanes can be alternately changed so that after a certain number of inspections, all lanes will serve as the main inspection lanes at least once. In this example, lane 2 and lane 5 are taken as the main inspection lanes, where lane 2 is the first main inspection lane and lane 5 is the second main inspection lane.

[0059] See Figure 2 As shown, the inspection vehicle 1 can generally obtain images within a certain range in the front and / or rear (or plus the side). For simplicity of description, in this example, only the case where the inspection vehicle 1 takes pictures forward is considered, and the actual shooting direction and range can be arbitrary. When the inspection vehicle 1 is driving in the first main inspection lane, the first inspection image includes the image data of the first main inspection lane (lane 2). At the same time, the first inspection image will also include other lanes around the first main inspection lane. For example, lanes 1, 3, 4, and 5 are also photographed to a certain extent; it is assumed that lane 6 cannot be recognized due to its relatively long distance (in actual applications, the lanes that can actually be recognized will be determined based on the actual shooting situation). Figure 2 The shooting range is indicated by a slant line in

[0060] In this example, when identifying road surface diseases based on the first inspection image, the road surfaces of lanes 1, 2, 3, 4, and 5 are all inspected and disease identification is carried out. Since in the image information of the first inspection image, the information of the remaining lanes is less than that of the first main inspection lane, and its accuracy is relatively low, the areas with doubtful identification results in the remaining lanes (lanes 1, 3, 4, 5) are marked. Figure 2 In the example of , the areas with doubtful identification results are indicated by circles.

[0061] Obtain the second inspection image obtained when the inspection vehicle is driving in the second main inspection lane. In actual applications, the second main inspection lane can be inspected by the same inspection vehicle. For example, after the inspection vehicle finishes inspecting and driving on the first main inspection lane, it turns around to the second main inspection lane and drives back in the reverse direction; it can also be inspected by different inspection vehicles for the second main inspection lane. For example, two inspection vehicles are sent out simultaneously for inspection respectively.

[0062] Based on the second inspection image itself or the positioning device (GPS) on the inspection vehicle, etc., the position when the second inspection image is taken can be determined. As Figure 2As shown, when the inspection vehicle is at a certain position in Lane 5, there will be areas where the recognition results of its second inspection image are in doubt. These areas where the recognition results are in doubt are marked after road disease recognition of the first inspection image. Based on the second inspection image, supplementary road disease recognition is carried out on the areas where the recognition results are in doubt in Lanes 3 and 4 to try to obtain more accurate recognition results. Since Lane 5 is the second main inspection lane, normal road disease recognition will be carried out on Lane 5, and the areas where the recognition results are in doubt in Lane 5 will not be referred to during this process.

[0063] If there are no areas where the recognition results are in doubt in the second inspection image, then road disease recognition will only be carried out on Lane 5 and Lane 6 that cannot be effectively photographed by the first inspection image.

[0064] Of course, in other embodiments, the driving direction of the second main inspection lane can be the same as that of the first main inspection lane. For example, the first main inspection lane and the second main inspection lane are Lane 1 and Lane 3 respectively.

[0065] In some embodiments, before obtaining the second inspection image, it further includes:

[0066] When the inspection vehicle is driving in the second main inspection lane, determine whether there are areas where the recognition results are in doubt at the current position of the inspection vehicle;

[0067] At the position where there are areas where the recognition results are in doubt, send an instruction to the image acquisition device on the inspection vehicle to obtain the second inspection image at a higher sampling frequency.

[0068] In some specific examples, the process of road disease recognition based on the second inspection image can be real-time or can be carried out afterwards based on the stored image data. And the above step of sending an instruction to the image acquisition device on the inspection vehicle to obtain the second inspection image at a higher sampling frequency is executed in real time during the inspection process of the inspection vehicle. Among them, the device for sending the instruction can be a terminal on the inspection vehicle for controlling devices such as the image acquisition device, or a server in the cloud; the road disease recognition method of this embodiment can be executed by a terminal carried on the inspection vehicle or by a server in the cloud.

[0069] In practical applications, if there are multiple image acquisition devices on the inspection vehicle, it is also possible to only control some of the image acquisition devices to obtain the second inspection image at a higher sampling frequency (here the second inspection image can be understood as the set of images obtained by multiple image acquisition devices). For example, only control the image acquisition device whose shooting direction is towards the lane where the area with doubtful recognition results is located to obtain the second inspection image at a higher sampling frequency.

[0070] Controlling the image acquisition device to acquire the second inspection image at a higher sampling frequency will be beneficial to the accurate identification of areas where the recognition results are in doubt, and only increasing the sampling frequency when there are areas where the recognition results are in doubt is beneficial to saving resources such as storage, communication, and computing power.

[0071] In some embodiments, the supplementary road surface disease identification for the areas where the recognition results are in doubt based on the second inspection image includes:

[0072] Performing fusion enhancement processing on the areas where the recognition results are in doubt in the first inspection image, the second inspection image, and the remote sensing image, where the fusion ratio of the remote sensing image is less than that of the first inspection image or the second inspection image.

[0073] In a specific example, the fusion enhancement processing includes, but is not limited to, fusing and splicing the clear parts of the three images, combining the complementary and redundant information in the first inspection image, the second inspection image, and the remote sensing image, and generating image features that contain all key details and have enhancement characteristics. Additionally, the shooting directions of the first inspection image and the second inspection image can be different (i.e., the inspection vehicle shoots in lanes in different driving directions), so there will be a large shooting angle difference between the first inspection image and the second inspection image. The first inspection image and the second inspection image obtained from two opposite directions can simulate the three-dimensional information of the area where the recognition results are in doubt, which can further increase the accuracy of road surface disease identification.

[0074] Performing road surface disease identification based on the image after fusion enhancement processing to obtain supplementary recognition results.

[0075] By comprehensively integrating the image information of the first inspection image, the second inspection image, and the remote sensing image, and using the trained road surface recognition model for road surface disease identification, the accuracy of road surface disease identification can be improved.

[0076] In some embodiments, the supplementary road surface disease identification for the areas where the recognition results are in doubt based on the second inspection image includes:

[0077] Obtaining a first recognition result obtained by performing road surface disease identification based on the first inspection image and a second recognition result obtained by performing road surface disease identification based on the second inspection image;

[0078] Determining the shooting distances between the shooting positions of the first inspection image and the second inspection image and the area where the recognition results are in doubt;

[0079] Determining the final recognition result according to the change trend of the recognition results in the order from far to near of the shooting distances.

[0080] In some embodiments, the step of determining the final recognition result according to the change trend of the recognition results in the order of the shooting distances from far to near includes:

[0081] If it is determined that in the order of the shooting distances from far to near, the possibility of no disease existing in the recognition results of the road surface disease recognition increases, then modify the recognition result of the area with doubtful recognition results to no disease existing;

[0082] On the contrary, modify the recognition result of the area with doubtful recognition results to disease existing.

[0083] As another implementation manner, as in the foregoing embodiments, when performing road surface disease recognition, probability scores output by a related model can be obtained. The above methods respectively obtain probability scores (i.e., the first recognition result and the second recognition result) output based on different inspection images. Assume that the probability score of disease existing in the first recognition result is greater than the probability score of disease existing in the second recognition result, and the shooting position of the first inspection image is farther from the area with doubtful recognition results. Then it indicates that as the distance gets closer, the change trend of the recognition result approaches no disease existing. Therefore, the recognition result of the area with doubtful recognition results can be modified to no disease existing. In this embodiment, the recognition result can be adjusted based on the change trend of the probability score, which has a lower requirement for the accuracy of the inspection image and saves more resources.

[0084] In some embodiments, if it is determined that there is no area with doubtful recognition results in the area outside the second main inspection lane at the position corresponding to the second inspection image, then remove the area outside the second main inspection lane in the second inspection image that coincides with the first inspection image, thereby saving resources. This embodiment is based on whether there is an area with doubtful recognition results for removal, without comparing and removing duplicates of images, so there is no need for a complex image processing process, occupying less computing power resources while saving communication and storage resources.

[0085] During the inspection process, there may be more than two main inspection lanes, and the driving directions of some main inspection lanes are the same. When performing road surface disease recognition for supplementing the area with doubtful recognition results, any one of the main inspection lanes other than the first main inspection lane can be selected as the second main inspection lane.

[0086] In practical applications, in order to further improve the complementary degree of the information of the areas with doubtful recognition results in the second inspection image, before obtaining the second inspection image obtained when the inspection vehicle is driving in the second main inspection lane, it includes: when there are multiple pre-planned main inspection lanes, if there is a lane with a driving direction opposite to that of the first main inspection lane, then select the lane with a driving direction opposite to that of the first main inspection lane as the second main inspection lane. Using the lane with the opposite driving direction as the second main inspection lane, that is, selecting the inspection image with the opposite shooting direction as the second inspection image, can supplement the information of the areas with doubtful recognition results from different shooting angles and reflect more three-dimensional information of the areas with doubtful recognition results in some implementation processes.

[0087] In some embodiments, before obtaining the second inspection image obtained when the inspection vehicle is driving in the second main inspection lane, the road surface inspection method further includes:

[0088] Identify the lane to which the area with doubtful recognition result belongs;

[0089] Dynamically adjust the pre-planned second main inspection lane according to the number of areas with doubtful recognition results in each lane.

[0090] For the convenience of explanation, as an example, assume that the first main inspection lane is lane 3 and the pre-planned second main inspection lane is lane 6. And there are relatively many areas with doubtful recognition results in lane 4. In order to provide better-quality supplementary road surface disease recognition for the areas with doubtful recognition results on lane 4 to improve the overall recognition accuracy, it can be selected to temporarily adjust the second main inspection lane to lane 5, so that the position of shooting the second inspection image is closer to lane 4, so as to improve the recognition effect of the areas with doubtful recognition results on lane 4.

[0091] In practical applications, specific judgment rules for dynamically adjusting the pre-planned second main inspection lane can be formulated according to requirements.

[0092] Send a notice of temporarily adjusting the main inspection lane to a predetermined device;

[0093] This device can be an in-vehicle device or a device such as the mobile phone of the inspection personnel. The purpose is to enable the inspection personnel to know that the main inspection lane has been temporarily adjusted, so that the inspection personnel can conduct inspections according to the adjusted main inspection lane.

[0094] Among them, dynamically adjusting the pre-planned second main inspection lane includes:

[0095] Assign different weights to the areas where the recognition results are in doubt according to the distance relationship between the lanes and the inspection vehicle, where the farther the distance, the greater the weight. Continuing with the example where the first main inspection lane is lane 3 and the pre-planned second main inspection lane is lane 6. For simplicity, based on the distance from lane 6, assume that the weight of the area where the recognition result is in doubt in lane 5 is 1, the weight of the area where the recognition result is in doubt in lane 4 is 2, there is no area where the recognition result is in doubt in lane 3 as the first main inspection lane, the weight of the area where the recognition result is in doubt in lane 2 is 4, and the weight of the area where the recognition result is in doubt in lane 1 is 5.

[0096] Determine the total weight of the areas where the recognition results are in doubt when the adjacent lanes of the pre-planned second main inspection lane are used as the main inspection lanes.

[0097] Suppose in a certain section of the road, the number of areas where the recognition results are in doubt in lanes 1, 2, 4, 5, and 6 are: 1, 5, 0, 2, and 3 respectively.

[0098] Then when lane 6 (the pre-planned second main inspection lane) is used as the main inspection lane, the total weight is: 1*5 + 5*4 + 0*2 + 2*1 + 3*0 = 27.

[0099] Similarly, when lane 5 (the adjacent lane of the pre-planned second main inspection lane) is used as the main inspection lane, the total weight is: 1*4 + 5*3 + 0*1 + 2*0 + 3*1 = 22.

[0100] In the case where there is an adjacent lane whose total weight is smaller than the total weight of the pre-planned second main inspection lane and the difference in the total weights exceeds the set threshold, use this adjacent lane as the actual second main inspection lane.

[0101] Suppose the set threshold is 3. In the above example, the difference between the total weight of lane 5 and the total weight of lane 6 is 5, which exceeds the set threshold of 3. Then use lane 5 as the actual second main inspection lane, that is, send a notice to the predetermined device to temporarily adjust the second main inspection lane to lane 5. It can be seen that this method can temporarily change the main inspection lane towards the lane close to the areas where there are more areas with doubtful recognition results when there are more areas with doubtful recognition results in the lanes farther from the second main inspection lane, so as to better obtain the image information of the areas with doubtful recognition results, thereby improving the accuracy of the supplementary road surface disease recognition.

[0102] In practical applications, with the change of the distance relationship between the lanes and the inspection vehicle, the weight change of the areas where the recognition results are in doubt in the other lanes can be non-linear. The above only takes the linear weight change as an example for understanding.

[0103] The above example can dynamically determine whether to adjust the main inspection lane to an adjacent lane of the pre-planned second main inspection lane based on the weight of the area where the recognition result is in doubt. It can not only improve the accuracy of supplementary recognition of the area where the recognition result is in doubt to a certain extent, but also prevent the new second main inspection lane from deviating too much from the pre-planned second main inspection lane, and still ensure relatively accurate road surface disease recognition of the pre-planned second main inspection lane.

[0104] In some embodiments, the set threshold is determined based on the interval since the pre-planned second main inspection lane was last used as the main inspection lane. Specifically, the longer the interval since the pre-planned second main inspection lane was last used as the main inspection lane, the larger the set threshold.

[0105] In a specific example, the set threshold can also be adjusted dynamically in actual applications. If the pre-planned second main inspection lane has not been selected as the main inspection lane for a long time for inspection, the value of the set threshold is increased to prevent the pre-planned second main inspection lane from changing easily. This can ensure that each lane can be selected as the main inspection lane after a certain period of time, and prevent some lanes from not being used as the main inspection lane for a long time, resulting in missing relatively accurate road surface disease recognition results. As an extreme case assumption, when the interval since the pre-planned second main inspection lane was last used as the main inspection lane exceeds the set time, the set threshold can be adjusted to a very large value to ensure that the next second main inspection lane will not be adjusted.

[0106] In the embodiment of the present application, in combination with remote sensing data, road surface inspection and disease recognition are realized for the other recognizable lanes around the first main inspection lane. When the second inspection image obtained when the inspection vehicle is driving on the second main inspection lane is acquired, it is determined whether there is an area where the recognition result is in doubt in the area outside the second main inspection lane at the position corresponding to the second inspection image; if so, supplementary road surface disease recognition is performed on the area where the recognition result is in doubt according to the second inspection image. The method of the present application realizes the inspection and supplementary recognition of the lanes that the inspection vehicle has not driven on, improves the completion degree of the overall road inspection, and improves the recognition accuracy of road diseases.

[0107] The embodiment of the present application also provides a road surface inspection device based on artificial intelligence, including a processor and a memory. A computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the above-mentioned road surface inspection method based on artificial intelligence are realized.

[0108] Moreover, although exemplary embodiments have been described herein, the scope includes any and all embodiments based on the present disclosure having equivalent elements, modifications, omissions, combinations (e.g., schemes that cross various embodiments), adaptations, or alterations. It is not limited to the examples described in this specification or during the implementation of this application, and the examples will be construed as non-exclusive.

[0109] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more aspects thereof) can be used in combination with each other. For example, those of ordinary skill in the art can use other embodiments when reading the above description.

[0110] The above embodiments are only exemplary embodiments of the present disclosure, and those skilled in the art can make various modifications or equivalent replacements to the present invention within the essence and protection scope of the present disclosure, and such modifications or equivalent replacements should also be regarded as falling within the protection scope of the present invention.

Claims

1. A road inspection method based on artificial intelligence, characterized in that: include: Pre-acquire remote sensing images of the road to be inspected under specified visual conditions; Acquire a first inspection image obtained when the inspection vehicle is traveling in the first main inspection lane; extracting remote sensing image features of the remote sensing image, and extracting first image features of the first inspection image; fusing the remote sensing image features and the first image features, and using the trained road surface recognition model to identify road surface defects on the first main inspection lane and other identifiable lanes, and marking areas in the other identifiable lanes where the recognition results are questionable; When a second inspection image obtained when the inspection vehicle is traveling in the second main inspection lane is obtained, determining whether there is an area outside the second main inspection lane where the recognition result is questionable at a position corresponding to the second inspection image; According to the second inspection image and the remote sensing image, and based on the road surface recognition model, road surface disease recognition is performed on the second main inspection lane and the remaining lanes that cannot be recognized in the first inspection image; wherein, if there is an area outside the second main inspection lane where the recognition result is questionable, supplementary road surface disease recognition is performed on the area where the recognition result is questionable based on the second inspection image; The road surface disease recognition of the area where the recognition result is questionable based on the second inspection image includes: Performing fusion enhancement processing on the area where the recognition result is questionable in the first inspection image, the second inspection image and the remote sensing image, wherein the fusion ratio of the remote sensing image is smaller than that of the first inspection image, or the second inspection image; Based on the fusion enhanced processed images, road surface disease identification is performed to obtain supplementary identification results; The road surface disease recognition of the area where the recognition result is questionable based on the second inspection image includes: Acquire a first recognition result obtained by identifying road surface defects based on the first inspection image, and a second recognition result obtained by identifying road surface defects based on the second inspection image; Determining the shooting distances between the shooting positions of the first inspection image and the second inspection image and the area where the recognition result is in doubt; The final recognition result is determined according to the change trend of the recognition results in the order of shooting distance from far to near; The step of determining the final recognition result according to the change trend of the recognition result in the order of shooting distance from far to near includes: If it is determined that the probability of no road disease in the road disease recognition result increases in the order of shooting distance from far to near, the recognition result of the area where the recognition result is questionable is modified to no disease; Otherwise, the recognition result of the area where the recognition result is questionable is modified to indicate that there is a disease.

2. The road inspection method based on artificial intelligence as claimed in claim 1, characterized in that: Before acquiring the second inspection image, the following steps are also included: When the inspection vehicle is traveling in the second main inspection lane, determining whether there is an area where the recognition result is questionable at the current position of the inspection vehicle; At the location where the area where the recognition result is questionable exists, an instruction is sent to the image acquisition device on the inspection vehicle to acquire the second inspection image at a higher sampling frequency.

3. The road inspection method based on artificial intelligence as claimed in claim 1, characterized in that: If it is determined that at the position corresponding to the second inspection image, there is no area outside the second main inspection lane where the recognition result is questionable, the area outside the second main inspection lane in the second inspection image that overlaps with the first inspection image is removed.

4. The road inspection method based on artificial intelligence as claimed in claim 1, characterized in that: Before acquiring the second inspection image obtained when the inspection vehicle is traveling in the second main inspection lane, the road inspection method further includes: When there are multiple pre-planned main inspection lanes, if there is a lane with a driving direction opposite to that of the first main inspection lane, the lane with a driving direction opposite to that of the first main inspection lane is selected as the second main inspection lane.

5. The road inspection method based on artificial intelligence as claimed in claim 1, characterized in that: Before acquiring the second inspection image obtained when the inspection vehicle is traveling in the second main inspection lane, the road inspection method further includes: Identifying the lane to which the area in question belongs; According to the number of areas in each lane where the recognition results are questionable, dynamically adjust the pre-planned second main inspection lane; Send a notice to the scheduled equipment to temporarily adjust the main inspection lane; The dynamic adjustment of the pre-planned second main inspection lane includes: Different weights are assigned to the areas where the recognition results are questionable according to the distance relationship between the lane and the inspection vehicle, wherein the farther the distance, the greater the weight; When determining the adjacent lane of the pre-planned second main inspection lane as the main inspection lane, the sum of weights of the area where the recognition result is in doubt; If there is an adjacent lane whose total weight is smaller than the total weight of the pre-planned second main inspection lane and whose total weight difference exceeds a set threshold, the adjacent lane is used as the actual second main inspection lane.

6. The road inspection method based on artificial intelligence as claimed in claim 5, characterized in that: The set threshold is determined based on the interval from the last time the pre-planned second main inspection lane was used as the main inspection lane, wherein the longer the interval from the last time the pre-planned second main inspection lane was used as the main inspection lane, the larger the set threshold.

7. A road inspection device based on artificial intelligence, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the road inspection method based on artificial intelligence are implemented as described in any one of claims 1 to 6.

Images