Road disease inspection method and system, electronic equipment and computer program product
By combining vehicle terminals and cloud servers, lightweight and high-precision detection models are used to adapt to the characteristics of different road sections, the problem of missed road disease inspections in the existing technology is solved, and more efficient and accurate road disease is achieved. Test.
Patent Information
- Application Number
- CN202411426830.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing computer vision-based road disease patrol technology has the problem of missed inspection and cannot meet the demand for comprehensive inspections of the rapidly developing road network.
By deploying a lightweight detection model on the vehicle terminal for preliminary detection, and sending the preliminary detection information to the cloud server for high-precision processing, combining the resources of the vehicle terminal and the cloud server, different detection models are used to adapt the characteristics of different road sections to improve the Comprehensiveness and accuracy of road disease inspection.
It effectively improves the recall rate and accuracy of road disease detection, reduces the computing pressure of cloud servers, and improves the comprehensiveness and reliability of road disease inspections.
Smart Images

Figure CN118942069A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and in particular relates to a road defect inspection method, a road defect inspection system, an electronic device and a computer program product. Background Art
[0002] With the advancement of urbanization, the road network is becoming increasingly large, and traditional manual road disease inspections can no longer meet the needs. At present, although the road disease inspection technology based on computer vision has developed, there is still a problem of missed detection.
[0003] Therefore, a method that can improve the comprehensiveness of road disease inspection needs to be proposed. Summary of the invention
[0004] The present application provides a road hazard inspection method, a road hazard inspection system, an electronic device and a computer program product, which can improve the comprehensiveness of road hazard inspection.
[0005] In a first aspect, the present application provides a road disease inspection method, which is applied to a vehicle-mounted terminal; the inspection method comprises: Collect road surface video of the current road section; If the classification characteristic of the road damage category of the current road section is that the difference between categories is large, the first road detection model configured by the vehicle terminal is used as the target detection model; the first road detection model is used to detect multiple road damage categories; If the classification characteristic of the road damage category of the current road section is that the difference between categories is small, the second road detection model configured by the vehicle terminal is used as the target detection model; the second road detection model is used to detect whether there is road damage; Perform preliminary detection on the road surface video based on the target model to obtain preliminary detection information; The preliminary detection information is sent to a cloud server configured with a high-precision model, so that the cloud server can process it based on the high-precision model to obtain the road disease inspection results corresponding to the current road section.
[0006] In a second aspect, the present application provides a road disease inspection method applied to a cloud server, the inspection method comprising: After the vehicle terminal calls the lightweight first road detection model or the second road detection model to perform preliminary detection on the road surface video of the current road section based on the classification characteristics of the road disease detection category of the current road section, the preliminary detection information uploaded by the vehicle terminal is received; Call the high-precision model to process the road surface video based on the preliminary detection information to obtain the road disease inspection results corresponding to the current road section; Among them, the classification characteristics include large differences between categories and small differences between categories; if the differences between categories are large, the first road detection model is called, and the first road detection model is used to detect multiple road disease categories; if the differences between categories are small, the second road detection model is called, and the second road detection model is used to detect whether there are road diseases.
[0007] In a third aspect, the present application provides a road disease inspection system. It includes a vehicle terminal configured with a lightweight first road detection model and a second road detection model, and a cloud server configured with a high-precision model; The vehicle terminal is used to collect the road surface video of the current road section, and based on the classification characteristics of the road disease category of the current road section, call the lightweight first road detection model or the second road detection model to perform preliminary detection on the road surface video of the current road section, obtain preliminary detection information, and send the preliminary detection information to the cloud server; the classification characteristics include large differences between categories and small differences between categories; if the difference between categories is large, the first road detection model is called, and the first road detection model is used to detect multiple road disease categories; if the difference between categories is small, the second road detection model corresponds, and the second road detection model is used to detect whether there is a road disease; The cloud server is used to call the high-precision model to process the road surface video based on the preliminary detection information to obtain the road disease inspection results corresponding to the current road section.
[0008] In a fourth aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first or second aspect when executing the computer program.
[0009] In a fifth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect or the second aspect are implemented.
[0010] In a sixth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method of the first aspect or the second aspect.
[0011] Compared with the prior art, the present application has the following beneficial effects: the present application makes full use of the resources of the vehicle terminal and the cloud server, and first performs preliminary detection on the collected road surface video through two lightweight detection models configured on the vehicle terminal, thereby reducing the computing pressure of the cloud server to a certain extent and improving the computing efficiency of the cloud server. Among them, different detection models will be used for preliminary detection of road sections with different resolution characteristics, and adaptive preliminary detection strategies can be provided for different road sections as much as possible to improve the comprehensiveness of road disease inspections on each road section. A high-precision model is deployed on the cloud server, which accurately processes the road surface video based on the preliminary detection results of the vehicle terminal to improve the comprehensiveness and accuracy of road disease detection and ensure the reliability of road disease inspections.
[0012] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0014] Figure 1 It is a flowchart of a road disease inspection method applied to a vehicle-mounted terminal provided in an embodiment of the present application; Figure 2 It is a flow chart of a road disease inspection method applied to a cloud server provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of a road disease inspection system provided in an embodiment of the present application; Figure 4 is an example diagram of road damage categories provided in an embodiment of the present application; Figure 5 It is a schematic diagram of image processing by the L1-RDNet-based inspection system provided in an embodiment of the present application; Figure 6 It is a schematic diagram of image processing by the L2-RDNet-based inspection system provided in an embodiment of the present application; Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0015] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0016] In related technologies, with the advancement of urbanization, the road network is becoming increasingly large, and traditional manual road disease inspections can no longer meet the needs. At present, although the road disease inspection technology based on computer vision has developed, there is still a problem of missed detection.
[0017] Therefore, a method that can improve the comprehensiveness of road disease inspection needs to be proposed.
[0018] In order to solve this problem, the present application studied related technologies and found that the reason why the road disease inspection method based on computer vision has the problem of missed detection is that when relying on the model deployed by the cloud server for road disease inspection and analysis, a large number of images need to be uploaded from the vehicle terminal to the cloud server, resulting in huge network overhead. In order to reduce network overhead, a fixed-distance image acquisition mode is often adopted, that is, at each interval, the cloud server collects a frame of road surface image from the vehicle terminal for road disease detection and analysis. Although this method can effectively reduce network overhead, it cannot cover all road areas, is prone to missed detection, and reduces the comprehensiveness of road disease inspection.
[0019] Based on this, the present application proposes a road disease inspection method, which can first perform preliminary detection through a lightweight detection model deployed on a vehicle terminal, and then accurately process the preliminary detection information of a high-precision detection model deployed on a cloud server, which can not only effectively improve the recall rate of road diseases and the comprehensiveness of road disease detection, but also improve the accuracy of road disease inspection. The control method proposed in the present application will be described below through specific embodiments.
[0020] The inspection method for road diseases provided in the embodiment of the present application can be applied to a vehicle-mounted terminal on the one hand, and can be applied to a cloud server on the other hand. Whether it is a vehicle-mounted terminal or a cloud server, it can be an electronic device, for example, it can be applied to a mobile phone, a tablet computer, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of electronic device.
[0021] On the one hand, in order to illustrate the technical solution proposed in this application, each embodiment will be described below with the vehicle-mounted terminal as the execution subject.
[0022] Figure 1 A schematic flow chart of a road disease inspection method provided by the present application is shown, and the road disease inspection method comprises: Step 110: The vehicle-mounted terminal collects road surface video of the current road section.
[0023] When the inspection vehicle is driving along the road to be inspected, in order to fully cover all the roads to be inspected, the vehicle terminal on the vehicle can use the camera to record the road surface video of the current road section in real time. Road surface video is the basic data for road disease detection. Detection and analysis based on road surface video can identify various road diseases on the road to be inspected as comprehensively as possible, thereby providing data support and decision-making basis for road repair work.
[0024] For example, in order to capture as much road disease detail information as possible in different environments, the vehicle-mounted terminal may be provided with an intelligent collection mode. Under specific conditions, such as light changes, abnormal road color, etc., the vehicle-mounted terminal may automatically increase the collection frequency and / or fill light to ensure the reliability of data collection.
[0025] Step 120: If the classification characteristic of the road damage category of the current road section is that the difference between categories is large, the vehicle-mounted terminal uses the first road detection model configured in the vehicle-mounted terminal as the target detection model.
[0026] Step 130: If the classification characteristic of the road damage category of the current road section is that the difference between categories is small, the vehicle-mounted terminal uses the second road detection model configured in the vehicle-mounted terminal as the target detection model.
[0027] Different road sections have different traffic volumes and types of vehicles passing through, which leads to differences in the road diseases that may occur on different road sections.
[0028] For example, various types of road damage are prone to occur on sections with heavy traffic and complex types of vehicles passing through. These road damage categories show large differences between categories, that is, different road damages have obvious characteristics. For example, subsidence and longitudinal cracks are two types of road damage. The characteristics of these two types of road damage are highly distinguishable from each other. Therefore, the classification characteristics of sections where these two types of road damage occur can be determined as having large differences between categories.
[0029] For example, for a road section with half the traffic volume and a single type of vehicle passing through, such as a restricted road section, all passing vehicles are small vehicles, and the road damage that is prone to occur is relatively single, such as transverse splits and longitudinal cracks. Both types of road damage are cracks, and the features of the two are weakly distinguishable from each other, and more refined features are needed to achieve the distinction. Therefore, the classification characteristics of the road section where these two types of road damage occur can be determined as small differences between categories.
[0030] For the classification characteristics of each road section, it can be divided into those with large differences or those with small differences by analyzing its corresponding traffic flow and vehicle type, so as to facilitate the subsequent adoption of targeted detection strategies to efficiently and comprehensively detect road defects on the road surface of different sections.
[0031] In order to deal with road sections with different distinguishing characteristics, the vehicle terminal can be configured with two road detection models, namely the first road detection model and the second road detection model. Among them, the first road detection model is used to detect multiple road disease categories, so the first road detection model is suitable for road sections with large differences between categories; the second road detection model is used to detect whether there are road diseases, and is suitable for detecting similar road diseases. Therefore, the vehicle terminal can call the first road detection model or the second road detection model as the target model for preliminary detection according to the distinguishing characteristics of the current road section.
[0032] The computing power of the vehicle terminal is limited, so the two road detection models configured on the vehicle terminal are both lightweight, namely the lightweight road detection model (Lite Road Detection Network, Lite-RDNet). Lite-RDNet has simple functions and can meet basic detection needs. It has the advantages of small size, small memory usage, and small computing resource requirements. For ease of description, the first road detection model and the second road detection model can be recorded as L1-RDNet and L2-RDNet respectively.
[0033] Step 140: The vehicle-mounted terminal performs preliminary detection on the road surface video based on the target model to obtain preliminary detection information.
[0034] The vehicle terminal performs preliminary detection on the road surface video by calling the adapted road detection model, that is, it performs preliminary detection and screening on each frame of the road surface image in the road surface video to filter out road surface images suspected of containing road defects, that is, to obtain preliminary detection information. The road surface video can cover all the roads in the current section. The terminal device performs preliminary detection and screening on the road surface video, which can not only reduce the probability of missed detection of road defects, but also improve the recall rate of road defect detection.
[0035] Step 150: The vehicle-mounted terminal sends the preliminary detection information to a cloud server configured with a high-precision model, so that the cloud server can perform accurate processing to obtain the road disease inspection results corresponding to the current road section.
[0036] The cloud server has strong computing power. In order to ensure the comprehensiveness and accuracy of road disease detection, the cloud server can be configured with a high-precision model. The high-precision model can use high-performance cloud server clusters and high-precision deep learning algorithms to further process the detection of vehicle terminals to improve the accuracy and comprehensiveness of road disease inspection results.
[0037] In this embodiment, the two Lite-RDNets deployed in the terminal device are mainly used to quickly perform preliminary screening of road surface images suspected of containing road diseases in road surface videos, so as to improve the recall rate of disease identification and minimize omissions. At the same time, the complex and resource-intensive high-precision model of the cloud server can deeply analyze the results after the initial screening, significantly improve the accuracy of identification, and ensure higher accuracy. Through the collaborative mode of the vehicle-mounted terminal-server, the advantages of the terminal and the cloud can be effectively integrated: the small model of the terminal device realizes rapid initial screening, reduces the burden of data transmission, and widely captures potential diseases (high recall rate); the large model in the cloud focuses on refined analysis and accurately confirms the category of diseases (high accuracy), which can effectively balance efficiency, cost and safety while optimizing the performance of road disease detection. It is an efficient and reliable road disease inspection method.
[0038] In some embodiments, if the classification characteristic of the current road section is that the difference between categories is large, the terminal device calls the target detection model L1-RDNet to perform preliminary detection on the road video, specifically including: Step A1: The vehicle terminal performs preliminary detection on each frame of the road surface image in the road surface video through L1-RDNet to determine the initial image containing suspected road damage.
[0039] Step A2: The vehicle-mounted terminal marks the suspected disease category in the initial image to obtain a first target image.
[0040] Step A3: The vehicle-mounted terminal determines the first target image as preliminary detection information.
[0041] In order to allow the inspection to cover all the roads in the current section, the vehicle terminal can detect each frame of the road surface image in the road surface video through L1-RDNet. From the above content, it can be seen that L1-RDNet aims to filter out the initial image containing suspected road diseases from all the road surface images in the road surface video, and mark the suspected disease category in the initial image to obtain preliminary detection information, that is, the first target image. Among them, Lite-RDNet is labeled based on the location of the suspected road disease, so the first target image includes the labeling box of the suspected road disease and the category of the suspected road disease.
[0042] The first target image is the basis for accurate processing by the high-precision model, so preliminary detection can not only reduce network overhead and the computing power of the cloud server, but also improve the recall rate of road disease detection.
[0043] In some embodiments, if the classification characteristic of the current road section is that the difference between categories is small, the terminal device calls the target detection model L2-RDNet to perform preliminary detection on the road video. Step B1: The vehicle terminal performs preliminary detection on each frame of the road surface image in the road surface video through L2-RDNet to determine the second target image containing suspected road damage.
[0044] Step B2: The vehicle-mounted terminal cuts out the suspected disease area from the second target image to obtain an image block.
[0045] Step B3: The vehicle-mounted terminal determines the second target image and the image block as preliminary detection information.
[0046] L2-RDNet is different from L1-RDNet. L2-RDNet mainly detects whether there are road damages in road surface images. Although L2-RDNet can also detect all types of road damages, considering the accuracy of detection, it is preferred to detect road damages with small differences between categories, such as transverse cracks or longitudinal cracks, which can improve the accuracy of detection.
[0047] Based on this, after L2-RDNet detects and filters each frame of the road surface image in the road surface video, it does not directly annotate the filtered second target image, but focuses on cropping the suspected disease area from the second target image to generate image patches. These image patches serve as input data to prepare for subsequent classification tasks of similar road diseases, thereby improving the accuracy and efficiency of disease classification. That is, the vehicle terminal calls L2-RDNet to improve the accuracy and processing efficiency of the subsequent processing stage through image segmentation and focusing.
[0048] L2-RDNet mainly distinguishes the foreground and background of road images and detects whether there are road damages in road images. The detection effect is particularly good when the detected road damage categories are similar. This targeted detection strategy reduces the burden on the model and can more effectively improve the recall rate of recognition compared to multi-category target detection that requires simultaneous identification of multiple damages.
[0049] It can be understood that the vehicle terminal can call the adapted Lite-RDNet according to the different resolution characteristics of the current road section, and effectively detect and screen all road images of the current road section to obtain preliminary detection information. L1-RDNet can detect and classify various road disease categories in road images, and L2-RDNet only detects and segments whether the road image contains road diseases to obtain rough detection information. Compared with transmitting road videos, the network overhead of transmitting preliminary detection information is smaller, that is, this inspection method can ensure full coverage of the current road section while minimizing the subsequent data transmission network overhead and reducing the computing burden of the cloud server. This terminal-side preliminary processing mechanism effectively improves the overall efficiency of the system and balances the detection effect and resource consumption.
[0050] On the other hand, in order to illustrate the technical solution proposed in this application, each embodiment will be described below using a cloud server as the execution entity.
[0051] Figure 2 A schematic flow chart of a road disease inspection method provided by the present application is shown, and the road disease inspection method comprises: Step 210: After the vehicle terminal calls lightweight L1-RDNet or L2-RDNet to perform preliminary detection on the road surface video of the current road section based on the classification characteristics of the road disease detection category of the current road section, the cloud server receives the preliminary detection information uploaded by the vehicle terminal.
[0052] Step 220: The cloud server calls a high-precision model to process the road surface video based on the preliminary detection information to obtain the road disease inspection results corresponding to the current road section.
[0053] From the above content, we can know that the classification characteristics include large differences between categories and small differences between categories; if the difference between categories is large, L1-RDNet is called, and L1-RDNet is used to detect multiple road disease categories; if the difference between categories is small, L2-RDNet is called, and L2-RDNet is used to detect whether there is road disease. Based on this, after the vehicle terminal implements preliminary detection by calling the corresponding target detection model, the preliminary detection information obtained can be uploaded to the cloud server.
[0054] After receiving the preliminary detection information, the cloud server can accurately process the road damage based on the preliminary detection information. Specifically, the deployed high-precision model can be called to achieve accurate processing, so as to obtain the road damage inspection results corresponding to the current road section. For different Lite-RDNets, the high-precision models that can be called by the cloud server include multi-category detection models and multi-category classification models. Among them, L1-RDNet corresponds to the multi-category detection model. The functions of the two are basically the same. The difference is that the multi-category detection model on the cloud server is more powerful and has higher detection precision and accuracy; L2-RDNet corresponds to the multi-category classification model. L2-RDNet is used to implement binary classification detection, that is, whether there is a road damage; the multi-category classification model is used to finely classify the image blocks corresponding to the road damage to obtain the specific road damage category corresponding to each image block.
[0055] In this embodiment, based on the preliminary detection results of the vehicle-mounted terminal, the cloud server can accurately process the road damage of the current road section with low network overhead and low computational complexity.
[0056] In some embodiments, different preliminary detection information represents different data processing bases, and accordingly, the high-precision models called by the cloud server are different. The preliminary detection information corresponding to L1-RDNet includes a first target image marked with a suspected road disease category. Based on this, the cloud server can call a multi-category detection model to perform the following steps: Step C1: The cloud server calls a multi-category detection model to verify the suspected road damage category and obtains the verification result.
[0057] Step C2: The cloud server detects the unmarked area in the first target image to obtain a first detection result.
[0058] Step C3: The cloud server determines the road disease inspection result corresponding to the current road section based on the verification result and the first detection result.
[0059] The first target image corresponding to L1-RDNet is marked with the category of suspected road damage. In order to determine whether the annotation is accurate, the cloud server can call the multi-category detection model to verify the suspected road damage category. One case is that the verification is successful, that is, the L1-RDNet annotation is correct. At this time, the multi-category detection model can retain or optimize the annotation information, such as making the annotation box more suitable for the corresponding road damage; another case is that the verification fails. At this time, the multi-category detection model can correct the annotation information, such as changing the annotated road damage category. Regardless of the verification result, the multi-category detection model can perform accurate detection and obtain inspection results with higher accuracy.
[0060] In addition, for the unlabeled areas in the first target image, the cloud server can perform further detection based on the multi-category detection model to obtain the first detection result. This can avoid missing road damage that is not detected by L1-RDNet in the first image and improve the comprehensiveness of road damage detection.
[0061] Finally, the multi-category detection model can obtain more accurate and comprehensive road disease inspection results based on the verification results and the first detection results.
[0062] In some embodiments, for some road images, if they only contain road damage that is difficult to detect and annotated by L1-RDNet, these road images may be missed. In order to solve this problem, before obtaining the road damage inspection result based on the first detection result, it also includes: Step D1: The cloud server collects corresponding images from the road surface video based on a preset sampling strategy to obtain a third target image.
[0063] Step D2: The cloud server calls the multi-category detection model to detect the third target image and obtains a second detection result.
[0064] In order to further improve the comprehensiveness of road disease detection, the server can extract and detect each frame of the road surface video. The sampling strategy includes fixed-distance sampling and timed sampling to obtain a third target image. In combination with the above-mentioned embodiments, this application adopts two image acquisition methods. One image acquisition method is to collect road surface videos for predictive analysis, and the other image acquisition method is to directly collect from road surface videos based on the sampling strategy. The combination of the two image acquisition methods is equivalent to achieving double coverage of the current road section, which can significantly improve the comprehensiveness of the inspection method.
[0065] Based on the third target image, the cloud server can call a multi-category detection model to perform accurate detection and obtain a second detection result.
[0066] Correspondingly, step C3 includes: the cloud server filters the verification result, the first detection result and the redundant data in the second detection result to obtain the road disease inspection result.
[0067] The two image acquisition methods can obtain two groups of detection results in total, one of which is obtained through the verification results and the first detection results, and the other is the second detection result. In order to avoid repeated labeling, redundant data can be filtered out to obtain high-precision and comprehensive road disease inspection results.
[0068] In some embodiments, the preliminary detection information corresponding to the L2-RDNet includes a second target image containing road damage and a second target image block corresponding to the road damage, and the second target image block is cropped from the second target image. Based on this, the cloud server can call the multi-category classification model to perform the following steps: Step E1: The cloud server calls a multi-category classification model to determine the road damage category corresponding to the image block.
[0069] Step E2: The cloud server marks the road damage category corresponding to the image block in the second target image to obtain the road damage inspection result.
[0070] L2-RDNet mainly distinguishes the foreground and background of the image and identifies similar road damage, such as whether there is a road damage such as cracks in the road surface image. Based on this, for the segmented image blocks containing suspected damage, the cloud server can call a multi-category classification model to accurately identify and analyze them. The multi-category classification model can further determine whether the cracks in the image blocks are horizontal cracks or vertical cracks. After the multi-category classification model identifies the road damage category corresponding to the image block, it can mark the road damage category corresponding to the image block in the second target image to obtain the road damage inspection results.
[0071] In some cases, if the multi-class classification model can identify road damage, it means that the detection of L2-RDNet is correct; in this case, the specific road damage category corresponding to the image block can be directly annotated into the second target image. In another case, if the multi-class classification model fails to identify road damage, the image block and the second target image can be deleted.
[0072] In other words, the main task of the multi-class classification model is to carefully classify the preliminary results of the pre-screening of vehicle-mounted equipment, that is, to focus on classification rather than comprehensive target detection in order to reduce the learning complexity of the model and improve the accuracy of classification.
[0073] In some embodiments, the cloud server can also provide feedback on the inspection results. Specifically, the cloud server generates a detailed disease report based on the inspection results, covering key information such as the type, location and severity of the disease. And through wireless communication technology, this data can be fed back to the management department in real time to ensure timely response. In addition, the cloud server can also be integrated with the GIS system to further enhance decision support, help managers intuitively view the distribution of diseases, and quickly dispatch maintenance resources to improve the efficiency and accuracy of road maintenance.
[0074] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0075] Corresponding to the inspection method for road damage in the above embodiment, Figure 3 A structural block diagram of a road defect inspection system 3 provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.
[0076] Reference Figure 3 , the road disease inspection system 3 includes a vehicle terminal 31 configured with L1-RDNet and L2-RDNet, and a cloud server 32 configured with a high-precision model; The vehicle terminal 31 is used to collect the road surface video of the current road section, and based on the classification characteristics of the road disease category of the current road section, call the lightweight L1-RDNet or L2-RDNet to perform preliminary detection on the road surface video of the current road section, obtain preliminary detection information, and send the preliminary detection information to the cloud server; the classification characteristics include large differences between categories and small differences between categories; if the difference between categories is large, L1-RDNet is called, and L1-RDNet is used to detect multiple road disease categories; small differences between categories correspond to L2-RDNet, and L2-RDNet is used to detect whether there is a road disease; The cloud server 32 is used to call a high-precision model to process the road surface video based on the preliminary detection information to obtain the road disease inspection results corresponding to the current road section.
[0077] Optionally, when the vehicle-mounted terminal calls the first road detection model, the high-precision model includes a multi-category detection model; The vehicle-mounted terminal calls the first road detection model to perform preliminary detection on each frame of road surface image in the road surface video through the first road detection model to determine an initial image containing suspected road damage; mark the suspected damage category in the initial image to obtain a first target image; and determine the first target image as preliminary detection information; The cloud server calls a multi-category detection model to verify the suspected road disease category and obtain a verification result; detects the unmarked area in the first target image to obtain a first detection result; and determines the road disease inspection result corresponding to the current road section based on the verification result and the first detection result.
[0078] Optionally, when the vehicle-mounted terminal calls the second road detection model, the high-precision model includes a multi-category classification model; The vehicle-mounted terminal calls the second road detection model to perform preliminary detection on each frame of the road surface image in the road surface video through the second road detection model to determine a second target image containing suspected road damage; crop the suspected damage area from the second target image to obtain an image block; and determine the second target image and the image block as preliminary detection information; The cloud server calls a multi-category detection model to classify the road damage corresponding to the image block; the road damage category corresponding to the image block is marked in the second target image to obtain the road damage inspection result.
[0079] It should be noted that the information interaction and execution process between the above-mentioned devices are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0080] In some embodiments, in order to train different detection / classification models, the present application has created a road damage dataset named RD, the construction of which relies on image acquisition work in a real vehicle environment.
[0081] like Figure 4 As shown in the figure, the RD dataset covers a variety of road disease categories, including: subsidence (labeled as sink, number 0), longitudinal cracks (crack_p, number 1), longitudinal crack repair area (patch_crack_p, number 2), transverse cracks (crack_t, number 3), transverse crack repair area (patch_crack_t, number 4), network cracks (net, number 5) and network crack repair area (patch_net, number 6), a total of seven categories. In addition, the dataset is characterized by the integration of complex scenarios from multiple cities, various road sections and different time periods to fully enhance its representativeness and challenges in practical applications. Figure 4 Shows some examples of road damage images in the dataset.
[0082] The RD dataset contains a total of 57,381 images, which are counted in detail according to different road damage categories, and the distribution of each category is intuitively displayed in the form of charts. In actual application, in addition to counting the traffic volume and vehicle types passing through each road section, the road damage categories that appear on the road section can also be counted, so as to more accurately select the appropriate Lite-RDNet for each road section for preliminary detection.
[0083] In order to conduct effective algorithm evaluation, the dataset can be divided into two parts: a training set containing 52,381 images for model learning and optimization; and a validation set containing 5,000 images to test the generalization ability and recognition performance of the model.
[0084] Exemplarily, both L1-RDNet and the multi-category detection model can be trained using training samples containing road damage categories; L1-RDNet and the multi-category classification model can be trained using training samples containing similar road damage categories, such as training samples containing longitudinal cracks or transverse cracks, or training samples containing longitudinal crack repair areas, transverse crack repair areas, and mesh crack repair areas.
[0085] Among them, L1-RDNet is a pre-trained Lite-RDNet that can identify multiple road disease categories. The schematic diagram of the image processing of the inspection system based on L1-RDNet can be found in Figure 5 . Figure 5 There are two image sampling methods that can be selected, so as to achieve double coverage of the inspection section and ensure the comprehensiveness of road disease detection as much as possible. The specific inspection method can refer to the description of the above embodiment and is not limited here.
[0086] L2-RDNet can distinguish between foreground and background and determine whether there are similar road damages, such as whether there are repair areas corresponding to different road damages. The schematic diagram of the image processing of the inspection system based on L2-RDNet can be found in Figure 6 . L2-RDNet distinguishes the foreground and background of each road image to obtain image blocks containing road damage. In this way, Pro-RDNet can perform detailed classification based on image blocks, that is, focus on classification rather than comprehensive target detection, and specifically distinguish which road damage category each repair area corresponds to, thereby reducing the learning complexity of the model and ensuring the reliability of road damage detection. The specific inspection method can be found in the description of the aforementioned embodiment and is not limited here.
[0087] In some embodiments, the loss function of the above detection model mainly consists of two parts: regression loss and classification loss. Among them, the classification loss is the binary cross entropy loss BCE Loss, which is used to determine the specific category in the anchor box. The regression loss is CIoU Loss and DFL loss, which are used to calculate the error between the predicted bounding box and the ground truth box. At the same time, the matching strategy between positive and negative samples adopts the TAL dynamic matching method.
[0088] Specifically, the formula of BCE Loss is as follows:
[0089] in, L BCE is the binary cross entropy loss, n is the number of image samples of all road diseases, y i For the i The true category of the image samples, is the predicted category of the i-th image sample.
[0090] The regression loss is used to calculate the error between the predicted bounding box and the ground truth box. The regression loss uses the CIOU loss (Complete Intersection over Union loss, L CIoU ), which is an improved version of commonly used loss functions such as L1, L2, Intersection over Union (IoU) and GIoU. L CIoU The penalty term for aspect ratio is added. When the center points of the predicted box and the labeled box coincide, the errors in different situations can be better distinguished and the scale is invariant.
[0091]
[0092] in, b and b gt They are the center points of the prediction box and the annotation box respectively; ρ is the Euclidean distance between the predicted box and the labeled box; c is the distance between the diagonal of the closed area of the prediction box and the annotation box; v To ensure the consistency of the relative proportions of the predicted box and the labeled box, IoU is the intersection-over-union ratio of the predicted box and the labeled box; α is the weight coefficient; w and w gt They are the width of the prediction box and the annotation box respectively; h and h gt They are the heights of the predicted box and the labeled box respectively.
[0093] The formula for DFL loss is as follows:
[0094] in, S i is the cross entropy loss between the target true value and the predicted value on the left, S i+1 It is the cross entropy loss between the true target value and the predicted value on the right.
[0095] The total loss function of the road detection model Loss The formula is as follows:
[0096] Among them, λ1 and λ1 are balance coefficients.
[0097] The loss function of the classification model is mainly CE Loss, which is applicable to multi-category classification models. Its formula is as follows:
[0098] in, x is the output of the model, y represents the true label, C represents the number of categories, and n represents the number of samples.
[0099] Different networks in the model can be trained using different loss functions to obtain a trained model.
[0100] In some embodiments, there may be multiple versions of a converged model (detection model or classification model). In order to comprehensively and accurately measure the performance of each version of the model, it can be evaluated through a validation set. Specifically, the performance of the model on the validation set can be evaluated according to preset conditions. For example, these preset conditions may include performance indicators such as the intersection-over-union ratio of the model to road damage, detection accuracy, and recall rate.
[0101] IoU is an indicator for evaluating the degree of overlap between the predicted box and the annotated box. It calculates the ratio of the intersection and union of the predicted box and the annotated box. It plays a key role in determining whether it is a correct detection. The calculation formula of IoU can be written as:
[0102] Precision (p), also known as the precision rate, refers to the proportion of correct positive predictions to all positive predictions, as shown in the formula:
[0103] Recall (R), also known as recall rate, refers to the proportion of correctly predicted positives to all actual positives, as shown in the formula:
[0104] Average Precision (AP) is calculated from precision and recall. A line graph of precision is drawn based on the recall value, and the area under the line is calculated, as shown in the formula:
[0105] Mean Average Precision (mAP) refers to the average value of the average precision AP of C different road disease categories, as shown in the formula:
[0106] That is to say, after verifying each version of the model through the validation set, the model of each version can be comprehensively evaluated based on the above indicators, so as to determine the model with the best performance from each version as the trained model.
[0107] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. The electronic device may be a vehicle-mounted terminal or a cloud server. Figure 7 As shown, the electronic device 7 of this embodiment includes: at least one processor 70 ( Figure 7 Only one processor is shown in the figure), a memory 71, and a computer program 72 stored in the memory 71 and executable on at least one processor 70. When the processor 70 executes the computer program 72, the steps in any of the above-mentioned road disease inspection method embodiments are implemented, for example: Figure 1 Steps 110-150 are shown.
[0108] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0109] In some embodiments, the memory 71 may be an internal storage unit of the electronic device 7, such as a hard disk or memory of the electronic device 7. In other embodiments, the memory 71 may also be an external storage device of the electronic device 7, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 7.
[0110] Furthermore, the memory 71 may also include both an internal storage unit of the electronic device 7 and an external storage device. The memory 71 is used to store operating devices, applications, boot loaders, data, and other programs, such as program codes of computer programs, etc. The memory 71 may also be used to temporarily store data that has been output or is to be output.
[0111] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0112] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0113] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0114] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.
[0115] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0116] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0117] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0118] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0119] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A road disease inspection method, characterized in that: Applied to vehicle terminals; The inspection method comprises: Collect road surface video of the current road section; If the classification characteristic of the road damage category of the current road section is that the difference between categories is large, the first road detection model configured by the vehicle terminal is used as the target detection model; the first road detection model is used to detect multiple road damage categories; If the classification characteristic of the road damage category of the current road section is that the difference between categories is small, the second road detection model configured by the vehicle terminal is used as the target detection model; the second road detection model is used to detect whether there is a road damage; Performing preliminary detection on the road surface video based on the target model to obtain preliminary detection information; The preliminary detection information is sent to a cloud server configured with a high-precision model, so that the cloud server processes it based on the high-precision model to obtain a road disease inspection result corresponding to the current road section.
2. The inspection method according to claim 1, characterized in that: The target detection model is the first road detection model; the preliminary detection of the road surface video based on the target model to obtain preliminary detection information includes: Performing preliminary detection on each frame of road surface image in the road surface video by using the first road detection model to determine an initial image containing suspected road damage; Marking the suspected disease category in the initial image to obtain a first target image; The first target image is determined as the preliminary detection information.
3. The inspection method according to claim 1, characterized in that: The target detection model is the second road detection model; the preliminary detection of the road surface video based on the target model to obtain preliminary detection information includes: Performing preliminary detection on each frame of road surface image in the road surface video by using the second road detection model to determine a second target image containing suspected road damage; Cutting out the suspected disease area from the second target image to obtain an image block; The second target image and the image block are determined as the preliminary detection information.
4. A road disease inspection method, characterized in that: Applied to a cloud server, the inspection method includes: After the vehicle terminal calls the lightweight first road detection model or the second road detection model to perform preliminary detection on the road surface video of the current road section based on the classification characteristics of the road disease detection category of the current road section, receiving the preliminary detection information uploaded by the vehicle terminal; Calling a high-precision model to process the road surface video based on the preliminary detection information to obtain a road disease inspection result corresponding to the current road section; Among them, the classification characteristics include large differences between categories and small differences between categories; if the differences between categories are large, the first road detection model is called, and the first road detection model is used to detect multiple road disease categories; if the differences between categories are small, the second road detection model is called, and the second road detection model is used to detect whether there are road diseases.
5. The inspection method according to claim 4, characterized in that: The preliminary detection information corresponding to the first road detection model includes a first target image marked with a suspected road damage category; The high-precision model includes a multi-category detection model, and the road surface video is processed based on the preliminary detection information to obtain a road disease inspection result corresponding to the current road section, including: Calling the multi-category detection model to verify the suspected road damage category and obtain a verification result; Detecting an unmarked area in the first target image to obtain a first detection result; The road defect inspection result corresponding to the current road section is determined based on the verification result and the first detection result.
6. The inspection method according to claim 5, characterized in that: Before obtaining the road damage inspection result based on the first detection result, the method further includes: Based on a preset sampling strategy, a corresponding image is collected from the road surface video to obtain a third target image; Calling the multi-category detection model to detect the third target image to obtain a second detection result; Accordingly, the determining of the road damage inspection result corresponding to the current road section based on the verification result and the first detection result includes: The redundant data in the verification result, the first detection result and the second detection result are filtered to obtain the road disease inspection result.
7. The inspection method according to claim 6, characterized in that: The preliminary detection information corresponding to the second road detection model includes a second target image containing road damage and a second target image block corresponding to the road damage; the second target image block is cropped from the second target image; the high-precision model includes a multi-category classification model; the road surface video is processed based on the preliminary detection information to obtain a road damage inspection result corresponding to the current road section, including: Calling the multi-category classification model to classify the road damage corresponding to the image block; The road damage category corresponding to the image block is marked in the second target image to obtain the road damage inspection result.
8. A road disease inspection system, characterized in that: It includes a vehicle terminal configured with a lightweight first road detection model and a second road detection model, and a cloud server configured with a high-precision model; The vehicle-mounted terminal is used to collect a road surface video of the current road section, and based on the classification characteristics of the road damage category of the current road section, call a lightweight first road detection model or a second road detection model to perform preliminary detection on the road surface video of the current road section to obtain preliminary detection information, and send the preliminary detection information to the cloud server; the classification characteristics include large differences between categories and small differences between categories; if the difference between categories is large, the first road detection model is called, and the first road detection model is used to detect multiple road damage categories; The small difference between the categories corresponds to a second road detection model, and the second road detection model is used to detect whether there is a road disease; The cloud server is used to call the high-precision model to process the road surface video based on the preliminary detection information to obtain the road disease inspection result corresponding to the current road section.
9. The inspection system according to claim 8, characterized in that: In the case where the vehicle-mounted terminal calls the first road detection model, the high-precision model includes a multi-category detection model; The vehicle-mounted terminal calls the first road detection model to perform preliminary detection on each frame of road surface image in the road surface video through the first road detection model to determine an initial image containing suspected road damage; Marking the suspected disease category in the initial image to obtain a first target image; Determining the first target image as the preliminary detection information; The cloud server calls the multi-category detection model to verify the suspected road damage category and obtain a verification result; An unmarked area in the first target image is detected to obtain a first detection result; and a road disease inspection result corresponding to the current road section is determined based on the verification result and the first detection result.
10. The inspection system according to claim 8, characterized in that: In the case where the vehicle-mounted terminal calls the second road detection model, the high-precision model includes a multi-category classification model; The vehicle-mounted terminal calls the second road detection model to perform preliminary detection on each frame of road surface image in the road surface video through the second road detection model to determine a second target image containing suspected road damage; Cutting out the suspected disease area from the second target image to obtain an image block; Determine the second target image and the image block as the preliminary detection information; The cloud server calls the multi-category detection model to classify the road damage corresponding to the image block; The road damage category corresponding to the image block is marked in the second target image to obtain the road damage inspection result.
11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the road damage inspection method as described in any one of claims 1 to 7 is implemented.
12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by one or more processors, the road damage inspection method as described in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
An automobile blind area intelligent early warning method based on image recognition
CN109583432A
Pavement disease detection method and device based on edge detection neural network
CN110349119A
Road surface complex disease identification method based on deep learning
CN116229392A
City brain cloud edge cooperative computing method and device based on video semantic driving
CN117037158A
Road surface inspection method, device and equipment and readable storage medium
CN117095316A
Cited By
Road disease patrol method and system
CN121259539A
Road disease detection method and device, electronic equipment and storage medium
CN121708559A
