Road disease detection method, device, electronic device and storage medium

Abstract

The present application relates to the field of computer technology, and in particular to the field of intelligent transportation technology, and provides a method, device, electronic device and storage medium for detecting road damage, so as to improve the accuracy of road damage detection. The method comprises: based on the intensity value of each pixel point in the initial road damage image, performing position detection on the road damage, and determining each first area not covered by the road damage; after denoising each first area, enhancing the first edge feature of the road damage in the initial road damage image, and obtaining an intermediate road damage image; performing position detection on the road damage in the intermediate road damage image, and determining each second area covered by the road damage; performing edge detection on the road damage in each second area, and obtaining the second edge feature corresponding to the intermediate road damage image; fusing the global feature of the intermediate road damage image with the second edge feature, and obtaining a global feature map; and determining the basic attribute information of the road damage based on the global feature map.

Description

Background Art

[0002] As time goes by, roads will develop road defects such as cracks, fissures, potholes, etc. In order to ensure driving safety, roads with defects need to be maintained in a timely manner, otherwise traffic accidents are likely to occur.

[0003] Timely road maintenance requires timely inspection of roads and discovery of road diseases. If this process is done manually, it will consume a lot of manpower, material resources and financial resources. In the related technology, there is a method for automatically detecting road diseases, which detects road diseases by inputting road images into a disease detection and recognition model, obtains the recognition results of road disease detection, and identifies the road disease type. However, this method requires filtering the road image, and when filtering, the entire road image needs to be smoothed, which easily leads to the reduction of road disease features and reduces the accuracy of detection.

[0004] In summary, how to improve the accuracy of road disease detection is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a road damage detection method, device, electronic device and storage medium to improve the accuracy of road damage detection.

[0006] A method for detecting road damage provided in an embodiment of the present application includes:

[0007] Based on the intensity value of each pixel point in the initial road damage image, the position of the road damage is detected to determine each first area not covered by the road damage;

[0008] After performing denoising on each first region in the initial road damage image, enhancing the first edge feature of the road damage in the initial road damage image to obtain an intermediate road damage image;

[0009] Based on the intensity value of each pixel point in the middle road damage image, position detection is performed on the road damage to determine each second area covered by the road damage;

[0010] Performing edge detection of road damage in each second region in the middle road damage image to obtain a second edge feature corresponding to the middle road damage image, and fusing the global feature of the middle road damage image with the second edge feature to obtain a global feature map;

[0011] Based on the global feature map, basic attribute information of the road damage is determined.

[0012] The present application provides a road disease detection device, comprising:

[0013] A first detection unit, configured to perform position detection on the road damage based on the intensity value of each pixel point in the initial road damage image, and determine each first area not covered by the road damage;

[0014] A processing unit, configured to enhance the first edge features of the road damage in the initial road damage image after performing denoising on each first region in the initial road damage image, so as to obtain an intermediate road damage image;

[0015] A second detection unit, configured to perform position detection on the road damage based on the intensity value of each pixel point in the middle road damage image, and determine each second area covered by the road damage;

[0016] a fusion unit, configured to perform edge detection of road damage in each second region in the middle road damage image, obtain a second edge feature corresponding to the middle road damage image, and fuse the global feature of the middle road damage image with the second edge feature to obtain a global feature map;

[0017] A determination unit is used to determine basic attribute information of the road damage based on the global feature map.

[0018] Optionally, the first detection unit is specifically used for:

[0019] Based on the preset values ​​corresponding to at least one preset detection template and the intensity value of each pixel point in each area of ​​the initial road damage image, the position of the road damage is detected to obtain the first detection value of each area; wherein each detection template is used to detect the road damage in different directions; the first detection value represents the probability that the corresponding area is the area covered by the road damage;

[0020] Based on the first detection values, first areas not covered by the road damage are determined.

[0021] Optionally, the first detection unit is specifically used for:

[0022] For each area in the initial road damage image, the following operations are performed:

[0023] Based on the respective preset values ​​corresponding to the at least one detection template and the respective intensity values ​​of a plurality of pixel points in an area, respectively obtaining the respective similarity feature values ​​of the one area and the at least one detection template;

[0024] Among the at least one similar feature value corresponding to the one area, a similar feature value is determined as the first detection value corresponding to the one area.

[0025] Optionally, the first detection unit is specifically used for:

[0026] In the initial road damage image, each region whose corresponding first detection value is less than a preset threshold is determined as each first region not covered by the road damage.

[0027] Optionally, the second detection unit is specifically used for:

[0028] Based on the preset values ​​corresponding to at least one preset detection template and the intensity value of each pixel point in each area of ​​the middle road damage image, the position of the road damage is detected to obtain the second detection value of each area; wherein each detection template is used to detect the road damage in different directions; the second detection value indicates the probability that the corresponding area is the area covered by the road damage;

[0029] Based on each second detection value, each second area covered by the road disease is determined.

[0030] Optionally, the second detection unit is specifically used for:

[0031] For each area in the middle road damage image, the following operations are performed:

[0032] Based on the respective preset values ​​corresponding to the at least one detection template and the respective intensity values ​​of a plurality of pixel points in an area, respectively obtaining the respective similarity feature values ​​of the one area and the at least one detection template;

[0033] Among the at least one similar feature value corresponding to the one area, a similar feature value is determined as the second detection value corresponding to the one area.

[0034] Optionally, the second detection unit is specifically used for:

[0035] In the middle road damage image, each region whose corresponding second detection value is not less than a preset threshold is determined as each second region covered by the road damage.

[0036] Optionally, the fusion unit is specifically used for:

[0037] Based on the intensity value of each pixel in each second region, acquiring each gradient value corresponding to each pixel in each second region in multiple directions;

[0038] Based on each gradient value, determining the edge of the road defect in each second area;

[0039] The determination result is input into the edge feature extraction model to obtain the second edge feature corresponding to the middle road disease image.

[0040] Optionally, the fusion unit is further used to obtain the global features of the intermediate road damage image in the following manner:

[0041] Based on the road damage detection model, the middle road damage image is downsampled, and semantic features are extracted from the downsampled middle road damage image to obtain global features of the middle road damage image.

[0042] Optionally, the basic attribute information includes type information and orientation information, and the determining unit is specifically configured to:

[0043] Generate a road damage rotation detection frame based on the rotation target detector in the road damage detection model;

[0044] The type information and orientation information of the road damage are determined based on the road damage rotation detection frame.

[0045] Optionally, the basic attribute information further includes a damage level, and the determining unit is further configured to:

[0046] After determining the type information and orientation information of the road damage based on the road damage rotation detection frame, obtaining a local feature map containing the road damage in the global feature map based on the minimum circumscribed rectangle of the road damage rotation detection frame, and performing upsampling processing on the local feature map based on the road damage detection model; the rotation angle of the minimum circumscribed rectangle is zero;

[0047] Based on the road damage detection model, obtaining a segmentation mask corresponding to the upsampled local feature map;

[0048] Based on the segmentation mask, the number of pixels occupied by the road damage in the upsampled local feature map is determined, and the damage level of the road damage is determined based on the number.

[0049] Optionally, the device further comprises:

[0050] An alarm unit is used to photograph the target path in each preset time period included in each of the multiple preset cycles to obtain multiple initial road disease images;

[0051] If road damage is detected in the target path among the multiple initial road damage images taken within a preset time period, a first-level pre-alarm state is set for the road damage;

[0052] If the road damage is in the first-level pre-alarm state in multiple preset time periods of a preset cycle, a second-level pre-alarm state is set for the road damage;

[0053] If within the plurality of preset periods, the road hazards are all in a level 2 pre-alarm state, an alarm is issued for the road hazards.

[0054] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-mentioned road hazard detection methods.

[0055] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the above-mentioned road hazard detection methods.

[0056] An embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the above-mentioned road disease detection methods.

[0057] The beneficial effects of this application are as follows:

[0058] The embodiments of the present application provide a method, device, electronic device and storage medium for detecting road damage. Before the initial road damage image is denoised, the present application first detects the position of road damage in the image based on the intensity value of the pixel points to determine the area in the image that does not contain road damage. Then, the initial road damage image is denoised only for this area, thereby retaining the features of the road damage part, rather than indiscriminately denoising the entire image. Then, the first edge feature of the road damage in the initial road damage image is further enhanced to obtain an intermediate road damage image. Then, the position of the road damage is detected from the intermediate road damage image to determine the area containing the road damage, and the edge detection of the road damage is performed only in this area to obtain the second edge feature corresponding to the intermediate road damage image, thereby improving the accuracy of edge detection. Then, the global feature map obtained by fusing the second edge feature with the global feature is used to extract basic attribute information of the road damage, thereby improving the accuracy of road damage detection.

[0059] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0061] Figure 1 A schematic diagram of an application scenario of a road disease detection method provided in an embodiment of the present application;

[0062] Figure 2 An overall flow chart of a road disease detection method provided in an embodiment of the present application;

[0063] Figure 3 A schematic diagram of a horizontal detection template provided in an embodiment of the present application;

[0064] Figure 4 A schematic diagram of a longitudinal detection template provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of a left oblique detection template provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of a right oblique detection template provided in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of a detection template movement method provided in an embodiment of the present application;

[0068] Figure 8 A flowchart of a denoising process provided in an embodiment of the present application;

[0069] Fig. 9 A flow chart for edge detection of a middle road disease image provided by an embodiment of the present application;

[0070] Fig.10 A comparison diagram of a normal detection frame and a rotation detection frame provided in an embodiment of the present application;

[0071] Fig.11 An alarm flow chart provided for an embodiment of the present application;

[0072] Fig.12 A structural diagram of a road disease detection module provided in an embodiment of the present application;

[0073] Fig.13 A schematic diagram of a disease analysis module provided in an embodiment of the present application;

[0074] Fig.14 An overall flow chart of another road disease detection method provided in an embodiment of the present application;

[0075] Fig.15 A schematic diagram of the structure of a road hazard detection device provided in an embodiment of the present application;

[0076] Fig.16 A schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present application;

[0077] Fig.17 A schematic diagram of the hardware structure of another electronic device provided for application of an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0079] The following is an introduction to some concepts involved in the embodiments of the present application.

[0080] Intensity value: represents the brightness of the pixel, which is independent of the pixel color. Its value is between 0 and 255, where 255 is the highest pixel brightness and 0 is the lowest brightness.

[0081] Road damage image: It is divided into initial road damage image and intermediate road damage image. The image obtained by shooting a certain position on the road is the initial road damage image, and the image obtained after denoising and edge feature enhancement of the initial road damage image is the intermediate road damage image.

[0082] Region: A road damage image can be divided into multiple regions. Depending on the different step sizes when the detection template traverses the road damage image, there may or may not be overlapping parts in the multiple regions. Furthermore, the present application also involves a first region and a second region, wherein the first region is the regions in the initial road damage image that are not covered by road damage; and the second region is the regions in the intermediate road damage image that are covered by road damage.

[0083] Detection value: represents the probability of the area covered by the road disease. In the present application, the detection value is divided into a first detection value and a second detection value, wherein the first detection value is the detection value of each area in the initial road disease image, and the second detection value is the detection value of each area in the intermediate road disease image.

[0084] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0085] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes two terminal devices 110 and a server 120.

[0086] In the embodiment of the present application, the terminal device 110 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals and other devices; a client related to road disease detection can be installed on the terminal device, which can be software (such as a browser, road disease detection software, etc.), or a web page, a small program, etc. The server 120 is a background server corresponding to the software or web page, small program, etc., or a server specifically used for road disease detection, which is not specifically limited in the present application. The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0087] It should be noted that the road defect detection method in each embodiment of the present application can be executed by an electronic device, which may be a terminal device 110 or a server 120, that is, the method can be executed by the terminal device 110 or the server 120 alone, or can be executed jointly by the terminal device 110 and the server 120. For example, when executed by the server 120, the server 120 determines each first area not covered by the road disease based on the intensity value of each pixel point in the initial road disease image; then, the server 120 performs denoising only on each first area in the initial road disease image, and enhances the first edge feature of the road disease in the denoised initial road disease image to obtain an intermediate road disease image; the server 120 determines each second area covered by the road disease based on the intensity value of each pixel point in the intermediate road disease image; then, the server 120 performs edge detection of the road disease only in each second area in the intermediate road disease image, obtains the second edge feature corresponding to the intermediate road disease image based on the edge detection result, and fuses the global feature of the intermediate road disease image with the second edge feature to obtain a global feature map; finally, the server 120 determines the basic attribute information of the road disease based on the global feature map.

[0088] In an optional implementation, the terminal device 110 and the server 120 may communicate with each other via a communication network.

[0089] In an optional implementation, the communication network is a wired network or a wireless network.

[0090] It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0091] In addition, the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation and other scenarios.

[0092] The following describes the road deterioration detection method provided by the exemplary embodiment of the present application in combination with the application scenario described above and with reference to the accompanying drawings. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this regard.

[0093] See also Figure 2 As shown, it is an implementation flow chart of a road disease detection method provided in an embodiment of the present application. The specific implementation process of the method is as follows S201-S205:

[0094] S201: The server performs position detection on the road damage based on the intensity value of each pixel point in the initial road damage image, and determines each first area not covered by the road damage.

[0095] In the above, the initial road damage image is an image obtained by fixed-point shooting or non-fixed-point shooting; among them, fixed-point shooting refers to the fixed-point detection task of road damage, and a camera or other equipment that can be used for shooting is installed at a fixed position on a certain section of the road, and the section is photographed regularly, and the initial road damage image obtained by shooting is detected to confirm whether there is road damage; non-fixed-point shooting means that the shooting position is not fixed, for example, the initial road damage image of a certain section is captured by the driving recorder in a moving car. The captured initial road damage image can be directly used for subsequent detection. If the resolution of the initial road damage image is low, the initial road damage image can also be reconstructed with super resolution and then used for subsequent detection.

[0096] There may be road damage in the initial road damage image. The server needs to perform position detection on the road damage in the initial road damage image to determine the position of the road damage in the image, or in other words, determine which part of the initial road damage image contains the road damage and which part does not contain the road damage.

[0097] An optional implementation is that the server performs position detection of road damage based on preset values ​​corresponding to at least one preset detection template and the intensity value of each pixel point in each area of ​​the initial road damage image, and obtains a first detection value for each area; based on each first detection value, determines each first area not covered by the road damage.

[0098] In the above, the intensity value of each pixel represents the brightness of each pixel, which has nothing to do with the hue of the pixel itself (different colors may have the same brightness). Its value is set between 0 and 255, 255 is the highest brightness (white), and 0 is the lowest brightness (black). Before performing position detection, the intensity value of each pixel needs to be normalized and mapped to the interval [0, 1]. After normalization, 0 represents the lowest brightness (black) and 1 represents the highest brightness (white); the first detection value represents the probability that the corresponding area is covered by road damage.

[0099] There are multiple detection templates, each of which is used to detect road damage in different directions, including horizontal, vertical, left oblique, right oblique, etc. Each direction corresponds to a detection template; for example, when the road damage is a crack, different detection templates can be used to detect the direction of the crack in the area. The size of the detection template can be m×n, where m and n are both integers greater than 1. The expression of the horizontal detection template is:

[0100]

[0101] In the formula, b and y are both positive integers, and x∈[1,n], y∈[1,m].

[0102] For ease of understanding, the above formula is explained below with reference to the accompanying drawings. Figure 3 FIG. 1 is a schematic diagram of a horizontal detection template provided in an embodiment of the present application. In the figure, it is assumed that the size of the horizontal detection template is 5×5, that is, m=5, n=5, g x (x,y)=0 is represented by a black square (white text on a black background), g x (x,y)=1 is represented by a white square (black text on a white background). And is a positive integer, so b can take the value of 1 or 2.

[0103] When b = 1, Figure 3 In part (a), there are two white square areas in the horizontal detection template, each with a width of 1 square. In the area of ​​1<y<5, g x The value of (x, y) is 0, which is the black square area in the horizontal detection template; in the area of ​​y=1 and y=5, g x The value of (x,y) is 1, which means the white square area in the horizontal detection template.

[0104] When b = 2, Figure 3 In part (b), there are two white square areas in the horizontal detection template, each with a width of 2 squares. In the area of ​​2<y<4, g x The value of (x, y) is 0, which is the black square area in the horizontal detection template; in the area where y=1, 2, 4, and 5, g x The value of (x,y) is 1, which means the white square area in the horizontal detection template.

[0105] The expression of the vertical detection template is:

[0106]

[0107] In the formula, b and x are both positive integers, and x∈[1,n], y∈[1,m].

[0108] The above formula is also explained with reference to the accompanying drawings. Figure 4 FIG. 1 is a schematic diagram of a longitudinal detection template provided in an embodiment of the present application. In the figure, it is assumed that the size of the longitudinal detection template is 5×5, that is, m=5, n=5, g y (x,y)=0 is represented by a black square (white text on a black background), g y (x,y)=1 is represented by a white square (black text on a white background). And is a positive integer, so b can take the value of 1 or 2.

[0109] When b = 1, Figure 4 In part (a), in the region 1<x<5, g y The value of (x, y) is 0, which is the black square area in the vertical detection template; in the area of ​​x = 1 and x = 5, g y The value of (x,y) is 1, which means the white square area in the vertical detection template.

[0110] When b = 2, Figure 4 In part (b), in the region 2<x<4, that is, in the region x=3, g y The value of (x, y) is 0, which is the black square area in the vertical detection template; in the area where x=1, 2, 4, and 5, g y The value of (x,y) is 1, which means the white square area in the vertical detection template.

[0111] The expression of the left oblique detection template is:

[0112]

[0113] In the formula, 0≤i≤m-2, i is an integer, x and y are both positive integers, and x∈[1,n], y∈[1,m].

[0114] The above formula is explained with reference to the accompanying drawings. Figure 5 FIG. 1 is a schematic diagram of a left oblique detection template provided in an embodiment of the present application. In the figure, it is assumed that the size of the left oblique detection template is 5×5, that is, m=5, n=5, g xy (x,y)=0 is represented by a black square (white text on a black background), g xy (x, y) = 1 is represented by a white square (black text on a white background). Since 0≤i≤m-2, i is an integer, so i can take the value 0, 1, 2, or 3.

[0115] When i = 0, Figure 5 In part (a), in the region y = 6-x, g yx The value of (x, y) is 0, which is the black square area in the right oblique detection template; while in other areas of the left oblique detection template, g y The value of (x,y) is 1, which is the white square area in the right oblique detection template.

[0116] When i = 1, such as Figure 5 In part (b), in the region 5-x≤y≤7+x, g yx The value of (x, y) is 0, which is the black square area in the right oblique detection template; while in other areas of the left oblique detection template, g yx The value of (x,y) is 1, which is the white square area in the right oblique detection template.

[0117] Similarly, when i = 2, such as Figure 5 In part (c), in the region 4-x≤y≤8+x, g yx The value of (x,y) is 0. In other areas of the left oblique detection template, g yx The value of (x, y) is 1; when i = 3, such as Figure 5 In part (d), in the region 3-x≤y≤9+x, g yx The value of (x,y) is 0. In other areas of the left oblique detection template, g yx The value of (x,y) is 1.

[0118] The expression of the right oblique detection template is:

[0119]

[0120] In the formula, 0≤i≤n-2, i is an integer, x and y are both positive integers, and x∈[1,n], y∈[1,m].

[0121] The above formula is explained with reference to the accompanying drawings. Figure 6 FIG. 1 is a schematic diagram of a right oblique detection template provided in an embodiment of the present application. In the figure, it is assumed that the size of the right oblique detection template is 5×5, that is, m=5, n=5, g yx (x,y)=0 is represented by a black square (white text on a black background), g yx (x,y) = 1 is represented by a white square (black text on a white background). Since 0≤i≤n-2, i is an integer, so i can take the value 0, 1, 2, or 3.

[0122] When i = 0, Figure 6 In part (a), in the region y=x, g yx The value of (x, y) is 0, which is the black square area in the right oblique detection template; while in other areas of the right oblique detection template, g y The value of (x,y) is 1, which is the white square area in the right oblique detection template.

[0123] When i = 1, such as Figure 6 In part (b), in the region x-1≤y≤x+1, g yx The value of (x, y) is 0, which is the black square area in the right oblique detection template; while in other areas of the right oblique detection template, g yx The value of (x,y) is 1, which is the white square area in the right oblique detection template.

[0124] Similarly, when i = 2, such as Figure 6 In part (c), in the region x-2≤y≤x+2, gyx The value of (x,y) is 0. In other areas of the right oblique detection template, g yx The value of (x, y) is 1; when i = 3, such as Figure 6 In part (d), in the region x-3≤y≤x+3, g yx The value of (x,y) is 0. In other areas of the right oblique detection template, g yx The value of (x,y) is 1.

[0125] The m, n, and b values ​​mentioned above can be adjusted according to different situations of different initial road damage images. When position detection is performed on the same initial road damage image, m, n, and b will not change after being determined.

[0126] The above detection templates are used to traverse the initial road damage image, such as Figure 7 As shown, it is a schematic diagram of a detection template movement method provided by an embodiment of the present application. In the figure, it is assumed that the template size is 3×3, and the initial road damage image size is 6×6 (pixels). The distance of each movement of the detection template, that is, the step size, can be set according to the actual situation. The minimum can be set to 1 and the maximum can be set to 3. The step size cannot exceed the length and width of the detection template itself. Each time the detection template moves, the 3×3 (pixel) part where it stays is an area of ​​the initial road damage image. Therefore, the number of areas divided by the initial road damage image is related to the step size. When the step size is 1, the detection template moves 1 pixel unit each time until the entire initial road damage image is traversed. The initial road damage image of 6×6 pixels can be divided into 16 areas, and there are overlapping parts between areas; when the step size is 3, the detection template moves 3 pixel units each time until the entire initial road damage image is traversed. The initial road damage image of 6×6 pixels can be divided into 4 areas, and there is no overlapping part between areas.

[0127] It should be noted that Figure 7 The size of the detection template is slightly reduced for the convenience of distinction. In practical applications, the detection template of m×n (for example, 3×3) size should be the same as the size of an m×n (for example, 3×3) pixel area in the initial road damage image.

[0128] For each area in the initial road damage image, the server obtains the first detection value corresponding to the area by performing the following operations: based on the respective preset values ​​corresponding to at least one detection template and the respective intensity values ​​of multiple pixels in an area, respectively obtain the similar feature values ​​of an area and at least one detection template; among the at least one similar feature value corresponding to an area, determine a similar feature value as the first detection value corresponding to an area.

[0129] In the above, each preset value corresponding to each detection template is 1 or 0, and the specific situation is determined by the expression corresponding to the detection template, for example Figure 3 The preset values ​​contained in different 5×5 sized horizontal detection templates are listed.

[0130] For each area in the initial road damage image, it is necessary to obtain its corresponding first detection value. For each area in the initial road damage image, the specific implementation method is to normalize the intensity values ​​of each pixel point contained in a certain area in the initial road damage image, and then calculate the similarity feature value between the detection template and the area according to the normalized intensity values ​​and the preset value corresponding to the detection template. The calculation method of the similarity feature value is not specifically limited in this application, and it can be Euclidean distance, Manhattan distance, etc. A plurality of detection templates are used to calculate the similarity feature values ​​of the area, and finally a similarity feature value is selected from the plurality of similarity feature values ​​as the first detection value corresponding to the area. The selection method can be to select the one with the largest similarity feature value as the first detection value corresponding to the area.

[0131] After obtaining the first detection value corresponding to each area, it is determined whether the area belongs to the first area based on the first detection value. An optional implementation is that the server determines each area in the initial road damage image whose corresponding first detection value is less than a preset threshold as the first areas not covered by road damage.

[0132] Since the first detection value represents the probability that the area is covered by road damage, a threshold can be set. When the first detection value is ≥ the preset threshold, it indicates that the area is covered by road damage. When the first detection value is < the preset threshold, it indicates that the area does not contain road damage. Ultimately, all areas that do not contain road damage are determined, and these areas are determined as first areas.

[0133] If it is detected through the above process that there is no road damage in the initial road damage image, the detection result that no road damage is found in the initial road damage image can be directly output, and the subsequent steps will not be continued for the initial road damage image.

[0134] Assume that the server now detects the position of road damage in an initial road damage image, uses 3×3 horizontal, vertical, left oblique and right oblique detection templates, sets the step size to 1, and calculates similar feature values ​​for each area in the initial road damage image. Finally, each area obtains 4 similar feature values. The largest one among the 4 similar feature values ​​is selected as the first detection value corresponding to the area, and compared with the preset threshold, and each area whose corresponding first detection value is less than the preset threshold is determined as the first area.

[0135] S202: After performing denoising processing on each first region in the initial road damage image, the server enhances the first edge features of the road damage in the initial road damage image to obtain an intermediate road damage image.

[0136] Since each first area in the initial road damage image does not contain road damage, the server only performs smoothing and denoising processing on the first area to reduce noise interference for subsequent work, rather than performing smoothing and denoising processing on the entire initial road damage image indiscriminately. This can prevent the road damage in the initial road damage image from being blurred and the characteristics of the road damage from being seriously weakened.

[0137] In summary, based on the denoising process in S201 and S202, as Figure 8 As shown, it is a flowchart of a denoising process provided by an embodiment of the present application, and the server specifically performs the following steps:

[0138] Step 801: The server normalizes the intensity value of each pixel in the initial road damage image.

[0139] Step 802: The server obtains a first detection value of an area based on the preset values ​​corresponding to the detection templates and the intensity value of each pixel in the area.

[0140] Specifically, the server needs to obtain similar feature values ​​corresponding to each detection template and the area of ​​the region, and select one of the multiple similar feature values ​​as the first detection value of the area.

[0141] Step 803: The server determines whether the first detection value is ≥ a preset threshold value, and if so, executes step 804, otherwise, executes step 805.

[0142] Step 804: The server determines that the area is not the first area and does not perform denoising processing on the area.

[0143] Step 805: The server determines that the area is the first area, and performs denoising processing on the area.

[0144] Step 806: The server slides the detection template to the next area in the initial road damage image based on the step distance.

[0145] In addition to determining whether to perform denoising on each area based on the first detection value of the area and executing the denoising process for each area detected in the above flowchart, the server can also complete the detection of all areas, and then determine whether each area is the first area based on the first detection value of each area, and perform denoising on each area in turn.

[0146] Afterwards, the denoised initial road damage image is sharpened to enhance the first edge feature of the road damage, thereby obtaining an intermediate road damage image. For example, when the road damage is a crack, sharpening can enhance the feature of the fine crack.

[0147] Following the assumption in S201, the server performs denoising processing only on the first area, and performs sharpening processing on the denoised initial road damage image to obtain an intermediate road damage image.

[0148] S203: The server performs position detection on the road damage based on the intensity value of each pixel point in the middle road damage image, and determines each second area covered by the road damage.

[0149] After sharpening, not only the features of the road damage part but also the features of the entire image are enhanced. Therefore, the server needs to perform another position detection on the road damage in the middle road damage image to determine the second areas covered by the road damage.

[0150] An optional implementation is that the server performs position detection on the road damage based on the preset values ​​corresponding to each of the at least one preset detection templates and the intensity value of each pixel point in each area of ​​the middle road damage image, and obtains a second detection value for each area; based on each second detection value, the server determines each second area covered by the road damage.

[0151] The above-mentioned detection templates are also used to detect road damage in different directions. They are the same templates as the detection templates used to detect the initial road damage image in S201. The area division method of the intermediate road damage image is also the same as that of the initial road damage image, and is related to the step size, which will not be repeated in this application; the second detection value represents the probability that the corresponding area in the intermediate road damage image is the area covered by road damage.

[0152] An optional method for obtaining the second detection value corresponding to each area of ​​the middle road damage image is that for each area in the middle road damage image, the server performs the following operations respectively: based on the respective preset values ​​corresponding to at least one detection template and the respective intensity values ​​of multiple pixel points in an area, respectively obtain the similar feature values ​​of an area and at least one detection template; and determine a similar feature value from at least one similar feature value corresponding to an area as the second detection value corresponding to an area.

[0153] Afterwards, the server determines each area in the middle road damage image whose corresponding second detection value is not less than a preset threshold as each second area covered by the road damage.

[0154] In the above, the method for obtaining the second detection value corresponding to each area of ​​the middle road damage image is the same as the method for obtaining the first detection value of each area of ​​the initial road damage image in S201. Taking an area of ​​the middle road damage image as an example, the server first normalizes the intensity values ​​of each pixel point contained in a certain area of ​​the middle road damage image, and then calculates the similar feature values ​​corresponding to the area and each detection template respectively according to the normalized intensity values ​​and the preset values ​​corresponding to the detection template. And select a similar feature value from multiple similar feature values ​​as the second detection value corresponding to the area, and the selection method can be to select the one with the largest similar feature value as the second detection value corresponding to the area.

[0155] The second detection value represents the probability that the area is covered by road damage. After obtaining the second detection value corresponding to each area, the server can determine each area in the middle road damage image whose corresponding second detection value is ≥ the preset threshold as each second area covered by road damage. If the second detection value of a certain area is less than the preset threshold, it indicates that the possibility of the existence of road damage in the area is low, and the server can directly determine that the area is not covered by road damage. The preset threshold in the above is the same as the preset threshold in S201.

[0156] Continuing with the assumption in S202, continue to use the 3×3 horizontal, vertical, left oblique, and right oblique detection templates to calculate similar feature values ​​for each area in the middle road disease image. Similarly, each area obtains 4 similar feature values, and the largest one is selected from the 4 similar feature values ​​as the second detection value corresponding to the area, and compared with the preset threshold, and each area corresponding to the second detection value ≥ the preset threshold is determined as each second area.

[0157] S204: The server performs edge detection of road damage in each second area in the middle road damage image, obtains a second edge feature corresponding to the middle road damage image, and fuses the global feature of the middle road damage image with the second edge feature to obtain a global feature map.

[0158] For each second area, the server detects the edge of the road damage therein. An optional implementation is that the server obtains the gradient values ​​corresponding to each pixel point in each second area in multiple directions based on the intensity value of each pixel point; and determines the edge of the road damage in each second area based on the gradient values; and inputs the determination result into the edge feature extraction model to obtain the second edge feature corresponding to the middle road damage image.

[0159] Since the essence of edge detection is actually to find points where the intensity value changes dramatically, and multiple points are connected to form edges, the edges of road damage can be determined based on the gradient values ​​of each pixel in the x and y directions. In this process, the non-maximum suppression method is also used, that is, the non-maximum pixel points are removed to achieve the effect of suppressing noise.

[0160] After that, the server also needs to use a bilateral threshold to connect the edges. Specifically, a low threshold and a high threshold are set. The edge pixel points with a gradient greater than the high threshold are regarded as strong edge points, and those with a gradient less than the low threshold are regarded as weak edge points, and the weak edge points are suppressed.

[0161] Finally, the server outputs a binary image, that is, the edge of the road damage in the image is set to 1 (foreground pixel), usually black, and all others are set to 0 (background pixel), usually white.

[0162] In the above, the specific implementation of edge detection can adopt the Canny edge detection operator, and can also use the Sobel, Prewitt, Roberts edge detection operators, etc., which are not specifically limited in this application.

[0163] In summary, the process of determining the second area and performing edge detection in S201 and S202 is combined, as shown in FIG. Fig. 9 As shown, it is a flowchart of edge detection of a middle road disease image provided by an embodiment of the present application. The server specifically performs the following steps for each area of ​​the middle road disease image:

[0164] Step 901: The server normalizes the intensity value of each pixel.

[0165] Step 902: The server obtains a second detection value of an area based on the preset values ​​corresponding to the detection templates and the intensity value of each pixel in the area.

[0166] Step 903: The server determines whether the second detection value is ≥ a preset threshold value, and if so, executes step 904, otherwise, executes step 905.

[0167] If the second detection value is greater than or equal to the preset threshold, the corresponding area is confirmed as the second area.

[0168] Step 904: The server obtains the gradient value of each pixel in the x and y directions.

[0169] Step 905: The server slides the detection template to the next area in the middle road damage image based on the step distance.

[0170] Step 906: The server uses a non-maximum suppression method to suppress pseudo-edge pixels.

[0171] Step 907: Use a bilateral threshold to connect edge pixels.

[0172] In the above, the process of obtaining the gradient value of each pixel in the x and y directions, the process of suppressing pseudo edge pixels, and the process of connection processing can occur after each second area is determined, or after the detection template traverses the entire intermediate road disease image. The normalization in step 901 can be performed on all areas together, and the subsequent calculation of the second detection value can be performed after all normalization is completed.

[0173] After the entire edge detection is completed, the server outputs a binary image.

[0174] Through the above process, selective edge detection is performed on different areas of the middle road disease image (edge ​​detection is performed only on the second area), which can have a good filtering effect on water stains, shadows, noise, etc.

[0175] After the binarized image is obtained, the binarized image is input into a pre-trained edge feature extraction model to obtain the second edge feature corresponding to the middle road disease image, that is, the edge semantic feature.

[0176] The global features can be obtained in the following way: the server downsamples the middle road damage image based on the road damage detection model, and extracts semantic features from the downsampled middle road damage image to obtain the global features of the middle road damage image.

[0177] The road damage detection model includes an encoder, i.e., a downsampling neural network, which can downsample the intermediate road damage image to obtain the global features of the intermediate road damage image.

[0178] Finally, the obtained second edge features are fused with the global features to obtain the global feature map.

[0179] Following the assumption in S202, for the pixel points in each second area, the gradient of the intensity value is calculated in the x and y directions, and the pixel points with non-maximum values ​​are removed to obtain the pixel points located at the edge of the road disease. Finally, the pixel points are connected using a bilateral threshold to obtain the edge of the road disease in the intermediate road disease image. The pixel points in the edge part are set to 1, and the other pixel points are all set to 0 to obtain the corresponding binary image.

[0180] The server inputs the binarized image into the edge feature extraction model to obtain the second edge feature; inputs the middle road damage image into the road damage detection model for downsampling, and extracts semantic features from the downsampled middle road damage image to obtain the global feature of the middle road damage image. The server fuses the global feature with the second edge feature to obtain a global feature map.

[0181] S205: The server determines basic attribute information of road hazards based on the global feature map.

[0182] In the above, the basic attribute information includes type information and orientation information, and the orientation information further includes position information and angle information; the road disease detection model also includes a rotating target detector, such as Fig.10 As shown, it is a comparison diagram of a common detection frame and a rotation detection frame provided by an embodiment of the present application. Assuming that the road damage is a crack, the relevant technology will use a common detection frame to detect its type. However, if the road damage is a crack that spans the diagonal of the entire road damage image, the common detection frame can only detect the type of crack. However, since the detection frame almost occupies the entire road damage image and has no angle, it is impossible to confirm the position and angle of the crack based on the size, position and other information of the detection frame. However, the present application adopts a rotating target detector, and the rotation detection frame of the rotating target detector can have a certain angle and will not occupy the entire image, which can reduce the area of ​​invalid features. Therefore, the rotation detection frame of the rotating target detector can not only detect the crack type as a transverse crack, a longitudinal crack, a crack, a pothole, a block crack, etc., as well as the confidence of the detection result, but also output the position and angle of the crack according to the position and angle of the rotation detection frame. The rotating target detector can adopt an end-to-end cascaded directed target detector (English: Refined Rotated RetinaNet Detector, R3Det) and the like.

[0183] In summary, the server can generate a road disease rotation detection frame based on the rotating target detector in the road disease detection model; and determine the type information and orientation information of the road disease based on the road disease rotation detection frame.

[0184] In addition, the rotating object detector can also predict the coordinates of the rotating detection frame based on the global feature map. Based on the coordinates, the server can draw the rotating detection frame on images such as the initial road damage image, the denoised initial road damage image, and the intermediate road damage image.

[0185] In addition to type information and orientation information, basic attribute information also includes damage level. The road disease detection model also includes a decoder, which is an upsampling neural network. After the server determines the type information and orientation information of the road disease based on the road disease rotation detection frame, it can also obtain a local feature map containing the entire road disease part in the global feature map based on the minimum circumscribed rectangle of the road disease rotation detection frame, and upsample the local feature map based on the decoder in the road disease detection model; and based on the road disease detection model, obtain the segmentation mask corresponding to the upsampled local feature map; finally, the server determines the number of pixels occupied by the road disease in the upsampled local feature map based on the segmentation mask, and determines the damage level of the road disease based on the number.

[0186] In the above, the rotation angle of the minimum circumscribed rectangle of the rotated detection frame is zero, that is, a right rectangle; the segmentation mask is to mark each pixel in the image with a number. For example, in the present application, the pixels covered by road damage in the image are marked as 1, and the background pixels are marked as 0. Assuming that the road damage is a crack, the server can calculate the number of pixels with the number 1 to obtain the width of the crack and then determine the degree of damage.

[0187] In the above, both the down-sampling neural network and the up-sampling neural network are deep learning models, and convolutional neural networks, Transformer models, etc. can be used.

[0188] The above is the overall process of road disease detection for an initial road disease image. In the related art, there is no strategy for setting road disease alarm. Once a road disease is detected in an image, an alarm will be directly triggered, resulting in a high false detection rate and false alarm rate of road diseases. Therefore, due to the complexity of actual road conditions, in order to reduce false alarms, this application proposes an alarm strategy. Only when the alarm conditions are met, an alarm will be triggered. It is mainly divided into fixed-point detection tasks and non-fixed-point detection tasks.

[0189] For the task of fixed-point detection, a fixed shooting device, such as a camera, is installed at a certain position on a certain road section. The target path is photographed respectively within each preset time period contained in multiple preset cycles to obtain multiple initial road damage images. Since it is a fixed-point shooting at a fixed time, the real-time requirement of this task is not high. If the server detects the existence of road damage in the target path from multiple initial road damage images taken within a preset time period, a first-level pre-alarm state can be set for the road damage. If the road damage is in the first-level pre-alarm state within multiple preset time periods of a preset cycle, a second-level pre-alarm state is set for the road damage. If the road damage is in the second-level pre-alarm state within multiple preset cycles, an alarm is issued for the road damage.

[0190] like Fig.11As shown, it is an alarm flow chart provided by an embodiment of the present application. Assume that the cycle is set to 24 hours, that is, one day, and each day is divided into three time periods, namely, 8:00-8:10 a.m., 12:00-12:10 p.m., and 6:00-6:10 p.m. In each time period, the shooting device continuously shoots p frames at the same position, that is, outputs p initial road disease images, and the server executes the following steps:

[0191] Step 1101: The server performs road damage detection.

[0192] Step 1102: The server determines whether road damage is detected in p consecutive initial road damage images. If so, step 1103 is executed; otherwise, the server returns to step 1101.

[0193] Step 1103: Enter the first level pre-alarm state.

[0194] Step 1104: The server determines whether it is in the first-level pre-alarm state in q different time periods. If so, execute step 1105, otherwise return to step 1101.

[0195] Step 1105: Enter the second level pre-alarm state.

[0196] Step 1106: The server determines whether the system has entered the second-level pre-alarm state for u consecutive days. If so, execute step 1107, otherwise return to step 1101.

[0197] Step 1107: Server alarm.

[0198] Through the above method, false alarms caused by shadows and water stains can be effectively filtered out, thereby improving the accuracy of alarms.

[0199] For non-fixed-point detection tasks, such as detecting the initial road damage images captured by the driving recorder in the car, this task has high real-time requirements. If p frames of initial road damage images are captured continuously and road damage is detected, an alarm is directly issued.

[0200] Following the assumptions in S204, the rotating target detector in the road damage detection model can generate a road damage rotating detection frame, detect the type, confidence, position and angle of the road damage according to the global feature map, and obtain a local feature map containing the entire road damage part in the global feature map based on the minimum circumscribed rectangle of the road damage rotating detection frame. The decoder in the road damage detection model upsamples the local feature map and obtains the segmentation mask corresponding to the local feature map. Finally, the server can determine the damage level of the road disease based on the segmentation mask.

[0201] Assume that the cycle is one day, and each day is divided into three time periods. In each time period, p frames are continuously detected. If road diseases are detected in all p frames, the system enters the first-level pre-alarm state. If the system is in the first-level pre-alarm state in all three time periods of the day, the system enters the second-level pre-alarm state. If the system is in the second-level pre-alarm state for two consecutive days, the system will issue an alarm. Further, assume that the server has continuously detected p frames and found road diseases, and the system enters the first-level pre-alarm state. In the next two time periods, the server continues to detect, and in both time periods, the system enters the first-level pre-alarm state, and the system is upgraded to the second-level pre-alarm state. If the server continues to detect in the next 24 hours and also enters the second-level pre-alarm state, the server will issue an alarm.

[0202] In summary, the road damage detection method proposed in this application can be completed by cooperating with the following modules: Fig.12 As shown, it is a structural diagram of a road disease detection module provided by an embodiment of the present application, which can directly input the initial road disease image, or input the initial road disease image after super-resolution reconstruction. The module includes a filtering module, which is responsible for detecting the first area in the initial road disease image that is not covered by road diseases, and denoising the first area; then the sharpening module can sharpen the denoised initial road disease image, enhance the first edge feature, and obtain the middle road disease image; the edge detection module is responsible for detecting the second area covered by road diseases in the middle road disease image, and detecting the road disease edge in the second area, and outputting a binary image; the edge feature extraction model obtains the second edge feature corresponding to the middle road disease image according to the binary image. The disease analysis module is used to obtain the basic attribute information of the road disease according to the middle road disease image and the second edge feature; the disease alarm module is responsible for alarming.

[0203] like Fig.13 As shown, it is a schematic diagram of a disease analysis module provided by an embodiment of the present application. The disease analysis module can downsample the middle road disease image, and extract semantic features from the downsampled middle road disease image to obtain the global features of the middle road disease image. The global features are fused with the second edge features to obtain a global feature map. Rotational target detection can determine the rotation detection frame of the road disease based on the global feature map. The rotation detection frame can derive the road disease type, confidence, and location and angle of the road disease. Based on the minimum circumscribed rectangle (rectangle) of the rotation detection frame, a local feature map containing the entire road disease portion can be obtained in the global feature map. The local feature map is upsampled to obtain the segmentation mask corresponding to the local feature map. Finally, based on the segmentation mask, the number of pixels occupied by the road disease can be determined to further determine the damage level of the road disease.

[0204] Based on the above process, Fig.14As shown, it is an overall flow chart of another road disease detection method provided by an embodiment of the present application, and the server specifically performs the following steps:

[0205] Step 1401: For each area in the initial road damage image, the server obtains a first detection value of each area based on each detection template.

[0206] For each area, the server obtains similarity feature values ​​between an area and at least one detection template based on respective preset values ​​corresponding to at least one detection template and respective intensity values ​​of multiple pixel points in an area; among at least one similarity feature value corresponding to an area, the one with the largest similarity feature value is taken as the first detection value corresponding to an area.

[0207] Step 1402: The server determines a first area in the initial road damage image according to first detection values ​​of each area of ​​the initial road damage image.

[0208] After obtaining the first detection value of an area, the server determines whether the first detection value is less than a preset threshold value. If so, the area is determined as the first area not covered by road damage.

[0209] Alternatively, after all areas are detected, whether each area is a first area is determined based on the first detection value of each area. That is, the present application does not limit whether the step of determining the first area occurs after each first detection value is obtained or after all areas are detected.

[0210] Step 1403: The server performs denoising processing on the first area, and performs sharpening processing on the denoised initial road damage image to obtain an intermediate road damage image.

[0211] Step 1404: The server determines a second area in the middle road damage image according to the second detection values ​​of each area of ​​the middle road damage image.

[0212] For each area in the middle road damage image, the server performs the following operations respectively: based on the respective preset values ​​corresponding to at least one detection template and the respective intensity values ​​of multiple pixel points in an area, respectively obtain the similarity feature values ​​between an area and at least one detection template; and determine a similar feature value among at least one similar feature value corresponding to an area as the second detection value corresponding to an area. After that, the server determines the areas in the middle road damage image whose corresponding second detection values ​​are not less than the preset threshold as the second areas covered by the road damage.

[0213] Similarly, the step of determining the second area may occur after each second detection value is obtained, or may occur after all areas in the middle road disease image are detected, and this application does not make any specific limitation.

[0214] Step 1405: The server performs edge detection of road damage in each second area in the middle road damage image to obtain a second edge feature corresponding to the middle road damage image.

[0215] Specifically, in each second area, the server determines the edge of the road disease according to the gradient value of each pixel in the x and y directions, and adopts the non-maximum suppression method, that is, after removing the non-maximum pixel points, the server uses the bilateral threshold to suppress the weak edge points, and uses a binary image to represent the edge and background, that is, the edge part of the road disease in the image is set to 1, and all others are 0.

[0216] Finally, the server inputs the binary image into a pre-trained edge feature extraction model to obtain the second edge feature corresponding to the middle road disease image.

[0217] Step 1406: The server downsamples the middle road damage image based on the road damage detection model, and extracts semantic features from the downsampled middle road damage image to obtain global features of the middle road damage image.

[0218] Step 1407: The server fuses the second edge feature with the global feature to obtain a global feature map.

[0219] Step 1408: The server determines the type information and orientation information of the road damage based on the rotating target detector.

[0220] Step 1409: The server extracts a local feature map including the entire road damage portion from the global feature map.

[0221] The server extracts the global feature map based on the minimum circumscribed rectangle (rectangle) of the road disease rotation detection frame generated by the rotating object detector.

[0222] Step 1410: The server performs upsampling processing on the local feature map based on the decoder; and obtains a segmentation mask corresponding to the upsampled local feature map.

[0223] Step 1411: The server determines the number of pixels occupied by the road damage in the upsampled local feature map based on the segmentation mask, and determines the damage level of the road damage based on the number.

[0224] Based on the same inventive concept, the present application also provides a road disease detection device. Fig.15As shown, it is a schematic diagram of the structure of a road disease detection device, which may include:

[0225] The first detection unit 1501 is used to perform position detection on the road damage based on the intensity value of each pixel point in the initial road damage image, and determine each first area not covered by the road damage;

[0226] The processing unit 1502 is used to enhance the first edge features of the road damage in the initial road damage image after performing denoising on each first region in the initial road damage image, so as to obtain an intermediate road damage image;

[0227] The second detection unit 1503 is used to detect the position of the road damage based on the intensity value of each pixel point in the middle road damage image, and determine each second area covered by the road damage;

[0228] A fusion unit 1504 is used to perform edge detection of road damage in each second area in the middle road damage image, obtain a second edge feature corresponding to the middle road damage image, and fuse the global feature of the middle road damage image with the second edge feature to obtain a global feature map;

[0229] The determination unit 1505 is used to determine basic attribute information of road damage based on the global feature map.

[0230] Optionally, the first detection unit 1501 is specifically used for:

[0231] Based on the preset values ​​corresponding to at least one preset detection template and the intensity value of each pixel point in each area of ​​the initial road damage image, the position of the road damage is detected to obtain the first detection value of each area; wherein each detection template is used to detect the road damage in different directions; the first detection value indicates the probability that the corresponding area is the area covered by the road damage;

[0232] Based on the first detection values, first areas not covered by road damage are determined.

[0233] Optionally, the first detection unit 1501 is specifically used for:

[0234] For each area in the initial road damage image, the following operations are performed:

[0235] Based on respective preset values ​​corresponding to at least one detection template and respective intensity values ​​of a plurality of pixel points in an area, respectively obtaining respective similarity feature values ​​of an area and at least one detection template;

[0236] Among at least one similar feature value corresponding to an area, a similar feature value is determined as a first detection value corresponding to the area.

[0237] Optionally, the first detection unit 1501 is specifically used for:

[0238] In the initial road damage image, each region whose corresponding first detection value is less than a preset threshold is determined as each first region not covered by the road damage.

[0239] Optionally, the second detection unit 1503 is specifically used for:

[0240] Based on the preset values ​​corresponding to at least one preset detection template and the intensity value of each pixel point in each area of ​​the middle road damage image, the position of the road damage is detected to obtain the second detection value of each area; wherein each detection template is used to detect the road damage in different directions; the second detection value indicates the probability that the corresponding area is the area covered by the road damage;

[0241] Based on the second detection values, second areas covered by the road damage are determined.

[0242] Optionally, the second detection unit 1503 is specifically used for:

[0243] For each area in the middle road damage image, perform the following operations:

[0244] Based on respective preset values ​​corresponding to at least one detection template and respective intensity values ​​of a plurality of pixel points in an area, respectively obtaining respective similarity feature values ​​of an area and at least one detection template;

[0245] Among at least one similar feature value corresponding to an area, a similar feature value is determined as a second detection value corresponding to the area.

[0246] Optionally, the second detection unit 1503 is specifically used for:

[0247] In the middle road damage image, each region whose corresponding second detection value is not less than a preset threshold is determined as each second region covered by the road damage.

[0248] Optionally, the fusion unit 1504 is specifically configured to:

[0249] Based on the intensity value of each pixel point in each second area, obtaining each gradient value corresponding to each pixel point in each second area in multiple directions;

[0250] determining the edge of the road defect in each second area based on each gradient value;

[0251] The determination result is input into the edge feature extraction model to obtain the second edge feature corresponding to the middle road disease image.

[0252] Optionally, the fusion unit 1504 is further configured to obtain the global features of the intermediate road damage image in the following manner:

[0253] Based on the road damage detection model, the middle road damage image is downsampled, and semantic features are extracted from the downsampled middle road damage image to obtain the global features of the middle road damage image.

[0254] Optionally, the basic attribute information includes type information and orientation information, and the determining unit 1505 is specifically configured to:

[0255] Generate a road damage rotation detection frame based on the rotation target detector in the road damage detection model;

[0256] The type and orientation information of the road damage are determined based on the road damage rotation detection frame.

[0257] Optionally, the basic attribute information also includes a damage level, and the determining unit 1505 is further configured to:

[0258] After determining the type information and orientation information of the road damage based on the road damage rotation detection frame, a local feature map containing the road damage is obtained in the global feature map based on the minimum bounding rectangle of the road damage rotation detection frame, and the local feature map is upsampled based on the road damage detection model; the rotation angle of the minimum bounding rectangle is zero;

[0259] Based on the road damage detection model, the segmentation mask corresponding to the upsampled local feature map is obtained;

[0260] Based on the segmentation mask, the number of pixels occupied by the road damage in the upsampled local feature map is determined, and the damage level of the road damage is determined based on the number.

[0261] Optionally, the device further comprises:

[0262] The alarm unit 1506 is used to photograph the target path in each preset time period included in each of the multiple preset cycles to obtain multiple initial road damage images;

[0263] If road damage is detected in the target path among the multiple initial road damage images taken within a preset time period, a first-level pre-alarm state is set for the road damage;

[0264] If the road damage is in the first-level early warning state in multiple preset time periods of a preset cycle, a second-level early warning state is set for the road damage;

[0265] If the road damage is in the second-level pre-alarm state within multiple preset periods, an alarm will be issued for the road damage.

[0266] After introducing the road damage detection method and device according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0267] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0268] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device may be a server, such as Figure 1 In this embodiment, the structure of the electronic device can be as follows: Fig.16 As shown, it includes a memory 1601 , a communication module 1603 and one or more processors 1602 .

[0269] The memory 1601 is used to store computer programs executed by the processor 1602. The memory 1601 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0270] The memory 1601 may be a volatile memory, such as a random-access memory (RAM); the memory 1601 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1601 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1601 may be a combination of the above memories.

[0271] The processor 1602 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1602 is used to implement the above-mentioned road disease detection method when calling the computer program stored in the memory 1601 .

[0272] The communication module 1603 is used to communicate with terminal devices and other servers.

[0273] The specific connection medium between the memory 1601, the communication module 1603 and the processor 1602 is not limited in the embodiment of the present application. Fig.16 In the embodiment, the memory 1601 and the processor 1602 are connected via a bus 1604. The bus 1604 is connected to the processor 1602 via a bus 1604. Fig.16 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1604 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.16 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0274] The memory 1601 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the road disease detection method of the embodiment of the present application. The processor 1602 is used to execute the above-mentioned road disease detection method, such as Figure 2 shown.

[0275] In another embodiment, the electronic device may also be other electronic devices, such as Figure 1 The terminal device 110 shown in FIG. 1 is a terminal device 110 shown in FIG. 1 . In this embodiment, the structure of the electronic device can be as follows: Fig.17 As shown, it includes: a communication component 1710, a memory 1720, a display unit 1730, a camera 1740, a sensor 1750, an audio circuit 1760, a Bluetooth module 1770, a processor 1780 and other components.

[0276] The communication component 1710 is used to communicate with the server. In some embodiments, a wireless fidelity (WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The electronic device can help the user to send and receive information through the WiFi module.

[0277] The memory 1720 can be used to store software programs and data. The processor 1780 executes various functions and data processing of the terminal device 110 by running the software programs or data stored in the memory 1720. The memory 1720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The memory 1720 stores an operating system that enables the terminal device 110 to run. In the present application, the memory 1720 can store an operating system and various application programs, and can also store a computer program for executing the road disease detection method of the embodiment of the present application.

[0278] The display unit 1730 may also be used to display information input by a user or information provided to a user and a graphical user interface (GUI) of various menus of the terminal device 110. Specifically, the display unit 1730 may include a display screen 1732 disposed on the front of the terminal device 110. The display screen 1732 may be configured in the form of a liquid crystal display, a light emitting diode, or the like.

[0279] The display unit 1730 can also be used to receive input digital or character information and generate signal input related to user settings and function control of the terminal device 110. Specifically, the display unit 1730 may include a touch screen 1731 arranged on the front of the terminal device 110, which can collect user touch operations on or near it, such as clicking a button, dragging a scroll box, etc.

[0280] The touch screen 1731 can be covered on the display screen 1732, or the touch screen 1731 and the display screen 1732 can be integrated to realize the input and output functions of the terminal device 110, and the integrated display screen can be referred to as a touch display screen. In this application, the display unit 1730 can display the application and the corresponding operation steps.

[0281] Camera 1740 can be used to capture static images, and users can publish images taken by camera 1740 through the application. Camera 1740 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the processor 1780 to convert it into a digital image signal.

[0282] The terminal device may further include at least one sensor 1750, such as an acceleration sensor 1751, a distance sensor 1752, a fingerprint sensor 1753, and a temperature sensor 1754. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0283] The audio circuit 1760, the speaker 1761, and the microphone 1762 can provide an audio interface between the user and the terminal device 110. The audio circuit 1760 can transmit the electrical signal converted from the received audio data to the speaker 1761, which is converted into a sound signal for output. The terminal device 110 can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1762 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1760 and converted into audio data, and then the audio data is output to the communication component 1710 to be sent to, for example, another terminal device 110, or the audio data is output to the memory 1720 for further processing.

[0284] The Bluetooth module 1770 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1770 to exchange data.

[0285] The processor 1780 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1720 and calling data stored in the memory 1720. In some embodiments, the processor 1780 may include one or more processing units; the processor 1780 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1780. In the present application, the processor 1780 can run the operating system, application programs, user interface display and touch response, as well as the road disease detection method of the embodiment of the present application. In addition, the processor 1780 is coupled to the display unit 1730.

[0286] In some possible implementations, various aspects of the road damage detection method provided by the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the road damage detection method according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device may execute the following steps: Figure 2 Follow the steps shown in .

[0287] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0288] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0289] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0290] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0291] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for detecting road damage, characterized in that: The method comprises: Based on the intensity value of each pixel point in the initial road damage image, the position of the road damage is detected to determine each first area not covered by the road damage; After performing denoising on each first region in the initial road damage image, enhancing the first edge feature of the road damage in the initial road damage image to obtain an intermediate road damage image; Based on the intensity value of each pixel point in the middle road damage image, position detection is performed on the road damage to determine each second area covered by the road damage; Performing edge detection of road damage in each second region in the middle road damage image to obtain a second edge feature corresponding to the middle road damage image, and fusing the global feature of the middle road damage image with the second edge feature to obtain a global feature map; Based on the global feature map, basic attribute information of the road damage is determined.

2. The method according to claim 1, characterized in that The method of detecting the position of the road damage based on the intensity value of each pixel point in the initial road damage image and determining each first area not covered by the road damage includes: Based on the preset values ​​corresponding to at least one preset detection template and the intensity value of each pixel point in each area of ​​the initial road damage image, the position of the road damage is detected to obtain the first detection value of each area; wherein each detection template is used to detect the road damage in different directions; the first detection value represents the probability that the corresponding area is the area covered by the road damage; Based on the first detection values, first areas not covered by the road damage are determined.

3. The method according to claim 2, characterized in that The method of performing position detection on the road damage based on the preset values ​​corresponding to the at least one preset detection template and the intensity value of each pixel point in each area of ​​the initial road damage image to obtain the first detection value of each area includes: For each area in the initial road damage image, the following operations are performed: Based on the respective preset values ​​corresponding to the at least one detection template and the respective intensity values ​​of a plurality of pixel points in an area, respectively obtaining the respective similarity feature values ​​of the one area and the at least one detection template; Among the at least one similar feature value corresponding to the one area, a similar feature value is determined as the first detection value corresponding to the one area.

4. The method according to claim 2, characterized in that The determining, based on each first detection value, each first area not covered by the road disease comprises: In the initial road damage image, each region whose corresponding first detection value is less than a preset threshold is determined as each first region not covered by the road damage.

5. The method according to claim 1, characterized in that The detecting the position of the road damage based on the intensity value of each pixel in the middle road damage image to determine each second area covered by the road damage includes: Based on the preset values ​​corresponding to at least one preset detection template and the intensity value of each pixel point in each area of ​​the intermediate road damage image, the position of the road damage is detected to obtain the second detection value of each area; wherein each detection template is used to detect the road damage in different directions; the second detection value indicates the probability that the corresponding area is the area covered by the road damage; Based on each second detection value, each second area covered by the road disease is determined.

6. The method according to claim 5, characterized in that The method of performing position detection on the road damage based on the preset values ​​corresponding to the at least one preset detection template and the intensity value of each pixel point in each area of ​​the middle road damage image to obtain the second detection value of each area includes: For each area in the middle road damage image, the following operations are performed: Based on the respective preset values ​​corresponding to the at least one detection template and the respective intensity values ​​of a plurality of pixel points in an area, respectively obtaining the respective similarity feature values ​​of the one area and the at least one detection template; Among the at least one similar feature value corresponding to the one area, a similar feature value is determined as the second detection value corresponding to the one area.

7. The method according to claim 5, characterized in that The determining, based on each second detection value, each second area covered by the road disease comprises: In the middle road damage image, each region whose corresponding second detection value is not less than a preset threshold is determined as each second region covered by the road damage.

8. The method according to claim 1, characterized in that The performing edge detection of road damage in the second area of ​​the middle road damage image to obtain a second edge feature corresponding to the middle road damage image includes: Based on the intensity value of each pixel in each second region, acquiring each gradient value corresponding to each pixel in each second region in multiple directions; Based on each gradient value, determining the edge of the road defect in each second area; The determination result is input into the edge feature extraction model to obtain the second edge feature corresponding to the middle road disease image.

9. The method according to claim 1, characterized in that The global features of the intermediate road damage image are obtained in the following manner: Based on the road damage detection model, the middle road damage image is downsampled, and semantic features are extracted from the downsampled middle road damage image to obtain global features of the middle road damage image.

10. The method according to claim 1, characterized in that The basic attribute information includes type information and orientation information. The basic attribute information of the road damage is determined based on the global feature map, including: Generate a road damage rotation detection frame based on the rotation target detector in the road damage detection model; The type information and orientation information of the road damage are determined based on the road damage rotation detection frame.

11. The method according to claim 10, characterized in that The basic attribute information also includes a damage level. After determining the type information and orientation information of the road damage based on the road damage rotation detection frame, the method further includes: Based on the minimum circumscribed rectangle of the road defect rotation detection frame, a local feature map containing the road defect is obtained in the global feature map, and the local feature map is upsampled based on the road defect detection model; the rotation angle of the minimum circumscribed rectangle is zero; Based on the road damage detection model, obtaining a segmentation mask corresponding to the upsampled local feature map; Based on the segmentation mask, the number of pixels occupied by the road damage in the upsampled local feature map is determined, and the damage level of the road damage is determined based on the number.

12. The method according to any one of claims 1 to 11, characterized in that: The method further comprises: In each preset time period included in each of the multiple preset cycles, the target path is photographed respectively to obtain multiple initial road damage images; If road damage is detected in the target path among the multiple initial road damage images taken within a preset time period, a first-level pre-alarm state is set for the road damage; If the road damage is in the first-level pre-alarm state in multiple preset time periods of a preset cycle, a second-level pre-alarm state is set for the road damage; If within the plurality of preset periods, the road hazards are all in a level 2 pre-alarm state, an alarm is issued for the road hazards.

13. A road disease detection device, characterized in that: include: A first detection unit, configured to perform position detection on the road damage based on the intensity value of each pixel point in the initial road damage image, and determine each first area not covered by the road damage; A processing unit, configured to enhance the first edge features of the road damage in the initial road damage image after performing denoising on each first region in the initial road damage image, so as to obtain an intermediate road damage image; A second detection unit, configured to perform position detection on the road damage based on the intensity value of each pixel point in the middle road damage image, and determine each second area covered by the road damage; a fusion unit, configured to perform edge detection of road damage in each second region in the middle road damage image, obtain a second edge feature corresponding to the middle road damage image, and fuse the global feature of the middle road damage image with the second edge feature to obtain a global feature map; A determination unit is used to determine basic attribute information of the road damage based on the global feature map.

14. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The method comprises a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the methods of claims 1 to 12.

16. A computer program product, characterized in that The method comprises a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of any one of the methods described in claims 1 to 12.

Images