Road hazard identification and management method, device and intelligent terminal

Abstract

The present application relates to the field of intelligent road detection, and in particular to a road disease identification management method, device and intelligent terminal. The road disease identification management method includes: determining a target disease image and a target road sign image based on a detection video of a detection section; determining the disease feature information of the detection section based on the target disease image; determining the detection position of the detection section based on the target road sign image; binding the disease feature information and the detection position, and entering them into a database. Utilizing the above technical solution, the disease feature information describing the specific characteristics of the road disease is obtained by analyzing the target disease image, and the detection position of the detection section is obtained by analyzing the target road sign image. The present application has the characteristics of reducing the workload of human eye identification of diseases and manual measurement of diseases, reducing traffic safety risks caused by identifying road diseases, and improving the work efficiency of road disease identification.

Description

Technical Field

[0001] The present application relates to the field of intelligent road detection, and in particular to a road hazard identification and management method, device and intelligent terminal. Background Art

[0002] As vehicles pass through the road, cracks, potholes, ruts, deep pits, collapses and other defects may appear. These defects are collectively referred to as road diseases. Road diseases affect the road's traffic capacity, road life and the probability of traffic accidents. In order to prevent the adverse effects of road diseases, it is necessary to detect road diseases in a timely manner and maintain the roads. In recent years, my country's road construction has developed by leaps and bounds, the mileage of open roads has continued to increase, and the number of motor vehicles has increased rapidly, resulting in faster road pavement damage and a greater risk of road diseases. The inspection of road diseases has become more and more onerous.

[0003] The inspection of road defects is mainly carried out by inspectors from road maintenance units who use handheld cameras to take photos, identify road defects with the human eye, measure the size of road defects with a ruler, and record and upload the shape, size, and type of road defects. This method of identifying road defects is inefficient and poses a risk to traffic safety during the photo-taking process. Summary of the invention

[0004] The embodiments of the present application provide a road damage identification and management method, device and intelligent terminal for improving the efficiency and accuracy of road damage identification.

[0005] According to one aspect of an embodiment of the present application, a road hazard identification and management method is provided.

[0006] A road hazard identification and management method, comprising:

[0007] Based on the inspection video of the inspection section, determine the target disease image and the target road sign image;

[0008] Determining the disease feature information of the detected road section based on the target disease image;

[0009] Determining a detection position of the detection section based on the target road sign image;

[0010] The disease characteristic information and the detection location are bound and entered into a database.

[0011] By adopting the above technical solution, the target disease image with road diseases and the target road sign image with road signs are extracted from the detection video, the disease feature information describing the specific features of the road diseases is obtained by analyzing the target disease image, and the detection position of the detection section is obtained by analyzing the target road sign image, thereby obtaining the location area of ​​the road disease, and recording the specific feature description and location area of ​​the road disease. By using the above technical solution, the workload of human eye identification of diseases and manual measurement of diseases can be reduced, the traffic safety risks caused by identifying road diseases can be reduced, and the work efficiency of road disease identification can be improved.

[0012] Optionally, the step of determining the disease feature information of the detected road section based on the target disease image includes:

[0013] Based on the target disease image, determining a detection point set, wherein the detection point set includes a plurality of detection corner points, and each of the detection corner points is distributed around a disease area in the target disease image;

[0014] Determining the defect size of the target defect image based on the detection point set;

[0015] Based on the defect size of the target defect image, the defect feature information of the detected road section is determined.

[0016] By adopting the above technical solution, the detection point set can obtain multiple detection corner points distributed around the diseased area. The disease size can be calculated using each detection corner point. The disease size can reflect the area size of the diseased area, which is convenient for subsequent hazard level assessment of road diseases.

[0017] Optionally, the step of determining a detection point set based on the target disease image includes:

[0018] The target disease image is input into an instance segmentation model to obtain a detection point set of the target disease image.

[0019] By adopting the above technical solution and using the instance segmentation model, a detection point set is extracted from the target disease image, thereby improving the efficiency and accuracy of the detection point set extraction.

[0020] Optionally, the step of determining the defect size of the target defect image based on the detection point set includes:

[0021] Determine the length of the disease based on the long side of the rectangular area surrounded by each of the detection corner points, and determine the width of the disease based on the short side of the rectangular area surrounded by each of the detection corner points;

[0022] The defect size of the target defect image is determined based on the defect length and the defect width.

[0023] By adopting the above technical solution, the long side length and short side length of the rectangular area corresponding to the road defect are calculated, the defect length and the defect width are obtained, and then the defect size is estimated, so that the estimated area of ​​the road defect can be quickly obtained.

[0024] Optionally, the step of determining the disease feature information of the detected road section based on the target disease image includes:

[0025] Inputting the target disease image into an instance segmentation model to obtain the disease type of the target disease image;

[0026] Based on the disease type of the target disease image, disease feature information of the detected road section is determined.

[0027] By adopting the above technical solution, the specific type of road damage in the target damage image is obtained using the instance segmentation model, which facilitates the subsequent classification and hazard level assessment of the road damage.

[0028] Optionally, the step of determining the target disease image and the target road sign image based on the detection video of the detection road section includes:

[0029] Based on the detection video of the detection road section, an initial disease image group and an initial road sign image group are determined, wherein each frame image in the initial disease image group is photographed with road diseases, and each frame image in the initial road sign image group is photographed with road signs;

[0030] Based on the initial disease image group, determining a target disease image, and based on the initial landmark image group, determining a target landmark image;

[0031] Before the step of inputting the target disease image into the instance segmentation model to obtain the disease type of the target disease image, the step further includes:

[0032] The confidence of the instance segmentation model is adjusted based on the frame ratio between the initial disease image group and the detection video.

[0033] By adopting the above technical solution, the confidence of the instance segmentation model is adjusted according to the frame ratio between the initial disease image group and the detection video, thereby adjusting the recall rate, precision rate and false recognition rate of the instance segmentation model to achieve a balance between model operation efficiency and output accuracy.

[0034] Optionally, the target road sign image is acquired by a binocular camera, and the target road sign image includes a first target road sign image and a second target road sign image; the step of determining the detection position of the detection section based on the target road sign image includes:

[0035] Determine a first same-name point of the first target road sign image and a second same-name point of the second target road sign image, wherein the first same-name point and the second same-name point reflect the same position of the road sign;

[0036] The longitude and latitude of the road sign are calculated based on the first point with the same name and the second point with the same name, and the detection position of the detection road section is determined.

[0037] By adopting the above technical solution, the longitude and latitude of the location of the road sign can be calculated using the pixel points of the first target road sign image and the second target road sign image corresponding to the same road sign position, thereby determining the detection position and obtaining the location area of ​​the road disease.

[0038] Optionally, the step of determining the target disease image and the target road sign image based on the detection video of the detection road section includes:

[0039] Based on the detection video of the detection road section, an initial disease image group and an initial road sign image group are determined, wherein each frame image in the initial disease image group is photographed with road diseases, and each frame image in the initial road sign image group is photographed with road signs;

[0040] Based on the initial disease image group, a similar disease image group is determined; based on the initial road sign image group, a similar road sign image group is determined, wherein adjacent frame images in each group of the similar disease image groups contain road diseases with similar shapes, and adjacent frame images in each group of the similar road sign image groups contain road signs with similar shapes;

[0041] Based on the similar disease image group, a target disease image is determined; based on the similar road sign image group, a target road sign image is determined.

[0042] By adopting the above technical scheme, similar disease image groups and similar road sign image groups are first screened out from the initial disease image group and the initial road sign image group, and then the similar disease image groups and similar road sign image groups are further screened to obtain target disease images and target road sign images, which can reduce repeated calculations when adjacent frame images in the detection video detect road diseases or road signs at the same position.

[0043] According to another aspect of an embodiment of the present application, a road hazard identification and management device is provided.

[0044] A road hazard identification and management device, comprising:

[0045] A target acquisition module is used to determine a target disease image and a target road sign image based on the detection video of the detection road section;

[0046] A disease analysis module, used to determine disease feature information of the detected road section based on the target disease image;

[0047] A position analysis module, used to determine the detection position of the detection section based on the target road sign image;

[0048] The information input module is used to bind the disease characteristic information and the detection location and input them into a database.

[0049] According to another aspect of an embodiment of the present application, a smart terminal is provided.

[0050] An intelligent terminal includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute a road hazard identification and management method as described in any of the above technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of the road hazard identification and management method of the present application.

[0052] Figure 2 It is a schematic diagram of the target disease image, where the dotted line part is the disease area.

[0053] Figure 3 It is a sub-process diagram of step S11 and step S12 in the road hazard identification and management method.

[0054] Figure 4 Schematic diagram of the initial disease image group, the similar disease image group and the target disease image.

[0055] Figure 5 It is a sub-process diagram of step S2 in the road disease identification and management method.

[0056] Figure 6 It is a sub-process diagram of step S3 in the road hazard identification and management method.

[0057] Figure 7 It is a module schematic diagram of the road hazard identification and management device of the present application.

[0058] In the figure, 1. Target acquisition module; 2. Disease analysis module; 3. Location analysis module; 4. Information entry module. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] The numbers of the steps in this embodiment are for convenience of explanation only and do not limit the order of movement of the steps. In practical applications, the order of movement of the steps can be adjusted as needed, or performed simultaneously. These adjustments or replacements are all within the scope of protection of the present invention.

[0061] The terms "first" and "second" are used for descriptive purposes only and should not be construed as expressing or implying relative importance.

[0062] The term "includes" or "includes" any term or combination thereof, and may include any or all of the terms "includes" or "includes" any term or combination thereof, or ...

[0063] The following is an introduction to some concepts involved in the embodiments of the present application.

[0064] 1. Frame: It is the smallest unit of a single image in an image animation. A frame is a still picture, and continuous frames form a video. The frame number is the number of frames of the picture transmitted in 1 second, which can also be understood as the number of times the graphics processor can refresh per second.

[0065] 2. Pixel: A pixel is a color point that constitutes an image and is the most basic unit of an image. A frame of an image is actually made up of a lot of pixels. A pixel coordinate system is constructed based on a visual system. The pixel coordinate system is a two-dimensional coordinate system built on the image. Each pixel has a pixel coordinate in the pixel coordinate system. The x coordinate of the pixel coordinate reflects the number of columns in the image where the pixel is located, and the y coordinate of the pixel coordinate reflects the number of rows in the image where the pixel is located.

[0066] 3. Normalized coordinates: refers to the coordinates on the normalized plane. Divide the coordinates of the three-dimensional space points by Z, and all the coordinates of the space points are transferred to the unit distance in front of the camera. This plane is called the normalized plane, and then multiply it by the focal length to return the normalized plane to the imaging plane. The world coordinates in the world coordinate system can be converted to the camera coordinates of the camera coordinate system through rigid body transformation, and the camera coordinates of the camera coordinate system can be converted to the normalized coordinates of the image plane coordinate system through perspective projection. The normalized coordinates can be converted to the pixel coordinates of the pixel coordinate system through origin translation and scale expansion.

[0067] 4. Convolutional Neural Networks (CNN): Convolutional Neural Networks is a deep neural network with a convolutional structure. The convolutional structure can reduce the amount of memory occupied by the deep network. Its three key operations are local receptive field, weight sharing, and pooling layer, which effectively reduce the number of network parameters and alleviate the overfitting problem of the model.

[0068] 5. Instance segmentation algorithm: Instance segmentation is a task of identifying the outline of an object at the pixel level. Compared with other related tasks, instance segmentation is one of the more difficult computer vision tasks to solve. The task of the semantic segmentation algorithm is to classify each pixel in the image by labeling it with a category, that is, to identify the pixels corresponding to different objects; the task of the target detection algorithm is to identify the content in the image and detect its location, that is, to identify the location areas corresponding to different objects; the instance segmentation algorithm is a combination of the semantic segmentation algorithm and the target detection algorithm. It can automatically frame different instances from the image using the target detection method, that is, to identify the location areas corresponding to different objects in the image, and then use the semantic segmentation method to mark pixels one by one in different instance areas, that is, to identify which objects these location areas belong to, thereby identifying different instances in the image.

[0069] 6. Binocular vision algorithm: Binocular vision is a method that simulates the principle of human vision and uses computers to passively perceive distance. Observe an object from two or more points, obtain images at different perspectives, and obtain the three-dimensional information of the object by calculating the offset between pixels based on the matching relationship between pixels in the images through the principle of triangulation. After obtaining the depth of field information of the object, the actual distance between the object and the camera, the 3D size of the object, and the actual distance between the two points can be calculated.

[0070] 7. Batch Size: The number of samples selected by the neural network model in one training. The size of Batch Size affects the optimization degree and speed of the model, and it directly affects the usage of GPU memory.

[0071] The road disease identification and management method provided in the embodiment of the present application is applied to scenarios such as road quality inspection and road maintenance. For example, after a road reaches a certain maintenance cycle, or after an accident (such as a traffic accident) occurs on the road, it is necessary to identify and record the road diseases on the road surface. On the one hand, the maintenance unit can be notified in time to repair the road diseases, and on the other hand, individual more serious road diseases can be broadcast to vehicles traveling on this road to reduce the impact of road diseases on traffic safety. For another example, the accumulated road diseases of each road are counted, the reasons for the frequent occurrence of road diseases are analyzed, and the future health of each road is estimated.

[0072] The basic concept of this application is introduced below.

[0073] In the related art, the inspection of road defects is usually carried out by inspectors of road maintenance units who take photos with handheld cameras, identify road defects with the human eye, measure the size of road defects with a ruler, and record and upload the shape, size, and type of road defects. This method of identifying road defects has the problems of low recognition efficiency and traffic safety risks.

[0074] Based on the above problems, the present application extracts images with road defects and images with road signs based on videos shot on the detected road section, uses an instance segmentation algorithm to extract information describing the road defects from the image, and uses the content of the road signs in the image to calculate the specific location of the image, thereby recording the specific feature description and location area of ​​the road defects, thereby reducing the workload of human eye identification of defects and manual measurement of defects, reducing traffic safety risks caused by identifying road defects, and improving the work efficiency of road defect identification.

[0075] The following is attached to the instruction manual Figure 1-Figure 7 The embodiments of the present application are described in further detail.

[0076] An embodiment of the present application provides a road hazard identification and management method, and the main process of the method is described as follows.

[0077] Reference Figure 1 and Figure 2 , S1. Based on the detection video of the detection road section, determine the target disease image and the target road sign image.

[0078] Among them, the inspection road section refers to the traffic road that needs to be inspected for road defects, and the inspection video refers to the video data captured by the camera equipment on the inspection road section. The target defect image refers to the image extracted from the inspection video and captured with road defects. The target road sign image refers to the image extracted from the inspection video and captured with road signs. Road signs include road signs, road stakes and other objects with specific shapes that can be clearly distinguished from the shooting background.

[0079] It is worth noting that the detection section in this application can be set based on the detection section. For example, in a specific application scenario, it is necessary to conduct random inspections of roads or large-scale inspections of roads. The shooting time of the entire inspection task is very long. In this case, it is necessary to divide the original shooting video into multiple detection videos. The shooting section corresponding to each detection video is the detection section. The detection video in this application can be set based on the detection section. For example, in a specific application scenario, the detection section is a designated section that needs to be inspected. The shooting time of the entire inspection task is short. In this case, the original shooting video obtained by shooting the detection section can be taken as the detection video.

[0080] In this embodiment, the camera device is a binocular camera. In one application scenario, the camera device is specifically a surveyor equipped with a binocular camera, and the camera device is installed on a driving vehicle dedicated to the road maintenance unit. The driving vehicle takes pictures of the road surface of the detection section while driving on the detection section. In another application scenario, the camera device is specifically a driving recorder equipped with a binocular camera, and the camera device is installed on a private vehicle. The private vehicle takes pictures of the road surface of the detection section while driving on the detection section, and the detection video is obtained from the video data uploaded by the camera device.

[0081] S2. Determine the disease feature information of the inspection section based on the target disease image.

[0082] Among them, by using the pixel area in the target disease image where the road disease is photographed, the disease feature information used to describe the road disease can be extracted.

[0083] S3. Determine the detection position of the detection section based on the target road sign image.

[0084] Among them, by using the pixel area where the road sign is photographed in the target disease image, the actual position coordinates of the road sign in the world coordinate system can be analyzed and calculated to obtain the detection position.

[0085] S4. Bind the disease characteristic information and detection location and enter them into the database.

[0086] Among them, the disease feature information and the detection location are obtained by analyzing the data of the detection video, which can reflect the road disease on the detection section and a certain position of the road sign. Since this position should be relatively close to the road disease, the detection location can be considered as the adjacent location area of ​​the road disease, which can be used to reflect the specific location area of ​​the road disease. The disease feature information and the detection location are bound to each other and entered into the database, so as to record the specific feature description and location area of ​​the road disease, and complete the identification and record management of the road disease.

[0087] Reference Figure 3Specifically, in the sub-steps of step S1, it includes:

[0088] S11. Determine an initial disease image group and an initial road sign image group based on the detection video of the detection road section.

[0089] The detection video includes several frames of detection images, and the initial defect image group and the initial road sign image group are image data obtained by preliminary screening of each frame of detection images in the detection video. The initial defect image group includes at least one initial defect image, in which at least one road defect is captured; the initial road sign image group includes at least one initial road sign image, in which at least one road sign is captured.

[0090] The specific method for preliminarily screening the detection video to obtain the initial disease image group and the initial road sign image group is:

[0091] A1. Input each frame of detection image into the trained recognition and screening model to obtain the initial recognition and screening information;

[0092] The disease screening model includes a target classification algorithm module, a target detection algorithm module and a semantic segmentation algorithm module. The target classification algorithm module distinguishes the types of objects to determine whether the current object is a road disease or a road sign; the target detection algorithm module identifies multiple objects in the detection image; the semantic segmentation algorithm module marks the pixels occupied by different objects and frames each object one by one through each circumscribed rectangular frame, which is obtained by each pixel occupied by the object.

[0093] Each identification object is a road damage or a road sign. The identification preliminary screening information includes multiple identification preliminary screening areas. The identification preliminary screening area is an external rectangular frame surrounding the road damage or road sign distribution in the detection image. In the same frame of the detection image, if there are multiple road damages or road signs, each road damage or road sign will correspond to an external rectangular frame, and each external rectangular frame is provided with a corresponding classification mark and identity number. The classification mark is used to indicate that the external rectangular frame represents a road damage or a road sign, and the identity number is used to distinguish different road damages or road signs.

[0094] A2. Classify each frame of detected image based on the disease individual information, and determine the initial disease image group and the initial road sign image group based on the classification result.

[0095] By identifying the preliminary screening information, the distribution of road diseases or road signs in each frame of the detection image can be obtained. For example, in one frame of the detection image, four preliminary screening areas with identity numbers d1, d2, d3, and d4 are identified, among which the classification labels of d1, d2, and d3 are all diseases, and the classification label of d4 is a road sign. Therefore, d1, d2, and d3 are road diseases at different locations, and d4 is a road sign.

[0096] By identifying the initial screening information to distinguish whether there are road damage or road signs in the detection image, each frame of the detection image can be classified into an initial damage image with road damage, an initial road sign image with road signs, and a non-target image without road damage or road signs. Then, the initial damage image frames are integrated to obtain an initial damage image group, and the initial road sign image frames are integrated to obtain an initial road sign image group. In addition, the frames of the initial damage image group and the initial road sign image group are arranged in sequence according to the order of shooting time.

[0097] The purpose of step S11 is to perform preliminary screening of the detection images of each frame in the detection video, filter out the detection images in which neither road damage nor road signs exist, reduce the impact of such detection images on subsequent analysis and calculation, and obtain detection images with road damage or road signs, providing a basis for subsequent similarity screening of road damage or road signs.

[0098] Reference Figure 3 and Figure 4 , S12, based on the initial disease image group, determine the target disease image, based on the initial landmark image group, determine the target landmark image.

[0099] Among them, during the driving process of the vehicle, the camera equipment can only capture the road disease when the vehicle passes through the vicinity of the road disease. Since the shooting of the detection video is carried out continuously on the timeline according to the preset frame rate, there is a high probability that there will be multiple frames of initial disease images of the same road disease in the initial disease image group. These initial disease images will cause the system to repeatedly calculate the same road disease in the subsequent analysis process. In order to reduce the impact of repeated calculations, these initial disease images need to be screened to determine the target disease images.

[0100] Similarly, there is a high probability that there will be multiple frames of initial road sign images continuously shot for the same road sign in the initial road sign image group, which will cause the system to repeatedly calculate the same road defect. In order to reduce the impact of repeated calculations, these initial road sign images need to be screened to determine the target road sign images.

[0101] The sub-steps of step S12 include:

[0102] S121, based on the initial disease image group, a similar disease image group.

[0103] Among them, by identifying the preliminary screening information, the identification preliminary screening area of ​​the road disease in each frame of the initial disease image can be obtained, and then it is determined whether the same road disease exists in multiple frames of initial disease images. If so, multiple frames of initial disease images corresponding to the same road disease are extracted and integrated into a group of similar disease image groups. A group of similar disease image groups corresponds to one road disease.

[0104] On the other hand, the initial disease image group is obtained by taking pictures by a binocular camera, and the left camera and the right camera are taken respectively, and two corresponding batches of images should be obtained. In this embodiment, the above two batches of images are used as independent initial disease image groups to participate in the calculation. In other embodiments, only one batch of images can be used as the initial disease image group to participate in the calculation.

[0105] In this embodiment, the method for determining whether the same road disease exists in multiple frames of initial disease images and determining a group of similar disease images may be:

[0106] B1. Projecting each identification and initial screening area of ​​multiple frames of initial disease images to the same two-dimensional coordinate system, so that the shooting fields of each frame of initial disease images overlap in the two-dimensional coordinate system;

[0107] B2. Calculate the interval distance between each identification and initial screening area;

[0108] B3, based on the comparison between the interval distance and the preset interval threshold, each identification initial screening area with an interval distance greater than or equal to the interval threshold is determined as a similar disease area corresponding to the same road disease;

[0109] B4. The initial damage images corresponding to the respective similar damage areas are used as similar damage images corresponding to the same road damage, and are integrated into a similar damage image group.

[0110] The principle of the above method is: using the distances between different identification and preliminary screening areas, it is determined whether each identification and preliminary screening area is the same road disease, thereby determining a group of similar disease images.

[0111] In another preferred example, the method for determining whether the same road disease exists in multiple frames of initial disease images and determining a group of similar disease images may also be:

[0112] C1. Projecting each identification and initial screening area of ​​multiple frames of initial disease images to the same two-dimensional coordinate system, so that the shooting fields of each frame of initial disease images overlap in the two-dimensional coordinate system;

[0113] C2. Calculate the overlap between each identification and initial screening area;

[0114] C3. Based on the comparison between the overlap degree and the preset overlap ratio threshold, each identified preliminary screening area with an overlap degree greater than or equal to the overlap ratio threshold is determined as a similar disease area corresponding to the same road disease;

[0115] C4. The initial damage images corresponding to the respective similar damage areas are used as similar damage images corresponding to the same road damage, and are integrated into a similar damage image group.

[0116] The principle of the above method is: using the degree of overlap between different identification and preliminary screening areas, it is judged whether each identification and preliminary screening area is the same road disease, so as to determine a group of similar disease images.

[0117] S122: Determine a target disease image based on the group of similar disease images.

[0118] The sizes of the areas of the similar disease regions in the similar disease image group are compared, and the similar disease image corresponding to the similar disease region with the largest area is taken as the target disease image.

[0119] S123: Based on the initial road sign image group, a similar road sign image group is generated.

[0120] Among them, by identifying the preliminary screening information, the identification preliminary screening area of ​​the road signs in each frame of the initial road sign image can be obtained, and then it is determined whether the same road sign exists in multiple frames of initial road sign images. If so, multiple frames of initial road sign images corresponding to the same road sign are extracted and integrated into a group of similar road sign images. A group of similar road sign images corresponds to one road sign.

[0121] In this embodiment, the method for determining whether the same road sign exists in multiple frames of initial road sign images and determining a group of similar road sign images may be:

[0122] D1. Projecting each identification initial screening area of ​​multiple frames of initial road sign images to the same two-dimensional coordinate system, so that the shooting fields of each frame of initial road sign images overlap in the two-dimensional coordinate system;

[0123] D2. Calculate the interval distance between each identification and initial screening area;

[0124] D3, based on the comparison between the interval distance and the preset interval threshold, determining each identification initial screening area with an interval distance greater than or equal to the interval threshold as a similar disease area corresponding to the same road sign;

[0125] D4. The initial road sign images corresponding to the similar diseased areas are used as similar road sign images corresponding to the same road sign, and are integrated into a similar road sign image group.

[0126] The principle of the above method is: using the distances between different identification preliminary screening areas to determine whether each identification preliminary screening area is the same road sign, thereby determining a group of similar road sign images.

[0127] In another preferred example, the method for determining whether the same road sign exists in multiple frames of initial road sign images and determining a group of similar road sign images may also be:

[0128] E1. Projecting each identification initial screening area of ​​multiple frames of initial road sign images to the same two-dimensional coordinate system, so that the shooting fields of each frame of initial road sign images overlap in the two-dimensional coordinate system;

[0129] E2. Calculate the overlap between each identification and initial screening area;

[0130] E3. Based on the comparison between the overlap degree and a preset overlap ratio threshold, each identified preliminary screening area with an overlap degree greater than or equal to the overlap ratio threshold is determined as a similar disease area corresponding to the same road sign;

[0131] E4. The initial road sign images corresponding to the respective similar diseased areas are used as similar road sign images corresponding to the same road sign, and are integrated into a similar road sign image group.

[0132] The principle of the above method is: using the degree of overlap between different identification preliminary screening areas, it is judged whether each identification preliminary screening area is the same road sign, so as to determine a group of similar road sign images.

[0133] S124: Determine a target road sign image based on the similar road sign image group.

[0134] The sizes of the similar diseased regions in the similar road sign image group are compared, and the similar road sign image corresponding to the similar diseased region with the largest area is taken as the target road sign image.

[0135] By utilizing steps S121 to S124, repeated calculations of the same road damage or the same road sign in subsequent analysis can be reduced, thereby reducing the cost of calculation and analysis.

[0136] In the specific implementation process of step S2, determining the defect characteristic information of the detected road section based on the target defect image, the defect characteristic information includes defect size and defect type, wherein the defect size includes the range size used to reflect the road defect, and the defect type includes cracks, potholes, ruts, deep pits, collapse and other road defect types.

[0137] Reference Figure 5 , the sub-steps of step S2 include:

[0138] S21. Determine the size and type of the disease based on the target disease image.

[0139] The specific implementation of step S21 is: inputting the target disease image into the instance segmentation model, and obtaining the detection point set and the disease type according to the output of the instance segmentation model.

[0140] The instance segmentation model is a convolutional neural network that has been established and trained based on the instance segmentation algorithm. The instance segmentation model includes a target detection algorithm module, a semantic segmentation algorithm module, and a disease classification algorithm module. The pixel area in the target disease image where road diseases are photographed is defined as the disease area. The target detection algorithm module can identify one or more road diseases in the target disease image, and one or more disease areas are marked by pixel point marking.

[0141] The semantic segmentation algorithm module can identify pixels belonging to different manifestations in the target disease image. In this embodiment, the labels of the semantic segmentation algorithm module include disease labels for indicating disease areas (multiple road diseases belong to the same disease label) and non-disease labels for non-disease areas. The non-disease labels include background labels for indicating shooting background areas (such as the sky), road sign labels for indicating road sign areas, light and shadow labels for indicating light and shadow areas in the environment, and object labels for indicating other object areas in the road (such as manhole covers and vehicles). The semantic segmentation algorithm module adds a label to each pixel point and uses the label corresponding to the pixel point to determine which of the above areas the pixel point belongs to.

[0142] By combining the target detection algorithm module and the semantic segmentation algorithm module, the total disease area, shooting background area, road sign area, light and shadow area and other object areas can be separated, and the total disease area can be distinguished according to the individual road diseases, and finally each disease area is marked, and each disease area corresponds to a road disease.

[0143] The specific way of marking the diseased area by the instance segmentation model is to mark the diseased area through an external rectangular box. Specifically, it is based on each pixel point in the diseased area and determines the normalized coordinates of each pixel point. An external rectangular box passing through the pixel point with the smallest x coordinate, the pixel point with the largest x coordinate, the pixel point with the smallest y coordinate and the pixel point with the largest y coordinate is taken. This external rectangular box is distributed around the diseased area, and the pixels located at the four corners of this external rectangular box are detection corner points. The detection corner points corresponding to the same road disease constitute a detection point set.

[0144] In this embodiment, the instance segmentation model can actually also identify non-diseased areas, that is, identify the shooting background area, road sign area, light shadow area and other object areas, thereby reducing the probability of confusion between diseased areas and non-diseased areas and reducing the misrecognition rate.

[0145] The disease classification algorithm module is used to identify the specific road disease type corresponding to the disease area and output the disease type. Each disease type corresponds to a detection point set. The combination of disease type and detection point set can reflect the specific location, range area and disease type of a road disease.

[0146] The instance segmentation model identifies the type of road damage through the damage classification algorithm module, and the recall rate of the instance segmentation model can be adjusted by the confidence of the instance segmentation model. In order to achieve dynamic adjustment of the confidence, before inputting the target damage image into the instance segmentation model in step S21, it also includes:

[0147] Adjust the confidence of the instance segmentation model based on the frame ratio between the initial disease image group and the detection video.

[0148] Among them, the smaller the proportion of frames, the lower the probability of road damage, and a lower confidence level should be set to achieve a higher recall rate, lower precision rate, and higher false positive rate; conversely, the higher the probability of road damage, the higher the confidence level should be set to achieve a lower recall rate, higher precision rate, and lower false positive rate.

[0149] In this embodiment, the instance segmentation model specifically adopts mask_rcnn_inception_resnet_v2. Compared with other models, the network of this model is deeper, and diseases can be detected after 20,000-30,000 steps of small data training.

[0150] In this embodiment, the training of the instance segmentation model is completed by joint training of the target detection algorithm module, the semantic segmentation algorithm module and the disease classification algorithm module. The specific training method includes:

[0151] F1. Obtain a training data set, wherein the training data set includes training disease images and annotated data sets.

[0152] The training damage images are obtained by photographing road damage with a photographing device at a shooting angle corresponding to that of a moving vehicle. The labeled dataset is used as a reference for the output obtained after the training damage images are input into the instance segmentation model.

[0153] The annotation data set includes an annotation point set, an annotation type, and an annotation area. The annotation point set includes the four corner points of a rectangular bounding box distributed around the diseased area based on the diseased area in the training diseased image that is manually annotated. The annotation type is the disease type of the diseased area corresponding to the annotation point set. The annotation area includes a variety of areas divided according to the various labels of the semantic segmentation algorithm module, specifically including a disease annotation area corresponding to the disease label, a background annotation area corresponding to the background label, a landmark annotation area corresponding to the landmark label, a light and shadow annotation area corresponding to the light and shadow label, and an object annotation area corresponding to the object label.

[0154] F2. Input the training disease image into the initial model and output the detection result set of the training disease image, wherein the detection result set includes a training point set, a training type and a training area.

[0155] Among them, the initial model is the predecessor of the instance segmentation model, which is a neural convolutional network established based on the instance segmentation algorithm. The initial model includes a target detection algorithm module, a semantic segmentation algorithm module and a disease classification algorithm module. The labels of the semantic segmentation algorithm module include disease labels, background labels, road sign labels, light and shadow labels and object labels. The training area includes a disease training area corresponding to the disease label, a background training area corresponding to the background label, a road sign training area corresponding to the road sign label, a light and shadow training area corresponding to the light and shadow label and an object training area corresponding to the object label.

[0156] F3. Determine the loss function of the initial model, and adjust the parameters of the loss function according to the labeled data set and the detection result set, until the labeled data set of the training diseased images and the detection result set of the training diseased images are within the allowable error range. The initial model is considered to have been trained, and the trained initial model is obtained as the instance segmentation model.

[0157] In this embodiment, when the amount of data is large and the distribution of different label data is uneven, the model training is prone to underfitting, and increasing the Batch Size is limited by the GPU memory size. In order to meet the condition that the BatchSize of the instance segmentation model is large and the GPU can adapt to the Batch Size, this embodiment preferably uses Google TPU for training to reduce the risk of loss oscillation and underfitting problems.

[0158] S22. Determine the defect size of the target defect image based on the detection point set.

[0159] Among them, a circumscribed rectangular frame including the defect area can be formed between each detection corner point in the detection point set. The defect size refers to the area of ​​this circumscribed rectangular frame, and is considered to be approximately the area of ​​the road defect on the detection section.

[0160] The specific method for determining the defect size of the target defect image is:

[0161] S221. Determine the length of the defect based on the long side of the rectangular area surrounded by each detection corner point, and determine the width of the defect based on the short side of the rectangular area surrounded by each detection corner point.

[0162] S222: Determine the defect size of the target defect image based on the product of the defect length and the defect width.

[0163] S23. Determine the defect feature information of the inspection section based on the defect size and defect type of the target defect image.

[0164] If the detected road section corresponds to multiple frames of target disease images, all the disease sizes and disease types obtained based on the multiple frames of target disease images belong to the current detected road section. The disease size can reflect the size of the disease area, which is convenient for subsequent risk level assessment of road diseases.

[0165] In the specific implementation process of determining the detection position of the detection section based on the target road sign image in step S3, as long as the same-name point measurement is adopted, the longitude and latitude of this position are calculated using the same-name points corresponding to the same position in the binocular image.

[0166] Reference Figure 6 , the sub-steps of step S3 include:

[0167] S31, determining a first point of the same name in the first target landmark image and a second point of the same name in the second target landmark image.

[0168] The target road sign image is obtained by taking pictures by binocular cameras, the left camera and the right camera respectively, and the target road sign image includes a first target road sign image corresponding to the left camera and a second target road sign image corresponding to the right camera. The first same-name point is a pixel point in the first target road sign image, and the second same-name point is a pixel point in the first target road sign image. The first target road sign image and the second target road sign image should have overlapping shooting fields of view, that is, the same road sign may be photographed, and the first same-name point and the second same-name point are the same position on this road sign.

[0169] S32, calculating the longitude and latitude of the road sign based on the first point with the same name and the second point with the same name, and determining the detection position of the detection section.

[0170] The first and second points of the same name can be used to obtain the two-dimensional coordinates of the road sign. The depth information can be obtained by using the principle of triangulation to obtain the three-dimensional coordinates of the road sign, thereby calculating the longitude and latitude of the road sign and obtaining the detection position.

[0171] In other embodiments, step S3 may also directly use an image recognition algorithm to identify and detect the road sign, and obtain the detection position of the road sign through specific building information or text information, such as identifying text information on the road sign through an image text recognition algorithm.

[0172] The implementation principle of a road disease identification management method provided by the present application is as follows: extracting a target disease image with road diseases and a target road sign image with road signs from a detection video, obtaining disease feature information describing the specific features of road diseases by analyzing the target disease image, obtaining the detection position of the detection section by analyzing the target road sign image, thereby obtaining the location area of ​​the road disease, and recording the specific feature description and location area of ​​the road disease. The above technical solution can reduce the workload of human eye identification of diseases and manual measurement of diseases, reduce the traffic safety risks caused by identifying road diseases, and improve the work efficiency of road disease identification.

[0173] The present application also provides a road hazard identification and management device, which corresponds to the above-mentioned road hazard identification and management method.

[0174] Reference Figure 7 , the road disease identification and management device includes:

[0175] The target acquisition module 1 is used to determine the target disease image and the target road sign image based on the detection video of the detection section, and output them to the disease analysis module 2;

[0176] The disease analysis module 2 is used to determine the disease feature information of the detected road section based on the target disease image and output it to the position analysis module 3;

[0177] The position analysis module 3 is used to determine the detection position of the detection section based on the target road sign image and output it to the information input module 4;

[0178] The information input module 4 is used to bind the disease characteristic information and the detection location and input them into the database.

[0179] The road damage identification and management device provided in this embodiment can implement the various steps of the aforementioned embodiment due to the functions of its modules themselves and the logical connections between each other, and can therefore achieve the same technical effects as the aforementioned method. The principle analysis can be found in the relevant description of the aforementioned road damage identification and management method steps, which will not be repeated here.

[0180] The present application also provides a smart terminal.

[0181] An intelligent terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory stores training data, algorithm formulas, and filtering mechanisms in a training model. The processor is used to provide computing and control capabilities, and a road hazard identification and management method is implemented when the processor executes the computer program.

[0182] The smart terminal provided in this embodiment can achieve the same technical effect as the aforementioned method because the computer program in its memory will implement the various steps of the aforementioned method after running on the processor. The principle analysis can refer to the relevant description of the aforementioned method steps, which will not be repeated here.

[0183] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A road hazard identification and management method, characterized in that: include: Based on the inspection video of the inspection section, determine the target disease image and the target road sign image; Determining the disease feature information of the detected road section based on the target disease image; Determining a detection position of the detection section based on the target road sign image; Binding the disease characteristic information and the detection location, and entering them into a database; The step of determining the disease feature information of the detected road section based on the target disease image comprises: Inputting the target disease image into an instance segmentation model to obtain the disease type of the target disease image; Determining the disease characteristic information of the detected road section based on the disease type of the target disease image; The step of determining the target disease image and the target road sign image based on the detection video of the detection road section includes: Based on the detection video of the detection road section, an initial disease image group and an initial road sign image group are determined, wherein each frame image in the initial disease image group is photographed with road diseases, and each frame image in the initial road sign image group is photographed with road signs; Based on the initial disease image group, determining a target disease image, and based on the initial landmark image group, determining a target landmark image; Before the step of inputting the target disease image into the instance segmentation model to obtain the disease type of the target disease image, the step further includes: The confidence of the instance segmentation model is adjusted based on the frame ratio between the initial disease image group and the detection video.

2. The road hazard identification and management method according to claim 1, characterized in that: The step of determining the disease feature information of the detected road section based on the target disease image comprises: Based on the target disease image, determining a detection point set, wherein the detection point set includes a plurality of detection corner points, and each of the detection corner points is distributed around a disease area in the target disease image; Determining the defect size of the target defect image based on the detection point set; Based on the defect size of the target defect image, the defect feature information of the detected road section is determined.

3. The road hazard identification and management method according to claim 2, characterized in that: The step of determining a detection point set based on the target disease image comprises: The target disease image is input into an instance segmentation model to obtain a detection point set of the target disease image.

4. The road hazard identification and management method according to claim 3, characterized in that: The step of determining the defect size of the target defect image based on the detection point set comprises: Determine the length of the disease based on the long side of the rectangular area surrounded by each of the detection corner points, and determine the width of the disease based on the short side of the rectangular area surrounded by each of the detection corner points; The defect size of the target defect image is determined based on the defect length and the defect width.

5. The road hazard identification and management method according to claim 1, characterized in that: The target road sign image is acquired by a binocular camera, and the target road sign image includes a first target road sign image and a second target road sign image; The step of determining the detection position of the detection section based on the target road sign image comprises: Determine a first same-name point of the first target road sign image and a second same-name point of the second target road sign image, wherein the first same-name point and the second same-name point reflect the same position of the road sign; The longitude and latitude of the road sign are calculated based on the first point with the same name and the second point with the same name, and the detection position of the detection road section is determined.

6. The road hazard identification and management method according to claim 1, characterized in that: The step of determining the target disease image and the target road sign image based on the detection video of the detection road section includes: Based on the detection video of the detection road section, an initial disease image group and an initial road sign image group are determined, wherein each frame image in the initial disease image group is photographed with road diseases, and each frame image in the initial road sign image group is photographed with road signs; Based on the initial disease image group, a similar disease image group is determined; based on the initial road sign image group, a similar road sign image group is determined, wherein adjacent frame images in each group of the similar disease image groups contain road diseases with similar shapes, and adjacent frame images in each group of the similar road sign image groups contain road signs with similar shapes; Based on the similar disease image group, a target disease image is determined; based on the similar road sign image group, a target road sign image is determined.

7. A road hazard identification and management device, characterized in that: A method for road hazard identification and management for executing any one of claims 1 to 6, comprising: A target acquisition module (1) is used to determine a target disease image and a target road sign image based on a detection video of a detection road section; A disease analysis module (2), used to determine disease feature information of the detected road section based on the target disease image; A position analysis module (3), used to determine the detection position of the detection section based on the target road sign image; An information input module (4) is used to bind the disease characteristic information and the detection location and input them into a database.

8. An intelligent terminal, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes a road hazard identification and management method as claimed in any one of claims 1 to 6.

Images