Road damage coding generation and image matching verification method and system based on bus patrol

Abstract

The present invention discloses a method and system for generating road disease codes and verifying image matching based on bus patrols. First, a road disease database is established. The road disease database is used to store road disease information recorded by multiple bus patrols, and a road disease time evolution sequence is formed for the same disease. Then, a road disease code is generated using hash coding according to the stored road disease information. The present invention realizes the construction of a road disease code generation and identification method and the generation of a road disease time evolution sequence using the characteristics of high-frequency bus patrols, and also realizes the analysis and tracing of the location change and evolution of road diseases, and can understand the development trend and influencing factors of road diseases. Not only can a road disease code be automatically generated according to the identified disease information, but also specific road diseases can be identified and associated with specific location information, thereby improving the automated closed-loop management capability of intelligent road inspections, and is suitable for being widely promoted and used.

Description

Technical Field

[0001] The present invention relates to the technical field of combined application of public transportation and computer vision, and in particular to a method and system for generating road hazard codes and image matching verification based on bus patrol. Background Art

[0002] Daily road maintenance inspections are important supervisory tasks to ensure road safety and smooth traffic. Through daily maintenance inspections, damage to road infrastructure and safety hazards can be discovered in a timely manner, especially road disease detection; road disease detection refers to the inspection of the road surface and structure to identify and assess road damage, which may be caused by traffic load, environmental factors, material aging or other reasons; timely detection and repair of these diseases are essential to ensure road safety, extend road life and reduce maintenance costs.

[0003] At present, most of the existing road defect detection methods adopt a method that combines manual walking inspection with professional road inspection vehicles equipped with image recognition and perception equipment. However, this method has many problems, such as low efficiency, unstable quality and great safety hazards of manual walking inspection. In addition, professional road inspection vehicles are expensive, have long inspection cycles, and are difficult to capture road defects in a timely manner. Therefore, it is necessary to design a road defect coding generation and image matching verification method and system based on bus patrol. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and to better and effectively solve the existing road disease detection methods, which mostly adopt a method combining manual walking inspection with professional road inspection vehicles equipped with image recognition and sensing equipment. However, this method has many problems, such as low efficiency, unstable quality and great safety hazards of manual walking inspection, and professional road inspection vehicles are expensive, have long inspection cycles, and are difficult to capture road diseases in a timely manner. A road disease coding generation and image matching verification method and system based on bus patrols are provided, which realizes the use of the characteristics of high-frequency bus patrols to construct a road disease coding generation and identification method and generate a road disease time evolution sequence, and also realizes the analysis and tracing of the location changes and evolution of road diseases, and can understand the development trend and influencing factors of road diseases. Not only can the road disease code be automatically generated according to the identified disease information, but also specific road diseases can be identified and associated with specific location information, thereby improving the automated closed-loop management capability of intelligent road inspections.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] The method and system for generating road hazard coding and image matching verification based on bus patrols include the following steps:

[0007] Step A, establishing a road damage database, wherein the road damage database is used to store road damage information recorded by multiple bus patrols and form a road damage time evolution sequence for the same damage;

[0008] Step B, generating a road damage code by using hash coding according to the stored road damage information, and then combining the road damage code with the road damage type information to form complete damage description information;

[0009] Step C, pre-screening the set of road damage databases that are close to the newly generated record location according to the location information and calculating the cosine similarity, then using the cosine similarity to set a threshold, and then screening out the damages with a similarity to the newly generated code higher than the threshold from the pre-screened set to form a historical similar damage set;

[0010] Step D, perform image matching on the historical similar disease set and extract feature points and descriptors to perform feature matching and obtain feature point similarity, then compare the feature point similarity and determine whether it is the same disease to obtain the disease judgment result, and then store the new complete disease description information into the road disease database based on the disease judgment result.

[0011] The above-mentioned road disease code generation and image matching verification method based on bus patrol, step A, establishes a road disease database, the road disease database is used to store road disease information recorded by multiple bus patrols, and forms a road disease time evolution sequence for the same disease, wherein the fields of the road disease database include road disease code, bus route, bus vehicle, record location, record timestamp, disease image, disease type and description, the road disease code is used to identify a road disease, the bus route is used to record the route where the bus is located, the bus vehicle is used to record the bus vehicle where the disease is observed, the record location is used to record the location information of the observed disease, and the record time is used to record the location information of the observed disease. The time stamp is used to record the time when the disease is observed, the disease picture is used to store the picture when the disease is observed, the disease type is used to describe the type of road disease, and the description is used for the description information of the disease; a single road disease record in the road disease database can reflect the road disease code, bus route, bus vehicle, record location, record timestamp, disease picture and disease type, and the formation of a road disease time evolution sequence for the same disease is specifically to arrange the single road disease record in chronological order, and then sort the records according to the road disease code by timestamp, so as to view the evolution of the same disease at different time points and locations, and then observe the disease identification history records during the bus patrol process.

[0012] The aforementioned road damage code generation and image matching verification method based on bus patrol, step B, generates a road damage code using hash coding according to the stored road damage information, and then combines the road damage code with the road damage type information to form a complete damage description information. The specific steps are as follows:

[0013] Step B1, using the front-end road disease collection device installed on the bus to extract disease information using an image processing algorithm during the driving of the bus, thereby collecting road disease types, road disease images, road disease recording time and road disease location information;

[0014] Step B2, integrating the road disease recording time, the road disease location information and the road disease picture to obtain integrated information, then using a hash function to perform a hash operation on the integrated information and generate a hash code, and then using the hash code as the road disease code;

[0015] Step B3, associating the road damage type information with the hash code and generating a single road damage record, thereby forming damage description information.

[0016] In the aforementioned method for generating road defect codes and image matching verification based on bus patrol, the specific steps of using a hash function to perform a hash operation on the comprehensive information and generate a hash code in step B2 are as follows:

[0017] Step B21, establish a hash table for each road damage category, specifically mapping the damage category to the code, that is, creating a mapping function Disease type Unique code, if the classification information of road damage is converted into road damage type code When , the mapping function in the established hash table is used;

[0018] Step B22, converting the pixel value of the road damage image into a string, specifically, assuming that the road damage image is a Images with pixels pixels, then the road damage image Can be used To express, Indicates the number of rows, Indicates the number of columns, and then connects all pixel values ​​into a picture pixel value string , as shown in formula (1),

[0019]

[0020] Step B23, convert the location information into a string, specifically, set the location information ,in Indicates longitude, is the latitude, and then the location information Concatenate into position string , as shown in formula (2),

[0021]

[0022] Step B24: Encode the road damage type , Image pixel value string and the location string Concatenate and form a combined string , as shown in formula (3),

[0023]

[0024] Step B25, hash operation, specifically the combined string Perform hash operations and generate hash codes , as shown in formula (4),

[0025]

[0026] in, Represents a hash function.

[0027] The above-mentioned road disease code generation and image matching verification method based on bus patrol, step C, pre-screens the set of road disease databases that are close to the newly generated record position according to the position information and calculates the cosine similarity, then uses the cosine similarity to set the threshold, and then screens out the diseases with similarity to the newly generated code higher than the threshold from the pre-screened set to form a historical similar disease set. The specific steps are as follows:

[0028] Step C1, pre-screening the set of road damage databases that are close to the location of the newly generated record according to the location information, the specific steps are as follows:

[0029] Step C11: record each road hazard , calculate road disease records The distance between the location and the location of the newly generated road disease record , as shown in formula (5),

[0030]

[0031]

[0032] in, Represents each road damage record in the road damage database The location consisting of longitude and latitude, Indicates the location composed of longitude and latitude of the newly generated road disease record, and R represents the radius of the earth;

[0033] Step C12, setting the radius threshold And record the road damage The distance between the location and the location of the newly generated road disease record With radius threshold For comparison, the radius threshold Represents the threshold range of position similarity. If , then the road disease record Join the pre-screening collection;

[0034] Step C2, calculate the cosine similarity, set the newly generated road disease code For vector , and each road disease record in the pre-screened set is encoded as a vector , and then use the cosine similarity formula to calculate the similarity between the newly generated road damage code and each pre-screened road damage record code, where the cosine similarity formula is shown in formula (6):

[0035]

[0036] in, represents cosine similarity;

[0037] Step C3, using cosine similarity to set a threshold, and then filter out the defects with similarity to the newly generated code higher than the threshold from the pre-screened set to form a historical similar defect set. Specifically, the cosine similarity threshold is set. , if the cosine similarity , then confirm that the road damage record is similar to the newly generated road damage code and add the road damage record to the historical similar damage set.

[0038] The above-mentioned road disease coding generation and image matching verification method based on bus patrol, step D, performs image matching on the historical similar disease set and extracts feature points and descriptors to perform feature matching and obtain feature point similarity, then compares the feature point similarity and determines whether it is the same disease to obtain the disease judgment result, and then stores the new complete disease description information into the road disease database according to the disease judgment result. The specific process includes feature extraction, feature matching, geometric correction, position alignment, similarity measurement and generation of road disease time evolution sequence. The specific steps are as follows:

[0039] Step D1, feature extraction, specifically, using the scale-invariant feature transform (SIFT) algorithm to extract features from the newly collected road damage image and each road damage image in the historical similar damage set to obtain corresponding feature points;

[0040] Step D2, feature matching, specifically, taking the newly collected road damage picture as the first picture, and respectively calculating the feature point distance between the newly collected road damage picture and each road damage picture in the historical similar damage set as the second picture;

[0041] Step D3, geometric correction, specifically, selecting homography transformation as the geometric transformation model based on the feature point matching result, and using the RANSAC algorithm to calculate the relative position and posture transformation relationship between the two images;

[0042] Step D4, position alignment, specifically, aligning the two images according to the result of geometric correction, and then aligning them to the same coordinate system;

[0043] Step D5, similarity measurement, specifically, using the structural similarity index SSIM in the aligned images to calculate the similarity between the two photos and set the index threshold , and then judge whether the two pictures represent the same road disease at the same location based on the similarity measurement results. If the similarity is higher than the index threshold , then it is judged as the same disease, otherwise it is judged as different diseases;

[0044] Step D6, generating a time evolution sequence of road defects, is specifically based on the result of defect judgment. If the newly collected road defect is not identified as the same defect in the historical similar defect set, the complete defect description information corresponding to the road defect is stored in the road defect database with the new road defect code as the primary key; if the same defect is identified, the newly collected road defect is assigned according to the matched historical similar defect road defect code and a time evolution sequence of road defects is generated.

[0045] The above-mentioned road disease code generation and image matching verification method based on bus patrol, step D1 has the following specific steps:

[0046] Step D11, image preprocessing, specifically converting the road damage image into a grayscale image;

[0047] Step D12, scale space extreme value detection, specifically using Gaussian difference function to detect extreme points in the image at different scales and positions, wherein the extreme points are used as candidates for key points;

[0048] Step D13, key point positioning, specifically, by accurately positioning the extreme points and extracting local feature points with significant changes, wherein the local feature points are used to generate descriptors;

[0049] Step D14, direction determination, specifically assigning a main direction to each key point and making the descriptor invariant to image rotation;

[0050] Step D15, feature description, specifically, constructing a descriptor in a local area around the key point using image gradient information and obtaining local features, wherein the representation of the local features includes vectors and histograms;

[0051] Step D16, feature point screening, specifically screening the generated feature points according to the stability and distinguishability of the features;

[0052] The specific steps of step D2 are as follows:

[0053] Step D21, using the nearest neighbor matching method, select the feature point in the second image that is closest to the current feature point in the first image as its best matching point and calculate the Euclidean distance between the feature points , as shown in formula (7),

[0054] (7)

[0055] in, and Represents the first eigenvector of two feature points respectively. dimension, Represents the dimension of the feature vector.

[0056] Step D22, screen all matching points and define the threshold , if the distance between two feature points is less than the threshold , then confirm that they are matched, otherwise remove them;

[0057] The specific steps of step D3 are as follows:

[0058] Step D31, homography transformation, specifically, selecting a homography transformation model to map the feature points in the first road damage image to the corresponding matching feature points in the second road damage image, and setting a 3x3 homography matrix To express;

[0059] Step D32, using the RANSAC algorithm for random sampling, specifically using the RANSAC algorithm to estimate the homography transformation matrix H and eliminate outliers and false matches in feature point matching, and estimating the homography transformation matrix H by randomly selecting a minimum sample set, and calculating the projection errors of all points according to the homography transformation matrix H;

[0060] Step D33, model evaluation, specifically calculating the fit between the model generated by each random sampling and all the points in the data set;

[0061] Step D34, parameter update, specifically, selecting the model with the most inliers as the best model through an iterative process, and re-estimating the homography transformation matrix using the inliers ;

[0062] Step D35, repeating the iteration, specifically repeating the random sampling step D32 and the model evaluation step D33 until the set convergence condition is met;

[0063] Step D36, final estimation, specifically obtaining the homography transformation matrix through the RANSAC algorithm , where the homography transformation matrix It is the estimation of the relative position and posture transformation relationship between two images.

[0064] The above-mentioned road disease code generation and image matching verification method based on bus patrol, step D4 has the following specific steps:

[0065] Step D41, applying a geometric transformation, specifically using the homography transformation matrix obtained in step D36 Transform one of the photos to obtain a transformed photo, and then align the transformed photo to the coordinate system of the other photo, wherein the transformation includes translation, rotation and scaling;

[0066] Step D42, pixel interpolation, specifically, performing pixel interpolation on the image using nearest neighbor interpolation during the geometric transformation process;

[0067] Step D43, cropping the image, specifically cropping the image to make the two images the same size and removing the blank area;

[0068] Step D44, outputting the aligned image, specifically outputting the alignment result of the two images in the same coordinate system;

[0069] The calculation formula of the specific structural similarity index SSIM in step D5 is shown in formula (8):

[0070]

[0071] in, and Represent the first and second pictures respectively. and Respectively expressed as and The mean brightness of and Respectively expressed as and The variance of and Respectively expressed as and The covariance of , and are constants used for stability, and their values ​​are , and , represents the maximum value of gray level, and and Represents a constant used to prevent zeros from appearing in the denominator.

[0072] The road damage code generation and image matching verification system based on bus patrol includes a database construction module, a road damage code generation module, a historical similar damage set formation module and a damage judgment module;

[0073] The database construction module is used to establish a road disease database, which is used to store road disease information recorded by multiple bus patrols and form a road disease time evolution sequence for the same disease;

[0074] The road damage code generation module is used to generate a road damage code using hash code according to the stored road damage information, and then combine the road damage code with the road damage type information to form complete damage description information;

[0075] The historical similar disease set forming module is used to pre-screen the set of road disease databases that is close to the newly generated record position according to the position information and calculate the cosine similarity, then use the cosine similarity to set the threshold, and then screen out the diseases with the newly generated code similarity higher than the threshold from the pre-screened set to form the historical similar disease set;

[0076] The disease judgment module is used to perform image matching on historical similar disease sets and extract feature points and descriptors for feature matching and obtain feature point similarity, then compare the feature point similarity and determine whether it is the same disease to obtain a disease judgment result, and then store the new complete disease description information into the road disease database based on the disease judgment result.

[0077] The aforementioned road defect coding generation and image matching verification system based on bus patrol, the specific formation process of the historical similar defect set formation module includes feature extraction, feature matching, geometric correction, position alignment, similarity measurement and generation of road defect time evolution sequence.

[0078] The beneficial effects of the present invention are:

[0079] (1) The present invention first establishes a road damage database, which is used to store road damage information recorded by multiple bus patrols, and forms a road damage time evolution sequence for the same damage. Then, a road damage code is generated using hash coding based on the stored road damage information, and the road damage code is combined with road damage type information to form complete damage description information. Then, a set of road damage databases that is close to the newly generated record position is pre-screened based on the position information and cosine similarity is calculated. Then, a threshold is set using the cosine similarity. Then, damages with a similarity to the newly generated code higher than the threshold are screened from the pre-screened set to form a historical similar damage set. Then, image matching is performed on the historical similar damage set to extract feature points and The descriptor performs feature matching and obtains the similarity of feature points, then compares the similarity of feature points and determines whether it is the same disease to obtain the disease judgment result, and finally stores the new complete disease description information into the road disease database according to the disease judgment result; the present invention fully considers the advantages of using urban buses for road maintenance inspections, and buses are an important part of urban public transportation, and their routes cover various areas of the city. At the same time, due to the large number of buses running every day, the intelligent sensing equipment equipped with road disease inspection can detect diseases in time on the one hand, and on the other hand, the time evolution trend of each road disease can be discovered through each running route every day, which solves the problems of single point and inefficiency faced by traditional road disease detection methods.

[0080] (2) The present invention utilizes the high frequency of bus patrols to construct a road hazard coding generation and identification method for bus inspections and generates a road hazard time evolution sequence, effectively improving the automated closed-loop management capability of intelligent road inspections.

[0081] (3) The present invention can analyze and trace the location changes and evolution of road diseases, and understand the development trend and influencing factors of road diseases. It can not only automatically generate road disease codes based on the identified disease information, but also identify specific road diseases and associate them with specific location information, effectively realizing the association and tracing functions of disease identities, and can verify the accuracy of disease information through similarity calculation and feature matching based on the existing road disease database. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is a flow chart of the method and system for generating road hazard codes and image matching verification based on bus patrol of the present invention;

[0083] Figure 2 It is a hash representation of the disease category in the embodiment of the present invention;

[0084] Figure 31 is a schematic diagram of matching feature points between newly collected road damage and historical road damage in an embodiment of the present invention, wherein (a) is a comparison diagram of newly collected road damage and historical road damage, and (b) is a schematic diagram of matching feature points between newly collected road damage and historical road damage. DETAILED DESCRIPTION

[0085] The present invention will be further described below in conjunction with the accompanying drawings.

[0086] like Figure 1 As shown, the method and system for generating road hazard codes and image matching verification based on bus patrol of the present invention include the following steps:

[0087] Step A, establishing a road disease database, the road disease database is used to store road disease information recorded by multiple bus patrols, and form a road disease time evolution sequence for the same disease, wherein the fields of the road disease database include road disease code, bus route, bus vehicle, recording location, recording timestamp, disease picture, disease type and description, the road disease code is used to identify a road disease, the bus route is used to record the route where the bus is located, the bus vehicle is used to record the bus vehicle where the disease is observed, the recording location is used to record the location information of the observed disease, the recording timestamp is used to record the time when the disease is observed, the disease picture is used to store the picture when the disease is observed, the disease type is used to describe the type of road disease, and the description is used to describe the disease;

[0088] A single road disease record in the road disease database can reflect the road disease code, bus route, bus vehicle, record location, record timestamp, disease picture and disease type. The road disease time evolution sequence formed for the same disease is specifically to arrange the single road disease record in chronological order, and then sort the records according to the road disease code by timestamp, so as to view the evolution of the same disease at different time points and locations, and then observe the disease identification history records during the bus patrol process.

[0089] Step B, using hash coding to generate a road damage code based on the saved road damage information, and then combining the road damage code with the road damage type information to form complete damage description information. The specific steps are as follows:

[0090] Step B1, using the front-end road disease collection device installed on the bus to extract disease information using an image processing algorithm during the driving of the bus, thereby collecting road disease types, road disease images, road disease recording time and road disease location information;

[0091] Step B2, integrating the road disease recording time, the road disease location information and the road disease picture to obtain integrated information, then using a hash function to perform a hash operation on the integrated information and generate a hash code, and then using the hash code as the road disease code;

[0092] The specific steps of using the hash function to perform hash operation on the comprehensive information and generate the hash code in step B2 are as follows:

[0093] Step B21, establish a hash table for each road damage category, specifically mapping the damage category to the code, that is, creating a mapping function Disease type Unique code, if the classification information of road damage is converted into road damage type code When , the mapping function in the established hash table is used;

[0094] Step B22, converting the pixel value of the road damage image into a string, specifically, assuming that the road damage image is a Images with pixels pixels, then the road damage image Can be used To express, Indicates the number of rows, Indicates the number of columns, and then connects all pixel values ​​into a picture pixel value string , as shown in formula (1),

[0095]

[0096] Step B23, convert the location information into a string, specifically, set the location information ,in Indicates longitude, is the latitude, and then the location information Concatenate into position string , as shown in formula (2),

[0097]

[0098] Step B24: Encode the road damage type , Image pixel value string and the location string Concatenate and form a combined string , as shown in formula (3),

[0099]

[0100] Step B25, hash operation, specifically the combined string Perform hash operations and generate hash codes , as shown in formula (4),

[0101]

[0102] in, Represents a hash function.

[0103] Step B3, associating the road damage type information with the hash code and generating a single road damage record, thereby forming damage description information.

[0104] Step C, pre-screening the set of road damage databases that are close to the newly generated record location according to the location information and calculating the cosine similarity, then using the cosine similarity to set the threshold, and then screening out the damages with a similarity to the newly generated code higher than the threshold from the pre-screened set to form a historical similar damage set. The specific steps are as follows:

[0105] Step C1, pre-screening the set of road damage databases that are close to the location of the newly generated record based on the location information, the specific steps are as follows:

[0106] Step C11: record each road hazard , calculate road disease records The distance between the location and the location of the newly generated road disease record , as shown in formula (5),

[0107]

[0108]

[0109] in, Represents each road damage record in the road damage database The location consisting of longitude and latitude, Indicates the location composed of longitude and latitude of the newly generated road disease record, and R represents the radius of the earth;

[0110] Step C12, setting the radius threshold And record the road damage The distance between the location and the location of the newly generated road disease record With radius threshold For comparison, the radius threshold Represents the threshold range of position similarity. If , then the road disease record Join the pre-screening collection;

[0111] Step C2, calculate the cosine similarity, set the newly generated road disease code For vector , and each road disease record in the pre-screened set is encoded as a vector , and then use the cosine similarity formula to calculate the similarity between the newly generated road damage code and each pre-screened road damage record code, where the cosine similarity formula is shown in formula (6):

[0112]

[0113] in, represents cosine similarity;

[0114] Step C3, using cosine similarity to set a threshold, and then filter out the defects with similarity to the newly generated code higher than the threshold from the pre-screened set to form a historical similar defect set. Specifically, the cosine similarity threshold is set. , if the cosine similarity , then confirm that the road damage record is similar to the newly generated road damage code and add the road damage record to the historical similar damage set.

[0115] Step D, image matching is performed on the historical similar disease set, and feature points and descriptors are extracted for feature matching and feature point similarity is obtained. The feature point similarity is then compared to determine whether it is the same disease to obtain the disease judgment result. Then, according to the disease judgment result, the new complete disease description information is stored in the road disease database. The specific process includes feature extraction, feature matching, geometric correction, position alignment, similarity measurement and generation of road disease time evolution sequence. The specific steps are as follows:

[0116] Step D1, feature extraction, specifically, the newly collected road disease pictures are compared with each road disease picture in the historical similar disease set using the scale-invariant feature transform SIFT algorithm to extract features and obtain corresponding feature points. The specific steps are as follows:

[0117] Step D11, image preprocessing, specifically converting the road damage image into a grayscale image;

[0118] Step D12, scale space extreme value detection, specifically using Gaussian difference function to detect extreme points in the image at different scales and positions, wherein the extreme points are used as candidates for key points;

[0119] Step D13, key point positioning, specifically, by accurately positioning the extreme points and extracting local feature points with significant changes, wherein the local feature points are used to generate descriptors;

[0120] Step D14, direction determination, specifically assigning a main direction to each key point and making the descriptor invariant to image rotation;

[0121] Step D15, feature description, specifically, constructing a descriptor in a local area around the key point using image gradient information and obtaining local features, wherein the representation of the local features includes vectors and histograms;

[0122] Step D16, feature point screening, specifically screening the generated feature points according to the stability and distinguishability of the features;

[0123] Step D2, feature matching, specifically, taking the newly collected road damage picture as the first picture, and calculating the feature point distances between the newly collected road damage picture and each road damage picture in the historical similar damage set as the second picture, the specific steps are as follows:

[0124] Step D21, using the nearest neighbor matching method, select the feature point in the second image that is closest to the current feature point in the first image as its best matching point and calculate the Euclidean distance between the feature points , as shown in formula (7),

[0125]

[0126] in, and Represents the first eigenvector of two feature points respectively. dimension, Represents the dimension of the feature vector.

[0127] Step D22, screen all matching points and define the threshold , if the distance between two feature points is less than the threshold , then confirm that they are matched, otherwise remove them;

[0128] Step D3, geometric correction, specifically, selects homography transformation as the geometric transformation model based on the feature point matching result, and uses the RANSAC algorithm to calculate the relative position and posture transformation relationship between the two images. The specific steps are as follows:

[0129] Step D31, homography transformation, specifically, selecting a homography transformation model to map the feature points in the first road damage image to the corresponding matching feature points in the second road damage image, and setting a 3x3 homography matrix To express;

[0130] Step D32, using the RANSAC algorithm for random sampling, specifically using the RANSAC algorithm to estimate the homography transformation matrix H and eliminate outliers and false matches in feature point matching, and estimating the homography transformation matrix H by randomly selecting a minimum sample set, and calculating the projection errors of all points according to the homography transformation matrix H;

[0131] Step D33, model evaluation, specifically calculating the fit between the model generated by each random sampling and all the points in the data set;

[0132] Step D34, parameter update, specifically, selecting the model with the most inliers as the best model through an iterative process, and re-estimating the homography transformation matrix using the inliers ;

[0133] Step D35, repeating the iteration, specifically repeating the random sampling step D32 and the model evaluation step D33 until the set convergence condition is met;

[0134] Step D36, final estimation, specifically obtaining the homography transformation matrix through the RANSAC algorithm , where the homography transformation matrix Estimation of the relative position and posture transformation relationship between two images;

[0135] Step D4, position alignment, specifically aligns the two images according to the results of geometric correction, and then aligns them to the same coordinate system. The specific steps are as follows:

[0136] Step D41, applying a geometric transformation, specifically using the homography transformation matrix obtained in step D36 Transform one of the photos to obtain a transformed photo, and then align the transformed photo to the coordinate system of the other photo, wherein the transformation includes translation, rotation and scaling;

[0137] Step D42, pixel interpolation, specifically, performing pixel interpolation on the image using nearest neighbor interpolation during the geometric transformation process;

[0138] Step D43, cropping the image, specifically cropping the image to make the two images the same size and removing the blank area;

[0139] Step D44, outputting the aligned image, specifically outputting the alignment result of the two images in the same coordinate system;

[0140] Step D5, similarity measurement, specifically, using the structural similarity index SSIM in the aligned images to calculate the similarity between the two photos and set the index threshold , and then judge whether the two pictures represent the same road disease at the same location based on the similarity measurement results. If the similarity is higher than the index threshold , then they are judged to be the same disease, otherwise they are judged to be different diseases. The calculation formula of the specific structural similarity index SSIM is shown in formula (8):

[0141]

[0142] in, and Represent the first and second pictures respectively. and Respectively expressed as and The mean brightness of and Respectively expressed as and The variance of and Respectively expressed as and The covariance of , and are constants used for stability, and their values ​​are , and , represents the maximum value of gray level, and and Represents a constant used to prevent zeros from appearing in the denominator.

[0143] Step D6, generating a time evolution sequence of road defects, is specifically based on the result of defect judgment. If the newly collected road defect is not identified as the same defect in the historical similar defect set, the complete defect description information corresponding to the road defect is stored in the road defect database with the new road defect code as the primary key; if the same defect is identified, the newly collected road defect is assigned according to the matched historical similar defect road defect code and a time evolution sequence of road defects is generated.

[0144] The road disease code generation and image matching verification system based on bus patrol includes a database construction module, a road disease code generation module, a historical similar disease set formation module and a disease judgment module; the database construction module is used to establish a road disease database, the road disease database is used to save the road disease information recorded by multiple bus patrols, and form a road disease time evolution sequence for the same disease; the road disease code generation module is used to generate a road disease code based on the saved road disease information using hash coding, and then combine the road disease code with road disease type information to form complete disease description information; the historical similar disease set formation ... The similar disease set forming module is used to pre-screen the set in the road disease database that is close to the newly generated record position according to the position information and calculate the cosine similarity, then use the cosine similarity to set the threshold, and then screen out the diseases with the newly generated coding similarity higher than the threshold from the pre-screened set to form a historical similar disease set; the disease judgment module is used to perform image matching on the historical similar disease set and extract feature points and descriptors to perform feature matching and obtain feature point similarity, then compare the feature point similarity and judge whether it is the same disease to obtain the disease judgment result, and then store the new complete disease description information into the road disease database according to the disease judgment result.

[0145] Specifically, the specific formation process of the historical similar disease set forming module includes feature extraction, feature matching, geometric correction, position alignment, similarity measurement and generation of road disease time evolution sequence.

[0146] In order to better illustrate the use effect of the present invention, an embodiment of the specific application of the road disease code generation and image matching verification method and system of the present invention is introduced below.

[0147] In this embodiment, the road defect database in step A establishes a database RoadDefectsDB, which includes a main table DefectsTable for storing each defect record, and the fields of DefectsTable are as follows:

[0148] DefectCode (road defect code, VARCHAR), used to identify a single defect;

[0149] BusRoute (bus route, VARCHAR), records the route of the bus when the disease is found;

[0150] BusVehicle (Bus Vehicle, VARCHAR), records the specific bus number where the disease is found;

[0151] Location (record location, VARCHAR), records the geographical location information of the disease (such as GPS coordinates);

[0152] Timestamp (record timestamp, DATETIME), the time when the disease was discovered;

[0153] DefectImage (disease image, BLOB), disease image, stored in the form of binary large object;

[0154] DefectType (Disease Type, VARCHAR), describes the type of disease, such as cracks and potholes;

[0155] Description (TEXT), detailed description of the disease.

[0156] Step B is to generate road damage codes using hash codes, including image acquisition, information integration, hash operations, code generation, and formation of damage description information;

[0157] Step B1, collecting road damage information, the collecting device continuously captures road images while the bus is traveling, and uses image processing algorithms, such as YOLOV8 and Fast-RCNN target detection algorithms, to extract the damage information;

[0158] Step B2, generating a road damage code, wherein a hash function is used to perform a hash operation on the comprehensive information, specifically including the following sub-steps:

[0159] Step B21, establish a disease category hash table, as shown in Table 1. If the disease type is "Crack", then ;

[0160] Table 1. Disease category hash table

[0161] .

[0162] Step B22, image pixel value conversion, convert the pixel value of the obtained disease image into a string, for example Figure 2 The process of stringifying the pixel values ​​of a picture with a size of 3*3 is finally obtained.

[0163] Step B23: convert the location information into a string. , for example, if the location information is ,but ;

[0164] Step B24, forming a combined string, the disease type code, the image pixel value string and the location information string are combined, for example ,

[0165] ,but

[0166] Step B25, hash operation, for example, applying a hash operation to the above combined string to obtain a unique hash code

[0167] .

[0168] Step C, pre-screen the set of road damage databases that are close to the newly generated record location based on the location information and calculate the cosine similarity, then use the cosine similarity to set the threshold, and then screen out the damages with a similarity to the newly generated code higher than the threshold from the pre-screened set to form a historical similar damage set; in this implementation case, the radius threshold 𝑟=5 meters and the cosine similarity threshold 𝜃=0.8 are set.

[0169] Step D, perform image matching on the historical similar disease set and extract feature points and descriptors to perform feature matching and obtain feature point similarity, then compare the feature point similarity and determine whether it is the same disease to obtain the disease judgment result, and then store the new complete disease description information into the road disease database according to the disease judgment result. The specific process includes feature extraction, feature matching, geometric correction, position alignment, similarity measurement and generation of road disease time evolution sequence. Figure 3 The following is an example of image matching of a specific road pothole.

[0170] Step D1, feature extraction, the number of feature points extracted from the newly collected road damage image through scale-invariant feature transformation is 823, and the number of feature points that can be extracted from the corresponding historical similar damage image is 850;

[0171] Step D2, feature matching, for each historical similar disease picture, the average number of matched feature points is 500;

[0172] Step D3, geometric correction, the number of inliers required to complete the homography transformation is 460;

[0173] Step D4: position alignment, the size of the aligned image is 800x600 pixels;

[0174] Step D5, similarity measurement, using the structural similarity index SSIM to calculate the similarity between the two images, the similarity index SSIM value is 0.65, and the index threshold is set is 0.8, so the two pictures are not of the same type of road damage at the same location.

[0175] Table 2 shows the comparison of the matching accuracy and calculation time of the method of the present invention and the image matching method alone on different data set sizes. According to the data set collected from actual buses, the sizes of the disease records are 100, 500 and 1000 respectively, and specific thresholds are set to filter the cosine similarity and feature point matching.

[0176] Table 2. Comparison of matching accuracy and computation time on different dataset sizes

[0177] .

[0178] In summary, the road defect code generation and image matching verification method and system based on bus patrol of the present invention realizes the construction of road defect code generation and identification method and generation of road defect time evolution sequence by utilizing the characteristics of high-frequency bus patrol, and also realizes the analysis and tracing of the location change and evolution of road defects, and can understand the development trend and influencing factors of road defects. It can not only automatically generate road defect codes according to the identified defect information, but also identify specific road defects and associate them with specific location information, thereby improving the automated closed-loop management capability of intelligent road inspection.

[0179] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A road hazard coding generation and image matching verification method based on bus patrol, characterized by: The following steps are included: Step A, establishing a road damage database, wherein the road damage database is used to store road damage information recorded by multiple bus patrols, and to form a road damage time evolution sequence for the same damage; Step B, using hash coding to generate a road damage code based on the saved road damage information, and then combining the road damage code with the road damage type information to form complete damage description information. The specific steps are as follows: Step B1, using the front-end road disease collection device installed on the bus to extract disease information using an image processing algorithm during the driving of the bus, thereby collecting road disease types, road disease images, road disease recording time and road disease location information; Step B2, the road disease recording time, the road disease location information and the road disease picture are integrated to obtain integrated information, and then a hash function is used to perform a hash operation on the integrated information and generate a hash code, and then the hash code is used as the road disease code, wherein the specific steps of using a hash function to perform a hash operation on the integrated information and generate a hash code are as follows: Step B21, establish a hash table for each road damage category, specifically mapping the damage category to the code, that is, creating a mapping function f type :Disease type → unique code, if the classification information of road diseases is converted into road disease type code S type When , the mapping function in the established hash table is used; Step B22, converting the pixel value of the road damage image into a string. Specifically, assuming that the road damage image I is an m×n image with m×n pixels, the road damage image I can be represented by p ij Represented as follows, where i represents the number of rows and j represents the number of columns, and then all pixel values ​​are concatenated into a picture pixel value string S pixels , as shown in formula (1), S pixels =p 11 p 12 …p 1n p 21 p 22 …p mn (1); Step B23, convert the location information into a character string, specifically, set the location information L = (x, y), where x represents longitude and y represents latitude, and then concatenate the location information L into a location character string S location , as shown in formula (2), S location =x+y(2); Step B24, code the road damage type S type , Image pixel value string S pixels and position string S location Concatenate and form a combined string S combined , as shown in formula (3), S combined =S type +S pixels +S location (3); Step B25, hash operation, specifically, the combined string S combined Perform hash operation and generate hash code H, as shown in formula (4), H=hash(S combined ) (4) Wherein, hash(·) represents a hash function; Step B3, associating the road damage type information with the hash code and generating a single road damage record, thereby forming damage description information; Step C, pre-screening the set of road damage databases that are close to the newly generated record location according to the location information and calculating the cosine similarity, then using the cosine similarity to set a threshold, and then screening out the damages with a similarity to the newly generated code higher than the threshold from the pre-screened set to form a historical similar damage set; Step D, perform image matching on the historical similar disease set and extract feature points and descriptors to perform feature matching and obtain feature point similarity, then compare the feature point similarity and determine whether it is the same disease to obtain the disease judgment result, and then store the new complete disease description information into the road disease database based on the disease judgment result.

2. The method for generating road hazard codes and image matching verification based on bus patrol according to claim 1 is characterized in that: Step A, establishing a road disease database, the road disease database is used to store road disease information recorded by multiple bus patrols, and form a road disease time evolution sequence for the same disease, wherein the fields of the road disease database include road disease code, bus route, bus vehicle, recording location, recording timestamp, disease picture, disease type and description, the road disease code is used to identify a road disease, the bus route is used to record the route where the bus is located, the bus vehicle is used to record the bus vehicle where the disease is observed, the recording location is used to record the location information of the observed disease, the recording timestamp is used to record the time when the disease is observed, the disease picture is used to store the picture when the disease is observed, the disease type is used to describe the type of road disease, and the description is used to describe the disease; A single road disease record in the road disease database can reflect the road disease code, bus route, bus vehicle, record location, record timestamp, disease picture and disease type. The road disease time evolution sequence formed for the same disease is specifically to arrange the single road disease record in chronological order, and then sort the records according to the road disease code by timestamp, so as to view the evolution of the same disease at different time points and locations, and then observe the disease identification history records during the bus patrol process.

3. The method for generating road hazard codes and verifying image matching based on bus patrol according to claim 2 is characterized in that: Step C, pre-screening the set of road damage databases that are close to the newly generated record location according to the location information and calculating the cosine similarity, then using the cosine similarity to set the threshold, and then screening out the damages with a similarity to the newly generated code higher than the threshold from the pre-screened set to form a historical similar damage set. The specific steps are as follows: Step C1, pre-screening the set of road damage databases that are close to the location of the newly generated record according to the location information, the specific steps are as follows: Step C11: for each road damage record i, calculate the distance d between the position of the road damage record i and the position of the newly generated road damage record i , as shown in formula (5), Among them, (x i ,y i ) represents the location of each road damage record i in the road damage database, which is composed of longitude and latitude. (x new ,y new ) represents the location composed of longitude and latitude of the newly generated road disease record, and R represents the radius of the earth; Step C12: set the radius threshold r and calculate the distance d between the position of the road disease record i and the position of the newly generated road disease record i Compared with the radius threshold r, where the radius threshold r represents the threshold range of position similarity, if d i ≤r, then add the road disease record i to the pre-screening set; Step C2, calculate and obtain the cosine similarity, and set the newly generated road disease code H new is vector A, and each road disease record in the pre-screened set is encoded as vector B i , and then use the cosine similarity formula to calculate the similarity between the newly generated road damage code and each pre-screened road damage record code, where the cosine similarity formula is shown in formula (6): Among them, similarity(i) represents cosine similarity; Step C3, using cosine similarity to set a threshold, and then screen out the diseases with similarity to the newly generated code higher than the threshold from the pre-screening set to form a historical similar disease set. Specifically, the cosine similarity threshold θ is set. If cosine similarity similarity(i)≥θ, it is confirmed that the road disease record is similar to the newly generated road disease code and the road disease record is added to the historical similar disease set.

4. The method for generating road hazard codes and verifying image matching based on bus patrol according to claim 3 is characterized in that: Step D, image matching is performed on the historical similar disease set, and feature points and descriptors are extracted for feature matching and feature point similarity is obtained. The feature point similarity is then compared to determine whether it is the same disease to obtain the disease judgment result. Then, according to the disease judgment result, the new complete disease description information is stored in the road disease database. The specific process includes feature extraction, feature matching, geometric correction, position alignment, similarity measurement and generation of road disease time evolution sequence. The specific steps are as follows: Step D1, feature extraction, specifically, using the scale-invariant feature transform (SIFT) algorithm to extract features from the newly collected road damage image and each road damage image in the historical similar damage set to obtain corresponding feature points; Step D2, feature matching, specifically, taking the newly collected road damage picture as the first picture, and respectively calculating the feature point distance between the newly collected road damage picture and each road damage picture in the historical similar damage set as the second picture; Step D3, geometric correction, specifically, selecting homography transformation as the geometric transformation model based on the feature point matching result, and using the RANSAC algorithm to calculate the relative position and posture transformation relationship between the two images; Step D4, position alignment, specifically, aligning the two images according to the result of geometric correction, and then aligning them to the same coordinate system; Step D5, similarity measurement, specifically, using the structural similarity index SSIM in the aligned images to calculate the similarity between the two photos, and setting the index threshold Δ, and then judging whether the two pictures represent the same road disease at the same location based on the result of the similarity measurement. If the similarity is higher than the index threshold Δ, they are judged to be the same disease, otherwise they are judged to be different diseases; Step D6, generating a time evolution sequence of road defects, is specifically based on the result of defect judgment. If the newly collected road defect is not identified as the same defect in the historical similar defect set, the complete defect description information corresponding to the road defect is stored in the road defect database with the new road defect code as the primary key; if the same defect is identified, the newly collected road defect is assigned according to the matched historical similar defect road defect code and a time evolution sequence of road defects is generated.

5. The method for generating road hazard codes and verifying image matching based on bus patrol according to claim 4 is characterized in that: The specific steps of step D1 are as follows: Step D11, image preprocessing, specifically converting the road damage image into a grayscale image; Step D12, scale space extreme value detection, specifically using Gaussian difference function to detect extreme points in the image at different scales and positions, wherein the extreme points are used as candidates for key points; Step D13, key point positioning, specifically, by accurately positioning the extreme points and extracting local feature points with significant changes, wherein the local feature points are used to generate descriptors; Step D14, direction determination, specifically assigning a main direction to each key point and making the descriptor invariant to image rotation; Step D15, feature description, specifically, constructing a descriptor in a local area around the key point using image gradient information and obtaining local features, wherein the representation of the local features includes vectors and histograms; Step D16, feature point screening, specifically screening the generated feature points according to the stability and distinguishability of the features; The specific steps of step D2 are as follows: Step D21, using the nearest neighbor matching method, selects the feature point in the second image that is closest to the current feature point in the first image as its best matching point and calculates the Euclidean distance D between the feature points, as shown in formula (7), Among them, j and p j They represent the j-th dimension of the feature vectors of the two feature points, and n represents the dimension of the feature vector. Step D22, screen all matching points and define a threshold τ. If the distance between two feature points is less than the threshold τ, they are confirmed to be matched, otherwise they are eliminated; The specific steps of step D3 are as follows: Step D31, homography transformation, specifically, selecting a homography transformation model to map feature points in the first road damage image to corresponding matching feature points in the second road damage image, and representing this by setting a 3x3 homography matrix T; Step D32, using the RANSAC algorithm for random sampling, specifically using the RANSAC algorithm to estimate the homography transformation matrix H and eliminate outliers and false matches in feature point matching, and estimating the homography transformation matrix H by randomly selecting a minimum sample set, and calculating the projection errors of all points according to the homography transformation matrix H; Step D33, model evaluation, specifically calculating the fit between the model generated by each random sampling and all the points in the data set; Step D34, parameter update, specifically, selecting the model with the most inliers as the best model through an iterative process, and re-estimating the homography transformation matrix using the inliers Step D35, repeating the iteration, specifically repeating the random sampling step D32 and the model evaluation step D33 until the set convergence condition is met; Step D36, final estimation, specifically obtaining the homography transformation matrix T through the RANSAC algorithm * , where the homography transformation matrix T * It is the estimation of the relative position and posture transformation relationship between two images.

6. The method for generating road hazard codes and image matching verification based on bus patrol according to claim 5 is characterized in that: The specific steps of step D4 are as follows: Step D41, applying geometric transformation, specifically using the homography transformation matrix T obtained in step D36 * Transform one of the photos to obtain a transformed photo, and then align the transformed photo to the coordinate system of the other photo, wherein the transformation includes translation, rotation and scaling; Step D42, pixel interpolation, specifically, performing pixel interpolation on the image using nearest neighbor interpolation during the geometric transformation process; Step D43, cropping the image, specifically cropping the image to make the two images the same size and removing the blank area; Step D44, outputting the aligned image, specifically outputting the alignment result of the two images in the same coordinate system; The calculation formula of the specific structure similarity index SSIM in step D5 is shown in formula (8): Among them, I1 and I2 represent the first picture and the second picture respectively. and μ I2 They are represented as the mean brightness of I1 and I2 respectively, and are expressed as the variance of I1 and I2 respectively, and They are the covariances of I1 and I2 respectively; C1, C2 and C3 are constants used for stability, where the value is C1 = (k1L) 2 、C2=(k2L) 2 and L represents the maximum value of the gray level, and k1 and k2 represent constants for preventing zero from appearing in the denominator.

7. A road damage coding generation and image matching verification system based on bus patrol, wherein the specific operation process of the road damage coding generation and image matching verification system is based on the road damage coding generation and image matching verification method according to any one of claims 1 to 6, and is characterized in that: It includes a database construction module, a road disease code generation module, a historical similar disease set formation module and a disease judgment module; The database construction module is used to establish a road disease database, which is used to store road disease information recorded by multiple bus patrols and form a road disease time evolution sequence for the same disease; The road damage code generation module is used to generate a road damage code using hash code according to the stored road damage information, and then combine the road damage code with the road damage type information to form complete damage description information; The historical similar disease set forming module is used to pre-screen the set of road disease databases that is close to the newly generated record position according to the position information and calculate the cosine similarity, then use the cosine similarity to set the threshold, and then screen out the diseases with the newly generated code similarity higher than the threshold from the pre-screened set to form the historical similar disease set; The disease judgment module is used to perform image matching on historical similar disease sets and extract feature points and descriptors for feature matching and obtain feature point similarity, then compare the feature point similarity and determine whether it is the same disease to obtain a disease judgment result, and then store the new complete disease description information into the road disease database based on the disease judgment result.

8. The road hazard code generation and image matching verification system based on bus patrol according to claim 7 is characterized by: The specific formation process of the historical similar disease set forming module includes feature extraction, feature matching, geometric correction, position alignment, similarity measurement and generation of road disease time evolution sequence.

Images