Information integration method within road anomaly monitoring system

Abstract

The present invention discloses an information integration method within a road anomaly monitoring system, the method comprising: collecting source data of video images through a multispectral camera carried by an unmanned aerial vehicle; collecting source data of geographic location information through a road surface sensor; collecting source data of road conditions through a vehicle-mounted laser tube and a road surface sensor, etc.; and setting up a comprehensive cloud system that collects multi-party data and provides information integration functions. The information integration method within a road anomaly monitoring system proposed by the present invention can effectively integrate the abnormal information returned by the four-party abnormality detection system, effectively improve the information data quality and information interaction efficiency, and largely avoid the obstacles to abnormality resolution and the occurrence of secondary accidents caused by low information interaction efficiency.

Description

Technical Field

[0001] The invention belongs to the field of electronic information technology, and relates to a road information integration method, and in particular to an information integration method within a road anomaly monitoring system. Background Art

[0002] With the advent of the "big data" era, how to handle various data that are highly dispersed, independent, heterogeneous, and multi-layered, and how to obtain the desired data results from them is not only beneficial to individuals and enterprises, but also related to the interests of the entire country. Therefore, a reasonable and efficient data integration process method is particularly important.

[0003] So far, with the development of my country's economy, the consumption of infrastructure has been very serious. Under high pressure, the maintenance of infrastructure is the basic guarantee for personal safety and maintaining national operations. The road network of each city is the top priority of infrastructure, so it is particularly important to establish multi-dimensional real-time management. Among them, one of the most important links is to integrate and feedback the data collected by multiple parties, which will also be an important part of the construction of smart cities in the future.

[0004] In general, foreign data integration started earlier and has a wider scope, including research in biological genes and geography, and its application is mainly oriented to public utilities. In contrast, domestic research on data integration is in a passive catching-up stage, generally lagging behind foreign countries, and the main application direction is still oriented to commercial services. It can be said that there are few actual application-oriented research, let alone data integration research for road network management. In addition, the research content focuses on relatively basic technical research.

[0005] In view of this, there is an urgent need to design a new information integration method to overcome the above-mentioned defects of existing information integration methods. Summary of the invention

[0006] The present invention provides an information integration method within a road anomaly monitoring system, which can better solve the problem of integrating and feeding back multi-dimensional information data obtained when maintaining a road traffic network during the process of building a green smart city.

[0007] To solve the above technical problem, according to one aspect of the present invention, the following technical solution is adopted:

[0008] A method for integrating information within a road anomaly monitoring system, the method comprising:

[0009] Source data collection of video images through multispectral cameras carried by drones;

[0010] Source data collection of geographic location information through road surface sensors;

[0011] Collect road condition source data through vehicle-mounted laser tubes and road surface sensors;

[0012] Set up a comprehensive cloud system that collects data from multiple parties and provides information integration functions;

[0013] The multi-spectral camera carried by the unmanned aerial vehicle is used to realize the shooting and transmission of ordinary visible light and infrared light video content; the multi-spectral camera carried by the unmanned aerial vehicle includes a remote sensing device, and the remote sensing device includes a main control module and an image display and transmission module;

[0014] The main control module is used to complete the control of ordinary visible light and infrared light cameras, including video focus adjustment, zoom setting and video gain control;

[0015] The image display and transmission module is used to realize the encoding and decoding of two video signals, screen display and data transmission functions; the specific process is: firstly, the CCD camera on the drone end is adjusted through the main control module, the corresponding target image is obtained through the two channels of visible light and infrared light, and the image is converted into digital form and transmitted back to the cloud storage unit;

[0016] The road surface sensor refers to a GPS sensor installed in a manhole cover on a road; the GPS sensor includes a positioning module and an information transmission module;

[0017] The positioning module is used to receive the location information of the manhole cover, including time, positioning status, and longitude and latitude data information; the information transmission module transmits the sensor collected data in a fixed format back to the cloud database;

[0018] The vehicle-mounted laser tube and road pavement sensor include a vehicle-mounted infrared laser tube module, a road surface temperature sensor, and an air temperature / humidity sensor; wherein the infrared laser tube module has the function of collecting a reflected echo signal obtained by emitting a specific wavelength light beam to the road surface; each sensor of the vehicle-mounted system sends the collected echo signal, road surface temperature, and air temperature / humidity data back to the cloud database;

[0019] The integrated cloud system includes a cloud storage unit and a cloud database; the cloud storage unit is used to store video images transmitted back by the multi-spectral camera of the drone remote sensing; the cloud database is used to store geographic location information data sent by the GPS sensor on the road, infrared echo, temperature, humidity information data collected by multiple sensors in the vehicle system, and information reported by maintenance personnel and users;

[0020] The data collected by the multispectral camera on the drone is integrated, including: disassembly and analysis of video images, extraction of coordinates of the drone's geographic location information, and updating of the cloud database and cloud storage unit; the key frames of the video images are extracted by combining the histogram method and the frame averaging method based on the lens boundary; the specific algorithm is as follows: first, the video is structured: each frame of the video is stored; the difference between frames in the lens is calculated by the histogram difference method, and the difference formula is as follows:

[0021]

[0022] Where h(i) represents the cumulative distribution of each grayscale of a frame in the video; m and n represent the previous frame and the next frame respectively; the max() function represents the larger value of the two; k represents all grayscale values ​​of this frame;

[0023] And through multiple screenings, the boundary frames of the adaptive threshold are judged to finally determine the boundary of the shot; the video content is segmented according to the boundary of the shot; finally, the key frames are extracted from each shot; the extracted key frame images are subsequently analyzed, including: hot metal coding processing, digital image processing and road disease image feature repair analysis of various types of images to obtain information such as the type of disease. The specific algorithm is as follows: for icing images, the grayscale area of ​​the relevant image is first divided into four parts: blue, green, yellow and red. The maximum inter-class variance method is used to obtain the binary threshold of the image. Finally, the grayscale of the four fixed areas of the hot metal coding is The range can be modified to four adaptive thresholds; for water and oil pollution images, the image is first grayed out, and then the local threshold segmentation method is used to obtain a clear border, and finally the area of ​​water and oil pollution is calculated based on the pixel ratio; for road disease images, the image is first deblurred, and then the window contrast feature extraction algorithm is used to extract the image features to obtain the results; the required longitude and latitude, and longitude and hemisphere data are extracted from the geographic location information transmitted back by the drone, and the longitude and latitude are converted into degree, minute, and second format, and the hemisphere mark is added; then the cloud database and cloud storage unit are updated, and the processed images and numerical data are stored;

[0024] The data collected by the GPS sensors installed in the manhole covers on the roads are integrated, including: extracting the coordinates of the manhole cover’s geographic location information, obtaining the road name of the diseased location through the coordinate information, and updating the cloud database; similar to the remote sensing system, the coordinate extraction of the manhole cover’s geographic location information extracts the required longitude and latitude, and hemisphere data from the datagrams already stored in the Transportation Bureau’s library, and converts the longitude and latitude into degree, minute, and second format, adding a hemisphere identifier. The specific algorithm is as follows: define the initial received data: ddmm.mmmm as data, and the degrees in the general longitude and latitude format are obtained by dividing data by 100 and rounding; the fractions in the general longitude and latitude format are obtained by rounding the difference (denoted as m1) between data and the product of the degrees and 100; the seconds in the general longitude and latitude format are obtained by rounding the difference between m1 and the fraction; thus, the general longitude and latitude format is obtained: d°m′s″; then the updated coordinate information is obtained through the coordinate offset correction algorithm, and the specific road name is obtained through the API interface; finally, the cloud database is updated and the processed images and numerical data are stored;

[0025] The data collected by the vehicle-mounted infrared laser tube module, road surface temperature sensor, and air temperature / humidity sensor are integrated, including: road surface temperature calculation; air humidity calculation; judgment of whether the road has ice and water accumulation; the infrared radiation energy received by the road surface temperature sensor, the scattering coefficient and the first and second radiation constants, combined with Planck's law, the first radiation constant is divided by the product of the fifth power of the scattering coefficient and the inverse of e to the power of f(x) minus 1, where f(x) is the second radiation constant divided by the product of the scattering coefficient and temperature; air humidity is related to temperature, and air humidity is calculated through relative humidity, absolute humidity and saturated humidity; then, the thickness of ice and water on the road surface and other road condition information are calculated through road surface temperature and air humidity; the updated data is then stored in the cloud database;

[0026] The information reported by the maintenance personnel and users includes two categories: one is the progress report of the maintenance personnel on the abnormal road maintenance tasks, and the other is the abnormal road conditions found and reported by users during driving; the two types of information are segmented by a word segmentation algorithm based on frequency and dynamic programming, and the road section, abnormality type, and maintenance progress are extracted; the multi-party data integration process includes data conflict processing, and the conflict is processed by the open addressing method, including four types of conflicts; the first type is the conflict between the key frame analysis of video image extraction based on ice judgment and the judgment result of actual numerical data; the second type is the conflict between the key frame analysis of video image extraction based on water accumulation judgment and the judgment result of data obtained by the vehicle-mounted laser tube; the third type is the conflict between the key frame analysis of video images taken by drones based on road disease type judgment and the key frame analysis of video pictures obtained by vehicle-mounted cameras; the fourth type is the conflict of feedback time based on the judgment of abnormal processing progress;

[0027] The multi-party data integration process also includes data and image integration processes; the corresponding relationships mainly include the following: ice pictures correspond to ice thickness and temperature; water accumulation pictures correspond to water depth and area; road disease pictures correspond to road names; all of the above correspondences include the correspondence between location coordinate information and road names, and the specific algorithms are as follows: For the first correspondence: extract road ice pictures extracted from videos shot by drones from the cloud database, and at the same time extract ice thickness and temperature calculated based on the pictures and store them in an external database for subsequent web page display; For the second correspondence, similar to the first one: extract road water accumulation pictures extracted from videos shot by drones from the cloud database, and at the same time extract water accumulation depth and area calculated based on the pictures and store them in an external database for subsequent web page display; For the third correspondence: extract road disease pictures extracted from videos shot by drones from the cloud database. Since the drone video provides real-time trajectory, the geographic location information of the picture is obtained through the position of the extracted key frames, and stored in the external database for subsequent web page display.

[0028] A method for integrating information within a road anomaly monitoring system, the method comprising:

[0029] Source data collection of video images through multispectral cameras carried by drones;

[0030] Source data collection of geographic location information through road surface sensors;

[0031] Collect road condition source data through vehicle-mounted laser tubes and road surface sensors;

[0032] Set up a comprehensive cloud system that collects data from multiple parties and provides information integration functions.

[0033] As an embodiment of the present invention, the multi-spectral camera carried by the unmanned aerial vehicle is used to realize the shooting and transmission of ordinary visible light and infrared light video content; the multi-spectral camera carried by the unmanned aerial vehicle includes a remote sensing device, and the remote sensing device includes a main control module and an image display and transmission module;

[0034] The main control module is used to complete the control of ordinary visible light and infrared light cameras, including video focus adjustment, zoom setting and video gain control;

[0035] The image display and transmission module is used to realize the encoding and decoding of two video signals, screen display and data transmission functions; the specific process is: firstly, the CCD camera on the drone end is adjusted through the main control module, the corresponding target image is obtained through the two channels of visible light and infrared light, and the image is converted into digital form and transmitted back to the cloud storage unit;

[0036] The road surface sensor refers to a GPS sensor installed in a manhole cover on a road; the GPS sensor includes a positioning module and an information transmission module;

[0037] The positioning module is used to receive the location information of the manhole cover, including time, positioning status, and longitude and latitude data information; the information transmission module transmits the sensor collected data in a fixed format back to the cloud database;

[0038] The vehicle-mounted laser tube and road pavement sensor include a vehicle-mounted infrared laser tube module, a road surface temperature sensor, and an air temperature / humidity sensor; wherein the infrared laser tube module has the function of collecting a reflected echo signal obtained by emitting a specific wavelength light beam to the road surface; each sensor of the vehicle-mounted system sends the collected echo signal, road surface temperature, and air temperature / humidity data back to the cloud database;

[0039] The comprehensive cloud system includes a cloud storage unit and a cloud database; the cloud storage unit is used to store video images transmitted back by the drone remote sensing multispectral camera; the cloud database is used to store geographic location information data sent by the GPS sensor on the road, infrared echo, temperature, humidity information data collected by multiple sensors in the vehicle system, and information reported by maintenance personnel and users.

[0040] As an embodiment of the present invention, data collected by a multispectral camera on a drone is integrated, including: disassembling and analyzing video images, extracting coordinates of the drone's geographic location information, and updating a cloud database and a cloud storage unit;

[0041] The key frames of the video images are extracted by combining the histogram method and the frame averaging method based on the shot boundary; the specific algorithm is as follows:

[0042] First, the video is structured: each frame of the video is stored; the difference between frames in the lens is calculated by the histogram difference method. The difference formula is as follows:

[0043]

[0044] Where h(i) represents the cumulative distribution of each grayscale of a frame in the video; m and n represent the previous frame and the next frame respectively; the max() function represents the larger value of the two; k represents all grayscale values ​​of this frame;

[0045] And through multiple screenings, the boundary frames of the adaptive threshold are judged to finally determine the boundary of the shot; the video content is segmented according to the boundary of the shot; finally, the key frames are extracted from each shot; the extracted key frame images are subsequently analyzed, including: hot metal coding processing, digital image processing and road disease image feature repair analysis of various types of images to obtain information such as the type of disease. The specific algorithm is as follows: for icing images, the grayscale area of ​​the relevant image is first divided into four parts: blue, green, yellow and red. The maximum inter-class variance method is used to obtain the binary threshold of the image. Finally, the grayscale of the four fixed areas of the hot metal coding is The range can be modified to four adaptive thresholds; for water and oil pollution images, the picture is first grayed out, and then the local threshold segmentation method is used to obtain a clear border, and finally the area of ​​water and oil pollution is calculated according to the pixel ratio; for road disease images, the image is first deblurred, and then the window contrast feature extraction algorithm is used to extract the features of the image to obtain the result; the required longitude and latitude, and longitude and hemisphere data are extracted from the geographic location information transmitted back by the drone, and the longitude and latitude are converted into degree, minute, and second format, and the hemisphere mark is added; then the cloud database and cloud storage unit are updated, and the processed pictures and numerical data are stored.

[0046] As an implementation method of the present invention, the data collected by the GPS sensor installed in the manhole cover on the road is integrated, including: extracting the coordinates of the manhole cover's geographical location information, obtaining the name of the road where the disease is located through the coordinate information, and updating the cloud database; similar to the remote sensing system, the coordinate extraction of the manhole cover's geographical location information extracts the required longitude and latitude, and longitude and latitude hemisphere data from the datagram already stored in the transportation bureau's library, converts the longitude and latitude into degree, minute, and second format, and adds a hemisphere mark. The specific algorithm is as follows:

[0047] Define the initial received data: ddmm.mmmm as data. The degree in the general latitude and longitude format is obtained by dividing data by 100 and rounding; the fraction in the general latitude and longitude format is obtained by rounding the difference between data and the product of the degree and 100 (recorded as m1); the second in the general latitude and longitude format is obtained by rounding the difference between m1 and the fraction; thus, the general latitude and longitude format is obtained: d°m′s″;

[0048] Then, the updated coordinate information is obtained through the coordinate offset correction algorithm, and the specific road name is obtained through the API interface; finally, the cloud database is updated to store the processed images and numerical data.

[0049] As an embodiment of the present invention, data collected by a vehicle-mounted infrared laser tube module, a road surface temperature sensor, and an air temperature / humidity sensor are integrated, including: road surface temperature calculation; air humidity calculation; judgment of whether ice and water are generated on the road; infrared radiation energy received by the road surface temperature sensor, scattering coefficient and the first and second radiation constants, combined with Planck's law, the first radiation constant is divided by the product of the fifth power of the scattering coefficient and the inverse of e to the power of f(x) minus 1, where f(x) is the second radiation constant divided by the product of the scattering coefficient and temperature; air humidity is related to temperature, and air humidity is calculated through relative humidity, absolute humidity and saturated humidity; then, road condition information such as the thickness of ice and water on the road surface is calculated through important parameters such as road surface temperature and air humidity; and the updated data is then stored in a cloud database.

[0050] As an implementation mode of the present invention, the information reported by the maintenance personnel and the user includes two categories: one is the progress report of the maintenance personnel on the road abnormality maintenance work task, and the other is the abnormal road condition found and reported by the user during driving; the two types of information are segmented by a word segmentation algorithm based on frequency and dynamic programming, and the road section, abnormality type, and maintenance progress are extracted;

[0051] The multi-party data integration process includes data conflict processing, and the conflicts are processed through an open addressing method, including four types of conflicts: the first type is a conflict between the key frame analysis extracted from video images based on ice judgment and the judgment result of the actual numerical data; the second type is a conflict between the key frame analysis extracted from video images based on water accumulation judgment and the judgment result of the data obtained by the vehicle-mounted laser tube; the third type is a conflict between the key frame analysis extracted from video images taken by drones based on road disease type judgment and the key frame analysis of video pictures obtained by vehicle-mounted cameras; the fourth type is a conflict in feedback time based on the judgment of the progress of exception processing.

[0052] As an implementation mode of the present invention, the multi-party data integration process also includes a data and picture integration process;

[0053] The corresponding relationships are mainly as follows: ice pictures correspond to ice thickness and temperature; water accumulation pictures correspond to water depth and area; road disease pictures correspond to road names; all of the above correspondences include the correspondence between location coordinate information and road names. The specific algorithms are as follows:

[0054] For the first correspondence: extract the road icing picture extracted from the video taken by the drone from the cloud database, and extract the ice thickness and temperature calculated based on the picture and store them in the external database for subsequent web page display; For the second correspondence, similar to the first one: extract the road waterlogging picture extracted from the video taken by the drone from the cloud database, and extract the waterlogging depth and area calculated based on the picture and store them in the external database for subsequent web page display; For the third correspondence: extract the road disease picture extracted from the video taken by the drone from the cloud database. Since the drone video provides real-time trajectory, the geographic location information of the picture is obtained through the position of the extracted key frame, and stored in the external database for subsequent web page display.

[0055] A method for integrating information within a road anomaly monitoring system, the method comprising:

[0056] Step 101, receiving the raw data information collected and sent back by the multispectral camera on the drone, the multi-party sensor based on the narrowband Internet of Things, the vehicle-mounted laser tube and the road surface sensor, the camera, and the web platform, and storing it in the relevant cloud database or cloud storage unit;

[0057] Step 102: Perform data cleaning and related algorithm processing on the obtained raw data to preliminarily obtain processed data information;

[0058] Step 103: further conflict resolution is performed on the preliminary processed data in the cloud information system. After the data information processing is completed, the corresponding cloud database and storage unit are merged, and the data and images are further integrated and stored in the comprehensive cloud system;

[0059] Step 104: Finally, the data information in the cloud system is presented and fed back to the relevant third party through a visualization port.

[0060] As an implementation mode of the present invention, in step 102, the system will first clean the obtained raw data to remove abnormal values ​​and dirty data; after the data cleaning is completed, the corresponding algorithm will be selected according to the data information type for further processing to preliminarily obtain relevant data conclusions and valid information;

[0061] For the raw data sent back by the GPS locator, the information module in the positioning sensor provides data information of various types. The information category of the recommended minimum positioning information is selected through the GPRMC identifier. This type of information provides: Greenwich Mean Time in hours, minutes, and seconds format, positioning status, longitude and latitude information in degrees and minutes format, longitude and latitude hemisphere identifier, ground speed, date in day-month-year format, magnetic declination, magnetic declination direction, and mode indication; the above information is processed, including:

[0062] First, determine the positioning status. If it is marked as 'A', keep this data and perform subsequent processing. If it is displayed as 'V', read the next data. Among them, A indicates valid positioning, and V indicates invalid positioning.

[0063] Convert Greenwich Mean Time in hours, minutes and seconds format to Beijing time. According to the time zone difference, the new time is hh+8, mm, ss;

[0064] Convert the latitude and longitude data. The specific steps are as follows:

[0065] The received data format is: ddmm.mmmm; the general longitude and latitude format is: d°m′s″;

[0066] define data = ddmm.mmmm;

[0067] degree d = int(data / 100);

[0068] m1 = data - d * 100;

[0069] m=int(m1);

[0070] Seconds s = int((m1-m)*60);

[0071] When data is exported, add the south / north latitude mark after the latitude and the east / west longitude mark after the longitude to standardize the format;

[0072] Keep the processed data, update the cloud database, and discard the redundant data;

[0073] Next, perform offset correction on the coordinates obtained in the previous step. The specific implementation method is as follows:

[0074] Input PR (lat, lng), which is the coordinate information extracted in the previous step;

[0075] Retrieve the matching coordinate value in the correction table and obtain PO (latoffset, lngoffset), which is the correction offset difference;

[0076] Finally, add / subtract the difference value to the original longitude and latitude values ​​to obtain the corrected coordinate data;

[0077] Obtain various POI data through the API interfaces of various map service applications, including name, latitude and longitude coordinates, and address information. The specific implementation methods are as follows:

[0078] Input coordinate information for automatic positioning, and return the road street, house number streetNumber, province province, and city city information;

[0079] Extract the street information;

[0080] return street;

[0081] In step 102, three wavelengths, λ1, λ2, and λ3, are selected for the original reflected wave data sent back by the vehicle-mounted sensor according to the water absorption spectrum, the ice absorption spectrum, and the oil absorption spectrum; due to the combined effects of the road surface reflection and the absorption and scattering of the light signal by water molecules and solid particles, the backscattering coefficients of water for the three wavelengths are λ1>λ3>λ2; the backscattering coefficients of ice for the three wavelengths are λ1>λ2>λ3; and the backscattering coefficients of oil for the three wavelengths are λ1>λ3>λ2; and the road surface condition is qualitatively analyzed according to the size and difference of the ratio of the three backscattered light signals to the light signal when the road surface is dry;

[0082] For temperature calculation, by receiving infrared radiation energy from the road surface, according to Planck's law:

[0083] W(λ,T)=C1 / {λ 5 [exp(C2 / (λT))-1] -1}

[0084] Where, T is temperature; C1 is the first radiation constant; C2 is the second radiation constant;

[0085] When the wavelength is constant, the infrared radiation energy W is measured and the temperature can be calculated; temperature measurement range: -60~+85℃; measurement accuracy: ≤±0.1℃; resolution: 0.1℃;

[0086] For the calculation of the thickness of the covering, based on the absorbance of the substance and the Lambert-Beer law, the thickness calculation formula is derived:

[0087] d=L n (I0 / I) / (εC)

[0088] Where d is the thickness of the light-absorbing material; I0 is the incident light intensity; I is the intensity of the transmitted light after passing through the material; L n (I0 / I) is absorbance; C is the concentration of the substance; ε is the proportional coefficient of light absorption, which is related to the properties of the absorbing substance and the wavelength of the incident light;

[0089] Measuring range: 0.00~2mm; Measuring accuracy: ±0.01mm; Resolution: 0.01mm;

[0090] In step 102, for the original video images transmitted back by the multi-spectral camera of the drone and the vehicle-mounted camera, the video content is firstly decomposed into images;

[0091] The key frames of the video content are extracted by combining the histogram method and the frame averaging method based on the shot boundary. The specific steps are as follows:

[0092] First, the video is structured, including:

[0093] Store each frame of the video;

[0094] Calculate the difference between frames in the shot using the histogram difference method and store the data in a structure;

[0095]

[0096] Where h(i) represents the cumulative distribution of each grayscale of a frame in the video; m and n represent the previous frame and the next frame respectively; the max() function represents the larger value of the two; k represents all grayscale values ​​of this frame;

[0097] Multiple screenings are performed to determine the boundary frames of the adaptive threshold, and finally the shot boundary is determined;

[0098] The video content is segmented by shot boundaries, that is, the video is segmented into multiple shots consisting of continuous frames by multiple shot boundaries, and the frame similarity between continuous frames is high;

[0099] Then extract key frames from each shot, including:

[0100] Calculate the average value of the difference between each frame and the previous frame in the shot, and the interpolation value is recorded as distance;

[0101] Traverse the frames in the shot, find the frames whose distance is greater than a multiple of the average value, and extract them as key frames;

[0102] If there is no frame that meets the conditions, the middle frame of the shot is selected as the key frame;

[0103] Then, the adaptive hot metal code infrared measurement image pseudo color coding method, digital image processing method, and road damage repair image feature analysis method were used to process the ice image, water and oil pollution image, and road damage image.

[0104] For the icing image, first divide the grayscale area of ​​the relevant image into four parts: blue represents low-temperature objects l0-l1, green represents medium-low temperature objects l1-l2, yellow represents medium-temperature objects l2-l3, and red represents high-temperature objects l3-l4; further, let the number of items with grayscale value i in the image be n i , the gray level of the image is L; let the gray level k be the threshold to divide the image into two areas, the pixels with gray levels of 1 to k belong to the foreground area, denoted as A, and the pixels with gray levels of k+1 to L belong to the background area, denoted as B; according to the maximum inter-class variance criterion, change k from l to L, and calculate the inter-class variance σ 2; Then, the maximum inter-class variance method is used to obtain the binarization threshold of the image as the second area l2. Then, the thresholds of the two parts are calculated respectively as the first threshold l1 and the third threshold l3. The grayscale ranges of the four fixed areas of hot metal coding are modified to four adaptive thresholds, and the desired special effect picture can be obtained;

[0105] For the image of accumulated water and oil pollution, in order to calculate the relevant area, the image is firstly grayed using the rgb2gray function provided by MATLAB software;

[0106] In order to obtain a clear border, the local threshold segmentation method is used to divide the image into 4 parts, and the histogram of each part is found. The inversion method is used to eliminate discontinuity to obtain a binary image, and a disk structure element with a radius of 5 pixels is selected to perform an opening operation on the binary image to extract the border; secondly, in order to ensure the measurement accuracy of the area, the image is geometrically corrected through three steps: Hough transform, spatial transform, and grayscale difference;

[0107] Finally, the area of ​​the water and oil pollution part can be calculated by referring to the border area and the ratio of the water and oil pollution part pixels to the total pixels;

[0108] For road damage images, the images are first deblurred, and then the window contrast feature extraction algorithm is used to extract features from the images. Finally, in order to ensure the accuracy of the analysis, the false target information of the images needs to be removed. Furthermore, after the road damage is identified, the area of ​​the damage is calculated through the image. The relevant steps are similar to the above-mentioned method for calculating the area of ​​accumulated water and oil pollution.

[0109] In step 102, the feedback text sent back by the individual user and the maintenance personnel from the web page port is segmented; the maintenance progress information reported by the maintenance personnel and the road abnormality information reported by the user are processed by a segmentation algorithm based on frequency and dynamic programming. The specific process is as follows:

[0110] First, the feedback text segment to be processed is roughly segmented to obtain all segmentation results including the correct segmentation results; the full segmentation method is used here, that is, starting from the first character, all possible segmentation methods are obtained, and the remaining characters are segmented in this way, and the segmentation results are stored in a segmentation word tree with each character as the root;

[0111] On this basis, a roughly segmented directed acyclic graph is constructed through the N-shortest path construction algorithm, and its shortest path is solved to improve the efficiency of word segmentation;

[0112] Then, the segmentation results are combined with a low-to-top greedy algorithm for dynamic planning to obtain the global optimal path;

[0113] Input the information of each node in the graph and the predecessor node of each node, and calculate the correct segmentation result based on their frequency;

[0114] Finally, based on the part of speech of the segmented chunks, noun and adjective chunks are extracted to obtain information about road sections, abnormality types, and maintenance progress.

[0115] As for the video information transmitted back, it is first disassembled into multiple images through the streaming video disassembly algorithm, and then preliminarily screened together with the transmitted images to preliminarily exclude images without abnormal features. After the screening is completed, the images are processed through relevant algorithms to obtain the required special effect images;

[0116] In step 103, there are four main types of conflicts that may occur:

[0117] The first type of conflict is the conflict of conclusions on whether the road is frozen or not. That is, the image features show that the road surface temperature is low at a certain location and ice may occur, but the data shows that there is no ice at that location. This conflict occurs between the video photos sent back by the drone and the vehicle and the data sent back by the vehicle road sensor. Since the video photos can only get special effects pictures after being processed by the algorithm, and the data information sent back by the road sensor includes not only the judgment value of whether the road surface is frozen, but also the specific thickness of the ice. Therefore, when there is a conflict between the two, the data sent back by the road sensor is mainly used for judgment. If there is no ice, the relevant abnormal alarm is lifted.

[0118] The second type of conflict is water accumulation or misjudgment conflict, that is, the abnormal type judged by the image features does not match the data judgment value. This conflict occurs between the video photos sent back by the drone and the vehicle and the data sent back by the vehicle road sensor. According to the working principle of the laser tube and the road sensor, the conclusion calculated by the difference in the reflected waves sent back by different materials is more accurate than the picture. Therefore, when there is a contradiction between the two, the conclusion obtained by the data information sent back by the road sensor should be the main one.

[0119] The third type of conflict is the conflict in judging the road damage type. This conflict occurs between the video images sent back by the drone and the video images sent back by the vehicle camera. That is, the two images are judged by the initial algorithm to reach different conclusions on the road anomaly type. Therefore, it is necessary to further process and compare the images of both parties through algorithms to reach the final conclusion.

[0120] The fourth type of conflict is the exception handling progress conflict, that is, the feedback information from the maintenance unit is inconsistent with the actual exception handling situation. This conflict occurs between the feedback information from the government and the maintenance unit and the abnormal situation sent back by the drone. First, the feedback time of the relevant information of both parties is extracted and the time series is sorted. If the feedback information time is earlier than the information sent back by the drone, the conflict is judged to exist, and the abnormal information sent back by the drone shall prevail over the feedback information from the government maintenance unit.

[0121] After the above four types of conflicts are resolved, the scattered cloud databases and storage units are merged, and the relevant data and images are further integrated. The relevant integrated data and images are divided into the following three categories according to the anomaly type:

[0122] When the anomaly is icing on the road surface, it is necessary to extract the icing picture, icing area, thickness and road surface temperature data from the cloud storage unit, and further mark the picture with relevant data;

[0123] When the anomaly is waterlogging on the road, it is necessary to extract waterlogging images, waterlogging area and depth data from the cloud storage unit, and further mark the images with relevant data;

[0124] When the anomaly is a road disease, it is necessary to extract the disease image, disease name and disease area data from the cloud storage unit, and further mark the image with relevant data;

[0125] In step 104, the integrated data finally obtained in the integrated cloud system will be presented to maintenance personnel and individual users through an external interface, and users can feedback relevant information to the cloud system through this interface.

[0126] The beneficial effect of the present invention is that the information integration method within the road anomaly monitoring system proposed in the present invention can effectively integrate the abnormal information returned by the four-party abnormality detection system, effectively improve the information data quality and information interaction efficiency, and to a large extent avoid the obstacles to abnormality resolution and the occurrence of secondary accidents caused by low information interaction efficiency.

[0127] The present invention integrates for the first time the data collected by multiple parties such as video images taken by multi-spectral cameras on drones, narrowband Internet of Things multi-party sensors, vehicle-mounted laser tubes and road surface sensors, cameras, web platforms, etc., and performs data cleaning, data analysis, data standardization and other processes on multi-party data classification to achieve the reorganization, fusion and interconnection between heterogeneous data to achieve the purpose of data integration. This information integration process method within the road abnormality mobile Internet of Things monitoring system based on vehicle-road collaboration provides a reasonable and effective data integration method for road anomaly management based on multiple data sources, which can more efficiently perform abnormal feedback and elimination work, and is an all-round three-dimensional road abnormality analysis method.

[0128] This information integration method within the road abnormality mobile IoT monitoring system based on vehicle-road collaboration provides timely feedback information to relevant third parties, tracks the progress of abnormality elimination in real time, and realizes closed-loop control. At the same time, combined with the triggered inspection mechanism, it improves the information quality and utilization rate, and improves the smart city traffic road anomaly detection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0129] Figure 1 It is a principle block diagram of the information integration method in one embodiment of the present invention.

[0130] Figure 2 The topological diagram is a schematic diagram of the external environment in which the information integration method is applied in one embodiment of the present invention.

[0131] Figure 3 FIG. 1 is a CAD structural diagram of a drone used in one embodiment of the present invention.

[0132] Figure 4 This is a schematic diagram of the detection principle of a vehicle-mounted sensor used in one embodiment of the present invention.

[0133] Figure 5 This is a general flow chart of the information integration method in one embodiment of the present invention. DETAILED DESCRIPTION

[0134] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0135] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0136] The description in this section is only for several typical embodiments, and the present invention is not limited to the scope of the embodiments. The same or similar prior art means and some technical features in the embodiments are mutually replaced within the scope of the present invention.

[0137] The present invention discloses an information integration method within a road anomaly monitoring system, the method comprising: collecting source data of video images through a multispectral camera carried by an unmanned aerial vehicle; collecting source data of geographic location information through a road surface sensor; collecting source data of road conditions through a vehicle-mounted laser tube and a road surface sensor, etc.; and setting up a comprehensive cloud system that collects data from multiple parties and provides information integration functions.

[0138] Figure 1 This is a principle block diagram of an information integration method within a road anomaly monitoring system in an embodiment of the present invention. Figure 1 In one embodiment of the present invention, the method of the present invention comprises:

[0139] [Step 101] Receive the raw data information collected and sent back by the multi-spectral camera on the drone, the narrow-band Internet of Things multi-party sensors, the vehicle-mounted laser tube and road surface sensors, cameras, and the web platform, and store it in a related cloud database or cloud storage unit.

[0140] [Step 102] The obtained raw data is cleaned and processed by related algorithms to preliminarily obtain processed data information.

[0141] In step 102, the system will first clean the raw data to remove outliers and dirty data. After the data cleaning is completed, the corresponding algorithm will be selected according to the data information type for further processing to initially obtain relevant data conclusions and valid information.

[0142] Further, in step 102, for the raw data transmitted back by the GPS locator, the information module in the positioning sensor provides data information of various information types, and the information category of the recommended minimum positioning information is selected through the GPRMC identifier. This type of information provides: Greenwich Mean Time in hour, minute, and second format, positioning status (A indicates valid positioning, V indicates invalid positioning), longitude and latitude information in degree and minute format, longitude and latitude hemisphere identifier (N for the northern hemisphere, S for the southern hemisphere, E for the east longitude, and W for the west longitude), ground speed, date in day, month, and year format, magnetic declination, magnetic declination direction, and mode indication.

[0143] Processing of the above information includes:

[0144] First, determine the positioning status. If it is marked as 'A', retain this data and perform subsequent processing. If it is displayed as 'V', read the next one.

[0145] Convert Greenwich Mean Time in hours, minutes and seconds format to Beijing time. According to the time zone difference, the new time is hh+8, mm, ss.

[0146] Convert the latitude and longitude data. The specific steps are as follows:

[0147] The received data format is: ddmm.mmmm; the general longitude and latitude format is: d°m′s″;

[0148] define data = ddmm.mmmm;

[0149] degree d = int(data / 100);

[0150] m1 = data - d * 100;

[0151] m=int(m1);

[0152] Seconds s=int((m1-m)*60).

[0153] When data is exported, add the south / north latitude identifier after the latitude and the east / west longitude identifier after the longitude to standardize the format.

[0154] Keep the above processed data, update the cloud database, and discard the redundant data.

[0155] Next, perform offset correction on the coordinates obtained in the previous step. The specific implementation method is as follows:

[0156] Input PR (lat, lng), which is the coordinate information extracted in the previous step;

[0157] Retrieve the matching coordinate value in the correction table and obtain PO (latoffset, lngoffset), which is the correction offset difference;

[0158] Finally, add / subtract the difference to the original longitude and latitude values ​​to obtain the corrected coordinate data.

[0159] Through the API interface of each map service application, various POI data such as name, longitude and latitude coordinates, address and other information are obtained. The specific implementation method is as follows:

[0160] Input coordinate information for automatic positioning, and return information such as road (street), house number (streetNumber), province (province), city (city), etc.

[0161] Extract the road (street) information;

[0162] Return to street.

[0163] Furthermore, in step 102, for the original reflected wave data sent back by the vehicle-mounted sensor, three wavelengths (λ1, λ2, λ3) are selected respectively according to the water absorption spectrum, ice absorption spectrum and oil absorption spectrum. Due to the combined effect of road surface reflection and absorption and scattering of light signals by water molecules and solid particles, the backscattering coefficients of water for the three wavelengths are λ1>λ3>λ2; the backscattering coefficients of ice for the three wavelengths are λ1>λ2>λ3; and the backscattering coefficients of oil for the three wavelengths are λ1>λ3>λ2. According to the size and difference of the ratio of the three backscattered light signals to the light signal when the road surface is dry, the road surface condition can be qualitatively analyzed.

[0164] For temperature calculation, by receiving infrared radiation energy from the road surface, according to Planck's law:

[0165] W(λ,T)=C1 / {λ 5 [exp(C2 / (λT))-1]- 1}

[0166] Where: T is temperature; C1 is the first radiation constant; C2 is the second radiation constant.

[0167] When the wavelength is constant, the infrared radiation energy W is measured and the temperature can be calculated. Temperature measurement range: -60~+85℃; measurement accuracy: ≤±0.1℃; resolution: 0.1℃.

[0168] For the calculation of the thickness of the covering, based on the absorbance of the substance and the Lambert-Beer law, the thickness calculation formula is derived:

[0169] d=L n (I0 / I) / (εC)

[0170] Where: d is the thickness of the light-absorbing material; I0 is the incident light intensity; I is the intensity of the transmitted light after passing through the material; L n (I0 / I) is the absorbance; C is the concentration of the substance; ε is the proportional coefficient of light absorption, which is related to the properties of the absorbing substance and the wavelength of the incident light.

[0171] Measuring range: 0.O0~2mm; measuring accuracy: ±0.01mm; resolution: 0.01mm.

[0172] Furthermore, in step 102, for the original video images transmitted back by the multi-spectral camera of the drone and the vehicle-mounted camera, the video content is first decomposed into images.

[0173] The key frames of the video content are extracted by combining the histogram method and the frame averaging method based on the shot boundary. The specific steps are as follows:

[0174] First, the video is structured, including:

[0175] Store each frame of the video;

[0176] Calculate the difference between frames in the shot using the histogram difference method and store the data in a structure;

[0177]

[0178] Where h(i) represents the cumulative distribution of each grayscale of a frame in the video; m and n represent the previous frame and the next frame respectively; the max() function represents the larger value of the two; k represents all grayscale values ​​of this frame;

[0179] Multiple screenings are performed to determine the boundary frames of the adaptive threshold, and finally the shot boundary is determined;

[0180] The video content is segmented by shot boundaries, that is, the video is segmented into multiple shots consisting of continuous frames by multiple shot boundaries, and the frame similarity between continuous frames is high;

[0181] Then extract key frames from each shot, including:

[0182] Calculate the average value of the difference between each frame and the previous frame in the shot (recorded as distance);

[0183] Traverse the frames in the shot, find the frames whose distance is greater than the multiple of the average value, and extract them as key frames;

[0184] If there is no frame that meets the conditions, the middle frame of the shot is selected as the key frame.

[0185] Then, the adaptive hot metal code infrared measurement image pseudo-color encoding method, digital image processing method and road damage repair image feature analysis method were used to process the ice image, water and oil pollution image and road damage image.

[0186] For the icing image, the grayscale area of ​​the relevant image is first divided into four parts, namely: low-temperature objects l0-l1 (blue), medium-low-temperature objects l1-l2 (green), medium-temperature objects l2-l3 (yellow), and high-temperature objects l3-l4 (red). Furthermore, let the number of items with grayscale value i in the image be n i , the gray level of the image is L. Assume that the gray level k is used as the threshold to divide the image into two regions. The pixels with gray levels of 1 to k belong to the foreground region, denoted as A, and the pixels with gray levels of k+1 to L belong to the background region, denoted as B. According to the maximum inter-class variance criterion, change k from l to L and calculate the inter-class variance σ 2 Then, the maximum inter-class variance method is used to obtain the binarization threshold of the image as the second area l2. Then, the thresholds of the two parts are calculated separately as the first threshold l1 and the third threshold l3. The grayscale range of the four fixed areas of hot metal coding is modified to four adaptive thresholds, and the desired special effect picture can be obtained.

[0187] For the image of accumulated water and oil pollution, in order to calculate the relevant area, the image is first converted to grayscale using the rgb2gray function provided by MATLAB software.

[0188] Furthermore, in order to obtain a clear border, the local threshold segmentation method is used to divide the image into 4 parts, and the histogram of each part is found. The inversion method is used to eliminate discontinuity to obtain a binary image, and a disk structure element with a radius of 5 pixels is selected to perform an opening operation on the binary image to extract the border. Secondly, in order to ensure the measurement accuracy of the area, the image is geometrically corrected through three steps: Hough transform, spatial transform, and grayscale difference.

[0189] Finally, the area of ​​the water and oil pollution part can be calculated by referring to the border area and the ratio of the pixels of the water and oil pollution part to the total pixels.

[0190] For road damage images, the images are first deblurred, and then feature extraction is performed using the window contrast feature extraction algorithm. Finally, in order to ensure the accuracy of the analysis, the false target information of the image needs to be removed. Furthermore, after identifying the road damage, the area of ​​the damage is calculated through the image. The relevant steps are similar to the above-mentioned method for calculating the area of ​​water and oil pollution.

[0191] Furthermore, in step 102, the feedback text sent back by individual users and maintenance personnel from the web portal is segmented. The maintenance progress information reported by maintenance personnel and the road anomaly information reported by users are processed by a segmentation algorithm based on frequency and dynamic programming. The specific process is as follows:

[0192] First, the feedback text segment to be processed is roughly segmented to obtain all segmentation results including the correct segmentation results;

[0193] Here, the full segmentation method is used, that is, starting from the first character, all possible segmentation methods are obtained, and the remaining characters are repeated in this way, and the segmentation results are stored in a segmentation word tree with each character as the root;

[0194] On this basis, a roughly segmented directed acyclic graph is constructed through the N-shortest path construction algorithm, and its shortest path is solved to improve the efficiency of word segmentation;

[0195] Then, the segmentation results are combined with a low-to-top greedy algorithm for dynamic planning to obtain the global optimal path;

[0196] Input the information of each node in the graph and the predecessor node of each node, and calculate the correct segmentation result based on their frequency;

[0197] Finally, based on the part of speech of the segmented chunks, noun and adjective chunks are extracted to obtain information such as road sections, anomaly types, and maintenance progress.

[0198] As for the video information transmitted back, it is first disassembled into multiple images through the streaming video disassembly algorithm, and then preliminarily screened together with the returned images to preliminarily exclude images that do not record abnormal features. After the screening is completed, the images are processed through relevant algorithms to obtain the required special effects images.

[0199] [Step 103] Further conflict resolution is performed on the preliminary processed data in the cloud information system. After the data information processing is completed, the corresponding cloud database and storage unit are merged, and the data and images are further integrated and stored in the comprehensive cloud system.

[0200] In step 103, there are four main types of conflicts that may occur:

[0201] The first type of conflict is the conflict of judgment on whether the road is frozen or not. That is, the image features show that the road surface temperature is low at a certain place, and ice may occur, but the data shows that there is no ice at that place. This conflict occurs between the video photos sent back by the drone and the vehicle and the data sent back by the vehicle road sensor. Since the video photos can only get special effects pictures after being processed by the algorithm, the data information sent back by the road sensor includes not only the judgment value of whether the road surface is frozen, but also the specific thickness of the ice. Therefore, when there is a conflict between the two, the data sent back by the road sensor is mainly used for judgment. If there is no ice, the relevant abnormal alarm is lifted.

[0202] The second type of conflict is water accumulation or misjudgment conflict, that is, the type of abnormality judged by the image features does not match the data judgment value. This conflict occurs between the video photos sent back by the drone and the vehicle and the data sent back by the vehicle road sensor. According to the working principle of the laser tube and the road sensor, the conclusion calculated by the difference in the reflected waves sent back by different materials is more accurate than the picture. Therefore, when there is a contradiction between the two, the conclusion obtained by the data information sent back by the road sensor should be the main one.

[0203] The third type of conflict is the conflict in judging the type of road damage. This conflict occurs between the video images sent back by the drone and the video images sent back by the vehicle-mounted camera. That is, the two images have different conclusions on the type of road anomaly judged through preliminary algorithm judgment. Therefore, it is necessary to further perform algorithm processing and comparison on the images of both sides to reach a final conclusion.

[0204] The fourth type of conflict is the exception handling progress conflict, that is, the feedback information from the maintenance unit is inconsistent with the actual exception handling situation. This conflict occurs between the feedback information from the government and the maintenance unit and the abnormal situation sent back by the drone. First, the feedback time of the relevant information of both parties is extracted and the time series is sorted. If the feedback information time is earlier than the information sent back by the drone, the conflict is judged to exist, and the abnormal information sent back by the drone shall prevail over the feedback information from the government maintenance unit.

[0205] After the above four types of conflicts are resolved, the scattered cloud databases and storage units are merged, and the relevant data and images are further integrated. The relevant integrated data and images are divided into the following three categories according to the anomaly type:

[0206] When the anomaly is icing on the road surface, it is necessary to extract the icing picture, icing area, thickness and road surface temperature data from the cloud storage unit, and further mark the picture with relevant data;

[0207] When the anomaly is waterlogging on the road, it is necessary to extract waterlogging images, waterlogging area and depth data from the cloud storage unit, and further mark the images with relevant data;

[0208] When the anomaly is a road disease, it is necessary to extract the disease image, disease name and disease area data from the cloud storage unit, and further mark the image with relevant data;

[0209] [Step 104] The data information in the cloud system is finally presented and fed back to the relevant third party through a visual port such as a web page, APP, etc.

[0210] In step 104, the integrated data finally obtained in the integrated cloud system will be presented to maintenance personnel and individual users through external interfaces such as web pages and apps, and users can also feedback relevant information to the cloud system through this interface.

[0211] The present invention can effectively integrate the abnormal information returned by the four-party abnormality detection system, effectively improve the information data quality and information interaction efficiency, and to a large extent avoid the abnormality resolution obstacles and secondary accidents caused by the low efficiency of information interaction.

[0212] Figure 2 It is an external environment topology diagram applied by the internal information integration process method principle of the road abnormal movement IoT monitoring system based on vehicle-road collaboration provided by an embodiment of the present invention, such as Figure 2 As shown in the figure, the external environment topology can be divided into three levels:

[0213] The first level: receiving the original data collected by three parties, including the video images sent back by the multi-spectral camera on the drone, the GPS data sent back by the GPS locator in the drone and the narrowband Internet of Things, the reflected wave data collected by the on-board laser tube in the car, and the data images sent back by the on-board camera.

[0214] Within the first level, further Figure 3 As shown, the multispectral camera carried by the UAV has the characteristics of light weight, small size and remote triggering, and is suitable for use with small and medium-sized UAVs. Each band is equipped with a 1.3-megapixel CMOS sensor (6.18mm×4.95mm), with an image resolution of 1 280 pixels×1024 pixels, an aperture of F 3.2, a focal length of 9.6mm, a mass of 1.3kg, and an adjustable shutter speed (1 to 3s), and can acquire image data of 12 bands (optional).

[0215] In the first level, further, the original GPS data transmitted back to the system by the GPS locator installed in the drone and the narrowband Internet of Things system has the following data format:

[0216] (1) UTC time, in the format of hhmmss.sss;

[0217] (2) Positioning status, A = data available, V = data unavailable;

[0218] (3) Latitude, in the format of degrees, minutes, minutes, and minutes (ddmm.mmmm);

[0219] (4) Latitude, Northern Hemisphere (N) or Southern Hemisphere (S);

[0220] (5) Longitude, in the format of degrees, minutes, minutes, and minutes (ddmm.mmmm);

[0221] (6) Latitude, Eastern Hemisphere (E) or Western Hemisphere (W);

[0222] (7) Relative displacement velocity, 0.0 to 1851.8 knots;

[0223] (8) Relative displacement direction, 000.0 to 359.9 degrees (actual value);

[0224] (9) Date, in the format of ddmmyy;

[0225] (10) magnetic pole variable, 000.00 to 180.0;

[0226] (11) Degree;

[0227] (12) Checksum.

[0228] Within the first level, further Figure 4 As shown in the figure, the vehicle-mounted sensor used is centered on the transmittance parameters of water, ice and oil, and uses infrared remote sensing, multi-light source and multi-spectral measurement technology to perform close-range detection of road surfaces. The detection device emits three infrared lights of different wavelengths to the detection point of the road surface every 10 minutes. Due to the combined effects of road surface reflection and the absorption and scattering of light signals by water molecules and solid particles, the detector will measure and send back three backscattered light signals with different absorption coefficients to the system. The camera shoots the corresponding road condition video and photos from the driver's angle at close range and sends them back to the system.

[0229] The second level: The obtained raw data is stored in the cloud information system and the data is clarified, processed by algorithms and integrated.

[0230] In the second level, the collected original information data is first divided according to type and stored in a separate cloud database or cloud storage unit. After the subsequent related algorithms and conflict processing are completed, the cloud database and storage unit are merged and finally stored in the integrated cloud system.

[0231] The third level: present the data information in the integrated cloud system in the second level to relevant third parties (such as individual users, maintenance personnel, etc.) through visual ports such as web pages and apps.

[0232] Furthermore, in the subsequent maintenance process, the third party will feedback relevant maintenance progress and other information to the system based on the actual situation. The system will extract valid information from it and further process, integrate and update the information, and feedback the relevant information to individual users to eliminate abnormal alarms in a timely manner.

[0233] Figure 5 : is a flow chart of the information integration process method within the road abnormal movement IoT monitoring system based on vehicle-road collaboration provided by an embodiment of the present invention, such as Figure 5 As shown, the steps include:

[0234] Step S301: Start.

[0235] Step S302: Determine the type of data collector. If it is a narrowband Internet of Things, proceed to step S303; if it is a vehicle-mounted system, proceed to step S308; if it is a drone, proceed to step S313; if it is feedback from an individual user or maintenance personnel, proceed to step S331.

[0236] Step S303: Accept GPS type data transmitted by the narrowband Internet of Things.

[0237] Step S304: Store the data in the cloud database.

[0238] Step S305: Perform offset processing on the data.

[0239] Step S306: extract the corresponding road name from the data obtained in step 305.

[0240] Step S307: Obtain the corresponding road name and specific coordinate information.

[0241] Step S308: Store the data in the cloud database.

[0242] Step S309: Determine the absorption coefficient.

[0243] Step S310: Calculate the thickness of the covering.

[0244] Step S311: Obtain the covering type and the corresponding thickness value.

[0245] Step S312: storing the corresponding data information into the cloud storage unit.

[0246] Step S313: Determine the information type. If it is a video image, proceed to step S314; if it is GPS data, proceed to step S326.

[0247] Step S314: storing the corresponding video images into a cloud storage unit.

[0248] Step S315: Determine whether it is a picture, if not, proceed to step S316, if so, proceed to step S317.

[0249] Step S316: extract key frames from the video.

[0250] Step S317: Filter pictures.

[0251] Step S318: Determine the abnormality type. If it is icing, proceed to step S319; if it is water accumulation or oil pollution, proceed to step S321; if it is road damage, proceed to step S323.

[0252] Step S319: Perform hot metal encoding processing on the image.

[0253] Step S320: Obtain a special effects picture.

[0254] Step S321: performing digital image processing on the picture.

[0255] Step S322: Obtain the area of ​​accumulated water and oil pollution.

[0256] Step S323: Perform image feature analysis on the picture.

[0257] Step S324: performing digital image processing on the picture.

[0258] Step S325: Obtain the disease type and corresponding area.

[0259] Step S326: Store the data in the cloud database.

[0260] Step S327: Perform offset processing on the data.

[0261] Step S328: extract the corresponding road name from the data obtained in step 327.

[0262] Step S329: Obtain the corresponding road name and specific coordinate information.

[0263] Step S330: Store the data and pictures obtained in steps S320, S322, S325 and S329 into a cloud storage unit.

[0264] Step S331: Get feedback text.

[0265] Step S332: Store the relevant information into the cloud database.

[0266] Step S333: Store the information data into the cloud database.

[0267] Step S334: word segmentation extraction.

[0268] Step S335: extract the road name and corresponding coordinate data in the cloud storage unit of step S312 and step S330.

[0269] Step S336: Determine whether the two abnormal information occur at the same location. If so, proceed to step S337.

[0270] Step S337: Store the corresponding information data into the cloud database.

[0271] Step S338: Determine the conflict type. If there is a conflict between the data and the picture on whether there is ice, go to step S339; if there is a conflict between the data and the picture on whether there is water accumulation or oil pollution, go to step S341; if there is a conflict between the picture and the picture on whether there is road damage, go to step S342.

[0272] Step S339: Determine whether ice is formed by the data value, if not, proceed to step S340.

[0273] Step S340: resolve the exception.

[0274] Step S341: Determine the accurate abnormality type through data value judgment.

[0275] Step S342: Algorithm processing comparison.

[0276] Step S343: Confirm the exact disease type.

[0277] Step S344: extract the road name and corresponding coordinate data in the step cloud storage unit.

[0278] Step S345: Determine whether the two abnormal information occur at the same location. If so, proceed to step S345.

[0279] Step S346: Store the corresponding information data into the cloud database.

[0280] Step S347: extract the information return time.

[0281] Step S348: Determine whether the feedback information time is earlier than the drone information return time, if so, proceed to step S346.

[0282] Step S349: Use the information sent back by the drone to cover the feedback information.

[0283] Step S350: Merge the cloud database and the storage unit.

[0284] Step S351: Determine the abnormality type. If it is icing, proceed to step S352; if it is water accumulation and oil pollution, proceed to step S354; if it is road damage, proceed to step S356.

[0285] Step S352: extracting ice pictures, ice area, thickness and road surface temperature data.

[0286] Step S353: Mark the picture.

[0287] Step S354: extracting waterlogging images, waterlogging area and depth data.

[0288] Step S355: Mark the picture.

[0289] Step S356: Extract road damage images, damage names and area data.

[0290] Step S357: Mark the picture.

[0291] Step S358: Integrate the relevant information and store it into the integrated cloud system.

[0292] Step S359: Web page port output.

[0293] Step S360: End.

[0294] In one embodiment of the present invention, the information integration method within the road anomaly monitoring system of the present invention includes: collecting source data of video images through a multi-spectral camera on an unmanned aerial vehicle; collecting source data of geographic location information through a road surface sensor; collecting source data of road conditions through a vehicle-mounted laser tube and a road surface sensor; and setting up a comprehensive cloud system that collects data from multiple parties and provides information integration functions.

[0295] In one embodiment of the present invention, the method of the present invention further includes: integrating data collected by the multispectral camera on the drone, including: disassembling and analyzing the video image, extracting the coordinates of the drone's geographic location information, and updating the cloud database and cloud storage unit; extracting the key frames of the video image based on the lens boundary combined with the histogram method and the frame averaging method; the specific algorithm is as follows: first, the video is structured: each frame of the video is stored; the difference between the frames in the lens is calculated by the histogram difference method, and the difference formula is as follows:

[0296]

[0297] Where h(i) represents the cumulative distribution of each grayscale of a frame in the video; m and n represent the previous frame and the next frame respectively; the max() function represents the larger value of the two; k represents all grayscale values ​​of this frame;

[0298] And through multiple screenings, the boundary frames of the adaptive threshold are judged to finally determine the boundary of the shot; the video content is segmented according to the boundary of the shot; finally, the key frames are extracted from each shot; the extracted key frame images are subsequently analyzed, including: hot metal coding processing, digital image processing and road disease image feature repair analysis of various types of images to obtain information such as the type of disease. The specific algorithm is as follows: for icing images, the grayscale area of ​​the relevant image is first divided into four parts: blue, green, yellow and red. The maximum inter-class variance method is used to obtain the binary threshold of the image. Finally, the grayscale of the four fixed areas of the hot metal coding is The range can be modified to four adaptive thresholds; for water and oil pollution images, the picture is first grayed out, and then the local threshold segmentation method is used to obtain a clear border, and finally the area of ​​water and oil pollution is calculated according to the pixel ratio; for road disease images, the image is first deblurred, and then the window contrast feature extraction algorithm is used to extract the features of the image to obtain the result; the required longitude and latitude, and longitude and hemisphere data are extracted from the geographic location information transmitted back by the drone, and the longitude and latitude are converted into degree, minute, and second format, and the hemisphere mark is added; then the cloud database and cloud storage unit are updated, and the processed pictures and numerical data are stored.

[0299] In one embodiment of the present invention, the method of the present invention further includes: integrating data collected by GPS sensors installed in manhole covers on roads, including: extracting the coordinates of the manhole cover's geographic location information, obtaining the name of the road at the diseased location through the coordinate information, and updating the cloud database; extracting the coordinates of the manhole cover's geographic location information is similar to the remote sensing system, extracting the required longitude and latitude, and longitude and latitude hemisphere data from the datagram already stored in the transportation bureau's library, and converting the longitude and latitude into degree, minute, and second format, adding a hemisphere identifier, and the specific algorithm is as follows: defining the initial received data: ddmm.mmmm as data, the degrees in the general longitude and latitude format are obtained by dividing data by 100 and rounding; the fraction in the general longitude and latitude format is obtained by rounding the difference (denoted as m1) between data and the product of the degrees and 100; the seconds in the general longitude and latitude format are obtained by rounding the difference between m1 and the fraction; thus, the general longitude and latitude format is obtained: d°m′s″; then the updated coordinate information is obtained through the coordinate offset correction algorithm, and the specific road name is obtained through the API interface; finally, the cloud database is updated and the processed images and numerical data are stored.

[0300] In one embodiment of the present invention, the method of the present invention further includes: integrating data collected by the vehicle-mounted infrared laser tube module, the road surface temperature sensor, and the air temperature / humidity sensor, including: calculating the road surface temperature; calculating the air humidity; judging whether ice and water are generated on the road; the infrared radiation energy received by the road surface temperature sensor, the scattering coefficient and the first and second radiation constants, combined with Planck's law, the first radiation constant is divided by the product of the fifth power of the scattering coefficient and the inverse of e to the power of f(x) minus 1, where f(x) is the second radiation constant divided by the product of the scattering coefficient and the temperature; air humidity is related to temperature, and the air humidity is calculated by relative humidity, absolute humidity and saturated humidity; then, the road condition information such as the thickness of ice and water on the road surface is calculated by important parameters such as road surface temperature and air humidity; and the updated data is then stored in a cloud database.

[0301] In one embodiment of the present invention, the method of the present invention further includes: the information reported by the maintenance personnel and users includes two categories: one is the progress report of the maintenance personnel on the abnormal road maintenance work tasks, and the other is the abnormal road conditions found and reported by the user during driving; the two types of information are segmented by a word segmentation algorithm based on a combination of frequency and dynamic programming, and the road section, abnormal type, and maintenance progress are extracted; the multi-party data integration process includes data conflict processing, and the conflict is processed by an open addressing method, including four types of conflicts; the first type is the conflict between the key frame analysis extracted from the video image based on ice judgment and the judgment result of the actual numerical data; the second type is the conflict between the key frame analysis extracted from the video image based on water accumulation judgment and the judgment result of the data obtained by the vehicle-mounted laser tube; the third type is the conflict between the key frame analysis extracted from the video image taken by the drone based on the road disease type judgment and the judgment result of the key frame analysis of the video picture obtained by the vehicle-mounted camera; the fourth type is the conflict of feedback time based on the judgment of the progress of abnormal processing.

[0302] In one embodiment of the present invention, the method of the present invention further includes: the multi-party data integration process also includes a data and image integration process; the corresponding relationships mainly include the following: ice pictures correspond to ice thickness and temperature; water accumulation pictures correspond to water depth and area; road disease pictures correspond to road names; all of the above correspondences include the correspondence between location coordinate information and road names, and the specific algorithm is as follows: For the first correspondence: extract road ice pictures extracted from videos shot by drones from the cloud database, and at the same time extract ice thickness and temperature calculated based on the pictures and store them in an external database for subsequent web page display; For the second correspondence, similar to the first: extract road water accumulation pictures extracted from videos shot by drones from the cloud database, and at the same time extract water accumulation depth and area calculated based on the pictures and store them in an external database for subsequent web page display; For the third correspondence: extract road disease pictures extracted from videos shot by drones from the cloud database. Since the drone video provides real-time trajectory, the geographic location information of the picture is obtained through the position of the extracted key frames, and stored in the external database for subsequent web page display.

[0303] To sum up, the information integration method within the road anomaly monitoring system proposed in the present invention can effectively integrate the abnormal information returned by the four-party abnormality detection system, effectively improve the information data quality and information interaction efficiency, and to a large extent avoid the obstacles to abnormality resolution and the occurrence of secondary accidents caused by low information interaction efficiency.

[0304] The present invention integrates for the first time the data collected by multiple parties such as video images taken by multi-spectral cameras on drones, narrowband Internet of Things multi-party sensors, vehicle-mounted laser tubes and road surface sensors, cameras, web platforms, etc., and performs data cleaning, data analysis, data standardization and other processes on multi-party data classification to achieve the reorganization, fusion and interconnection between heterogeneous data to achieve the purpose of data integration. This information integration process method within the road abnormality mobile Internet of Things monitoring system based on vehicle-road collaboration provides a reasonable and effective data integration method for road anomaly management based on multiple data sources, which can more efficiently perform abnormal feedback and elimination work, and is an all-round three-dimensional road abnormality analysis method.

[0305] This information integration method within the road abnormality mobile IoT monitoring system based on vehicle-road collaboration provides timely feedback information to relevant third parties, tracks the progress of abnormality elimination in real time, and realizes closed-loop control. At the same time, combined with the triggered inspection mechanism, it improves the information quality and utilization rate, and improves the smart city traffic road anomaly detection system.

[0306] The description and application of the present invention here are illustrative and are not intended to limit the scope of the present invention to the above-described embodiments. The deformation and changes of the embodiments disclosed here are possible, and the replacement of the embodiments and the various components equivalent to those of ordinary skill in the art are well known. It should be clear to those skilled in the art that the present invention can be realized in other forms, structures, arrangements, proportions, and with other components, materials and parts without departing from the spirit or essential characteristics of the present invention. Other deformations and changes can be made to the embodiments disclosed here without departing from the scope and spirit of the present invention.

Claims

1. A method for integrating information within a road anomaly monitoring system, characterized in that: The method comprises: Source data collection of video images through multispectral cameras carried by drones; Source data collection of geographic location information through road surface sensors; Collect road condition source data through vehicle-mounted laser tubes and road surface sensors; Set up a comprehensive cloud system that collects data from multiple parties and provides information integration functions; The multi-spectral camera carried by the unmanned aerial vehicle is used to realize the shooting and transmission of ordinary visible light and infrared light video content; the multi-spectral camera carried by the unmanned aerial vehicle includes a remote sensing device, and the remote sensing device includes a main control module and an image display and transmission module; The main control module is used to complete the control of ordinary visible light and infrared light cameras, including video focus adjustment, zoom setting and video gain control; The image display and transmission module is used to realize the encoding and decoding of two video signals, screen display and data transmission functions; the specific process is: firstly, the CCD camera on the drone end is adjusted through the main control module, the corresponding target image is obtained through the two channels of visible light and infrared light, and the image is converted into digital form and transmitted back to the cloud storage unit; The road surface sensor refers to a GPS sensor installed in a manhole cover on a road; the GPS sensor includes a positioning module and an information transmission module; The positioning module is used to receive the location information of the manhole cover, including time, positioning status, and longitude and latitude data information; the information transmission module transmits the sensor collected data in a fixed format back to the cloud database; The vehicle-mounted laser tube and road pavement sensor include a vehicle-mounted infrared laser tube module, a road surface temperature sensor, and an air temperature / humidity sensor; wherein the infrared laser tube module has the function of collecting a reflected echo signal obtained by emitting a specific wavelength light beam to the road surface; each sensor of the vehicle-mounted system sends the collected echo signal, road surface temperature, and air temperature / humidity data back to the cloud database; The integrated cloud system includes a cloud storage unit and a cloud database; the cloud storage unit is used to store video images transmitted back by the multi-spectral camera of the drone remote sensing; the cloud database is used to store geographic location information data sent by the GPS sensor on the road, infrared echo, temperature, humidity information data collected by multiple sensors in the vehicle system, and information reported by maintenance personnel and users; The data collected by the multispectral camera on the drone is integrated, including: disassembly and analysis of video images, extraction of coordinates of the drone's geographic location information, and updating of the cloud database and cloud storage unit; the key frames of the video images are extracted by combining the histogram method and the frame averaging method based on the lens boundary; the specific algorithm is as follows: first, the video is structured: each frame of the video is stored; the difference between frames in the lens is calculated by the histogram difference method, and the difference formula is as follows: Where h(i) represents the cumulative distribution of each grayscale of a frame in the video; m and n represent the previous frame and the next frame respectively; the max() function represents the larger value of the two; k represents all grayscale values ​​of this frame; And through multiple screenings, the boundary frames of the adaptive threshold are judged to finally determine the boundary of the shot; the video content is segmented according to the boundary of the shot; finally, the key frames are extracted from each shot; the extracted key frame images are subsequently analyzed, including: hot metal coding processing, digital image processing and road disease image feature repair analysis of various types of images to obtain disease type information. The specific algorithm is as follows: for icing images, the grayscale area of ​​the relevant image is first divided into four parts: blue, green, yellow and red. The maximum inter-class variance method is used to obtain the binary threshold of the image. Finally, the grayscale of the four fixed areas of the hot metal coding is The range can be modified to four adaptive thresholds; for water and oil pollution images, the image is first grayed out, and then the local threshold segmentation method is used to obtain a clear border, and finally the area of ​​water and oil pollution is calculated based on the pixel ratio; for road disease images, the image is first deblurred, and then the window contrast feature extraction algorithm is used to extract the image features to obtain the results; the required longitude and latitude, and longitude and hemisphere data are extracted from the geographic location information transmitted back by the drone, and the longitude and latitude are converted into degree, minute, and second format, and the hemisphere mark is added; then the cloud database and cloud storage unit are updated, and the processed images and numerical data are stored; The data collected by the GPS sensors installed in the manhole covers on the roads are integrated, including: extracting the coordinates of the manhole cover’s geographic location information, obtaining the road name of the diseased location through the coordinate information, and updating the cloud database; similar to the remote sensing system, the coordinate extraction of the manhole cover’s geographic location information extracts the required longitude and latitude, and hemisphere data from the datagrams already stored in the Transportation Bureau’s library, converts the longitude and latitude into degree, minute, and second format, and adds a hemisphere identifier. The specific algorithm is as follows: define the initial received data: ddmm.mmmm as data, and the degrees in the longitude and latitude format are obtained by dividing data by 100 and rounding; the fraction in the longitude and latitude format is obtained by rounding the difference between data minus the product of the degrees and 100 as m1; the seconds in the longitude and latitude format are obtained by rounding the difference between m1 and the fraction; thus, the longitude and latitude format is obtained: d°m′s″; then the updated coordinate information is obtained through the coordinate offset correction algorithm, and the specific road name is obtained through the API interface; finally, the cloud database is updated and the processed images and numerical data are stored; The data collected by the vehicle-mounted infrared laser tube module, road surface temperature sensor, and air temperature / humidity sensor are integrated, including: road surface temperature calculation; air humidity calculation; judgment of whether the road has ice and water accumulation; the infrared radiation energy received by the road surface temperature sensor, the scattering coefficient and the first and second radiation constants, combined with Planck's law, the first radiation constant is divided by the product of the fifth power of the scattering coefficient and the inverse of e to the power of f(x) minus 1, where f(x) is the second radiation constant divided by the product of the scattering coefficient and temperature; air humidity is related to temperature, and air humidity is calculated through relative humidity, absolute humidity and saturated humidity; then, the thickness of ice and water on the road surface is calculated through road surface temperature and air humidity; and the updated data is then stored in the cloud database; The information reported by the maintenance personnel and users includes two categories: one is the progress report of the maintenance personnel on the abnormal road maintenance tasks, and the other is the abnormal road conditions found and reported by users during driving; the two types of information are segmented by a word segmentation algorithm based on frequency and dynamic programming, and the road section, abnormality type, and maintenance progress are extracted; the multi-party data integration process includes data conflict processing, and the conflict is processed by the open addressing method, including four types of conflicts; the first type is the conflict between the key frame analysis of video image extraction based on ice judgment and the judgment result of actual numerical data; the second type is the conflict between the key frame analysis of video image extraction based on water accumulation judgment and the judgment result of data obtained by the vehicle-mounted laser tube; the third type is the conflict between the key frame analysis of video images taken by drones based on road disease type judgment and the key frame analysis of video pictures obtained by vehicle-mounted cameras; the fourth type is the conflict of feedback time based on the judgment of abnormal processing progress; The multi-party data integration process also includes the data and image integration process; the corresponding relationships include the following: ice pictures correspond to ice thickness and temperature; water accumulation pictures correspond to water depth and area; road disease pictures correspond to road names; all of the above correspondences include the correspondence between location coordinate information and road names, and the specific algorithms are as follows: For the first correspondence: extract road ice pictures extracted from videos shot by drones from the cloud database, and at the same time extract ice thickness and temperature calculated based on the pictures and store them in an external database for subsequent web page display; For the second correspondence, similar to the first one: extract road water accumulation pictures extracted from videos shot by drones from the cloud database, and at the same time extract water accumulation depth and area calculated based on the pictures and store them in an external database for subsequent web page display; For the third correspondence: extract road disease pictures extracted from videos shot by drones from the cloud database. Since the drone video provides real-time trajectory, the geographic location information of the picture is obtained through the position of the extracted key frames, and stored in the external database for subsequent web page display.

2. A method for integrating information within a road anomaly monitoring system, characterized in that: The method comprises: Source data collection of video images through multispectral cameras carried by drones; Source data collection of geographic location information through road surface sensors; Collect road condition source data through vehicle-mounted laser tubes and road surface sensors; Set up a comprehensive cloud system that collects data from multiple parties and provides information integration functions; The multi-spectral camera carried by the unmanned aerial vehicle is used to realize the shooting and transmission of ordinary visible light and infrared light video content; the multi-spectral camera carried by the unmanned aerial vehicle includes a remote sensing device, and the remote sensing device includes a main control module and an image display and transmission module; The main control module is used to complete the control of ordinary visible light and infrared light cameras, including video focus adjustment, zoom setting and video gain control; The image display and transmission module is used to realize the encoding and decoding of two video signals, screen display and data transmission functions; the specific process is: firstly, the CCD camera on the drone end is adjusted through the main control module, the corresponding target image is obtained through the two channels of visible light and infrared light, and the image is converted into digital form and transmitted back to the cloud storage unit; The road surface sensor refers to a GPS sensor installed in a manhole cover on a road; the GPS sensor includes a positioning module and an information transmission module; The positioning module is used to receive the location information of the manhole cover, including time, positioning status, and longitude and latitude data information; the information transmission module transmits the sensor collected data in a fixed format back to the cloud database; The vehicle-mounted laser tube and road pavement sensor include a vehicle-mounted infrared laser tube module, a road surface temperature sensor, and an air temperature / humidity sensor; wherein the infrared laser tube module has the function of collecting a reflected echo signal obtained by emitting a specific wavelength light beam to the road surface; each sensor of the vehicle-mounted system sends the collected echo signal, road surface temperature, and air temperature / humidity data back to the cloud database; The integrated cloud system includes a cloud storage unit and a cloud database; the cloud storage unit is used to store video images transmitted back by the multi-spectral camera of the drone remote sensing; the cloud database is used to store geographic location information data sent by the GPS sensor on the road, infrared echo, temperature, humidity information data collected by multiple sensors in the vehicle system, and information reported by maintenance personnel and users; Integrate the data collected by the multispectral camera on the drone, including: disassembly and analysis of video images, coordinate extraction of the drone's geographic location information, and update the cloud database and cloud storage unit; The key frames of the video images are extracted by combining the histogram method and the frame averaging method based on the shot boundary; the specific algorithm is as follows: First, the video is structured: each frame of the video is stored; the difference between frames in the lens is calculated by the histogram difference method. The difference formula is as follows: Where h(i) represents the cumulative distribution of each grayscale of a frame in the video; m and n represent the previous frame and the next frame respectively; the max() function represents the larger value of the two; k represents all grayscale values ​​of this frame; The boundary frames of the adaptive threshold are judged through multiple screenings to finally determine the shot boundaries; the video content is segmented according to the shot boundaries; finally, the key frames are extracted from each shot; the extracted key frame images are subsequently analyzed, including: hot metal coding processing, digital image processing, and road disease image feature repair analysis for each type of image to obtain disease type information.

3. The information integration method within the road anomaly monitoring system according to claim 2, characterized in that: The specific algorithm is as follows: for ice images, first divide the grayscale area of ​​the relevant image into four parts: blue, green, yellow, and red. Use the maximum inter-class variance method to obtain the image binarization threshold, and finally modify the grayscale range of the four fixed areas of hot metal coding to four adaptive thresholds; for water and oil pollution images, first gray the image, then use the local threshold segmentation method to get a clear border, and finally calculate the area of ​​water and oil pollution based on the pixel ratio; for road disease images, first deblur the image, and then use the window contrast feature extraction algorithm to extract features from the image to obtain the result; extract the required longitude and latitude and longitude hemisphere data from the geographic location information transmitted back by the drone, and convert the longitude and latitude into degree, minute, and second format, and add a hemisphere mark; The cloud database and cloud storage unit are then updated to store the processed images and numerical data.

4. The information integration method within the road anomaly monitoring system according to claim 2, characterized in that: The data collected by the GPS sensors installed in the manhole covers on the roads are integrated, including: extracting the coordinates of the manhole cover’s geographic location information, obtaining the road name of the damaged location through the coordinate information, and updating the cloud database; similar to the remote sensing system, the coordinate extraction of the manhole cover’s geographic location information extracts the required longitude and latitude, and hemisphere data from the datagrams already stored in the transportation bureau’s library, converts the longitude and latitude into degrees, minutes, and seconds, and adds a hemisphere identifier. The specific algorithm is as follows: Define the initial received data: ddmm.mmmm as data. The degree in the longitude and latitude format is obtained by dividing data by 100 and rounding. The fraction in the longitude and latitude format is obtained by rounding the difference m1 of data minus the product of the degree and 100. The seconds in the longitude and latitude format are obtained by rounding the difference between m1 and the fraction. So far, the longitude and latitude format is obtained: d° m′s″; Then, the updated coordinate information is obtained through the coordinate offset correction algorithm, and the specific road name is obtained through the API interface; finally, the cloud database is updated to store the processed images and numerical data.

5. The information integration method within the road anomaly monitoring system according to claim 2, characterized in that: The data collected by the vehicle-mounted infrared laser tube module, road surface temperature sensor, and air temperature / humidity sensor are integrated, including: road surface temperature calculation; air humidity calculation; judgment of whether the road has ice and water accumulation; the infrared radiation energy received by the road surface temperature sensor, the scattering coefficient and the first and second radiation constants, combined with Planck's law, the first radiation constant is divided by the product of the fifth power of the scattering coefficient and the inverse of e to the power of f(x) minus 1, where f(x) is the second radiation constant divided by the product of the scattering coefficient and temperature; air humidity is related to temperature, and the air humidity is calculated through relative humidity, absolute humidity and saturated humidity; then, the thickness of ice and water on the road surface is calculated through road surface temperature and air humidity; the updated data is then stored in the cloud database.

6. The information integration method within the road anomaly monitoring system according to claim 2, characterized in that: The information reported by maintenance personnel and users includes two categories: one is the progress report of maintenance personnel on abnormal road maintenance tasks, and the other is the abnormal road conditions found and reported by users during driving. The two types of information are segmented by a word segmentation algorithm based on frequency and dynamic programming, and the road section, abnormality type, and maintenance progress are extracted. The multi-party data integration process includes data conflict processing, which is handled through an open addressing method, including four types of conflicts; The first one is that the key frame analysis extracted from the video image based on ice judgment conflicts with the judgment result of the actual numerical data; the second one is that the key frame analysis extracted from the video image based on water accumulation judgment conflicts with the judgment result of the data obtained by the vehicle-mounted laser tube; The third type is that the key frame analysis of the video images taken by the drone based on the road damage type judgment conflicts with the judgment results of the key frame analysis of the video images obtained by the vehicle-mounted camera; The fourth type is the feedback time conflict based on the judgment of exception handling progress.

7. The information integration method within the road anomaly monitoring system according to claim 2, characterized in that: The multi-party data integration process also includes data and image integration process; The corresponding relationships are as follows: ice pictures correspond to ice thickness and temperature; water accumulation pictures correspond to water depth and area; road disease pictures correspond to road names; all of the above correspondences include the correspondence between location coordinate information and road names. The specific algorithms are as follows: For the first correspondence: extract the road icing picture extracted from the video taken by the drone from the cloud database, and extract the ice thickness and temperature calculated based on the picture and store them in the external database for subsequent web page display; For the second correspondence, similar to the first one: extract the road waterlogging picture extracted from the video taken by the drone from the cloud database, and extract the waterlogging depth and area calculated based on the picture and store them in the external database for subsequent web page display; For the third correspondence: extract the road disease picture extracted from the video taken by the drone from the cloud database. Since the drone video provides real-time trajectory, the geographic location information of the picture is obtained through the position of the extracted key frame, and stored in the external database for subsequent web page display.

8. A method for integrating information within a road anomaly monitoring system, characterized in that: The information integration method comprises: Step 101, receiving the raw data information collected and sent back by the multispectral camera on the drone, the multi-party sensor based on the narrowband Internet of Things, the vehicle-mounted laser tube and the road surface sensor, the camera, and the web platform, and storing it in the relevant cloud database or cloud storage unit; Step 102: Perform data cleaning and related algorithm processing on the obtained raw data to preliminarily obtain processed data information; Step 103: further conflict resolution is performed on the preliminary processed data in the cloud information system. After the data information processing is completed, the corresponding cloud database and storage unit are merged, and the data and images are further integrated and stored in the comprehensive cloud system; Step 104: Finally, the data information in the cloud system is presented and fed back to the relevant third party through a visualization port; In step 102, the system will first clean the raw data to remove outliers and dirty data. After the data cleaning is completed, the corresponding algorithm will be selected according to the data information type for further processing to initially obtain relevant data conclusions and valid information. For the raw data sent back by the GPS locator, the information module in the positioning sensor provides data information of various types. The information category of the recommended minimum positioning information is selected through the GPRMC identifier. This type of information provides: Greenwich Mean Time in hours, minutes, and seconds format, positioning status, longitude and latitude information in degrees and minutes format, longitude and latitude hemisphere identifier, ground speed, date in day-month-year format, magnetic declination, magnetic declination direction, and mode indication; the above information is processed, including: First, determine the positioning status. If it is marked as 'A', keep this data and perform subsequent processing. If it is displayed as 'V', read the next data. Among them, A indicates valid positioning, and V indicates invalid positioning. Convert Greenwich Mean Time in hours, minutes and seconds format to Beijing time. According to the time zone difference, the new time is hh+8, mm, ss; Convert the latitude and longitude data. The specific steps are as follows: The format of received data is: ddmm.mmmm; the format of longitude and latitude is: d°m′s″; define data = ddmm.mmmm; degree d = int(data / 100); m1 = data - d * 100; m=int(m1); Seconds s = int((m1-m)*60); When data is exported, add the south / north latitude mark after the latitude and the east / west longitude mark after the longitude to standardize the format; Keep the processed data, update the cloud database, and discard the redundant data; Next, perform offset correction on the coordinates obtained in the previous step. The specific implementation method is as follows: Input PR (lat, lng), which is the coordinate information extracted in the previous step; Retrieve the matching coordinate value in the correction table and obtain PO (latoffset, lngoffset), which is the correction offset difference; Finally, add / subtract the difference value to the original longitude and latitude values ​​to obtain the corrected coordinate data; Obtain various POI data through the API interfaces of various map service applications, including name, latitude and longitude coordinates, and address information. The specific implementation methods are as follows: Input coordinate information for automatic positioning, and return the road street, house number streetNumber, province province, and city city information; Extract the street information; return street; In step 102, three wavelengths, λ1, λ2, and λ3, are selected for the original reflected wave data sent back by the vehicle-mounted sensor according to the water absorption spectrum, the ice absorption spectrum, and the oil absorption spectrum; due to the combined effects of the road surface reflection and the absorption and scattering of the light signal by water molecules and solid particles, the backscattering coefficients of water for the three wavelengths are λ1>λ3>λ2; the backscattering coefficients of ice for the three wavelengths are λ1>λ2>λ3; and the backscattering coefficients of oil for the three wavelengths are λ1>λ3>λ2; and the road surface condition is qualitatively analyzed according to the size and difference of the ratio of the three backscattered light signals to the light signal when the road surface is dry; For temperature calculation, by receiving infrared radiation energy from the road surface, according to Planck's law: W(λ,T)=C1 / {λ 5 [exp(C2 / (λT))-1] -1 } Where, T is temperature; C1 is the first radiation constant; C2 is the second radiation constant; When the wavelength is constant, the infrared radiation energy W is measured and the temperature can be calculated; temperature measurement range: -60~+85℃; measurement accuracy: ≤±0.1℃; resolution: 0.1℃; For the calculation of the thickness of the covering, based on the absorbance of the substance and the Lambert-Beer law, the thickness calculation formula is derived: d=L n (I0 / I) / (εC) Where d is the thickness of the light-absorbing material; I0 is the incident light intensity; I is the intensity of the transmitted light after passing through the material; L n (I0 / I) is absorbance; C is the concentration of the substance; ε is the proportional coefficient of light absorption, which is related to the properties of the absorbing substance and the wavelength of the incident light; Measuring range: 0.00~2mm; Measuring accuracy: ±0.01mm; Resolution: 0.01mm; In step 102, for the original video images transmitted back by the multi-spectral camera of the drone and the vehicle-mounted camera, the video content is firstly decomposed into images; The key frames of the video content are extracted by combining the histogram method and the frame averaging method based on the shot boundary. The specific steps are as follows: First, the video is structured, including: Store each frame of the video; Calculate the difference between frames in the shot using the histogram difference method and store the data in a structure; Where h(i) represents the cumulative distribution of each grayscale of a frame in the video; m and n represent the previous frame and the next frame respectively; the max() function represents the larger value of the two; k represents all grayscale values ​​of this frame; Multiple screenings are performed to determine the boundary frames of the adaptive threshold, and finally the shot boundary is determined; The video content is segmented by shot boundaries, that is, the video is segmented into multiple shots consisting of continuous frames by multiple shot boundaries, and the frame similarity between continuous frames is high; Then extract key frames from each shot, including: Calculate the average value of the difference between each frame and the previous frame in the shot, and the interpolation value is recorded as distance; Traverse the frames in the shot, find the frames whose distance is greater than the multiple of the average value, and extract them as key frames; If there is no frame that meets the conditions, the middle frame of the shot is selected as the key frame; Then, the adaptive hot metal code infrared measurement image pseudo color coding method, digital image processing method, and road damage repair image feature analysis method were used to process the ice image, water and oil pollution image, and road damage image. For the icing image, first divide the grayscale area of ​​the relevant image into four parts: blue represents low-temperature objects l0-l1, green represents medium-low temperature objects l1-l2, yellow represents medium-temperature objects l2-l3, and red represents high-temperature objects l3-l4; further, let the number of items with grayscale value i in the image be n i , the gray level of the image is L; let the gray level k be the threshold to divide the image into two areas, the pixels with gray levels of 1 to k belong to the foreground area, denoted as A, and the pixels with gray levels of k+1 to L belong to the background area, denoted as B; according to the maximum inter-class variance criterion, change k from l to L, and calculate the inter-class variance σ 2 ; Then, the maximum inter-class variance method is used to obtain the binarization threshold of the image as the second area l2. Then, the thresholds of the two parts are calculated respectively as the first threshold l1 and the third threshold l3. The grayscale ranges of the four fixed areas of hot metal coding are modified to four adaptive thresholds, and the desired special effect picture can be obtained; For the image of accumulated water and oil pollution, in order to calculate the relevant area, the image is firstly grayed using the rgb2gray function provided by MATLAB software; In order to obtain a clear border, the local threshold segmentation method is used to divide the image into 4 parts, and the histogram of each part is found. The inversion method is used to eliminate discontinuity to obtain a binary image, and a disk structure element with a radius of 5 pixels is selected to perform an opening operation on the binary image to extract the border; secondly, in order to ensure the measurement accuracy of the area, the image is geometrically corrected through three steps: Hough transform, spatial transform, and grayscale difference; Finally, the area of ​​the water and oil pollution part can be calculated by referring to the border area and the ratio of the water and oil pollution part pixels to the total pixels; For road damage images, the images are first deblurred, and then the window contrast feature extraction algorithm is used to extract features from the images. Finally, in order to ensure the accuracy of the analysis, the false target information of the images needs to be removed. Furthermore, after the road damage is identified, the area of ​​the damage is calculated through the image. The relevant steps are similar to the above-mentioned method for calculating the area of ​​accumulated water and oil pollution. In step 102, the feedback text sent back by the individual user and the maintenance personnel from the web page port is segmented; the maintenance progress information reported by the maintenance personnel and the road abnormality information reported by the user are processed by a segmentation algorithm based on frequency and dynamic programming. The specific process is as follows: First, the feedback text segment to be processed is roughly segmented to obtain all segmentation results including the correct segmentation results; the full segmentation method is used here, that is, starting from the first character, all possible segmentation methods are obtained, and the remaining characters are segmented in this way, and the segmentation results are stored in a segmentation word tree with each character as the root; On this basis, a roughly segmented directed acyclic graph is constructed through the N-shortest path construction algorithm, and its shortest path is solved to improve the efficiency of word segmentation; Then, the segmentation results are combined with a low-to-top greedy algorithm for dynamic planning to obtain the global optimal path; Input the information of each node in the graph and the predecessor node of each node, and calculate the correct segmentation result based on their frequency; Finally, based on the part of speech of the segmented chunks, noun and adjective chunks are extracted to obtain information about road sections, abnormality types, and maintenance progress. As for the video information transmitted back, it is first disassembled into multiple images through the streaming video disassembly algorithm, and then preliminarily screened together with the transmitted images to preliminarily exclude images without abnormal features. After the screening is completed, the images are processed through relevant algorithms to obtain the required special effect images; In step 103, four types of conflicts may be generated: The first type of conflict is the conflict of conclusions on whether the road is frozen or not. That is, the image features show that the road surface temperature is low at a certain location and ice may occur, but the data shows that there is no ice at that location. This conflict occurs between the video photos sent back by the drone and the vehicle and the data sent back by the vehicle road sensor. Since the video photos can only get special effects pictures after being processed by the algorithm, and the data information sent back by the road sensor includes not only the judgment value of whether the road surface is frozen, but also the specific thickness of the ice. Therefore, when there is a conflict between the two, the data sent back by the road sensor is mainly used for judgment. If there is no ice, the relevant abnormal alarm is lifted. The second type of conflict is water accumulation or misjudgment conflict, that is, the type of abnormality judged by the image features is inconsistent with the data judgment value. This conflict occurs between the video photos sent back by the drone and the vehicle and the data sent back by the vehicle road sensor. According to the working principle of the laser tube and the road sensor, the conclusion calculated by the difference in reflected waves sent back by different materials is more accurate than the picture. Therefore, when there is a contradiction between the two, the conclusion obtained by the data information sent back by the road sensor should be given priority. The third type of conflict is the road disease type judgment conflict, which occurs between the video pictures sent back by the drone and the video pictures sent back by the vehicle camera, that is, the two pictures have different conclusions on the road abnormality type judged by the preliminary algorithm, so it is necessary to further perform algorithm processing and comparison on the pictures of both sides to reach the final conclusion. The fourth type of conflict is the exception handling progress conflict, that is, the feedback information from the maintenance unit is inconsistent with the actual exception handling situation. This conflict occurs between the feedback information from the government and the maintenance unit and the abnormal situation sent back by the drone. First, the feedback time of the relevant information of both parties is extracted and the time series is sorted. If the feedback information time is earlier than the information sent back by the drone, the conflict is judged to exist, and the abnormal information sent back by the drone shall prevail over the feedback information from the government maintenance unit. After the above four types of conflicts are resolved, the scattered cloud databases and storage units are merged, and the relevant data and images are further integrated. The relevant integrated data and images are divided into the following three categories according to the anomaly type: When the anomaly is icing on the road surface, it is necessary to extract the icing picture, icing area, thickness and road surface temperature data from the cloud storage unit, and further mark the picture with relevant data; When the anomaly is waterlogging on the road, it is necessary to extract waterlogging images, waterlogging area and depth data from the cloud storage unit, and further mark the images with relevant data; When the anomaly is a road disease, it is necessary to extract the disease image, disease name and disease area data from the cloud storage unit, and further mark the image with relevant data; In step 104, the integrated data finally obtained in the integrated cloud system will be presented to maintenance personnel and individual users through an external interface, and users can feedback relevant information to the cloud system through this interface.

Images