Intelligent identification system for urban road diseases and facilities based on bus vehicle mobile sensing

Abstract

The application relates to the field of intelligent traffic and computer vision, in particular to an urban road disease and facility intelligent identification system based on bus mobile sensing, which comprises a road camera device, a satellite positioning device, a vehicle-mounted computing terminal, a wireless communication device and a server. The vehicle-mounted computing terminal calculates and accumulates displacement according to the vehicle position, selects equidistant observation images, forms overall road coordinates, lower end road coordinates and upper profile road coordinates of a candidate target, determines road disease candidates and road facility candidates in combination with overall road coordinate deviation, lower end road coordinate deviation, allowed projection deviation and continuous same-direction deviation of the upper profile road coordinates, generates identification records and sends the identification records to the server. According to the application, equidistant observation and road coordinate deviation constraint candidates are used to keep the same candidate target corresponding in position, reduce repeated candidates and position deviation, and make the identification result correspond to road coordinates and representative images.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation and computer vision technology, and in particular to an intelligent identification system for urban road defects and facilities based on the motion perception of public transport vehicles. Background Technology

[0002] In the fields of intelligent transportation and computer vision, the collection, transmission, identification, positioning, judgment, storage, and display of image data related to vehicles, roads, traffic facilities, and road traffic scenarios are typically carried out. This usually involves the coordinated operation of vehicle-mounted cameras, positioning devices, vehicle terminals, wireless communication links, and servers. These systems correlate and process road images, video frames, location coordinates, and time information acquired during vehicle movement, and record the identification results based on image features, target categories, and spatial locations. Intelligent identification of urban road defects and facilities refers to a system that performs image recognition and location recording for road surface damage and road ancillary facilities. Typically, public buses equipped with cameras and positioning devices continuously collect road scene images and vehicle locations. After correlating the images with time and coordinate information, the data is transmitted to an onboard terminal or server. The program extracts, compares, and classifies the appearance features of road surface cracks, potholes, traffic signs, guardrails, and other objects, storing the identified categories, corresponding locations, and image records in a database for terminal query and display.

[0003] Existing technologies for identifying urban road defects and facilities rely on continuous acquisition of road images by public transport vehicles and the combination of these images with location information to form identification records. As the vehicle travels, the viewpoint changes with its position and heading, and the position scale and perspective shape of the same road object in the continuous images change accordingly. When identifying based on the features of a single image, duplicate candidates or positional deviations are easily formed. The boundary shape of road defects and three-dimensional road facilities in the images is also affected by the observation angle. There is a lack of stable spatial correspondence between continuous observation results, and the consistency of the association between the identification category and the road location is limited. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent identification system for urban road defects and facilities based on the motion perception of public transport vehicles. Addressing the problem that the viewpoint for road image acquisition changes with the vehicle's position and heading during operation, causing variations in the position, scale, and perspective of the same road object in consecutive road images, the system establishes a correspondence between vehicle displacement, isometric observation images, candidate target road coordinates, and continuous observation results to differentiate and classify road defects and facilities. This results in generated identification records with corresponding categories, road coordinates, and representative images.

[0005] To achieve the above objectives, the present invention provides an intelligent identification system for urban road defects and facilities based on the mobile perception of public transport vehicles, comprising: a road camera device, a satellite positioning device, an on-board computing terminal, a wireless communication device, and a server; Road cameras capture road images, while satellite positioning equipment collects vehicle position, heading, and positioning error. The vehicle-mounted computing terminal calculates the cumulative displacement using continuously collected vehicle positions and determines the benchmark sampling distance by combining installation parameters. When the cumulative displacement reaches the benchmark sampling distance, it determines the equidistant observation image from the road image. The vehicle-mounted computing terminal extracts the grayscale difference between adjacent pixels in the equidistant observation image, connects the pixels that meet the edge determination conditions to form candidate boundaries, and determines the candidate target and the overall boundary, lower boundary and upper contour of the candidate target by the candidate boundaries; Based on the vehicle position, heading, and installation parameters corresponding to the isometric observation images, the overall boundary, lower boundary, and upper contour of the candidate target are converted into overall road coordinates, lower road coordinates, and upper contour road coordinates, respectively. The overall road coordinates are used to determine the predicted position of the candidate target in the next isometric observation image, and the same candidate target is matched according to the predicted position. The vehicle-mounted computing terminal compares the overall road coordinates and the lower road coordinates of the same candidate target in three consecutive equidistant observation images to determine the overall road coordinate deviation and the lower road coordinate deviation, and determines the allowable projection deviation based on the positioning error. Road defect candidates are determined when the overall road coordinate deviation does not exceed the allowable projection deviation and the boundary scale of the candidate targets remains stable in three consecutive observations. Candidate road facilities are determined when the coordinate deviation of the lower road does not exceed the allowable projection deviation and the coordinates of the upper contour road remain offset in the same direction in three consecutive observations. The vehicle-mounted computing terminal performs category identification on road defect candidates and road facility candidates, determines the corresponding road coordinates and representative images, generates an identification record containing the category, road coordinates and representative images, and sends the identification record to the server through a wireless communication device.

[0006] Furthermore, when determining the reference sampling distance, the vehicle-mounted computing terminal reads the installation height, installation angle, focal length, and number of vertical pixels of the road camera device. It uses the installation height, installation angle, and focal length to determine the forward distance corresponding to each vertical pixel row on the road plane. The road length between the upper and lower limits of the forward distance interval corresponding to the effective vertical imaging range that can be mapped to the road plane is taken as the effective vertical road observation length. The effective vertical road observation length is divided into four equal-length segments, and the road length corresponding to a single equal-length segment is taken as the reference sampling distance.

[0007] Furthermore, the vehicle-mounted computing terminal allocates the position increment between two adjacent vehicle positions to the corresponding acquisition time according to the acquisition time of the road image, and determines the vehicle position corresponding to each road image; taking the vehicle position corresponding to the previous equidistant observation image as the cumulative starting point, the driving distance between consecutive vehicle positions is accumulated. When the accumulated displacement first reaches the reference sampling distance, the road image with the smallest time difference with the time of reaching the reference sampling distance is selected as the next equidistant observation image, and the vehicle position corresponding to the next equidistant observation image is used as the new cumulative starting point.

[0008] Furthermore, the vehicle-mounted computing terminal reads the color channel brightness of each pixel in the equidistant observation image and converts it into grayscale value, calculates the grayscale difference between each pixel and its adjacent pixels, and determines the maximum grayscale difference from the grayscale difference corresponding to each pixel. The maximum grayscale difference within the same equidistant observation image is arranged in numerical order. The maximum grayscale difference corresponding to the preset arrangement position is selected as the edge determination threshold. Pixels that reach the edge determination threshold and are in continuous position are connected to form candidate boundaries.

[0009] Furthermore, the vehicle-mounted computing terminal divides the candidate boundary into regions according to the continuous relationship of pixel positions, determines the boundary pixels belonging to the same continuous region as a candidate target, records the horizontal and vertical boundary positions of the candidate target in the equidistant observation image, and determines the lower boundary by the vertical boundary position; Using the midpoint of the vertical range of the candidate target as the dividing position, the boundary pixels above the dividing position are defined as the upper contour, and the overall boundary is determined by the horizontal boundary position and the vertical boundary position.

[0010] Furthermore, the vehicle-mounted computing terminal reads the vehicle position, heading, and installation parameters corresponding to the isometric observation image. It uses the vertical pixel position of each boundary pixel in the overall boundary, lower boundary, and upper contour, combined with the installation height, installation tilt angle, and focal length, to determine the forward road distance corresponding to each boundary pixel. It also uses the horizontal pixel offset of each boundary pixel relative to the image center line to determine the corresponding horizontal road distance. Using vehicle position and heading as the conversion reference, the forward road distance and lateral road distance are converted to the road coordinate system, forming the overall road coordinates, lower road coordinates, and upper contour road coordinates, respectively.

[0011] Furthermore, the vehicle-mounted computing terminal reads the vehicle position, heading, and installation parameters corresponding to the next equidistant observation image, and performs an inverse transformation of the overall road coordinates relative to the vehicle position and heading corresponding to the next equidistant observation image to determine the predicted position of the overall road coordinates in the next equidistant observation image. The search area is determined with the predicted location as the center. The position difference between the center of each candidate boundary and the predicted location is calculated from the candidate boundaries formed within the search area. The candidate boundary that satisfies the preset matching range and whose boundary range corresponds to the candidate target in the previous equidistant observation image is selected as the continuous observation result of the same candidate target.

[0012] Furthermore, the vehicle-mounted computing terminal determines the overall road coordinate center and the lower road coordinate center of the same candidate target in three consecutive equidistant observation images, calculates the road plane distance between the corresponding overall road coordinate centers and the road plane distance between the corresponding lower road coordinate centers, and uses them as the overall road coordinate deviation and the lower road coordinate deviation, respectively. The positioning error is converted into the road length corresponding to adjacent pixels in the central road area of ​​the image to determine the allowable projection deviation.

[0013] Furthermore, the vehicle-mounted computing terminal continuously acquires the overall road coordinates, lower road coordinates, and upper contour road coordinates of the same candidate target in three consecutive equidistant observation images. When the deviation between each overall road coordinate does not exceed the allowable projection deviation, and the boundary scale of the candidate target remains stable in the three consecutive observations, the same candidate target is identified as a road defect candidate. When the deviation between the coordinates of each lower road does not exceed the allowable projection deviation, and the coordinates of the upper contour road maintain the same offset direction along the continuous observation sequence, the same candidate target is identified as a candidate road facility.

[0014] Furthermore, the vehicle-mounted computing terminal determines the category of road defects by utilizing the extension relationship, closure relationship, and gray-scale difference within the boundary corresponding to the overall boundary for road defect candidates. For road facility candidates, the road facility category is determined by using the stable relationship of road coordinates corresponding to the lower boundary, the longitudinal extension relationship and the lateral extension relationship of the upper contour; For each equidistant observation image corresponding to the same candidate target, calculate the overall road coordinate deviation, select the equidistant observation image with the first value of the overall road coordinate deviation as the representative image, and write the category, road coordinates and representative image into the recognition record.

[0015] The working principle and advantages of this invention are as follows: In this invention, by calculating the cumulative displacement using continuous vehicle positions and determining the reference sampling distance using installation parameters, road images are formed into equidistant observation images based on the actual driving distance of vehicles, reducing the impact of vehicle speed changes on continuous observation intervals. The overall boundary, lower boundary, and upper contour of the candidate target are converted into corresponding road coordinates. The predicted position in the subsequent equidistant observation image is determined by the overall road coordinates, which can maintain the correspondence of the same candidate target in continuous observation. The judgment is made by combining the deviation of the overall road coordinates with the deviation of the lower road coordinates and the allowable projection deviation. The continuous same-direction offset of the upper contour road coordinates is combined to distinguish between road defect candidates and road facility candidates, so that the category recognition result forms a corresponding recognition record with the road coordinates and representative images. Attached Figure Description

[0016] Figure 1 This is a flowchart of the bus vehicle movement perception and road recognition process of the present invention; Figure 2 This is a diagram illustrating the system data interaction relationships of the present invention. Figure 3 This is a flowchart of the isometric observation image selection process of the present invention; Figure 4 This is a flowchart illustrating the candidate target road coordinate transformation process of the present invention. Figure 5 This is a flowchart of the candidate determination process for three consecutive observations according to the present invention. Detailed Implementation

[0017] The following detailed explanation illustrates the specific implementation methods: This embodiment provides an intelligent identification system for urban road defects and facilities based on the mobile perception of public transport vehicles. It includes a road camera device, a satellite positioning device, an onboard computing terminal, a wireless communication device, and a server. The road camera device is installed in the public transport vehicle to continuously acquire road images during the vehicle's operation. The satellite positioning device is used to acquire the vehicle's position, heading, and positioning error. The onboard computing terminal receives the road images and the corresponding vehicle position, heading, and positioning error, and performs tasks such as determining the reference sampling distance, selecting equidistant observation images, forming candidate boundaries, determining candidate targets, transforming road coordinates, continuous observation matching, judging road defect candidates and road facility candidates, and classifying them. The wireless communication device transmits the identification records generated by the onboard computing terminal to the server.

[0018] The road camera device employs an onboard CMOS digital camera. The CMOS image sensor converts the optical image of the road scene into a digital image signal and outputs an RGB road image. In this embodiment, the road image resolution is 1920×1080px, each color channel uses 8-bit digital luminance representation, the image acquisition frequency is 25fps, and the nominal acquisition time interval between two adjacent road images is 0.040s. When the road camera outputs the road image, it simultaneously generates a frame timestamp corresponding to the road image.

[0019] After the road camera device is installed, it is calibrated using a planar calibration plate. The planar calibration plate with known grid size is set at different positions and angles within the field of view of the road camera device. Based on the actual geometric position and corresponding pixel position of the corner point of the calibration plate, the pixel scale corresponding to the lateral focal length, the pixel scale corresponding to the longitudinal focal length, the lateral position of the principal point, the longitudinal position of the principal point, the radial distortion parameter, and the tangential distortion parameter are determined.

[0020] In this embodiment, the horizontal focal length corresponds to the pixel scale. Pixel scale corresponding to vertical focal length All are 1450px, main point horizontal position The value is 960px, and the vertical position of the main point is... The value is 540px. Before the road image enters subsequent pixel position processing, the onboard computing terminal performs distortion correction processing on the road image according to the calibrated radial and tangential distortion parameters. The horizontal and vertical pixel positions mentioned below refer to the pixel positions in the distortion-corrected road image.

[0021] The installation height of the road camera device is measured using a laser rangefinder, which measures the vertical distance from the optical center of the camera device to the road surface where the vehicle is parked. In this embodiment, the installation height is 2.80m. The installation tilt angle is measured using a digital tilt meter, which measures the downward tilt angle of the optical axis of the road camera device relative to the local road surface. In this embodiment, the installation tilt angle is 12°. The pixel scale corresponding to the lateral focal length, the pixel scale corresponding to the longitudinal focal length, the principal point position, the installation height, the installation tilt angle, and the number of vertical pixels in the image are stored as installation parameters in the vehicle-mounted computing terminal.

[0022] Local road plane refers to the continuous road surface treated as a local plane within the effective longitudinal road observation range corresponding to an equidistant observation image. The mounting angle is determined relative to the local road plane. For urban road sections where there are no obvious steps or abrupt changes in slope on the road surface, road coordinate transformation is performed according to the local road plane. When there are obvious abrupt changes in slope on the road surface, the corresponding road image is not used to form the road plane coordinate determination result; processing continues after subsequent road images meet the local road plane conditions again.

[0023] The satellite positioning equipment employs a dual-antenna, multi-frequency GNSS positioning system. The vehicle's position is represented by its longitude and latitude calculated from the GNSS positioning data. The vehicle's heading is calculated from the carrier phase baseline between the two GNSS antennas. The positioning error is the horizontal positioning accuracy estimate output by the GNSS positioning equipment. In this embodiment, the data output frequency of the satellite positioning equipment is 10Hz, the nominal acquisition time interval between two adjacent sets of positioning data is 0.100s, and the horizontal positioning accuracy estimate used is 0.80m.

[0024] Each set of satellite positioning data includes at least GNSS time, vehicle longitude, vehicle latitude, vehicle heading, horizontal positioning accuracy estimate, and positioning validity status. If the positioning validity status is valid, and the vehicle longitude, vehicle latitude, vehicle heading, and horizontal positioning accuracy estimate are all available, the corresponding data will be used as valid vehicle spatial status data. If the positioning validity status is invalid or the aforementioned fields are missing, the corresponding data will not be used to update the cumulative displacement, nor will it be used to establish a road coordinate transformation benchmark.

[0025] The road camera device and the satellite positioning equipment establish a correspondence based on a unified time reference. The satellite positioning equipment outputs GNSS time and a second pulse, and the vehicle-mounted computing terminal uses the second pulse to synchronize the system clock. When the road camera device acquires each road image, the vehicle-mounted computing terminal records the corresponding frame timestamp, so that the road image acquisition time and the satellite positioning data acquisition time can be converted to the same time axis.

[0026] Please see Figures 1 to 5 The processing procedure in this embodiment is as follows: Collect road images and vehicle spatial status data.

[0027] Road camera devices continuously capture road images while the bus is in motion and record the capture time for each image. Satellite positioning equipment continuously captures the vehicle's position, heading, and positioning error, and records the corresponding capture time. The onboard computing terminal determines the vehicle's position and heading corresponding to the road image capture time.

[0028] For vehicle locations with latitude and longitude directly output by satellite positioning equipment, the ground distance between adjacent valid vehicle locations is determined using the Haversine spherical distance relationship: ; (1) in, This represents the ground distance between two consecutive valid vehicle positions, in meters (m). The reference radius of the Earth is 6,378,137 m in this implementation method; and These represent the latitudes of the previous and next vehicle positions, respectively. and These represent the longitude of the previous and next vehicle positions, respectively. Longitude and latitude are converted to rad before trigonometric calculations. It is a dimensionless intermediate quantity.

[0029] After conversion using formula (1), the vehicle displacements of three adjacent positioning intervals are 0.63m, 0.78m, and 0.66m, respectively. Taking the vehicle position corresponding to the previous equidistant observation image as the cumulative starting point, the cumulative displacement after the end of the first positioning interval is 0.63m, the cumulative displacement after the end of the second positioning interval is 1.41m, and the cumulative displacement after the end of the third positioning interval is 2.07m.

[0030] When vehicle longitude and latitude need to be involved in road coordinate transformation, the onboard computing terminal converts them into local eastward and northward coordinates based on a selected reference point in the area where the inspected road is located through map planar projection. For road images between two adjacent valid vehicle positions, the eastward and northward coordinates are linearly interpolated according to the acquisition time ratio.

[0031] For example, the acquisition times of two consecutive sets of valid vehicle positions are 12.500s and 12.600s, respectively, which correspond to a road plane displacement of 0.80m. When the road image acquisition time is 12.540s, the road image is located at 40% of the position between the two positioning times, and the corresponding vehicle position is the position after the previous vehicle position has moved by 40% along the eastward coordinate increment and the northward coordinate increment, respectively.

[0032] Vehicle headings are interpolated according to the same time scale. When adjacent headings cross the 0° and 360° boundaries, interpolation is performed based on the minimum angle difference between the two headings. For example, if the previous heading is 358° and the next heading is 2°, it is treated as a 4° heading change.

[0033] When the valid positioning status of any adjacent satellite positioning data becomes invalid, vehicle position interpolation and heading interpolation will not be performed across the corresponding invalid positioning interval. The corresponding road image will still be retained as a road image captured by the road camera device, but it will not be used as an isometric observation image that requires vehicle position and heading to participate in road coordinate transformation.

[0034] Determine the baseline sampling distance.

[0035] The vehicle-mounted computing terminal reads the installation height, installation angle, pixel scale corresponding to the horizontal focal length, pixel scale corresponding to the vertical focal length, principal point position, and number of vertical pixels in the image of the road camera device. It then maps the vertical pixel rows in the road image to the local road plane and determines the forward road distance corresponding to each vertical pixel row.

[0036] Define normalized pixel coordinates: ; The forward road distance and lateral road distance corresponding to each image pixel are determined by the following formula: (2) in, This indicates the horizontal pixel position in the distortion-corrected road image, in pixels (px). Indicates the vertical pixel position, in pixels (px). and These represent the horizontal and vertical positions of the main point, respectively, in pixels (px). and These represent the pixel scale corresponding to the horizontal focal length and the pixel scale corresponding to the vertical focal length, respectively, in pixels (px). and These are dimensionless normalized pixel coordinates; This indicates the installation height, in meters (m). Indicates the installation tilt angle; This represents the forward road distance at the intersection of the pixel imaging ray and the local road plane, in meters. The distance from the intersection point to the vehicle's forward centerline is expressed in meters (m).

[0037] In formula (2), the trigonometric functions and normalized pixel coordinates are dimensionless quantities. and The length dimension is determined by the installation height. Provided.

[0038] In this embodiment, , , , , The effective longitudinal imaging range is defined as row 620 to row 1020. According to formula (2), the forward road distance corresponding to row 620 is about 10.336m, the forward road distance corresponding to row 1020 is about 4.788m, and the effective longitudinal road observation length is about 5.547m.

[0039] The effective longitudinal road observation length of 5.547m is divided into four equal-length sections, each corresponding to approximately 1.387m. In this implementation method, 1.387m is used as the benchmark sampling distance.

[0040] When using the same installation parameters, the forward road distance corresponding to the 700th pixel row is approximately 8.468m, the forward road distance corresponding to the 800th pixel row is approximately 6.873m, and the forward road distance corresponding to the 900th pixel row is approximately 5.755m.

[0041] Determine the equidistant observation images.

[0042] The onboard computing terminal uses the vehicle position corresponding to the previous equidistant observation image as the accumulation starting point to accumulate the travel distance between consecutive vehicle positions. When a previous equidistant observation image has not yet been formed, the currently determined starting vehicle position is used as the accumulation starting point.

[0043] When the cumulative displacement has not yet reached the reference sampling distance, the system continues to receive subsequent valid vehicle positions and accumulate the travel distance; when the cumulative displacement reaches the reference sampling distance for the first time, the system determines the time corresponding to reaching the reference sampling distance, compares the time difference between this time and the road image acquisition time, and selects the road image with the smallest time difference as the next equidistant observation image.

[0044] Taking the aforementioned 1.387m baseline sampling distance as an example, when a bus travels at 5m / s, the time interval between two adjacent equidistant observation positions is approximately 0.277s, corresponding to approximately 6.94 road image acquisition cycles; when traveling at 8m / s, the time interval is approximately 0.173s, corresponding to approximately 4.33 road image acquisition cycles; and when traveling at 10m / s, the time interval is approximately 0.139s, corresponding to approximately 3.47 road image acquisition cycles.

[0045] Taking the two consecutive vehicle displacements of 0.63m and 0.78m as examples, the first positioning interval is from 12.400s to 12.500s, and the second positioning interval is from 12.500s to 12.600s. After the first positioning interval ends, the cumulative displacement is 0.63m; after entering the second positioning interval, it needs to move about 0.757m to reach the reference sampling distance of 1.387m. With the position in the second positioning interval changing approximately linearly with time, the moment when the cumulative displacement reaches 1.387m is approximately 12.597s.

[0046] If the nearby road images were acquired at 12.560s and 12.600s respectively, the time differences between the two road images and 12.597s are approximately 0.037s and 0.003s respectively. Therefore, the road image corresponding to 12.600s is selected as the next equidistant observation image, and the vehicle position corresponding to 12.600s is used as the new cumulative starting point.

[0047] Before a road image is identified as an isometric observation image, the road image data, frame timestamp, corresponding vehicle position, and corresponding heading are checked. If the road image data is incomplete, the frame timestamp is invalid, or valid vehicle position and heading cannot be obtained, the road image is not identified as an isometric observation image, and the next road image that meets the conditions is processed.

[0048] Form candidate boundaries.

[0049] The vehicle-mounted computing terminal reads the color channel brightness of each pixel in the equidistant observation image, converts the RGB color channel brightness into grayscale values, and calculates the grayscale difference between the current pixel and its adjacent pixels above, below, to the left, and to the right.

[0050] The grayscale value and the maximum grayscale difference are determined by the following formula: ; (3) in, , and These represent the brightness of the red channel, green channel, and blue channel of the current pixel, respectively. This represents the grayscale value of the current pixel; This represents the grayscale value of the pixel to be evaluated. , , and These represent the grayscale values ​​of the pixels above, below, to the left, and to the right of the current pixel, respectively. This represents the maximum grayscale difference corresponding to the current pixel.

[0051] For example, if a pixel has a red channel brightness of 112, a green channel brightness of 118, and a blue channel brightness of 121, its grayscale value is approximately 116.548 according to formula (3). When the grayscale values ​​of its upper, lower, left, and right adjacent pixels are 115, 144, 119, and 113 respectively, the corresponding grayscale differences are approximately 1.548, 27.452, 2.452, and 3.548 respectively. The maximum grayscale difference of this pixel is 27.452.

[0052] For a 1920×1080px equidistant observation image, determine the maximum grayscale difference for each pixel with a complete four-neighborhood. Arrange all maximum grayscale differences in ascending order, and take the maximum grayscale difference corresponding to the 85th percentile as the edge determination threshold for the equidistant observation image. For example, when the maximum grayscale difference corresponding to the 85th percentile is 26, pixels with a maximum grayscale difference of not less than 26 are determined as edge pixels.

[0053] Edge pixels are connected using an 8-neighborhood connectivity relationship. Formula (3) is used to obtain the maximum grayscale difference between the current pixel and its four adjacent pixels (upper, lower, left, and right). After a pixel is determined to be an edge pixel, the connectivity relationship between the current edge pixel and its surrounding horizontal, vertical, and diagonal adjacent edge pixels is then determined.

[0054] Edge pixels belonging to the same 8-neighbor contiguous region form the same candidate boundary. For example, when a contiguous region has 86 edge pixels, these 86 edge pixels form one candidate boundary; when another group of 43 edge pixels does not have an 8-neighbor connection with this contiguous region, they form another candidate boundary.

[0055] If the current equidistant observation image does not contain edge pixel regions that can form continuous candidate boundaries, no candidate target is formed, and the next equidistant observation image is processed.

[0056] The candidate target, overall boundary, lower boundary, and upper contour are determined by the candidate boundary.

[0057] The vehicle-mounted computing terminal divides the candidate boundary into regions according to the continuity of edge pixel positions, and determines the edge pixels belonging to the same continuous region as a candidate target.

[0058] The lateral boundary position of a candidate target is determined by the lateral pixel positions of all edge pixels of the same candidate target, and the vertical boundary position is determined by the vertical pixel positions of all edge pixels. The overall boundary is composed of all candidate boundary pixels of the same candidate target.

[0059] The lower boundary is the sequence of continuously connected boundary pixels at the lower vertical position of the candidate target. The upper contour is the midpoint between the upper and lower vertical positions of the candidate target, and the boundary pixels above the upper boundary that belong to the same candidate boundary are considered as the upper contour.

[0060] Taking a candidate target as an example, its boundary pixels are distributed horizontally from 720px to 1040px and vertically from 610px to 950px. The horizontal boundary range is 320px, and the vertical boundary range is 340px. The midpoint of the vertical range is 780px. Boundary pixels above 780px that belong to the same candidate boundary form the upper contour.

[0061] Assuming the overall boundary consists of 86 boundary pixels, with 18 consecutive boundary pixels forming the lower boundary and 31 boundary pixels forming the upper contour, then the positions of the 86 overall boundary pixels, the 18 lower boundary pixels, and the 31 upper contour pixels are stored respectively.

[0062] The 320px and 340px mentioned above only represent the imaging range of candidate targets in the current isometric observation image. When a vehicle approaches the same road object, the pixel scale of the candidate target in the image changes with the observation distance. Therefore, the boundary scale stability in three consecutive observations is not directly judged according to the pixel width and pixel height in different road images, but according to the road space boundary scale formed by subsequent road coordinate transformation.

[0063] The candidate targets are converted into overall road coordinates, lower road coordinates, and upper contour road coordinates.

[0064] The vehicle-mounted computing terminal reads the vehicle position, heading, and installation parameters corresponding to the current isometric observation image, and obtains the boundary pixel positions in the overall boundary, lower boundary, and upper contour, respectively.

[0065] For each boundary pixel, the forward road distance and the lateral road distance are determined using formula (2). For road defects located on the road plane, the intersection of the imaging ray corresponding to the overall boundary pixel and the local road plane is used to represent the position of the road defect boundary in the road plane.

[0066] For road infrastructure, the upper contour is a three-dimensional structure that is separated from the road plane; the upper contour itself is not located on the road plane. In this embodiment, the upper contour road coordinates are the projected road coordinates obtained by extending the imaging ray corresponding to the upper contour pixels onto the local road plane. These upper contour road coordinates are used to characterize the changes in the projection of the upper contour imaging ray onto the road plane at different observation positions, and do not represent the true three-dimensional spatial coordinates of the upper contour of the road infrastructure.

[0067] For example, when the horizontal position of a boundary pixel is 1040px and the vertical position is 800px, the forward road distance is approximately 6.873m and the lateral road distance is approximately 0.403m using formula (2). When the horizontal position of another boundary pixel is 820px and the vertical position is 900px, the forward road distance is approximately 5.755m and the lateral road distance is approximately -0.600m, where the negative sign indicates that the road position is located on the other side of the vehicle's forward centerline.

[0068] Vehicle longitude and latitude are converted to local east and north coordinates via map projection. The first coordinate axis of the road coordinate system is oriented eastward, the second axis is oriented northward, and the vehicle heading... Increase clockwise from due north.

[0069] The transformation of the vehicle's local road position to a unified road coordinate system, and the transformation of the unified road coordinate system to the predicted position in the subsequent equidistant observation image, are determined according to the following formula: (4) In the formula, This represents the lateral road distance obtained from formula (2), in meters. This indicates the forward road distance, in meters (m). and These represent the current vehicle's eastbound and northbound road coordinates, respectively, in meters (m). Indicates the current vehicle heading; and These represent the eastbound and northbound road coordinates after boundary pixel transformation, respectively, in meters (m). and These represent the vehicle and road coordinates corresponding to the subsequent equidistant observation image, in meters; Indicates the vehicle's heading corresponding to the next equidistant observation image; and These represent the lateral road distance and forward road distance of the same road coordinate point relative to the position of the next vehicle, respectively, in meters; and These represent the predicted horizontal pixel position and the predicted vertical pixel position, respectively, in pixels (px).

[0070] For example, if the current vehicle road coordinates are 100.00m and 50.00m, and the vehicle heading is 0°, a boundary pixel can be obtained by formula (2) with a lateral road distance of 0.403m and a forward road distance of 6.873m, corresponding to road coordinates of approximately 100.403m and 56.873m.

[0071] All boundary pixels in the overall boundary are transformed into overall road coordinates after road coordinate transformation. All boundary pixels in the lower boundary are transformed into lower road coordinates after road coordinate transformation. All boundary pixels in the upper contour are transformed into upper contour road coordinates after the aforementioned projection processing.

[0072] The coordinate center of the overall road is taken as the average of the east and north coordinates of all road coordinate points on the overall boundary; the coordinate center of the lower road is taken as the average of the east and north coordinates of all lower road coordinate points; the coordinate center of the upper contour road is taken as the average of the east and north coordinates of all upper contour road coordinate points.

[0073] Determine the predicted location of the candidate target in the subsequent isometric observation image and match the same candidate target.

[0074] The vehicle-mounted computing terminal reads the vehicle position, heading, and installation parameters corresponding to the next isometric observation image, and obtains the overall road coordinates of the candidate targets in the previous isometric observation image.

[0075] Using the overall road coordinates in the previous equidistant observation image and the vehicle position and heading corresponding to the next equidistant observation image as input, the overall road coordinates are reverse-converted to the predicted position in the next equidistant observation image using formula (4).

[0076] For example, in the previous observation, the lateral road distance of a certain road position relative to the vehicle was about 0.403m and the forward road distance was about 6.873m. When the bus moves forward 1.387m along the current direction and the heading remains unchanged, the forward road distance of the same road position in the next observation position is about 5.486m and the lateral road distance is still about 0.403m. According to formula (4), the predicted lateral pixel position is about 1058px and the predicted vertical pixel position is about 930px.

[0077] The preset matching range is determined based on the positioning error range of the satellite positioning equipment and the projection parameters of the road camera device. The positioning error of two consecutive observations is 0.80m, and after combining the road lengths corresponding to adjacent pixels in the central road area of ​​the image, approximately 1.613m is used as the road position error coverage between the two consecutive observations.

[0078] Taking a predicted lateral road distance of approximately 0.403m and a forward road distance of approximately 5.486m as an example, a road position error coverage of 1.613m is applied to the lateral and forward directions respectively, and the error range is mapped to the subsequent equidistant observation image using formula (4). The corresponding lateral pixel positions are approximately distributed between 559px and 1629px, and the vertical pixel positions are approximately distributed between 783px and 1182px.

[0079] Centered on the predicted positions of 1058px and 930px, this implementation uses a horizontal ±600px and a vertical ±260px as the preset matching range. The initial search area is 458px to 1658px horizontally and 670px to 1190px vertically. The actual vertical pixel range of the road image is 0px to 1079px, therefore the actual search area is taken as 458px to 1658px horizontally and 670px to 1079px vertically.

[0080] The vehicle-mounted computing terminal forms candidate boundaries within the search area and determines the center of each candidate boundary. The center of the candidate boundary is calculated by averaging the horizontal and vertical positions of all edge pixels of the candidate boundary.

[0081] For example, three candidate boundaries are formed within the search area, with center values ​​of 1048px, 940px, 1320px, 1000px, and 1710px, 990px, respectively. The first candidate boundary has a horizontal difference of approximately 10px and a vertical difference of approximately 10px relative to the predicted position; the second candidate boundary has a horizontal difference of approximately 262px and a vertical difference of approximately 70px; and the third candidate boundary has a horizontal difference of approximately 652px and a vertical difference of approximately 60px. The horizontal position difference of the third candidate boundary exceeds the preset matching range and is not included in the subsequent boundary range verification.

[0082] Boundary range verification considers both the number of boundary pixels and the overall boundary range. Taking a candidate target in a previous equidistant observation image with 86 boundary pixels as an example, the number of boundary pixels in the subsequent candidate is considered to be between 43 and 172 as a boundary range condition. When the first candidate boundary has 92 boundary pixels and the second candidate boundary has 190 boundary pixels, the first candidate boundary meets the boundary range condition, while the second candidate boundary does not.

[0083] When two or more candidate boundaries simultaneously satisfy the prediction location condition and the boundary range condition, the pixel distance between the center of the candidate boundary and the prediction location is determined respectively, and the candidate boundary with the smaller pixel distance is selected as the continuous observation result of the same candidate target in the subsequent equidistant observation image.

[0084] When no candidate boundary meeting the conditions exists within the search area, the continuous observation relationship of the current candidate target terminates at the equidistant observation image, without using other candidate boundaries to complete the three consecutive observations. Newly formed candidate boundaries in the subsequent equidistant observation image re-establish the continuous observation relationship according to the new candidate target.

[0085] After obtaining the matching results of the same candidate target in three adjacent equidistant observation images through the above process, three consecutive observation results are formed.

[0086] The road coordinate deviation of the same candidate target in three consecutive equidistant observation images is compared to determine the candidates for road defects or road facilities.

[0087] Three consecutive observations are designated as the first observation, the second observation, and the third observation, respectively. The vehicle-mounted computing terminal obtains the overall road coordinate center and the lower road coordinate center corresponding to the three observations, and compares the differences in road coordinates between the first and second observations, between the second and third observations, and between the first and third observations.

[0088] Road coordinate deviation and allowable projection deviation are determined according to the following formula: (5) in, and This indicates two different sequence numbers from three consecutive observations; Indicates the first The coordinate center of the overall road or the lower road coordinate center for each observation is in meters. Indicates the first The coordinates of the overall road or the lower road corresponding to each observation are in meters. This indicates the overall road coordinate deviation or the lower road coordinate deviation between two observations, expressed in meters (m). and These represent the positioning errors at the two observation times, in meters (m). This represents the upper bound of the road length corresponding to adjacent pixels in the central road region of the image, in meters. This represents the allowable projection deviation between two observations, expressed in meters (m).

[0089] The central road region of the image is taken as the middle of the effective road imaging range. In this embodiment, the horizontal pixel position of 960px and the vertical pixel position of 800px are used as the road length conversion position. According to formula (2), the forward road distance corresponding to the 800th pixel row is about 6.8730m, the forward road distance corresponding to the 801st pixel row is about 6.8598m, and the corresponding vertical road length between them is about 0.0131m; at the same vertical position, the horizontal road length corresponding to the adjacent 1px is about 0.0050m. Therefore, 0.0131m is taken as the upper limit of the road length corresponding to adjacent pixels in the central road region of the image. .

[0090] When the positioning error corresponding to two consecutive observations is 0.80m, the allowable projection deviation is: ; The corresponding allowable projection deviation is taken as 1.613m. When the positioning errors in the three observations are not the same, the allowable projection deviation for each group is determined by the positioning errors at the corresponding two observation times.

[0091] For road defect candidates, the overall road coordinate deviations of the three groups are compared first. If the overall road coordinate deviations between the first and second, second and third, and first and third observations do not exceed the corresponding allowable projection deviations, then it is determined whether the boundary scale of the candidate target remains stable in the three consecutive observations.

[0092] The boundary scale is determined by the minimum area bounding rectangle of the overall road coordinate point set. The longer side of the minimum area bounding rectangle is used as the first boundary scale, and the shorter side is used as the second boundary scale.

[0093] For example, the overall road coordinate centers formed by three consecutive observations were 100.00m, 50.00m, 100.28m, 50.11m, and 100.19m, 49.94m, respectively. The overall road coordinate deviation between the first and second observations was approximately 0.301m, between the second and third observations it was approximately 0.192m, and between the first and third observations it was approximately 0.199m, all of which are below the allowable projection deviation of 1.613m.

[0094] The longest side of the circumscribed rectangle corresponding to the smallest area in the three observations is 1.16m, 1.13m, and 1.18m, and the shortest side is 0.82m, 0.80m, and 0.84m, respectively. Using the boundary scale of the first observation as a comparison benchmark, if the changes in the longest and shortest sides in the second and third observations do not exceed 10%, it is determined that the boundary scale remains stable in the three consecutive observations.

[0095] In the above data, the second change in the long side is approximately 2.59%, and the change in the short side is approximately 2.44%; the third change in the long side is approximately 1.72%, and the change in the short side is approximately 2.44%, all satisfying the 10% boundary scale stability condition. Therefore, given that the overall road coordinate deviations in all three sets do not exceed the allowable projection deviation, this candidate target is identified as a candidate for road distress.

[0096] If the short side of the third observation becomes 0.94m, which is about 14.63% different from the first observation of 0.82m, then the boundary scale stability condition is not met. Even if the deviation of the overall road coordinates of the three sets does not exceed the allowable projection deviation, the corresponding candidate target will not be identified as a candidate for road defect.

[0097] If the candidate conditions for road defects are not met, the candidate conditions for road facilities shall be evaluated.

[0098] For candidate road facilities, the lower road coordinate deviations are compared between the first and second observations, the second and third observations, and the first and third observations. If the lower road coordinate deviations in all three sets do not exceed the corresponding allowable projection deviations, then the offset direction of the upper contour road coordinates in the three consecutive observations is determined.

[0099] The first displacement vector is formed from the coordinate center of the first upper contour road to the coordinate center of the second upper contour road, and the second displacement vector is formed from the coordinate center of the second upper contour road to the coordinate center of the third upper contour road. The eastward and northward components of the two displacement vectors are multiplied separately, and then the two products are added together. If the result is greater than 0, the angle between the two consecutive displacement vectors is less than 90°, confirming that the upper contour road coordinates maintain the same direction of shift in the three consecutive observations; if the result is equal to or less than 0, the condition of continuous same-direction shift is not met.

[0100] For example, the coordinate centers of the lower road formed by three consecutive observations were 205.00m, 82.00m, 205.18m, 82.09m, and 205.27m, 82.17m, respectively. The coordinate deviation of the lower road between the first and second observations was approximately 0.201m, between the second and third observations it was approximately 0.120m, and between the first and third observations it was approximately 0.319m, all of which are below the allowable projection deviation of 1.613m.

[0101] The corresponding upper contour road coordinate centers are 204.42m, 84.20m, 204.77m, 84.30m and 205.09m, 84.39m, respectively. The displacement vectors from the first to the second displacement are 0.35m and 0.10m, and the displacement vectors from the second to the third displacement are 0.32m and 0.09m. The sum of the products of the corresponding components of the two displacement vectors is positive, thus satisfying the condition of continuous same-direction offset. Therefore, the corresponding candidate targets are identified as road facility candidates.

[0102] If the sum of the products of the corresponding components of two consecutive displacement vectors is equal to or less than 0, the corresponding candidate target will not be identified as a road facility candidate.

[0103] If neither the candidate conditions for road defects nor the candidate conditions for road facilities are met, but the same candidate target can still be successfully matched in the next equidistant observation image, the current first observation is removed, and a new three-consecutive-observation window is formed using the current second observation, the current third observation, and the newly acquired fourth observation, and then the judgment is made according to the same processing relationship.

[0104] If the same candidate target cannot be matched in the next equidistant observation image, the continuous observation relationship of the candidate target is terminated, and the interrupted observation data is not used to combine with the subsequently reappearing candidate target to form three consecutive observations.

[0105] The candidate road defects and candidate road facilities are categorized and an identification record is generated.

[0106] For road defect candidates, the on-board computing terminal reads the overall boundary corresponding to the road defect candidate, determines the extension and closure relationship of the overall boundary, and reads the grayscale difference of the pixels inside the overall boundary.

[0107] When the overall boundary can form a continuous closed boundary with its beginning and end connected, and there are no open endpoints connected to the closed body, the overall boundary is determined to be closed; when it cannot form a continuous closed region, the overall boundary is determined to be unclosed.

[0108] For candidate road defects with unclosed overall boundaries, the main extension length and boundary width are determined using the minimum area bounding rectangle of the overall road coordinate point set. The longer side of the minimum area bounding rectangle is used as the main extension length, and the shorter side is used as the boundary width.

[0109] When the main extension length reaches 5 times the boundary width, and there are continuous gray-scale change areas that reach the edge determination threshold along the main extension direction inside the overall boundary, the corresponding road defect candidate will be identified as a crack-type road defect.

[0110] For example, if a group of candidate targets has a main extension length of 1.06m and a boundary width of 0.18m, and the main extension length is approximately 5.89 times the boundary width, and the overall boundary is not closed and there is a continuous gray-scale variation area inside the boundary, then it meets the criteria for determining crack-type road defects.

[0111] For road defect candidates that form a closed relationship, count the total number of pixels inside the closed candidate boundary, and the number of pixels whose maximum grayscale difference reaches the current edge determination threshold of the equidistant observation image.

[0112] When the number of internal pixels reaching the edge determination threshold exceeds one-quarter of the number of internal pixels on the boundary, and the overall road coordinate stability condition is met for three consecutive observations, the corresponding road defect candidate will be identified as a pothole-type road defect.

[0113] For example, the closed boundary contains 1240 pixels, of which 348 pixels have a maximum grayscale difference that reaches the edge determination threshold. These 348 pixels account for approximately 28.1% of the 1240 internal pixels, which is more than a quarter, thus satisfying the grayscale difference condition within the boundary of pothole-type road defects.

[0114] If a road defect candidate does not meet the criteria for either crack or pothole type, no corresponding crack or pothole category result will be generated. If three more consecutive observations can be generated for the same candidate target, the category will be re-identified using the new consecutive observation results.

[0115] For candidate road facilities, the onboard computing terminal reads the lower boundary and upper contour. When the deviation of the lower road coordinates in three consecutive observations does not exceed the allowable projection deviation, a stable relationship of road coordinates corresponding to the lower boundary is formed.

[0116] For road facility candidates whose upper contours form closed planar boundaries, the lateral and longitudinal extension ranges of the upper contours are determined. If the absolute difference between the lateral and longitudinal extension ranges does not exceed one-quarter of the larger of the two, and the lower road coordinates meet stability conditions, the corresponding road facility candidate is identified as a traffic sign road facility.

[0117] For example, the horizontal extension range of the upper outline is 116px, and the vertical extension range is 108px. The difference between the two is 8px, which is about 6.9% of 116px, less than a quarter. At the same time, the coordinate deviation of the lower road in the three consecutive observations did not exceed the allowable projection deviation, thus meeting the judgment criteria for traffic sign road facilities.

[0118] For road facility candidates with multiple consecutive lower boundary road coordinate points, the lower boundary road coordinate points are arranged according to the forward position of the road. When multiple lower boundary road coordinate points originate from the same consecutive lower boundary and are continuously distributed along the road direction, and the lateral extension range of the upper contour is greater than the longitudinal extension range, the corresponding road facility candidate is identified as a guardrail type road facility.

[0119] For example, the upper outline extends 410px laterally and 96px vertically, with the lateral extension being greater than the vertical extension. Furthermore, multiple lower boundary road coordinate points are continuously arranged along the road direction, and the lower road coordinate deviations in three consecutive observations are all lower than the allowable projection deviations, thus meeting the criteria for determining guardrail-type road facilities.

[0120] If a candidate road facility does not meet the criteria for either traffic signs or guardrails, no corresponding road facility category result will be generated. If the same candidate target can continue to generate valid continuous observations, the new continuous observation results will be used for category identification.

[0121] After completing category recognition, the vehicle-mounted computing terminal determines the corresponding road coordinates and representative image.

[0122] For road defect candidates, the road coordinates in the identification record are taken from the overall road coordinate center of the road defect candidate; for road facility candidates, the road coordinates in the identification record are taken from the lower road coordinate center of the road facility candidate.

[0123] The representative image is selected from the equidistant observation images corresponding to three consecutive observations. The overall road coordinate deviation between the candidate target corresponding to each equidistant observation image and the other two equidistant observation images is calculated, and the arithmetic mean of the two sets of overall road coordinate deviations is taken as the ranking value of the equidistant observation image.

[0124] Taking the aforementioned three road defect observations as examples, the overall road coordinate deviation between the first and second observations was approximately 0.301m, between the first and third observations it was approximately 0.199m, and between the second and third observations it was approximately 0.192m.

[0125] The sorting value of the first equidistant observation image is: ; The sorting value of the second equidistant observation image is: ; The sorting value of the third equidistant observation image is: ; The third equidistant observation image has the smallest sort value, so it is selected as the representative image.

[0126] The identification record is composed of category, road coordinates, and representative image. The category in the road defect identification record is the road defect category obtained through category identification, and the road coordinates use the overall road coordinate center; the category in the road facility identification record is the road facility category obtained through category identification, and the road coordinates use the lower road coordinate center.

[0127] The wireless communication device receives the identification record generated by the vehicle-mounted computing terminal and sends the category, road coordinates, and representative image from the same identification record to the server. The categories, road coordinates, and representative images corresponding to different candidate targets maintain a correspondence.

[0128] Under the aforementioned implementation parameters, the road camera device can continuously capture images at 25fps for 30 seconds, generating approximately 750 road images; the satellite positioning device can continuously output at 10Hz for 30 seconds, generating approximately 300 sets of positioning data. With an average bus speed of 8m / s, the distance traveled in 30 seconds is approximately 240m.

[0129] Based on a baseline sampling distance of 1.387m, 240m contains approximately 173 complete baseline sampling intervals. Without considering missing road images and invalid satellite positioning data, approximately 173 equidistant observation positions triggered by subsequent cumulative displacement can be formed; when the initial observation image corresponding to the cumulative starting point is included in the complete equidistant observation image sequence, approximately 174 equidistant observation images are obtained.

[0130] The three consecutive equidistant observations include three adjacent observation positions. There are two reference sampling intervals between the first and third observations. Therefore, based on the reference sampling distance of 1.387m, the corresponding bus movement distance is approximately 2.774m.

[0131] When the bus speed is 6 m / s, the time corresponding to the 1.387 m reference sampling distance is approximately 0.231 s, which corresponds to approximately 5.78 image acquisition cycles at an image acquisition frequency of 25 fps. When the bus speed is 9 m / s, the corresponding time is approximately 0.154 s, which corresponds to approximately 3.85 image acquisition cycles. As the vehicle speed changes, the number of road images contained between adjacent equidistant observation images changes, but the road spatial interval corresponding to the equidistant observation images is still determined by the 1.387 m reference sampling distance.

[0132] For example, if the first candidate target is successfully matched at the 38th, 39th, and 40th consecutive equidistant observation positions, the three observation results form a three-consecutive-observation window. If the same candidate target can still be matched at the 41st equidistant observation position, and the previous three consecutive observations have not yet formed a final category, then the 39th, 40th, and 41st observation positions are used to form a new three-consecutive-observation window.

[0133] When the second candidate target is successfully matched at the 55th, 56th, and 57th consecutive equidistant observation positions, a separate observation window is formed for the second candidate target for three consecutive times. The overall road coordinates, lower road coordinates, upper contour road coordinates, and the deviations of each road coordinate corresponding to different candidate targets are saved and processed separately and are not mixed with each other.

[0134] In this embodiment, the 85th percentile is used to form the edge determination threshold; the number of candidate boundary pixels is between one-half and two times the number of candidate target boundary pixels in the previous step, used for boundary range verification of the same candidate target; the boundary scale change corresponding to the overall road coordinate point set does not exceed 10%, used for boundary scale stability judgment of road defect candidates; the main extension length reaches 5 times the boundary width, used for category judgment of crack-type road defects; the number of pixels reaching the edge determination threshold inside the closed boundary exceeds one-quarter of the number of internal pixels, used for category judgment of pothole-type road defects; the difference between the horizontal extension range and the vertical extension range of the upper contour does not exceed one-quarter of the larger extension range, used for category judgment of traffic sign-type road facilities.

[0135] 0.80m is the horizontal positioning accuracy estimate corresponding to a set of satellite positioning data, 0.0131m is the road length corresponding to the adjacent vertical pixels near the 800th pixel row, 1.613m is the allowable projection deviation formed by the positioning error of two consecutive observations and the road resolution length of the image, and 1.387m is the benchmark sampling distance formed by dividing the effective vertical road observation length into four equal parts.

[0136] The above parameters are used for describing positioning errors, converting road image scales, judging road coordinate deviations, and selecting equidistant observation images, respectively.

[0137] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A smart identification system for urban road defects and facilities based on the motion perception of public transport vehicles, characterized in that, The system includes: road camera device, satellite positioning equipment, vehicle-mounted computing terminal, wireless communication equipment, and server; The road camera device collects road images, the satellite positioning device collects vehicle position, heading and positioning error, the vehicle-mounted computing terminal uses the continuously collected vehicle position to calculate the cumulative displacement, and determines the reference sampling distance in combination with the installation parameters. When the cumulative displacement reaches the reference sampling distance, the equidistant observation image is determined from the road image. The vehicle-mounted computing terminal extracts the grayscale difference between adjacent pixels in the equidistant observation image, connects pixels that meet the edge determination conditions to form candidate boundaries, and determines the candidate target and the overall boundary, lower boundary and upper contour of the candidate target by the candidate boundaries; The vehicle-mounted computing terminal, based on the vehicle position, heading, and installation parameters corresponding to the isometric observation image, converts the overall boundary, lower boundary, and upper contour of the candidate target into overall road coordinates, lower road coordinates, and upper contour road coordinates, respectively. It then uses the overall road coordinates to determine the predicted position of the candidate target in the next isometric observation image and matches the same candidate target according to the predicted position. The vehicle-mounted computing terminal compares the overall road coordinates and the lower road coordinates of the same candidate target in three consecutive equidistant observation images to determine the overall road coordinate deviation and the lower road coordinate deviation, and determines the allowable projection deviation based on the positioning error. When the overall road coordinate deviation does not exceed the allowable projection deviation, and the boundary scale of the candidate target remains stable in three consecutive observations, the road defect candidate is determined; When the lower road coordinate deviation does not exceed the allowable projection deviation, and the upper contour road coordinates remain offset in the same direction in three consecutive observations, road facility candidates are determined; The vehicle-mounted computing terminal performs category identification on the road defect candidates and the road facility candidates, determines the corresponding road coordinates and representative images, generates an identification record containing the category, road coordinates and representative images, and sends the identification record to the server through the wireless communication device.

2. The intelligent identification system for urban road defects and facilities based on the mobile perception of public transport vehicles according to claim 1, characterized in that, The process by which the vehicle-mounted computing terminal determines the reference sampling distance includes: The installation height, installation angle, focal length, and number of vertical pixels of the road camera device are read. The forward distance of each vertical pixel row on the road plane is determined using the installation height, installation angle, and focal length. The road length between the upper and lower limits of the forward distance interval corresponding to the effective vertical imaging range that can be mapped to the road plane is taken as the effective vertical road observation length. The effective vertical road observation length is divided into four equal-length segments, and the road length corresponding to a single equal-length segment is taken as the reference sampling distance.

3. The intelligent identification system for urban road defects and facilities based on the mobile perception of public transport vehicles according to claim 2, characterized in that, The process by which the vehicle-mounted computing terminal determines the isometric observation image includes: Based on the acquisition time of the road images, the position increment between two adjacent acquisition times is allocated to the corresponding acquisition time to determine the vehicle position corresponding to each road image. Using the vehicle position corresponding to the previous isometric observation image as the cumulative starting point, the travel distance between consecutive vehicle positions is accumulated. When the accumulated displacement first reaches the reference sampling distance, the road image with the smallest time difference from the time when the reference sampling distance is reached is selected as the next isometric observation image, and the vehicle position corresponding to the next isometric observation image is used as the new cumulative starting point.

4. The intelligent identification system for urban road defects and facilities based on the motion perception of public transport vehicles according to claim 3, characterized in that, The process by which the vehicle-mounted computing terminal forms candidate boundaries includes: Read the color channel brightness of each pixel in the equidistant observation image and convert it into grayscale value. Calculate the grayscale difference between each pixel and its adjacent pixels. Determine the maximum grayscale difference from the grayscale differences corresponding to each pixel. The maximum grayscale difference within the same equidistant observation image is arranged in numerical order, and the maximum grayscale difference corresponding to the preset arrangement position is selected as the edge determination threshold. Pixels that reach the edge determination threshold and are in continuous position are connected to form candidate boundaries.

5. The intelligent identification system for urban road defects and facilities based on the motion perception of public transport vehicles according to claim 4, characterized in that, The process by which the vehicle-mounted computing terminal determines candidate targets from the candidate boundaries includes: The candidate boundary is divided into regions according to the continuous relationship of pixel positions, and the boundary pixels belonging to the same continuous region are determined as a candidate target. Record the lateral and longitudinal boundary positions of the candidate target in the isometric observation image, and determine the lower boundary from the longitudinal boundary position; Using the midpoint of the vertical range of the candidate target as the dividing position, the boundary pixels above the dividing position are determined as the upper contour, and the overall boundary is determined by the horizontal boundary position and the vertical boundary position.

6. The intelligent identification system for urban road defects and facilities based on the motion perception of public transport vehicles according to claim 5, characterized in that, The process by which the vehicle-mounted computing terminal converts the candidate target into road coordinates includes: Read the vehicle position, heading and installation parameters corresponding to the isometric observation image, use the vertical pixel position of each boundary pixel in the overall boundary, the lower boundary and the upper contour, and combine the installation height, installation tilt angle and focal length to determine the forward road distance corresponding to each boundary pixel, and use the horizontal pixel offset of each boundary pixel relative to the image center line to determine the corresponding horizontal road distance. Using vehicle position and heading as the conversion reference, the forward road distance and lateral road distance are converted to the road coordinate system, forming the overall road coordinates, lower road coordinates, and upper contour road coordinates, respectively.

7. The intelligent identification system for urban road defects and facilities based on the motion perception of public transport vehicles according to claim 6, characterized in that, The process by which the vehicle-mounted computing terminal matches the same candidate target includes: Read the vehicle position, heading, and installation parameters corresponding to the next isometric observation image, and reverse-convert the overall road coordinates relative to the vehicle position and heading corresponding to the next isometric observation image to determine the predicted position of the overall road coordinates in the next isometric observation image. The search area is determined with the predicted location as the center. The position difference between the center of each candidate boundary and the predicted location is calculated from the candidate boundaries formed within the search area. The candidate boundary that satisfies the preset matching range and whose boundary range corresponds to the candidate target in the previous equidistant observation image is selected as the continuous observation result of the same candidate target.

8. The intelligent identification system for urban road defects and facilities based on the mobile perception of public transport vehicles according to claim 1, characterized in that, The process by which the vehicle-mounted computing terminal determines the overall road coordinate deviation, the lower road coordinate deviation, and the allowable projection deviation includes: Determine the overall road coordinate center and the lower road coordinate center of the same candidate target in three consecutive equidistant observation images, and calculate the road plane distance between the corresponding overall road coordinate centers and the road plane distance between the corresponding lower road coordinate centers, which are respectively used as the overall road coordinate deviation and the lower road coordinate deviation. The positioning error is converted into the road length corresponding to adjacent pixels in the central road region of the image to determine the allowable projection deviation.

9. The intelligent identification system for urban road defects and facilities based on the mobile perception of public transport vehicles according to claim 1, characterized in that, The process by which the on-board computing terminal determines the road defect candidates and the road facility candidates includes: The overall road coordinates, lower road coordinates, and upper contour road coordinates of the same candidate target are continuously obtained in three consecutive equidistant observation images. When the deviation between the coordinates of each overall road does not exceed the allowable projection deviation, and the boundary scale of the candidate target remains stable in three consecutive observations, the same candidate target is identified as the road defect candidate. When the deviation between the coordinates of each lower road does not exceed the allowable projection deviation, and the coordinates of the upper contour road maintain the same offset direction along the continuous observation sequence, the same candidate target is determined as the road facility candidate.

10. The intelligent identification system for urban road defects and facilities based on the mobile perception of public transport vehicles according to claim 1, characterized in that, The process by which the vehicle-mounted computing terminal generates the identification record includes: For road defect candidates, the road defect category is determined by utilizing the extension relationship, closure relationship, and gray-scale difference within the boundary corresponding to the overall boundary. For road facility candidates, the road facility category is determined by using the stable relationship of road coordinates corresponding to the lower boundary, the longitudinal extension relationship and the lateral extension relationship of the upper contour; The overall road coordinate deviation is calculated for each equidistant observation image corresponding to the same candidate target. The equidistant observation image with the first value of the overall road coordinate deviation is selected as the representative image, and the category, road coordinates and the representative image are written into the recognition record.

Images