Road warning column missing and lodging detection method and system based on dynamic patrol

Abstract

The application discloses a road warning column missing and falling detection method and system based on dynamic patrol. The method comprises the following steps: acquiring a road image sequence and obtaining a vehicle pose through synchronous positioning and inertial data fusion; searching for a warning column expected site projection in a field of view to generate a candidate area and performing target detection; establishing a patrol window in units of expected sites, calculating target existence scores, line-of-sight obstruction scores and observation coverage degrees by associating multiple frame detection results; when the target existence is confirmed, restoring a three-dimensional main axis vector and comparing the three-dimensional main axis vector with a normal installation reference direction to determine a posture state; establishing a space-time occupation grid model to update occupation, idle or unknown states and aggregate scores; outputting a facility state by comprehensively considering the scores and the posture state, and generating a pre-warning record or a maintenance work order for an abnormality. Through the method, real missing, temporary invisibility and falling abnormalities can be distinguished, and technical problems, such as insufficient coverage, late reporting, low single-frame recognition reliability and ambiguous state determination in a dynamic scene in the prior art, are solved.

Description

Technical Field

[0001] This invention relates to computer vision, and more specifically to a method and system for detecting missing and fallen road warning posts based on dynamic inspection. Background Technology

[0002] Roadside warning posts are typically installed along urban roadsides, in central medians, at ramp entrances, and in construction diversion areas to indicate driving boundaries, separate traffic flows, and guide drivers' vision. However, these posts are prone to loss, collapse, or severe tilting due to vehicle collisions, aging infrastructure, loose bases, inclement weather, and construction. If not detected promptly, this reduces road safety guidance and increases traffic risks. Current inspection methods rely primarily on manual patrols, fixed video surveillance, or ordinary vehicle-mounted image detection. These methods suffer from insufficient coverage, reporting delays, susceptibility to occlusion in single-frame recognition, variations in lighting, complex backgrounds, and vehicle vibrations, making it difficult to reliably determine the missing or collapsed state of warning posts in dynamic scenarios.

[0003] With the widespread adoption of Geographic Information System (GIS) asset ledgers and digital twin technologies, the spatial location information of road facilities has been gradually digitized, providing a data foundation for accurate detection based on the expected locations of warning posts. Simultaneously, the increasing sophistication of sensors such as positioning units, vehicle-mounted cameras, and inertial measurement units equipped on mobile vehicles like patrol cars, buses, and sanitation vehicles makes it possible to map the expected locations of warning posts to candidate image regions and conduct continuous observation. However, current technologies have not effectively integrated the constraints of GIS-based warning post expected locations with the spatiotemporal alignment capabilities of multiple sensors, and lack a systematic method for determining the state of warning posts in dynamic driving scenarios. This results in insufficient reliability in distinguishing between actual absence, temporary invisibility, and collapse anomalies.

[0004] Therefore, it is necessary to design a new method to distinguish between real missing data, temporarily invisible data, and collapsed anomalies, in order to solve the technical problems in the existing technology, such as insufficient coverage, delayed reporting, low reliability of single-frame recognition, and ambiguous state determination in dynamic scenes. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for detecting missing and fallen road warning posts based on dynamic inspection.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for detecting missing and collapsed road warning posts based on dynamic inspection, comprising: A sequence of road images collected during vehicle operation is acquired, and the road image sequence is time-synchronized. The road image sequence includes multiple frames of road images, positioning data, and inertial measurement data. The vehicle pose is obtained by fusing the positioning data and the inertial measurement data. Based on the vehicle pose, the expected location of the warning post in the field of view of the current frame road image is retrieved, and its world coordinates are projected onto the current frame road image to generate a candidate region. Target detection is performed within the candidate area to determine key points of the target and generate detection results; a current inspection window is established for each warning post's expected location as a unit, and the detection results of multiple frames for the same warning post's expected location are temporally correlated to calculate the target presence score, line-of-sight occlusion score, and observation coverage; When the detection result confirms the existence of the target, the three-dimensional principal axis vector of the warning post is recovered based on the key points of the target and compared with the normal installation reference direction determined by the vehicle pose. The attitude state is determined based on the angle between the two. A spatiotemporal occupancy grid model is established for each warning post's expected location. The occupancy, vacancy, or unknown status of the grid cells is updated based on the detection results of each frame of road images. The grid occupancy score and grid vacancy score are then aggregated. The system integrates the target presence score, line-of-sight occupancy score, observation coverage, grid occupancy score, grid idle score, and attitude status to output facility status, and generates early warning records or maintenance work orders for abnormal states.

[0007] The further technical solution is as follows: the expected locations of the warning posts are recorded in the warning post expected location database. The warning post expected location database is established based on the geographic information system digital twin base and map, and records the facility number, world coordinates, road, facility type, design height, installation benchmark, historical inspection status and maintenance records; in areas where the expected locations of the warning posts have not yet been established, the locations where the warning posts have been confirmed to be in normal condition during historical inspections are used as the expected locations of the warning posts.

[0008] The further technical solution is as follows: the step of projecting its world coordinates onto the current frame road image to generate candidate regions includes: Based on the camera's intrinsic and extrinsic parameters and the vehicle's pose, the expected base mounting point corresponding to the expected location of the warning post is projected perspective onto the current frame of the road image. The top point is determined based on the design height. A candidate region is generated by combining the projection of the expected base mounting point and the top point. The candidate region is then expanded based on positioning error, map error, and camera calibration error.

[0009] The further technical solution is as follows: the target detection also determines the probability of target presence and occlusion in each frame of road image; The target presence score and the line-of-sight occlusion score are the weighted sums of the target presence probability and occlusion probability of each frame in the current inspection window, respectively. The observation coverage is the ratio of the number of effective observation frames to the number of theoretically observable frames. The weights are determined by image quality, candidate region projection error, target distance, observation angle, and vehicle speed. When the weight sum is zero, the number of theoretically observable frames is zero, or the candidate region is completely outside the image range, insufficient observation is output.

[0010] The further technical solution is as follows: the target key points include at least one of the top key points, bottom key points, and column key points; The step of recovering the three-dimensional principal axis vector of the warning column based on the target key points includes: Combining prior dimensions of the warning post, camera intrinsic parameters, road plane constraints, and bottom grounding point constraints, the target key points are restored from image coordinates to three-dimensional coordinates. Specifically, the bottom three-dimensional point is obtained by intersecting the observation line of sight corresponding to the bottom key point with the road plane. The top three-dimensional point is determined based on the bottom three-dimensional point and a preset height prior, or the three-dimensional coordinates are determined using multi-frame parallax, binocular depth, or point cloud data. The normalized vector of the difference between the top three-dimensional point and the bottom three-dimensional point is taken as the three-dimensional principal axis vector.

[0011] The further technical solution is as follows: the comparison with the normal installation reference direction determined by the vehicle's position, and the determination of the attitude state based on the included angle between the two, includes: The three-dimensional principal axis vector is compared with the normal installation reference direction determined by the vehicle pose to determine the included angle between the two. When the included angle is less than the normal posture threshold, the posture is determined to be normal; when the included angle is between the normal posture threshold and the collapse determination threshold, the tilt is determined to be suspected; when the detection results confirm the existence of the target, the confidence of the key points is not lower than the confidence threshold, the length of the three-dimensional principal axis vector is not less than the preset length threshold, and the included angle is not less than the collapse determination threshold, the collapse is determined to be abnormal.

[0012] The further technical solution is as follows: A spatiotemporal occupancy grid model is established for each expected location of the warning post, and the occupancy, vacancy, or unknown status of the grid cells is updated based on the detection results of each frame of road images, including: A spatiotemporal occupancy grid model is established for each warning post's expected location; Based on the vehicle pose and the projection, the detection results of each frame of road image are mapped to the spatiotemporal occupancy grid model. When a warning post structure is detected, the occupancy status of the corresponding grid cell is increased. When the observation is complete and no warning post structure is detected, the idle status of the corresponding grid cell is increased. When the observation line of sight is blocked by an obstruction, the unknown status is maintained or the weight of the road image in the update is reduced. Log-probability update, Bayesian update, Kalman filter, or temporal probability model is used for the update.

[0013] The further technical solution is as follows: the inertial measurement data is collected by an inertial measurement unit; The time synchronization of the road image sequence includes: Each frame of road image is bound to a timestamp, vehicle pose, camera intrinsic parameters, camera extrinsic parameters, inertial measurement unit extrinsic parameters, and image quality score; the inertial measurement data is filtered to estimate the direction of gravity; frames with image quality scores below a quality threshold, abnormal positioning, or turbulence exceeding a turbulence threshold are weighted or removed; and the data that is not removed is cached.

[0014] The further technical solution is as follows: The facility status includes normal, abnormal collapse, high confidence missing, undeterminable occlusion or insufficient observation, suspected map deviation or facility migration; when the target presence score is lower than the presence threshold, the line-of-sight occlusion score is lower than the occlusion threshold, the observation coverage is higher than the coverage threshold, the grid occupancy score is lower than the occupancy threshold and the grid idle score is higher than the idle threshold, it is determined to be high confidence missing; when the target presence score is lower than the presence threshold and the line-of-sight occupancy score is higher than the occupancy threshold, or the observation coverage is lower than the coverage threshold, it is determined to be undeterminable occlusion or insufficient observation; when the target presence score is not lower than the presence threshold and the deviation between the detected target location and the expected location of the warning post exceeds the deviation threshold, it is determined to be suspected map deviation or facility migration, and a correction suggestion for the expected location of the warning post is generated.

[0015] This invention also provides a road warning post missing and fallen detection system based on dynamic inspection, including: The data acquisition unit is used to acquire a sequence of road images collected by the vehicle during driving, and to synchronize the road image sequence in time. The road image sequence includes multiple frames of road images, positioning data and inertial measurement data. The vehicle pose is obtained by fusing the positioning data and the inertial measurement data. The site projection unit is used to retrieve the expected site of the warning post in the field of view of the current frame road image according to the vehicle pose, and project its world coordinates onto the candidate region of the current frame road image. The existence verification unit is used to perform target detection in the candidate area to determine target key points and form detection results. It establishes the current inspection window with each warning post expected location as a unit, performs time correlation on multiple frames of detection results of the same warning post expected location, and calculates the target existence score, line-of-sight occlusion score and observation coverage. The attitude determination unit is used to recover the three-dimensional principal axis vector of the warning post based on the key points of the target when the detection result confirms the existence of the target, compare it with the normal installation reference direction determined by the vehicle posture, and determine the attitude state based on the angle between the two. The grid fusion unit is used to establish a spatiotemporal occupancy grid model for each warning post's expected location. It updates the occupancy, vacancy, or unknown status of the grid unit based on the detection results of each frame of road image, and aggregates them to obtain grid occupancy scores and grid vacancy scores. The status judgment unit is used to integrate the target presence score, line-of-sight occupancy score, observation coverage, grid occupancy score, grid idle score, and attitude status output facility status, and generate early warning records or maintenance work orders for abnormal states.

[0016] The beneficial effects of this invention compared with the prior art are as follows: This invention obtains the vehicle pose by fusing positioning and inertial measurement data, projects world coordinates onto the image candidate region based on the warning post expected location library to limit the detection range, performs target detection within the candidate region and establishes a patrol window to perform temporal correlation on the detection results of multiple frames, calculates the target presence score, line-of-sight occlusion score and observation coverage, and simultaneously uses key points to recover the three-dimensional principal axis vector and compares it with the normal installation reference direction to determine the attitude state. Then, it combines the spatiotemporal occupancy grid model to aggregate occupancy and idle scores, and finally outputs the facility status by comprehensively considering the above scores and attitude state, thereby effectively distinguishing between real missing, temporarily invisible and collapsed anomalies, and solving the problems of insufficient coverage, delayed reporting, low reliability of single-frame recognition and ambiguous status determination in dynamic scenes in the prior art.

[0017] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the method for detecting missing and collapsed road warning posts based on dynamic inspection, as provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the expected location projection of the warning post provided in an embodiment of the present invention; Figure 3 A schematic diagram of the three-dimensional principal axis of the recovery warning column provided in an embodiment of the present invention; Figure 4 A schematic diagram of the updated grid cell provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the generation of a work order as provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the current frame pose determination process provided in an embodiment of the present invention; Figure 7 A schematic diagram of the inspection window and historical reference images provided in an embodiment of the present invention; Figure 8 A schematic block diagram of a road warning post missing and fallen detection system based on dynamic patrol provided in an embodiment of the present invention; Figure 9 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0022] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] Please see Figure 1 , Figure 1This is a flowchart illustrating the method for detecting missing and fallen road warning posts based on dynamic patrol, as provided in an embodiment of the present invention. This method is applied in a server. Vehicle pose is obtained by fusing positioning and inertial measurement data. World coordinates are projected onto candidate regions of the image based on a warning post expected location library to limit the detection range. Target detection is performed within the candidate regions, and a patrol window is established to perform temporal correlation on multi-frame detection results. Target presence score, line-of-sight occupancy score, and observation coverage are calculated. Simultaneously, the three-dimensional principal axis vector is recovered using key points and compared with the normal installation reference direction to determine the attitude state. Then, the occupancy and idle scores are aggregated using a spatiotemporal occupancy grid model. Finally, the facility status is output by comprehensively considering the above scores and attitude state, thereby effectively distinguishing between true missing posts, temporarily invisible posts, and fallen posts, solving the problems of insufficient coverage, delayed reporting, low reliability of single-frame recognition, and ambiguous status determination in dynamic scenes in existing technologies.

[0025] Figure 1 This is a flowchart illustrating the method for detecting missing and collapsed road warning posts based on dynamic inspection, provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110 to S160.

[0026] S110. Acquire a sequence of road images collected during vehicle operation, and synchronize the road image sequence in time. The road image sequence includes multiple frames of road images, positioning data, and inertial measurement data. The vehicle pose is obtained by fusing the positioning data and the inertial measurement data.

[0027] In this embodiment, the vehicle can utilize municipal patrol vehicles, buses, or sanitation vehicles scattered throughout the city as a mobile data collection platform, achieving high-frequency road coverage without the need for dedicated patrol routes. During vehicle operation, an onboard high-definition camera collects road images, a GPS / GNSS module collects the vehicle's position, and an inertial measurement unit collects the vehicle's acceleration, angular velocity, and attitude change data.

[0028] In this embodiment, vehicle pose refers to the position and orientation of the vehicle in a preset world coordinate system. The world coordinate system can be a universal transverse Mercator projection coordinate system or a unified urban coordinate system. The location database, high-precision map, and vehicle positioning results are all calculated under this coordinate system. The transformation T from the vehicle coordinate system to the world coordinate system can be performed. wv (t) indicates that the vehicle pose is obtained by fusing positioning data collected by GPS / GNSS and inertial measurement data collected by the inertial measurement unit. The fusion method can employ extended Kalman filtering, complementary filtering, or factor graph optimization, among other methods. Correspondingly, T vw (t)=T wv (t) -1This represents the transformation from the world coordinate system to the vehicle coordinate system. An inertial measurement unit (IMU) is a sensor module that integrates an accelerometer and a gyroscope to measure the motion state of a vehicle.

[0029] Specifically, the inertial measurement data is collected by an inertial measurement unit; The time synchronization of the road image sequence includes: Each frame of road image is bound to a timestamp t and a vehicle pose T. wv (t), camera intrinsic parameter K, camera extrinsic parameter T vc IMU external parameters T vi Image quality score Q t Among them, camera extrinsic parameters T vc T represents the transformation from the camera coordinate system to the vehicle coordinate system. cv =T vc -1 This represents the transformation from the vehicle coordinate system to the camera coordinate system, and the IMU extrinsic parameter T. vi This represents the transformation from the IMU coordinate system to the vehicle coordinate system. Since the sampling frequencies of the camera, GPS / GNSS module, and inertial measurement unit are different, time synchronization can be achieved by using the exposure time of each frame of road image as a reference, selecting the positioning data closest to the exposure time, and interpolating the inertial measurement data to align each frame of road image, positioning data, and inertial measurement data to the same time reference. The inertial measurement data is filtered to estimate the gravity direction. Frames with image quality scores below a quality threshold, abnormal positioning, or turbulence exceeding a turbulence threshold are weighted less or removed. Data that is not removed is cached and uploaded to the analysis platform. Image quality score Q t The result can be calculated by comprehensively considering the image's sharpness, exposure, and blurriness. The aforementioned time synchronization, filtering, and quality inspection processing can be completed at the acquisition end by the vehicle-mounted edge computing unit. When the vehicle is in a weak network or network outage environment, the cached data can be resumed after the network is restored, avoiding the loss of inspection data.

[0030] Time synchronization ensures that each frame of road image, positioning data, and inertial measurement data is aligned under a unified time reference, guaranteeing the accuracy of subsequent world coordinate projection. Quality inspection removes or downweights low-quality frames, reducing misjudgments caused by vehicle bumps and image blurring at the source, which helps improve the accuracy of missing and overturned detection.

[0031] S120. Based on the vehicle pose, retrieve the expected location of the warning post within the field of view of the current frame road image, and project its world coordinates onto the current frame road image to generate a candidate region.

[0032] In this embodiment, the expected locations of the warning posts are recorded in a warning post expected location database. This database is built based on a geographic information system digital twin and map, recording facility number, world coordinates, road affiliation, facility type, design height, installation reference, historical inspection status, and maintenance records. In areas where expected locations for warning posts have not yet been established, locations where warning posts have been confirmed to be in normal condition during historical inspections are used as expected locations. Confirmation of the warning post's normal condition during historical inspections means that the warning post at that location was confirmed to exist and be in normal condition during previous inspections. This ensures that all locations included in the database have actually had facilities, avoiding the mistaken inclusion of locations without facilities as expected locations. The warning post expected location database is dynamically updated based on maintenance work and the conclusions of each round of inspections. When the maintenance department installs, relocates, or removes warning posts, the location records in the database are updated accordingly, ensuring that subsequent inspections are always based on the latest facility records.

[0033] Specifically, during vehicle operation, the system determines the vehicle's position T. wv (t) Retrieve the expected locations of warning posts within the current camera's field of view. During the retrieval, the world coordinates of each point in the location database can be transmitted via T. vw (t) and T cv Transform to the camera coordinate system and determine the expected warning post locations within the camera's field of view and at a distance less than the search radius as the expected locations of the warning posts within the current frame's road image field of view. The search range can be dynamically adjusted according to vehicle speed, road grade, and positioning accuracy. For example, the search range expands accordingly when the vehicle speed is higher or the positioning error is greater.

[0034] In this embodiment, the candidate region refers to the image region in the current frame road image corresponding to the expected location of the warning post. It is used to limit the search range of target detection, so that the system only verifies the existence and orientation of the facility within the candidate region, rather than searching the entire image.

[0035] In one embodiment, step S120 described above may include steps S121 to S122.

[0036] S121. Based on the camera intrinsic parameters, camera extrinsic parameters, and the vehicle pose, project the expected base mounting point corresponding to the expected location of the warning post onto the current frame road image.

[0037] In this embodiment, based on camera intrinsic parameters, camera extrinsic parameters, and the vehicle pose, the expected base mounting point corresponding to the expected location of the warning post is perspective-projected onto the current frame road image. Specifically, for any expected location i of the warning post, its world coordinates are denoted as P. i w P i w To represent the expected mounting points recorded in the digital twin base of a geographic information system, a feasible projection relationship is: pi (t)=π(K·T_ cv ·T_ vw (t)·P_ i ^ w ), where T vw (t) represents the transformation from the world coordinate system to the vehicle coordinate system, T cv This represents the transformation from the vehicle coordinate system to the camera coordinate system, where K is the camera intrinsic parameter, π is the perspective projection function, and p... i (t) represents the predicted pixel position of the warning post in the current frame. For wide-angle or fisheye cameras, the image coordinates can be distorted before perspective projection based on the distortion parameters obtained from camera calibration to ensure the accuracy of the projection position.

[0038] S122. Determine the top point based on the design height, generate a candidate region by combining the projection of the expected base mounting point and the top point, and expand the candidate region based on positioning error, map error and camera calibration error.

[0039] Specifically, the height H of the warning column is designed based on the location database records. i Set the bottom point P i w and top point P i w +H i ·b i w Simultaneously projected into the image, where b i w The normal installation reference direction in the world coordinate system; a rectangular candidate region (ROI) is generated based on the projection results of the bottom and top points. i (t), and expand the candidate region outward according to the positioning error, map error and camera calibration error. The expansion amount can be the maximum value of the pixel projection offset caused by the above three types of errors, so that the expanded candidate region can contain the projection deviation caused by the error.

[0040] By using the above method, the search of the entire image is narrowed down to the verification within the candidate region corresponding to the expected site, which reduces the interference of complex backgrounds and similar targets on detection, while also reducing the amount of computation, which is conducive to improving the efficiency of dynamic patrol and the road coverage.

[0041] S130. Perform target detection within the candidate area to determine key points of the target and generate detection results; establish a current inspection window based on the expected location of each warning post, perform time correlation on multiple frames of detection results for the same expected location of the warning post, and calculate the target presence score, line-of-sight occlusion score, and observation coverage.

[0042] In this embodiment, the detection result refers to the set of results output after performing target detection on the candidate region, including at least the target presence probability p_ det (i,t), occlusion probability p_ occ The target region (i,t) and key points on the top, bottom, or pillar for pose estimation; optionally, the detection results may also include auxiliary features such as segmentation masks, pillar edge lines, and reflective film areas to enrich the evidence for subsequent discrimination. Target detection can be implemented by a convolutional neural network target detector, which can be trained using road image samples labeled with the location and key points of the warning pillars. The samples can cover different lighting conditions, weather, shooting angles, and partial occlusion. Target detection can also be achieved by combining traditional image features with template matching.

[0043] In this embodiment, the current inspection window refers to the time window W established on a per-prediction basis for each warning post's expected location. i This is used to aggregate multiple frames of detection results obtained during the current patrol of the same expected site. The start and end of the previous patrol window can be determined based on the time when the vehicle enters and leaves the retrieval range of the expected site. The theoretical number of observable frames can be determined by the camera frame rate and the duration of the current patrol window. Since target detection itself is performed in the candidate area corresponding to each expected site, the detection results of the same expected site in each frame naturally belong to that expected site. Based on this, the system performs time correlation on the detection results in the window and aggregates multiple frames of the same expected site under the same facility temporary identifier.

[0044] In this embodiment, the target presence score refers to the comprehensive score E that characterizes the probability of a warning bar being present at the expected location. i The line-of-sight obstruction score is a comprehensive score that characterizes the degree to which the line of sight at the expected location is blocked by obstructions such as vehicles, pedestrians, guardrails, or greenery. i Observation coverage refers to the indicator C that characterizes the adequacy of observations of the expected sites. i .

[0045] Specifically, the target detection also determines the target presence probability and occlusion probability of each frame of the road image; the target presence score and the line-of-sight occlusion score are respectively the weighted sum of the target presence probability and occlusion probability of each frame within the current inspection window, i.e., E i =Σ(w t ·p det (i,t)) / Σw t O i =Σ(w t ·p occ (i,t)) / Σw t The observation coverage is the ratio of the number of effective observation frames to the theoretically observable number of frames, i.e., C. i= Number of effective observation frames / Number of theoretically observable frames. Where, the weight w of each frame... t Determined by image quality, candidate region projection error, target distance, observation angle, and vehicle speed, it can be written as w t =w q ·w proj ·w angle ·w speed Frames with higher image quality, smaller projection error, closer target distance, more positive observation angle, and more stable vehicle speed have greater weights; each weight can be normalized to the interval between 0 and 1; target presence score, line-of-sight occlusion score, observation coverage, and each probability are all normalized to the interval between 0 and 1. When the sum of weights is zero, the theoretically observable number of frames is zero, or the candidate region is completely outside the image range, insufficient observation is output, and missing correlation score is not calculated.

[0046] Furthermore, if a valid current frame confirms the existence of a target and the pose key points are reliable, then the pose judgment process in step S140 is entered; if the target has a low score but the line-of-sight occlusion score is high or the observation coverage is insufficient, then the missing point is not directly determined, but is marked as insufficient observation or suspected occlusion; only when the target has a low score, the line-of-sight occlusion score is low and the observation coverage is high, then the missing point confirmation process is entered.

[0047] In this embodiment, the target key points include at least one of top key points, bottom key points, and column key points. Each key point carries a confidence score from the target detection output, which is used for confidence judgment in subsequent attitude determination.

[0048] By jointly characterizing the target presence score, line-of-sight occlusion score, and observation coverage, the system can distinguish between "not detected" and "not visible," avoiding false alarms due to absence based solely on single-frame detection. This helps reduce false alarms caused by occlusion and complex backgrounds.

[0049] S140. When the detection result confirms the existence of the target, the three-dimensional principal axis vector of the warning post is recovered based on the key points of the target, and compared with the normal installation reference direction determined by the vehicle posture. The attitude state is determined based on the angle between the two.

[0050] In this embodiment, the three-dimensional principal axis vector refers to the unit vector a that represents the axial direction of the warning post in three-dimensional space. i The normal installation reference direction refers to the direction in which the axis of the warning post should be located when it is installed according to design requirements and installation reference. i For warning posts that should normally be installed vertically, their direction b in the world coordinate system i wThe vector can be a vertically upward unit vector in the world coordinate system, or it can be given by the installation orientation library in the geographic information system base; after transforming it to the camera coordinate system where the three-dimensional principal axis vector is located, we get b. i c If more detailed installation orientation information is not provided, option b can also be used. i c =-g c That is, the opposite direction of gravity. Attitude status refers to the state characterizing the tilt of the warning post relative to its normal installation reference direction, including normal attitude, suspected tilt, and abnormal collapse.

[0051] Specifically, by combining prior knowledge of the warning post size, camera intrinsic parameters, road plane constraints, and bottom grounding point constraints, the target key points are restored from image coordinates to three-dimensional coordinates. The road plane used for the road plane constraints can be provided by a high-precision map, or fitted from roadside point clouds or vehicle sensor data using a random sampling consistency method, or estimated based on the vehicle pose and a preset road slope model. The bottom key points are written as homogeneous coordinates u. bottom =[x bottom y bottom ,1] T Calculate the direction r of its observation line of sight. bottom =normalize(K -1 ·u bottom Let the road surface be n. T X+d=0; if n T r bottom The value of λ is not zero and the obtained λ bottom =-d / (n T r bottom If the value is greater than 0, then the bottom three-dimensional point P is obtained. bottom c =λ bottom ·r bottom The top 3D points can be obtained through a depth sensor, multi-frame parallax, or a preset height prior. If a monocular implementation is used, the bottom 3D points are combined with the preset height prior H. i The top 3D point P is determined by searching for reprojection errors and column continuity. top cSpecifically, the search can proceed along a straight line passing through the bottom 3D point and parallel to the direction of gravity. The search criteria include the degree of overlap between the line connecting the search point and the bottom 3D point projected back into the image and the detected edge of the pillar, as well as the reprojection error of the search point constrained by a preset height. Points whose reprojection error meets the error condition and whose overlap with the pillar edge meets the overlap condition are selected as the top 3D points. Optionally, multi-frame parallax, binocular depth, or point cloud data generated by vehicle motion can be used to determine or correct the 3D coordinates of the target key points. The normalized vector of the difference between the top 3D point and the bottom 3D point is taken as the 3D principal axis vector, i.e., a. i =normalize(P top c -P bottom c ).like Figure 3 As shown, it illustrates the process of restoring the three-dimensional principal axis vector of the warning column.

[0052] In one embodiment, step S140 described above may include steps S141 to S142.

[0053] S141. Compare the three-dimensional principal axis vector with the normal installation reference direction determined by the vehicle pose, and determine the included angle between the two.

[0054] Specifically, the included angle θ i =arccos(clamp(|a i ·b i c | / (||a i ||·||b i c ||), -1, 1), where clamp means limiting the input to the range of -1 to 1 to avoid out-of-bounds errors caused by numerical errors.

[0055] S142. When the included angle is less than the normal posture threshold, the posture is determined to be normal; when the included angle is between the normal posture threshold and the collapse determination threshold, the tilt is suspected; when the detection results confirm the existence of the target, the confidence of the key points is not lower than the confidence threshold, the length of the three-dimensional principal axis vector is not less than the preset length threshold, and the included angle is not less than the collapse determination threshold, the collapse is determined to be abnormal.

[0056] Specifically, when the included angle is less than the normal posture threshold θ1, the posture is considered normal; when the included angle is between the normal posture threshold θ1 and the collapse judgment threshold θ2, a suspected tilt is determined; when the detection results confirm the existence of the target, the confidence level of the key points is not lower than the confidence level threshold, the length of the three-dimensional principal axis vector is not less than the preset length threshold, and the included angle is not less than the collapse judgment threshold θ2, a collapse anomaly is determined. The determination of a collapse anomaly does not require occlusion verification or continuous multi-frame verification as a necessary condition; multiple valid posture observation results within the current inspection window can be used to calculate the posture stability S. i Attitude stability can be characterized as the consistency of multiple angle calculation results within the current inspection window. The smaller the difference between the angle results, the higher the attitude stability, thus improving the confidence of the collapse judgment. θ1 and θ2 are configured according to the facility type, installation scenario, and maintenance standards. For example, θ1 can be set to 15° and θ2 can be set to 45°. The specific values ​​can also be adjusted according to the on-site calibration results.

[0057] Furthermore, if the confidence level of the bottom or top key points is low, the denominator of the intersection of the road planes is 0, and λ bottom If the value is not greater than 0, the length of the 3D principal axis vector is close to 0, or the angle calculation input exceeds the limit, the output pose cannot be confirmed or is insufficiently observed. If the top keypoint is not visible but the bottom keypoint and the column still have sufficient effective features, multiple column skeleton points can be used to fit the 3D principal axis vector; specifically, the 3D coordinates of multiple column skeleton points can be fitted with straight lines, for example, the least squares method can be used to fit the column axis, and the direction of the fitted line can be used as the 3D principal axis vector; if the effective features in the current image are insufficient to form a reliable 3D principal axis vector, the output pose cannot be confirmed or is insufficiently observed. The above checks are used to ensure the validity of the pose calculation, rather than requiring separate occlusion verification for each clearly visible fallen target.

[0058] like Figure 6 As shown, the left side represents the historical normal upright posture, and the right side represents the currently observed fallen posture. After confirming the presence of the warning post in the current image, the system extracts the key points of the posture and constructs a three-dimensional principal axis vector 'a'. i And calculate its relationship with the normal installation reference direction b. i c The included angle θ i When the posture evidence and threshold conditions are met, the collapse anomaly is directly output.

[0059] By comparing the three-dimensional principal axis vector with the normal installation reference direction determined by the vehicle's pose in the same three-dimensional coordinate system, the error caused by judging solely by the tilt angle of the image plane is avoided. Even if the vehicle rolls, pitches, or is traveling on a slope, the system can still restore the tilt state of the warning post relative to the real physical reference, which helps to reduce false alarms caused by bumps and complex road conditions.

[0060] S150. Establish a spatiotemporal occupancy grid model for each warning post's expected location, update the occupancy, vacancy, or unknown status of the grid cells based on the detection results of each frame of road images, and aggregate to obtain grid occupancy scores and grid vacancy scores.

[0061] In this embodiment, the spatiotemporal occupancy grid model refers to a three-dimensional grid data structure established with the world coordinates P_i^w of the expected location of the warning post as the center. The grid cells have three states: occupied, idle, or unknown, and are cumulatively updated over time as the vehicle passes through multiple frames of observation. The coverage area of ​​the grid model can be a cubic region centered on the expected location, covering a size several times the base size of the warning post. The side length of the grid cells can be configured according to the base size of the warning post and the detection accuracy. The initial state of each grid cell can be set to unknown, updated to occupied or idle with observation, and remains unknown when observation is blocked. The coverage area of ​​the grid model can be a cubic region centered on the expected location, covering a size several times the base size of the warning post. The side length of the grid cells can be configured according to the base size of the warning post and the detection accuracy.

[0062] Grid occupancy score refers to a score G that aggregates the occupancy status of several grid cells around a target location, representing the degree to which the physical space at that target location is occupied by the warning column structure. i ; Grid vacancy score refers to the score F obtained by aggregating the vacancy status of several grid cells around a target site, which characterizes the degree to which the physical space at the target site can be reliably observed to be empty. i .

[0063] Specifically, G can be obtained by weighted aggregation of the occupancy probabilities of several grid cells surrounding the expected site. i The idle probability of each grid cell is weighted and aggregated to obtain F. i The aggregate weight α of each grid cell c The aggregation weight can be determined based on the number of times the grid cell is effectively observed and the distance between the grid cell and the expected site. The more times the grid cell is effectively observed and the closer it is to the expected site, the greater its aggregation weight.

[0064] In one embodiment, step S150 described above may include steps S151 to S152.

[0065] S151. Establish a spatiotemporal occupancy grid model for the expected location of each warning post; S152. Based on the vehicle pose and the projection, the detection results of each frame of road image are mapped to the spatiotemporal occupancy grid model. When a warning post structure is detected, the occupancy status of the corresponding grid cell is increased. When the observation is complete and no warning post structure is detected, the idle status of the corresponding grid cell is increased. When the observation line of sight is blocked by an obstruction, the unknown status is maintained or the weight of the road image in the update is reduced. Log-probability update, Bayesian update, Kalman filter or temporal probability model is used during the update.

[0066] Based on the vehicle pose and the projection, the detection results of each frame of road image are mapped to the spatiotemporal occupancy grid model, such as... Figure 4 As shown. When a warning post structure is detected, the occupancy status of the corresponding grid cell is increased; when the corresponding grid cell is fully observed and no warning post structure is detected, its idle status is increased; when the observation line of the corresponding grid cell is blocked by obstructions such as vehicles, pedestrians, guardrails, or greenery, its unknown status is maintained or the weight of that frame of road image in the update is reduced.

[0067] Specifically, a log-odds update method can be used during the update: L t (c)=L (t-1) (c)+log(P(c|z t ) / (1-P(c|z t ))), P occ_grid (c)=1-1 / (1+exp(L t (c)), where c is the mesh element, z t For the current frame observation, L t (c) represents the logarithmic probability of occupancy of the grid cell, P. occ_grid (c) represents the occupancy probability of the grid cell. The idle probability of the grid cell can be obtained by subtracting the occupancy probability from one. Besides the log-odds update method, Bayesian updates, Kalman filtering, or time-series probabilistic models can also be used to update the state of the grid cells. When there are no valid observations in the grid cells surrounding the expected site, i.e., the sum of the aggregate weights of all grid cells Σα... c When the value is zero, the output is unknown, and grid occupancy score and grid idle score are not calculated.

[0068] Furthermore, the system aggregates several grid cells near the expected site to obtain a grid occupancy score G. i And grid idle rating F i Then, if the following conditions are met simultaneously within a continuous observation window: the target exists (score E) i Below the existence threshold τ det Score for visual obstruction: O i Below the occlusion threshold τ occ Observation coverage C i Above the coverage threshold τ covGrid occupancy score G i Below the occupancy threshold τ grid And the grid idle score F i Above the idle threshold τ free If so, it is confirmed that the warning post corresponding to the expected location is missing. The above thresholds are configured based on road grade, equipment accuracy, weather, and warning post type, for example, τ det It can be set to 0.30, τ occ It can be set to 0.40, τ cov It can be set to 0.70, τ grid It can be set to 0.30, τ free It can be set to 0.70, and the specific value can be adjusted after calibration of normal, occluded, and missing samples. If the target has a low score but the line-of-sight occlusion score is high, or the grid status is unknown for a long time, a missing work order will not be generated, and a suspected occlusion or pending review will be output; a high-confidence missing conclusion will only be generated when the target has a low score and the local space is sufficiently observed to be empty.

[0069] like Figure 7 As shown, Figure 7 This is a schematic diagram of the patrol window and historical reference images provided in an embodiment of the present invention. The current patrol frames t1 and t2 are representative frames obtained when the same vehicle passes the expected location of the same warning post. After the vehicle position and shooting angle change, no warning posts are detected in the candidate area, and the expected installation area remains observable. The system aggregates all valid frames in the current patrol window, and outputs a high-confidence missing value when each score meets the missing value condition. Figure 7 The historical reference images are only used to prove that a warning post previously existed at that location and are not included in the current multi-frame statistics.

[0070] By using spatial modeling with three states—occupied, idle, and unknown—the system can distinguish between three situations: missing, occluded, and insufficient observation. This avoids misreporting normal warning posts that are obscured by large vehicles as missing, which helps improve the accuracy of missing detection and the reliability of the conclusions.

[0071] S160. Based on the target presence score, line-of-sight occupancy score, observation coverage, grid occupancy score, grid idle score, and attitude status, output the facility status and generate early warning records or maintenance work orders for abnormal states.

[0072] In this embodiment, the facility status includes five categories: normal, abnormal collapse, high-confidence missing, obstruction making it undeterminable or insufficient observation, suspected map deviation, or facility migration. The decision can also be made in conjunction with historical inspection status H. i The deviation D between the detected target location and the expected location of the warning post iIn the case of a collapsed branch, evidence of existence can be obtained from the target presence probability or the confirmation result of the current frame; in the case of a missing branch, the target presence score of the current inspection window is used. Suspected tilt can be used as an intermediate state before the collapse judgment threshold, and attitude stability S... i It can be used as an enhanced indicator of the confidence level of lodging.

[0073] Specifically, rule-based thresholds are used for judgment: when the target is confirmed to exist in the current frame, the included angle is less than the normal attitude threshold, and the confidence of the key point is not lower than the confidence threshold, it is judged as normal; when the target is confirmed to exist in the current frame, the confidence of the key point is not lower than the confidence threshold, the length of the three-dimensional principal axis vector is not less than the preset length threshold, and the included angle is not less than the collapse judgment threshold, it is judged as collapse anomaly. Multiple stable attitude observation results in the current inspection window can improve the confidence of collapse judgment, but it is not a necessary condition; when the target existence score is lower than the existence threshold, the line-of-sight occupancy score is lower than the occupancy threshold, the observation coverage is higher than the coverage threshold, the grid occupation score is lower than the occupation threshold, and the grid idle score is higher than the idle threshold, it is judged as high confidence missing; when the target existence score is lower than the existence threshold and the line-of-sight occupancy score is higher than the occupancy threshold, or the observation coverage is lower than the coverage threshold, or the weight sum is zero, it is judged as occupancy cannot be judged or insufficient observation; when the target existence score is not lower than the existence threshold and the deviation between the detected target position and the expected location of the warning post exceeds the deviation threshold, it is judged as suspected map deviation or facility migration, and a correction suggestion for the expected location of the warning post is generated. The system retains key intermediate evidence during the judgment process for manual review.

[0074] In this embodiment, the early warning record refers to a structured record generated for abnormal states, used to prompt relevant personnel for review or handling; the maintenance work order refers to a structured work order generated for confirmed abnormal targets, used to arrange maintenance and handling. For example... Figure 5As shown, a work order must include at least the following basic information: facility number, anomaly type, anomaly confidence level, GPS coordinates, road location, and data collection time; on-site screenshots, candidate area projection results, current multi-frame inspection results, occlusion and coverage results, attitude angle, and grid cell occupancy status; and disposal fields such as suggested disposal level and review status. Maintenance work orders can be pushed to the maintenance management terminal. Reviewers can confirm or reject anomaly conclusions based on the inspection evidence retained in the work order. Review results can be written back to the historical inspection status and maintenance records of the warning column expected location database, forming a closed-loop management system of detection, dispatch, review, and cancellation. The recommended treatment level can be determined based on the type and confidence level of the anomaly. For example, lodging anomalies correspond to the emergency treatment level, high confidence missing corresponds to the expedited treatment level, and suspected tilting corresponds to the routine treatment level. Maintenance work orders can be pushed to the maintenance management terminal, and reviewers can confirm or reject the anomaly conclusion based on the test evidence retained in the work order. The review results can be written back to the historical inspection status and maintenance records of the warning column expected site database, forming a closed-loop management of detection, dispatch, review, and cancellation.

[0075] Through multi-dimensional state judgment and structured work order generation, the abnormal conclusions output by the system are traceable, verifiable, and can be directly connected to the maintenance process, reducing the cost of manual inspection and the delay in handling.

[0076] To achieve the objectives of this invention, various alternative solutions exist for the above steps, all of which fall within the scope of protection of this invention. Regarding data acquisition, in addition to municipal patrol vehicles, buses, or sanitation vehicles, dedicated inspection vehicles, inspection robots, drones, or roadside fixed data collection units can also be used. For positioning methods, in addition to GPS / GNSS and IMU fusion, real-time dynamic differential positioning, wheel speed odometer, visual odometer, or a fusion of instant positioning and map building technologies can also be used. Regarding the location database, the geographic information system digital twin base can be replaced with a location database constructed from high-precision maps, building information models, or historical patrol point clouds. Regarding projection, in addition to perspective projection, a distortion-corrected camera model or a fisheye camera model can also be used. Regarding scoring, weighted sums can be replaced with arithmetic mean, median, time-decayed weighted average, or the output of a trained time-series model. Regarding target detection, in addition to neural network detectors... In addition to traditional visual methods that combine edge features with template matching, depth acquisition can also be achieved using stereo vision, lidar point clouds, structured light, or time-of-flight ranging, besides monocular height priors. For mesh modeling, the 3D occupancy mesh can be replaced with a 2D planar grid or a columnar neighborhood model centered on the expected location; log-probability updates can be replaced with Bayesian updates, Kalman filtering, temporal probability models, or counting voting. For attitude references, the normal installation reference direction can be the opposite direction of gravity, a direction given by the GIS installation direction library, or the road plane normal. For state decisions, rule-based threshold decisions can be replaced with decision trees, Bayesian networks, or other interpretable classifiers, but hard criteria for missing and collapse should not be replaced by purely statistical ranking results.

[0077] The aforementioned method for detecting missing and fallen road warning posts based on dynamic patrols obtains vehicle pose by fusing positioning and inertial measurement data. Based on the expected location library of warning posts, world coordinates are projected onto candidate regions of the image to limit the detection range. Target detection is performed within the candidate regions, and a patrol window is established to perform temporal correlation on the detection results of multiple frames. The method calculates target presence score, line-of-sight occupancy score, and observation coverage. Simultaneously, the method uses key points to recover the three-dimensional principal axis vector and compares it with the normal installation reference direction to determine the attitude state. Then, it combines the occupancy and idle scores with a spatiotemporal occupancy grid model. Finally, it outputs the facility status by comprehensively considering the above scores and attitude state, thereby effectively distinguishing between true missing posts, temporarily invisible posts, and fallen posts, solving the problems of insufficient coverage, delayed reporting, low reliability of single-frame recognition, and ambiguous status determination in dynamic scenes in existing technologies.

[0078] Figure 8 This is a schematic block diagram of a road warning post missing and fallen detection system 300 based on dynamic patrol provided in an embodiment of the present invention. Figure 8As shown, corresponding to the above-described method for detecting missing and fallen road warning posts based on dynamic patrol, this invention also provides a system 300 for detecting missing and fallen road warning posts based on dynamic patrol. This system 300 includes a unit for executing the aforementioned method for detecting missing and fallen road warning posts based on dynamic patrol, and the system can be configured in a server. Specifically, please refer to... Figure 8 The road warning post missing and fallen detection system 300 based on dynamic inspection includes a data acquisition unit 301, a site projection unit 302, an existence verification unit 303, an attitude determination unit 304, a grid fusion unit 305, and a state decision unit 306.

[0079] The data acquisition unit 301 is used to acquire a sequence of road images collected during vehicle operation and to synchronize the road image sequence in time. The road image sequence includes multiple frames of road images, positioning data, and inertial measurement data. The vehicle pose is obtained by fusing the positioning data and the inertial measurement data. The site projection unit 302 is used to retrieve the expected locations of warning posts within the field of view of the current frame of the road image based on the vehicle pose and project their world coordinates onto the candidate region generated from the current frame of the road image. The existence verification unit 303 is used to perform target detection within the candidate region to determine target key points and generate detection results. A current inspection window is established for each expected warning post location, and the detection results of multiple frames for the same expected warning post location are temporally correlated to calculate the target's existence. The system includes a scoring system, a line-of-sight occupancy score, and an observation coverage system. A posture determination unit 304, when the detection results confirm the existence of a target, recovers the three-dimensional principal axis vector of the warning post based on the target's key points, compares it with the normal installation reference direction determined by the vehicle's pose, and determines the posture state based on the angle between the two. A mesh fusion unit 305, for establishing a spatiotemporal occupancy mesh model for each warning post's expected location, updates the occupancy, vacancy, or unknown state of the mesh units based on the detection results of each frame of road images, and aggregates the mesh occupancy score and mesh vacancy score. A status judgment unit 306, for comprehensively analyzing the target existence score, line-of-sight occupancy score, observation coverage, mesh occupancy score, mesh vacancy score, and posture state, outputs the facility status and generates early warning records or maintenance work orders for abnormal states.

[0080] In one embodiment, the site projection unit 302 includes: The projection subunit is used to project the expected base mounting point corresponding to the expected location of the warning post onto the current frame road image based on the camera intrinsic parameters, camera extrinsic parameters and the vehicle pose. The expansion subunit is used to determine the top point based on the design height, generate a candidate region by combining the projection of the expected base mounting point and the top point, and expand the candidate region based on the positioning error, map error and camera calibration error.

[0081] In one embodiment, the attitude determination unit 304 is used to combine the prior dimensions of the warning post, camera intrinsic parameters, road plane constraints, and bottom grounding point constraints to restore the target key points from image coordinates to three-dimensional coordinates. Specifically, the bottom three-dimensional point is obtained by intersecting the observation line of sight corresponding to the bottom key point with the road plane, and the top three-dimensional point is determined based on the bottom three-dimensional point and a preset height prior, or the three-dimensional coordinates are determined using multi-frame parallax, binocular depth, or point cloud data; the normalized vector of the difference between the top three-dimensional point and the bottom three-dimensional point is taken as the three-dimensional principal axis vector.

[0082] In one embodiment, the attitude determination unit 304 is used to compare the three-dimensional principal axis vector with the normal installation reference direction determined by the vehicle pose to determine the included angle between the two; when the included angle is less than the normal attitude threshold, the attitude is determined to be normal; when the included angle is between the normal attitude threshold and the collapse determination threshold, the tilt is determined to be suspected; when the detection results confirm that the target exists, the confidence of the key point is not lower than the confidence threshold, the length of the three-dimensional principal axis vector is not less than the preset length threshold, and the included angle is not less than the collapse determination threshold, the collapse is determined to be abnormal.

[0083] In one embodiment, the grid fusion unit 305 includes: an establishment subunit for establishing a spatiotemporal occupancy grid model for each expected location of a warning post; and an update subunit for mapping the detection results of each frame of road image to the spatiotemporal occupancy grid model based on the vehicle pose and the projection, increasing the occupancy status of the corresponding grid unit when a warning post structure is detected, increasing the idle status of the corresponding grid unit when the observation is complete and no warning post structure is detected, and maintaining an unknown status or reducing the weight of the road image in the update when the observation line of sight is blocked by an obstruction; the update uses log-probability update, Bayesian update, Kalman filtering, or a temporal probability model.

[0084] In one embodiment, the data acquisition unit 301 is used to bind a timestamp, vehicle pose, camera intrinsic parameters, camera extrinsic parameters, inertial measurement unit extrinsic parameters, and image quality score to each frame of road image; filter the inertial measurement data to estimate the gravity direction; reduce the weight or remove frames with image quality scores below a quality threshold, abnormal positioning, or turbulence exceeding a turbulence threshold; and cache the data that has not been removed.

[0085] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned road warning post missing and fallen detection system 300 based on dynamic inspection and each unit can be referred to the corresponding description in the aforementioned method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0086] The aforementioned road warning post missing and fallen detection system 300 based on dynamic inspection can be implemented as a computer program, which can, for example... Figure 9 It runs on the computer device shown.

[0087] Please see Figure 9 , Figure 9 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0088] See Figure 9 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0089] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a method for detecting missing and fallen road warning posts based on dynamic inspection.

[0090] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0091] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a method for detecting missing and fallen road warning posts based on dynamic inspection.

[0092] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0093] The processor 502 is used to run the computer program 5032 stored in the memory to implement all the steps of the method for detecting missing and fallen road warning posts based on dynamic inspection.

[0094] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0095] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0096] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform all the steps of the dynamic patrol-based method for detecting missing and fallen road warning posts.

[0097] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0098] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0099] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0100] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the system of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0101] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0102] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting missing and collapsed road warning posts based on dynamic inspection, characterized in that, include: A sequence of road images collected during vehicle operation is acquired, and the road image sequence is time-synchronized. The road image sequence includes multiple frames of road images, positioning data, and inertial measurement data. The vehicle pose is obtained by fusing the positioning data and the inertial measurement data. Based on the vehicle pose, the expected location of the warning post in the field of view of the current frame road image is retrieved, and its world coordinates are projected onto the current frame road image to generate a candidate region. Target detection is performed within the candidate area to determine key points of the target and generate detection results; a current inspection window is established for each warning post's expected location as a unit, and the detection results of multiple frames for the same warning post's expected location are temporally correlated to calculate the target presence score, line-of-sight occlusion score, and observation coverage; When the detection result confirms the existence of the target, the three-dimensional principal axis vector of the warning post is recovered based on the key points of the target and compared with the normal installation reference direction determined by the vehicle pose. The attitude state is determined based on the angle between the two. A spatiotemporal occupancy grid model is established for each warning post's expected location. The occupancy, vacancy, or unknown status of the grid cells is updated based on the detection results of each frame of road images. The grid occupancy score and grid vacancy score are then aggregated. The system integrates the target presence score, line-of-sight occupancy score, observation coverage, grid occupancy score, grid idle score, and attitude status to output facility status, and generates early warning records or maintenance work orders for abnormal states.

2. The method for detecting missing and collapsed road warning posts based on dynamic inspection according to claim 1, characterized in that, The expected locations of the warning posts are recorded in the warning post expected location database, which is established based on the geographic information system digital twin base and map, and records the facility number, world coordinates, road, facility type, design height, installation benchmark, historical inspection status and maintenance records; in areas where the expected locations of the warning posts have not yet been established, the locations where the warning posts have been confirmed to be in normal condition during historical inspections are used as the expected locations of the warning posts.

3. The method for detecting missing and collapsed road warning posts based on dynamic inspection according to claim 2, characterized in that, The step of projecting its world coordinates onto the current frame road image to generate candidate regions includes: Based on the camera's intrinsic and extrinsic parameters and the vehicle's pose, the expected base mounting point corresponding to the expected location of the warning post is projected perspective onto the current frame of the road image. The top point is determined based on the design height. A candidate region is generated by combining the projection of the expected base mounting point and the top point. The candidate region is then expanded based on positioning error, map error, and camera calibration error.

4. The method for detecting missing and collapsed road warning posts based on dynamic inspection according to claim 1, characterized in that, The target detection also determines the probability of target presence and occlusion in each frame of the road image; The target presence score and the line-of-sight occlusion score are the weighted sums of the target presence probability and occlusion probability of each frame in the current inspection window, respectively. The observation coverage is the ratio of the number of effective observation frames to the number of theoretically observable frames. The weights are determined by image quality, candidate region projection error, target distance, observation angle, and vehicle speed. When the weight sum is zero, the number of theoretically observable frames is zero, or the candidate region is completely outside the image range, insufficient observation is output.

5. The method for detecting missing and collapsed road warning posts based on dynamic inspection according to claim 1, characterized in that, The target key points include at least one of the top key points, bottom key points, and column key points; The step of recovering the three-dimensional principal axis vector of the warning column based on the target key points includes: Combining prior dimensions of the warning post, camera intrinsic parameters, road plane constraints, and bottom grounding point constraints, the target key points are restored from image coordinates to three-dimensional coordinates. Specifically, the bottom three-dimensional point is obtained by intersecting the observation line of sight corresponding to the bottom key point with the road plane. The top three-dimensional point is determined based on the bottom three-dimensional point and a preset height prior, or the three-dimensional coordinates are determined using multi-frame parallax, binocular depth, or point cloud data. The normalized vector of the difference between the top three-dimensional point and the bottom three-dimensional point is taken as the three-dimensional principal axis vector.

6. The method for detecting missing and collapsed road warning posts based on dynamic inspection according to claim 1, characterized in that, The comparison with the normal installation reference direction determined by the vehicle's position, and the determination of the attitude state based on the included angle, includes: The three-dimensional principal axis vector is compared with the normal installation reference direction determined by the vehicle pose to determine the included angle between the two. When the included angle is less than the normal posture threshold, the posture is determined to be normal; when the included angle is between the normal posture threshold and the collapse determination threshold, the tilt is determined to be suspected; when the detection results confirm the existence of the target, the confidence of the key points is not lower than the confidence threshold, the length of the three-dimensional principal axis vector is not less than the preset length threshold, and the included angle is not less than the collapse determination threshold, the collapse is determined to be abnormal.

7. The method for detecting missing and collapsed road warning posts based on dynamic inspection according to claim 1, characterized in that, The process of establishing a spatiotemporal occupancy grid model for each expected location of a warning post, and updating the occupancy, vacancy, or unknown status of grid cells based on the detection results of each frame of road images, includes: A spatiotemporal occupancy grid model is established for each warning post's expected location; Based on the vehicle pose and the projection, the detection results of each frame of road image are mapped to the spatiotemporal occupancy grid model. When a warning post structure is detected, the occupancy status of the corresponding grid cell is increased. When the observation is complete and no warning post structure is detected, the idle status of the corresponding grid cell is increased. When the observation line of sight is blocked by an obstruction, the unknown status is maintained or the weight of the road image in the update is reduced. Log-probability update, Bayesian update, Kalman filter, or temporal probability model is used for the update.

8. The method for detecting missing and collapsed road warning posts based on dynamic inspection according to claim 1, characterized in that, The inertial measurement data is collected by the inertial measurement unit; The time synchronization of the road image sequence includes: Each frame of road image is bound to a timestamp, vehicle pose, camera intrinsic parameters, camera extrinsic parameters, inertial measurement unit extrinsic parameters, and image quality score; the inertial measurement data is filtered to estimate the direction of gravity; frames with image quality scores below a quality threshold, abnormal positioning, or turbulence exceeding a turbulence threshold are weighted or removed; and the data that is not removed is cached.

9. The method for detecting missing and collapsed road warning posts based on dynamic inspection according to claim 1, characterized in that, The facility status includes normal, abnormal collapse, high confidence missing, undeterminable occlusion or insufficient observation, suspected map deviation or facility migration; when the target presence score is lower than the presence threshold, the line-of-sight occlusion score is lower than the occlusion threshold, the observation coverage is higher than the coverage threshold, or the grid occupancy score is lower than the occupancy threshold and the grid idle score is higher than the idle threshold, it is judged as high confidence missing; when the target presence score is lower than the presence threshold and the line-of-sight occupancy score is higher than the occupancy threshold, or the observation coverage is lower than the coverage threshold, it is judged as undeterminable occupancy or insufficient observation; when the target presence score is not lower than the presence threshold and the deviation between the detected target location and the expected location of the warning post exceeds the deviation threshold, it is judged as suspected map deviation or facility migration, and a correction suggestion for the expected location of the warning post is generated.

10. A road warning post missing and fallen detection system based on dynamic inspection, characterized in that, include: The data acquisition unit is used to acquire a sequence of road images collected by the vehicle during driving, and to synchronize the road image sequence in time. The road image sequence includes multiple frames of road images, positioning data and inertial measurement data. The vehicle pose is obtained by fusing the positioning data and the inertial measurement data. The site projection unit is used to retrieve the expected site of the warning post in the field of view of the current frame road image according to the vehicle pose, and project its world coordinates onto the candidate region of the current frame road image. The existence verification unit is used to perform target detection in the candidate area to determine target key points and form detection results. It establishes the current inspection window with each warning post expected location as a unit, performs time correlation on multiple frames of detection results of the same warning post expected location, and calculates the target existence score, line-of-sight occlusion score and observation coverage. The attitude determination unit is used to recover the three-dimensional principal axis vector of the warning post based on the key points of the target when the detection result confirms the existence of the target, compare it with the normal installation reference direction determined by the vehicle posture, and determine the attitude state based on the angle between the two. The grid fusion unit is used to establish a spatiotemporal occupancy grid model for each warning post's expected location. It updates the occupancy, vacancy, or unknown status of the grid unit based on the detection results of each frame of road image, and aggregates them to obtain grid occupancy scores and grid vacancy scores. The status judgment unit is used to integrate the target presence score, line-of-sight occupancy score, observation coverage, grid occupancy score, grid idle score, and attitude status output facility status, and generate early warning records or maintenance work orders for abnormal states.

Images