Signboard defect detection method and device, electronic equipment and storage medium

Abstract

The application discloses a sign defect detection method and device, electronic equipment and a storage medium, and belongs to the technical field of image detection. The method comprises the following steps: acquiring a road scene image comprising a target sign, wherein the target sign comprises a panel and a stand; performing validity detection on the panel according to the geometric features of the panel in the road scene image to obtain a panel validity detection result; in the case that the panel validity detection result represents that the validity detection is passed, performing inclination detection on the stand according to the road scene image with the panel as a reference to obtain a stand inclination detection result; and determining a defect detection result of the target sign according to the panel validity detection result and the stand inclination detection result. The application realizes the sign defect detection with the panel geometric features of the sign as a reference and without the dependence on the ground plane reference, and improves the accuracy of the sign defect detection.

Description

Technical Field

[0001] This application belongs to the field of image detection technology, and in particular relates to a method, apparatus, electronic device and storage medium for detecting defects in signs. Background Technology

[0002] Rectangular panel traffic signs with upright posts are a core component of road traffic safety facilities. Their posts are constantly exposed to the outdoor environment, making them highly susceptible to tilting or bending deformation due to factors such as vehicle collisions, strong wind loads, foundation settlement, and material fatigue. This can cause the sign panels to become skewed, twisted, or even detached, severely affecting drivers' ability to recognize road signs and endangering driving safety. Existing solutions (such as traffic light pole detection) rely on the ground plane as a reference point, which becomes unreliable on slopes, curves, and uneven road surfaces, leading to decreased detection accuracy. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the related art. To this end, this application proposes a method, apparatus, electronic device, and storage medium for detecting sign defects, which realizes sign defect detection that can be performed with reference to the geometric features of the sign panel and without relying on a ground plane reference, thereby improving the accuracy of sign defect detection.

[0004] In a first aspect, this application provides a method for detecting defects in signs, the method comprising: Acquire a road scene image including a target sign, the target sign including a panel and a post; The panel is validated based on its geometric features in the road scene image to obtain the panel validation result; If the panel validity detection result indicates that the validity detection is passed, the tilt detection of the column is performed based on the road scene image, with the panel as a reference, to obtain the column tilt detection result. Based on the panel validity test results and the column tilt test results, the defect test results of the target sign are determined.

[0005] In the above technical solution, the panel's validity is checked by performing self-verification and calibration on the panel using the geometric features of the panel in the road scene image, and the corresponding detection result of the panel is obtained. After completing the panel reference validity, the tilt detection of the column is performed based on the relative structure of the panel and the column. Through the pre-panel validity verification, the reference error caused by panel deformation or tilt is avoided, and the column tilt detection is completed under the premise of reliable panel reference. Then, the overall defect detection result of the target sign can be determined based on the detection results of the panel and the column. Compared with related technologies that use the ground plane as a reference reference, there is a problem of unreliable reference reference in the scenarios of slopes, curves and uneven road surfaces. By using the panel geometric features to complete the reference self-verification and self-calibration to build a reliable reference and complete the column tilt detection, the defect detection of the sign can be performed without relying on the ground plane reference, which can be performed with the geometric features of the sign panel as a reference. This improves the accuracy and adaptability of sign detection in complex road scenarios such as no standard reference and inconsistent shooting posture.

[0006] Secondly, this application provides a sign defect detection device, the device comprising: The acquisition module is used to acquire a road scene image including a target sign, the target sign including a panel and a post; A panel detection module is used to perform validity detection on the panel based on the geometric features of the panel in the road scene image, and obtain the panel validity detection result; The column detection module is used to perform tilt detection on the column based on the road scene image, with the panel as a reference, when the panel validity detection result indicates that the validity detection has passed, and to obtain the column tilt detection result. The determination module is used to determine the defect detection result of the target sign based on the panel validity detection result and the column tilt detection result.

[0007] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the sign defect detection method as described in the first aspect above.

[0008] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sign defect detection method as described in the first aspect above.

[0009] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the sign defect detection method as described in the first aspect.

[0010] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the sign defect detection method as described in the first aspect above.

[0011] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0012] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is one of the flowcharts illustrating a sign defect detection scheme provided in some embodiments of this application; Figure 2 This is a schematic diagram of panel distortion provided in some embodiments of this application; Figure 3 This is a schematic diagram illustrating the tilt detection of a column provided in some embodiments of this application; Figure 4 This is a second schematic flowchart of a sign defect detection method provided in some embodiments of this application; Figure 5 This is one of the schematic diagrams of a sign defect detection system provided in some embodiments of this application; Figure 6 This is a second schematic diagram of a sign defect detection system provided in some embodiments of this application; Figure 7 This is a schematic diagram of the structure of a sign defect detection device provided in some embodiments of this application; Figure 8 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments that can be directly obtained based on the embodiments of this application are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0015] The following description, in conjunction with the accompanying drawings, details the sign defect detection method, apparatus, electronic device, and storage medium provided in this application through specific embodiments and application scenarios.

[0016] Among them, the sign defect detection method can be applied to the terminal, and can be executed by the hardware or software in the terminal.

[0017] The sign defect detection method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the sign defect detection method. The following description uses an electronic device as the execution subject to illustrate the sign defect detection method provided in this application embodiment.

[0018] Figure 1 This is one of the flowcharts illustrating a sign defect detection scheme provided in some embodiments of this application. For example... Figure 1 As shown, the defect detection method for signs includes steps 110, 120, 130 and 140.

[0019] Step 110: Obtain a road scene image including the target sign, wherein the target sign includes a panel and a post.

[0020] As an example, road scene images can be acquired in real time by image acquisition devices such as road inspection vehicle cameras and fixed monitoring cameras, including at least sign panels and posts. Alternatively, road scene images corresponding to target signs can be read in batches by accessing a historical inspection image database.

[0021] In some embodiments, the method for acquiring road scene images is determined according to the road scene type, and road scene images including target signs are acquired according to the acquisition method corresponding to the road scene type.

[0022] As an example, road scene types can include highway scenes, expressway scenes, and provincial highway scenes. In highway scenes, road scene images can be automatically acquired through monitoring cameras mounted on gantry frames. In national highway scenes, road scene images can be acquired through vehicle-mounted cameras on inspection vehicles. In provincial highway scenes, manual inspection can be conducted to assist in acquiring road scene images through handheld terminals. Road scene images are acquired based on the environmental characteristics of the real-world environment, adapting to actual road scenes rather than controlled shooting scenarios.

[0023] Step 120: Perform validity detection on the panel based on the geometric features of the panel in the road scene image to obtain the panel validity detection result.

[0024] The geometric features of a sign panel refer to the geometric attributes of the sign panel as represented by a road scene image. Examples include the geometric features of the panel corners and the geometric features of the outline. These features can be used to reflect information such as the regularity of the panel's shape and its spatial orientation.

[0025] Panel validity testing is a compliance verification performed on the panel. In this application embodiment, in addition to performing routine defect detection on the panel, validity testing also includes determining whether the panel can be used as a measurement benchmark for tilt detection of the column.

[0026] In some embodiments, validating the panel based on its geometric features in the road scene image includes: The panel outline is fitted based on the panel's geometric features. The degree of panel distortion is quantified by the deviation of the panel's interior corners to determine whether the panel has deformed. The panel's baseline edge is extracted based on the panel outline. Combined with scene vertical references or historical steady-state data, the overall posture of the panel is verified to check whether it has tilted.

[0027] In this embodiment, the reliability of the pre-verification benchmark is achieved by performing validity detection on the panel, thus avoiding misjudgment of the column detection that might result from using a deformed or skewed panel as the measurement benchmark.

[0028] Step 130: If the panel validity detection result indicates that the validity detection has passed, using the panel as a reference, perform tilt detection on the pillar according to the road scene image to obtain the pillar tilt detection result.

[0029] Column tilt detection is used to detect the relative attitude deviation of the column with respect to the current effective panel reference.

[0030] In some embodiments, using a panel that has passed the validity test as a reference, the straight line features of the column edge are extracted, and the relative tilt angle between the column and the panel reference is calculated; it is also possible to combine multiple frames of continuous road scene images or historical panel references to count the long-term tilt offset of the column relative to the panel and identify the gradual tilt of the column.

[0031] Using the panel as a reference, the tilt of the pillar is detected based on the road scene image to obtain the pillar tilt detection result. Compared with the conventional method of using external global references such as the ground or camera coordinate system as references, which are unreliable in scenarios such as slopes, curves, and uneven road surfaces, this embodiment uses a reference constructed based on the geometric posture of the panel itself as a measurement reference. This allows the tilt detection of the pillar to be completed based solely on the structure of the sign itself, which helps to solve the problems of poor scene adaptability and uncontrollable reference.

[0032] Step 140: Determine the defect detection result of the target sign based on the panel validity detection result and the column tilt detection result.

[0033] In some embodiments, the method further includes: generating maintenance decision information that matches the defect type based on the defect detection results, thereby achieving differentiated operation and maintenance.

[0034] For example, the defect detection results of the target sign may include panel distortion detection results, panel tilt detection results, and column tilt detection results; if the defect detection result is that the panel itself is twisted or deformed, the maintenance decision information includes replacing the panel; if the defect detection result is that the panel is tilted as a whole, the maintenance decision information includes triggering a benchmark re-inspection; if the defect detection result is that the panel is tilted due to the column tilt, the maintenance decision information includes straightening the column, etc.

[0035] Furthermore, since image processing technology can be used to complete sign defect detection (such as extracting the edge lines of the posts in the road scene image using the Canny algorithm, extracting vertical lines using the Hough transform, and detecting panel corners using the Harris algorithm), no deep learning hardware support is required, and ordinary industrial control computers can run it in real time. Data can be collected using existing surveillance cameras, eliminating the need to install physical sensors on each sign, thus reducing actual deployment and execution costs.

[0036] The sign defect detection method provided in this application uses the geometric features of the panel in the road scene image to perform panel self-verification and calibration for validity detection, and obtains the corresponding detection result of the panel. After completing the panel reference validity, the tilt detection of the column is performed based on the relative structure of the panel and the column. Through the pre-panel validity verification, the reference error caused by panel deformation or tilt is avoided, and the column tilt detection is completed under the premise of reliable panel reference. Then, the overall defect detection result of the target sign can be determined based on the detection results of the panel and the column. Compared with the related technology that uses the ground plane as a reference reference, there is a problem of unreliable reference reference in the scenarios of slope, curve and uneven road surface. The method uses the panel geometric features to complete the reference self-verification and self-calibration to build a reliable reference and complete the column tilt detection. It realizes the sign defect detection that can be performed without relying on the ground plane reference, using the geometric features of the sign panel as a reference. This improves the accuracy and adaptability of sign detection in complex road scenarios such as no standard reference and inconsistent shooting posture.

[0037] In some embodiments of this application, the validity detection results include at least panel distortion detection results and panel tilt detection results; The step of performing validity detection on the panel based on the geometric features of the panel in the road scene image to obtain the panel validity detection result includes: The panel is twisted based on its geometric features to obtain the panel twist detection result; If the panel distortion detection result indicates that the distortion detection has passed, a panel baseline representing the panel orientation is determined. The panel is tilted based on the panel baseline to obtain the panel tilt detection result; If the panel tilt detection result indicates that the tilt detection has passed, the panel validity detection result is determined to be passed.

[0038] In this embodiment, the panel distortion detection result is the panel shape regularity detection result determined based on the panel geometric features, which can be used to characterize whether the sign panel has undergone geometric deformation such as stretching or warping.

[0039] Figure 2 This is a schematic diagram of panel distortion provided in some embodiments of this application. For example... Figure 2As shown, taking a rectangular panel (the panel can also be triangular, etc.) as an example, for a panel without distortion (a normal panel), its four interior angles should all be 90° in a road scene image. However, for a distorted panel, its interior angles usually deviate from 90°. Taking a triangular panel as an example, for a panel without distortion (a normal panel), its three interior angles should all be 60° in a road scene image. However, for a distorted panel, its aspect ratio and parallelism parameters may change. Therefore, if the panel's own geometry is damaged or distorted, it is necessary to immediately terminate the use of that panel as a reference and subsequent inspection processes. This allows for the early removal of deformation defects in the panel itself, avoiding the generation of invalid references by distorted panels.

[0040] In some embodiments, the degree of panel distortion is quantified by calculating the deviation of each interior angle of the sign panel in the road scene image from the corresponding standard angle; alternatively, parameters such as the aspect ratio and parallelism of opposite sides of the panel can be determined based on the panel outline, and combined with the structural constraints that the panel should meet, to determine whether the panel has non-rigid torsional deformation and obtain the distortion detection result.

[0041] The panel baseline is a reference line segment generated based on the panel passing the torsion detection, which can be used to characterize the overall posture of the panel. As an example, when the sign panel is rectangular, the longitudinal side line segment of the panel can be directly used as the panel baseline, or the central axial baseline of the panel can be determined by fitting the upper and lower contour endpoints of the panel. When the sign panel is triangular, the line connecting the vertex of the panel and the midpoint of the bottom edge can be used as the panel baseline. This embodiment does not specifically limit this. Unlike external fixed references such as the ground, the panel baseline is constructed based on the panel's own structure and changes synchronously with the posture of the panel, adapting to any shooting angle or any road slope scenario, which can solve the dependence of sign defect identification on ground plane reference.

[0042] The purpose of performing tilt detection on the panel based on the panel baseline is to determine whether the overall installation posture of the sign is tilted, thereby avoiding the problem that the panel surface is not twisted, but the panel is tilted as a whole, which would cause the baseline to tilt and fail.

[0043] As an example, one could identify natural vertical reference objects such as utility poles or building edges in a road scene image and compare them with the panel baseline to perform tilt detection. Alternatively, one could retrieve the baseline of the target sign in its historical qualified state and compare it with the current panel baseline to perform tilt detection.

[0044] The sign defect detection method provided in this application performs validity detection on the panel, including panel distortion detection and panel tilt detection. If the panel distortion detection passes, a panel baseline is generated and the panel tilt is checked. The panel deformation defect and the overall tilt of the panel are checked sequentially for two types of baseline failure. Only after the double-layer inspection is passed can the panel baseline be used for subsequent column tilt detection. The distortion detection is based on the contour of the sign panel itself as an intrinsic geometric reference, and the tilt detection does not require the ground plane as a reference. This realizes the defect detection of the sign panel based on the geometric features of the sign panel and can be performed without relying on the ground plane reference.

[0045] In some embodiments of this application, the geometric features include corner features; the step of performing distortion detection on the panel based on the geometric features of the panel to obtain the panel distortion detection result includes: The geometric contour of the panel is fitted based on the corner features, and the angle values ​​of each interior corner of the panel are determined based on the geometric contour. The distortion of the panel is determined based on the deviation between the angle values ​​of each interior angle and their corresponding preset angle values. If the degree of distortion does not exceed a preset degree of distortion threshold, the panel distortion detection result is determined to be passed.

[0046] Corner points are points in an image where grayscale intensity changes discontinuously, often described as extreme points or edge intersections. They are typically image feature points corresponding to the vertices of a panel. In some embodiments, the Harris corner detection algorithm is used to detect grayscale changes to identify the corner or corner positions of the panel. Corner features refer to the pixel features of each corner point of the panel in a road scene image.

[0047] Taking a rectangular panel as an example, the panel's geometric contour is a closed quadrilateral outline obtained by fitting the four corner points of the panel. This can be used to reconstruct the rectangular panel's shape boundary in a road scene. For example, a corner detection algorithm can be used to extract the four vertices of the panel and connect them in sequence to generate a closed quadrilateral contour. Then, the angle between adjacent vectors of the four sides can be calculated based on the vector angle formula to obtain the four interior angles.

[0048] The preset angle values ​​are the angles that each inner corner of a normal, defect-free panel should be at. For example, the preset angle value for each inner corner of a defect-free panel is 90°. The preset angle values ​​can be adjusted to suit the actual shape of the sign.

[0049] In some embodiments, taking a rectangular panel as an example, quadrilateral contours are fitted to the four corner points of the panel to calculate the four interior angles α1, α2, α3, and α4; the panel's twist degree D is defined as Σ|α i 90°| / 4; The preset distortion threshold can be set to 5 degrees. If the distortion degree D>5°, the panel is determined to have distortion deformation, and the panel distortion detection result is not passed. The preset distortion threshold is the preset upper limit of deformation allowance. If the distortion degree is lower than this upper limit, the panel can be determined to pass the distortion detection and can be used to generate panel baselines.

[0050] By calculating the average deviation of each interior angle and treating the average deviation as the overall panel distortion, the impact of random noise from a single corner on the overall distortion detection can be reduced. The distortion measure quantifies the degree of deviation between each actual interior angle of the sign panel and its corresponding standard interior angle. The larger the distortion value, the more severe the distortion, such as panel deformation, warping, or corner offset.

[0051] The sign defect detection method provided in this application fits the geometric contour of the panel based on corner features and determines the angle value of each interior corner of the panel. Then, it quantifies the degree of distortion by comparing the deviation between the actual interior corner and the standard interior corner. Finally, it compares the result with a preset threshold to determine whether the panel passes the distortion detection. It only identifies four corner points, resulting in low overall image processing overhead. By performing distortion detection on the panel to intercept deformed panels, it prevents the generation of invalid baselines from distorted contours, effectively ensuring the accuracy of subsequent column tilt detection.

[0052] In some embodiments of this application, the step of performing tilt detection on the panel based on the panel baseline to obtain the panel tilt detection result includes: Based on the vertical structures in the road scene image other than the target sign, determine the scene vertical line corresponding to the road scene image; If the angle between the panel baseline and the scene vertical line does not exceed the first angle threshold, the panel tilt detection result is determined to be passed.

[0053] Vertical lines in a scene refer to straight lines defined by vertical structures other than signs within a road scene image. Common vertical structures in road scene images include lampposts, guardrail posts, building sides, tree trunks, and other solid structures that are nearly perpendicular to the ground in real-world scenarios. These can be used as external vertical reference standards when the ground is uneven.

[0054] For example, determining the scene vertical line corresponding to a road scene image can be done by filtering vertical straight line segments in the road scene image, removing the lines of the sign itself, and taking the straight line with the highest confidence as the scene vertical line; or by extracting multiple candidate vertical structure straight lines, calculating the average direction of the multiple straight lines, and fitting the scene vertical line to avoid reference errors caused by the tilt of a single vertical structure.

[0055] The offset angle between the panel baseline and the scene's vertical line represents the degree of tilt of the sign panel relative to the actual vertical direction. The first angle threshold is a pre-set upper limit for the allowable tilt angle, such as 2°; if the angle is less than this threshold, the panel tilt detection is considered successful, and the panel baseline can be used as a reference.

[0056] In some embodiments, the method further includes: if the panel tilt detection result is passed, recording and storing the angle between the current panel baseline and the scene vertical line to provide raw data for subsequent time-series comparison and long-term attitude monitoring.

[0057] The sign defect detection method provided in this application, after the panel undergoes distortion inspection to generate a qualified panel baseline, extracts scene vertical lines based on vertical structures other than signs in the road scene, and compares the angle between the panel baseline and the scene vertical lines; when the tilt angle is within the allowable range, the overall posture of the panel is qualified and the baseline is valid; when there are vertical components such as guardrails, light poles, and buildings in the road scene image, there is no need to add additional calibration reference objects or use the ground plane as a reference, which improves the accuracy and adaptability of sign detection in complex road scenes such as unfixed shooting postures.

[0058] In some embodiments of this application, determining the scene vertical line corresponding to the road scene image based on other vertical structures in the road scene image besides the target sign includes: Extract multiple straight lines from the road scene image, and filter out straight lines that are within a preset vertical angle range from the multiple straight lines to obtain a candidate vertical line set; Exclude straight lines within the panel area and pillar area of ​​the target sign from the candidate vertical line set to obtain the scene vertical line set; The average slope of each straight line in the set of vertical lines in the scene is taken as the slope of the vertical line in the scene.

[0059] The preset vertical angle range corresponds to the angle interval of a straight line in the real vertical direction; it is used to filter candidate vertical lines that are approximately vertical line segments in the image coordinate system of road scene images, and the specific settings can be combined with the actual application scenario.

[0060] The candidate vertical line set is a collection of line segments whose direction is close to vertical after angle filtering. For example, the Hough line detection algorithm can be used to detect lines in a road scene image, obtain parameters such as the inclination angle and slope of each line segment, filter line segments whose inclination angle falls into the vertical angle range, and integrate them to obtain the candidate vertical line set.

[0061] The scene vertical line set consists of candidate straight lines formed by environmental vertical structures after excluding straight lines within the sign body area. This effectively avoids mistaking the side of the sign pillar or the vertical edge of the panel as environmental vertical lines, and prevents tilt detection failure caused by the reference vertical line and the panel baseline being from the same source.

[0062] In some embodiments, the average slope of multiple environmental vertical lines in the scene vertical line set is calculated and fitted to obtain a denoised standard reference vertical line to represent the vertical orientation of the real scene. As an example, one could directly calculate the arithmetic mean of the slopes of all valid scene vertical lines and construct a standard scene vertical line for comparison based on the average slope; another could be to first remove abnormal straight lines with excessively large slope deviations from the scene vertical line set to avoid the influence of a skewed reference.

[0063] Since the slope of the scene's vertical lines is obtained by averaging multiple environmental vertical lines rather than relying on a single reference object, even if there is a slight tilt in a single vertical structure such as a railing or tree trunk in the road, the error caused by a single object can be offset by averaging, resulting in a more accurate vertical reference standard.

[0064] Here is a specific example: The Hough transform is used to extract all lines that are close to the vertical direction, excludes the lines of the panel itself and the columns, retains the lines of other vertical structures, and calculates the average slope to obtain the slope k_cam of the vertical lines of the scene. Calculate the angle β between the panel baseline slope k_ref and the scene vertical line slope k_cam: β=arctan(|(k_ref k_cam) / (1+k_ref·k_cam)|) If the included angle β > 2°, the panel itself is determined to be tilted and cannot be used as a reference for tilt detection of the upright.

[0065] The sign defect detection method provided in this application generates a standard scene vertical line by averaging the slopes of multiple external vertical lines corresponding to other vertical structures besides the target sign. This standard line is then used for subsequent tilt angle comparison with the panel baseline. By averaging the directions of multiple environmental vertical structures, compared to related technologies that rely on the ground plane or a single reference object as a tilt detection reference, this solution ensures that the reference object is independent of the target sign by excluding certain areas and averaging the values. Furthermore, the averaging process offsets the errors caused by a single object, improving the accuracy of determining the slope of the scene vertical line and thus contributing to the reliability of panel tilt detection.

[0066] In some embodiments of this application, the step of performing tilt detection on the panel based on the panel baseline to obtain the panel tilt detection result includes: Using the panel baseline as a reference, the relative tilt angle of the column with respect to the panel baseline is determined based on the road scene image; During the historical defect detection process of the target sign, the historical relative tilt angle is determined with reference to the historical panel baseline. If the difference between the relative tilt angle and the historical relative tilt angle does not exceed the second angle threshold, the panel tilt detection result is determined to be passed.

[0067] The relative tilt angle of the column represents the degree of tilt of the column relative to the panel (panel baseline); and in this embodiment, the panel baseline is generated based on the panel that has passed the validity test. The panel distortion does not exceed the preset distortion threshold and the angle between the panel baseline and the vertical line of the scene does not exceed the first angle threshold. Therefore, the relative tilt angle can further represent the degree of tilt of the column relative to the road scene.

[0068] For example, one could extract straight lines from both sides of the column and fit a reference line for the column center, then calculate the angle between the column center line and the panel baseline as the real-time relative tilt angle; or one could extract multiple column edge segments, calculate the average slope of multiple column straight lines, and then calculate the angle between the average slope and the panel baseline to reduce noise interference caused by stains and paint chips on the column edges.

[0069] The historical panel baseline is the panel baseline determined during historical tilt detection of the panel, and the historical relative tilt angle is the relative tilt angle determined during historical tilt detection of the panel. As an example, the tilt parameters corresponding to each panel can be stored in a local or cloud database, including parameters such as the baseline slope and relative tilt angle determined in each tilt detection.

[0070] The second angle threshold is a preset angle deviation range, which can be used to constrain the maximum acceptable offset between the current attitude and the historical attitude of the panel.

[0071] If the difference between the relative tilt angle and the historical relative tilt angle does not exceed the second angle threshold, the panel tilt detection result is determined to be passed. That is, using historical detection data (historical relative tilt angle) of the sign as the detection standard is more suitable for the inherent slight tilt angle of the sign's installation, rather than forcing the sign to strictly meet the geographical vertical orientation, thus better reflecting the actual road scenario. If the panel's attitude offset does not exceed an acceptable range, the panel tilt detection result can be considered passed.

[0072] The time-series pole tilt verification provided in this application embodiment does not rely on the ground plane or other vertical objects in the external environment. By comparing with the pole's own historical data, it determines whether the current panel baseline has undergone overall deflection.

[0073] It is understood that the scheme for tilt detection of the panel based on historical panel baselines and historical relative tilt angles provided in this application embodiment, and the scheme for tilt detection of the panel based on the vertical structure in the road scene image, can be executed in parallel and can be performed simultaneously to conduct dual tests.

[0074] Here is a specific example: Once the panel baseline is valid during the first tilt test of the panel, or the accurate panel baseline is confirmed by manual calibration, determine and save the relative tilt angle θ of the column relative to the panel baseline. When performing subsequent tilt detection on the panel, the difference between the current relative tilt angle θ_current and the historical relative tilt angle θ_history is calculated: Δθ=|θ_current θ_history| If the difference between the relative tilt angle and the historical relative tilt angle does not exceed the second angle threshold (e.g., 1°) |Δθ|≤1°, then the panel baseline is considered valid. For panels with multiple historical data sets, sliding window trend analysis can be used instead of single threshold comparison to further improve reliability.

[0075] The sign defect detection method provided in this application first calculates the relative tilt angle of the current post relative to the panel baseline, then compares it with historical relative tilt angles to determine the offset difference. If the angle fluctuation is less than a second angle threshold, the current panel baseline posture is determined to be stable and the tilt detection is qualified. Even when no usable vertical structure exists in the road scene image, the panel baseline posture can still be verified based on historical time-series data, broadening the applicability of sign defect detection. The scheme for tilt detection based on historical panel baselines and historical relative tilt angles, and the scheme for tilt detection based on vertical structures in the road scene image, complement each other, ensuring the reliability of the panel validity detection process. In some embodiments of this application, the step of using the panel as a reference to perform tilt detection on the pillar based on the road scene image to obtain the pillar tilt detection result includes: Extract the edge lines of the pillars in the road scene image; The tilt angle of the column is determined based on the angle between the panel baseline and the edge line of the column. The column tilt detection result is determined based on the comparison between the column tilt angle and the third angle threshold.

[0076] Figure 3 This is a schematic diagram illustrating the tilt detection of a column according to some embodiments of this application. For example... Figure 3 As shown, the column edge line is the vertical outline of the sign column extracted from the road scene image, used to characterize the column's spatial posture and orientation. The column tilt angle is the relative angle between the column edge line and the panel baseline that has passed the validity detection, used to quantify the degree of tilt between the column and the panel.

[0077] The third angle threshold is a preset upper limit for column tilt deviation, used to determine whether the column has a structural tilt defect. If the relative tilt angle θ between the panel baseline and the edge line of the column does not exceed the third angle threshold, the column is determined to be in a normal state, and the column tilt detection result is passed; if the relative tilt angle exceeds θ and exceeds the third angle threshold, the column is determined to be tilted, and the column tilt detection result is failed.

[0078] The sign defect detection method provided in this application extracts the edge line of the post and calculates the relative tilt angle between the post and the panel baseline. It then automatically determines the tilt defect of the post by combining the calculated angle threshold. The entire process is based on the structural relationship between the sign panel and the post to complete the relative detection of the post. It does not need to rely on the ground plane as a reference and is not affected by complex working conditions such as road surface undulations and camera installation deviations. This solves the problems of poor adaptability and easy false detection in related technologies that rely on the ground plane. Furthermore, since the panel contour is extracted synchronously by using the panel self-calibration reference mechanism, the post detection and panel detection share the same reference reference and the same input image, resulting in strong data consistency.

[0079] Figure 4 This is a second schematic flowchart of a sign defect detection method provided in some embodiments of this application. For example... Figure 4 As shown, the sign defect detection method also includes: Acquire road scene images containing target signs The imaging quality of the acquired images is checked; if the images are unqualified, the image acquisition process is repeated. If the image is qualified, extract the four corner points of the sign panel, fit the geometric contour of the panel quadrilateral, solve for each interior angle, and calculate the panel distortion. The panel distortion is compared with a preset distortion threshold. If the panel distortion exceeds the preset distortion threshold, the defect is determined to be panel distortion. If the panel distortion does not exceed the preset distortion threshold, perform image edge detection after preprocessing the image; Extract the effective corner points of the panel, fit the side contour of the panel, and generate a panel baseline representing the orientation of the panel; Two optional methods are used to detect the baseline tilt angle of the panel: filter the scene vertical lines of the road environment and calculate the angle between the panel baseline and the scene vertical lines as the baseline tilt angle; or, retrieve the historical tilt angle of the sign and calculate the offset difference between the current relative tilt angle and the historical value as the baseline tilt angle. Compare the tilt angle of the baseline with the first angle threshold. If the tilt angle of the baseline exceeds the first angle threshold, the defect is determined to be panel tilt. If the angle of inclination of the baseline does not exceed the first angle threshold, extract the edge line of the column; Calculate the column tilt angle between the column edge line and the qualified panel baseline, compare the tilt angle with the third angle threshold, and complete the column tilt detection. The overall defect detection of the sign was completed by integrating the panel twist detection results, panel tilt detection results, and column tilt detection results.

[0080] In some embodiments of this application, the method further includes performing damage detection on the panel; the damage detection on the panel includes: After obtaining the geometric contour of the panel, an edge density analysis is performed within the panel area defined by the geometric contour to obtain the edge pixel ratio. The edge pixel ratio is compared with a preset damage threshold to obtain the panel damage detection result; Specifically, if the proportion of edge pixels is greater than the preset damage threshold, the panel is determined to be damaged.

[0081] This application embodiment determines the detection area based on the fitted panel geometric contour, calculates the edge pixel density and edge pixel ratio by statistically analyzing the panel's internal edge pixel density, and combines it with a preset damage threshold to quantitatively determine whether the panel has cracks, peeling, or breakage, thus realizing damage detection for sign panels.

[0082] In some embodiments of this application, the defect detection results of the target sign include at least the panel distortion detection results, the panel tilt detection results, and the column tilt detection results.

[0083] In some embodiments, when the panel is subjected to damage detection, the defect detection results of the target sign also include the panel damage detection results.

[0084] In some embodiments of this application, determining the defect detection result of the target sign based on the validity detection result of the panel and the defect detection result of the column includes: If the panel validity test fails and the panel distortion test result indicates that the test fails, then the defect test result of the target sign is determined to be the distortion and deformation of the panel itself. If the panel validity test fails, and the panel distortion test result indicates that the test passed but the panel tilt test result indicates that the test failed, then the defect test result of the target sign is determined to be that the panel is tilted as a whole or the panel baseline is drifted. If the panel validity test passes and the column tilt test result indicates that the column is tilted, then the defect test result of the target sign is determined to be that the tilt of the column causes the panel to be crooked. If the panel validity test passes and the column tilt test result indicates that the column is not tilted, then the defect test result of the target sign is determined to be that the sign is in normal condition. Wherein, if the panel damage detection result indicates that damage exists, the defect detection result also includes panel damage information.

[0085] This application embodiment can differentiate between various types of signs, such as panel distortion, overall panel tilt / baseline drift, and column tilt linkage; and can be combined with panel damage detection results to realize a multi-dimensional defect judgment system including structural deformation, posture deviation, assembly abnormality, and appearance damage; it constructs a comprehensive quantitative evaluation index system, improves the objectivity of sign condition assessment, and helps support subsequent scientific maintenance decisions.

[0086] In some embodiments of this application, the method further includes: generating maintenance decision information that matches the defect type based on the defect detection results.

[0087] As an example, if the defect detection result is that the panel itself is twisted or deformed, the maintenance decision information includes replacing the panel; if the defect detection result is that the panel is tilted as a whole or the panel baseline is drifted, the maintenance decision information includes triggering a baseline re-inspection; if the defect detection result is that the panel is tilted due to the column tilting, the maintenance decision information includes straightening the column; if the defect detection result also includes panel damage information, the maintenance decision information also includes repairing or replacing the panel.

[0088] Based on defect detection results, maintenance decision information is generated for the target signage. This intelligent generation of maintenance decisions and full automation of the process from image acquisition to defect identification and decision generation is achieved. Furthermore, automatically recording and storing each detection result in a database allows for long-term trend analysis of the column / panel status, transforming passive alarms into proactive warnings. This enables maintenance to be scheduled in advance before severe bending deterioration, thus supporting predictive maintenance.

[0089] Figure 5This is one of the schematic diagrams of a sign defect detection system provided in some embodiments of this application. For example... Figure 5 As shown, the sign defect detection system is divided into three layers, from top to bottom: data acquisition layer, image recognition layer, and application layer. The data acquisition layer is used to collect on-site images of road signs through various inspection terminal devices and transmit the image data to the image recognition layer. Specifically, video streams can be collected through road monitoring cameras and transmitted in the form of RTSP video streams, road scene images can be collected through vehicle-mounted inspection cameras and transmitted via USB or 4G-5G, and road scene images can be collected through handheld inspection terminals and transmitted via 5G or Wi-Fi. The image recognition layer includes a panel status detection module and a column verticality detection module. The panel comprehensive status detection module sequentially performs panel torsion detection, panel tilt verification, and panel damage detection to complete the detection of all defects in the sign panel. The column tilt detection module calculates the relative tilt angle of the column based on the verified panel baseline to identify column tilt defects. The application layer corresponds to the sign maintenance management system, which receives defect detection results from the image recognition layer and performs business functions such as defect classification and storage, disease recording, inspection ledger management, and maintenance alarm push.

[0090] Figure 6 This is a second schematic diagram of a sign defect detection system provided in some embodiments of this application. For example... Figure 6 As shown, the sign defect detection system is also equipped with a display interface. The left side of the interface has a function menu bar for managing inspection tasks, historical inspection records, maintenance work orders, and defect statistical analysis. The interface includes a GIS map section for marking the locations of signs; The right side of the interface displays the complete inspection parameters of the selected sign, including the column tilt angle, panel twist, panel damage results, overall operating status, and the most recent inspection time. The inspection record table at the bottom of the interface stores multiple historical inspection records. Based on the results of various inspection indicators, it marks the status as normal or alarm, so that maintenance personnel can carry out defect detection and maintenance work on the signs.

[0091] The sign defect detection method provided in this application can be executed by a sign defect detection device. This application uses the example of a sign defect detection device executing the sign defect detection method to illustrate the sign defect detection device provided in this application.

[0092] Figure 7 This is a schematic diagram of the structure of a sign defect detection device provided in some embodiments of this application. For example... Figure 7As shown, the sign defect detection device 700 includes: The acquisition module 701 is used to acquire a road scene image including a target sign, wherein the target sign includes a panel and a post; The panel detection module 702 is used to perform validity detection on the panel based on the geometric features of the panel in the road scene image, and obtain the panel validity detection result; The column detection module 703 is used to perform tilt detection on the column based on the road scene image, with the panel as a reference, when the panel validity detection result indicates that the validity detection is passed, and to obtain the column tilt detection result. The determination module 704 is used to determine the defect detection result of the target sign based on the panel validity detection result and the column tilt detection result.

[0093] In some embodiments, the validity detection results include at least panel distortion detection results and panel tilt detection results; the panel detection module 702 is used for: The panel is twisted based on its geometric features to obtain the panel twist detection result; If the panel distortion detection result indicates that the distortion detection has passed, a panel baseline representing the panel orientation is determined. The panel is tilted based on the panel baseline to obtain the panel tilt detection result; If the panel tilt detection result indicates that the tilt detection has passed, the panel validity detection result is determined to be passed.

[0094] In some embodiments, the geometric features include corner features; the step of performing distortion detection on the panel based on the geometric features of the panel to obtain the panel distortion detection result includes: The geometric contour of the panel is fitted based on the corner features, and the angle values ​​of each interior corner of the panel are determined based on the geometric contour. The distortion of the panel is determined based on the deviation between the angle values ​​of each interior angle and their corresponding preset angle values. If the degree of distortion does not exceed a preset degree of distortion threshold, the panel distortion detection result is determined to be passed.

[0095] In some embodiments, the step of performing tilt detection on the panel based on the panel baseline to obtain the panel tilt detection result includes: Based on the vertical structures in the road scene image other than the target sign, determine the scene vertical line corresponding to the road scene image; If the angle between the panel baseline and the scene vertical line does not exceed the first angle threshold, the panel tilt detection result is determined to be passed.

[0096] In some embodiments, determining the scene vertical line corresponding to the road scene image based on other vertical structures in the road scene image besides the target sign includes: Extract multiple straight lines from the road scene image, and filter out straight lines that are within a preset vertical angle range from the multiple straight lines to obtain a candidate vertical line set; Exclude straight lines within the panel area and pillar area of ​​the target sign from the candidate vertical line set to obtain the scene vertical line set; The average slope of each straight line in the set of vertical lines in the scene is taken as the slope of the vertical line in the scene.

[0097] In some embodiments, the step of performing tilt detection on the panel based on the panel baseline to obtain the panel tilt detection result includes: Using the panel baseline as a reference, the relative tilt angle of the column with respect to the panel baseline is determined based on the road scene image; During the historical defect detection process of the target sign, the historical relative tilt angle is determined with reference to the historical panel baseline. If the difference between the relative tilt angle and the historical relative tilt angle does not exceed the second angle threshold, the panel tilt detection result is determined to be passed.

[0098] In some embodiments, the column detection module 703 is used for: Extract the edge lines of the pillars in the road scene image; The tilt angle of the column is determined based on the angle between the panel baseline and the edge line of the column. The column tilt detection result is determined based on the comparison between the column tilt angle and the third angle threshold.

[0099] The sign defect detection device in this application embodiment can be an electronic device or a component of an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific type of device.

[0100] The sign defect detection device in this application embodiment can be a device with an operating system. This operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems; this application embodiment does not specifically limit it.

[0101] The sign defect detection device provided in this application embodiment can realize all the processes implemented in the above-described sign defect detection method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0102] Figure 8 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application. In some embodiments, such as Figure 8 As shown, this application embodiment also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, it implements the various processes of the above-described sign defect detection method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0103] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0104] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described sign defect detection method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0105] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0106] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described sign defect detection method.

[0107] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0108] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described sign defect detection method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0109] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0110] It should be noted that the information (including but not limited to road scene images), data (including but not limited to data used for analysis, data stored, and data displayed) and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0111] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0113] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0114] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0115] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for detecting defects in signboards, characterized in that, The method includes: Acquire a road scene image including a target sign, the target sign including a panel and a post; The panel is validated based on its geometric features in the road scene image to obtain the panel validation result; If the panel validity detection result indicates that the validity detection is passed, the tilt detection of the column is performed based on the road scene image, with the panel as a reference, to obtain the column tilt detection result. Based on the panel validity test results and the column tilt test results, the defect test results of the target sign are determined.

2. The method according to claim 1, characterized in that, The validity test results include at least the panel distortion test results and the panel tilt test results; The step of performing validity detection on the panel based on the geometric features of the panel in the road scene image to obtain the panel validity detection result includes: The panel is twisted based on its geometric features to obtain the panel twist detection result; If the panel distortion detection result indicates that the distortion detection has passed, a panel baseline representing the panel orientation is determined. The panel is tilted based on the panel baseline to obtain the panel tilt detection result; If the panel tilt detection result indicates that the tilt detection has passed, the panel validity detection result is determined to be passed.

3. The method according to claim 2, characterized in that, The geometric features include corner features; The step of performing distortion detection on the panel based on its geometric features to obtain the panel distortion detection result includes: The geometric contour of the panel is fitted based on the corner features, and the angle values ​​of each interior corner of the panel are determined based on the geometric contour. The distortion of the panel is determined based on the deviation between the angle values ​​of each interior angle and their corresponding preset angle values. If the degree of distortion does not exceed a preset degree of distortion threshold, the panel distortion detection result is determined to be passed.

4. The method according to claim 2 or 3, characterized in that, The process of performing tilt detection on the panel based on the panel baseline to obtain the panel tilt detection result includes: Based on the vertical structures in the road scene image other than the target sign, determine the scene vertical line corresponding to the road scene image; If the angle between the panel baseline and the scene vertical line does not exceed the first angle threshold, the panel tilt detection result is determined to be passed.

5. The method according to claim 4, characterized in that, The step of determining the scene vertical line corresponding to the road scene image based on other vertical structures in the road scene image besides the target sign includes: Extract multiple straight lines from the road scene image, and filter out straight lines that are within a preset vertical angle range from the multiple straight lines to obtain a candidate vertical line set; Exclude straight lines within the panel area and pillar area of ​​the target sign from the candidate vertical line set to obtain the scene vertical line set; The average slope of each straight line in the set of vertical lines in the scene is taken as the slope of the vertical line in the scene.

6. The method according to claim 2 or 3, characterized in that, The process of performing tilt detection on the panel based on the panel baseline to obtain the panel tilt detection result includes: Using the panel baseline as a reference, the relative tilt angle of the column with respect to the panel baseline is determined based on the road scene image; During the historical defect detection process of the target sign, the historical relative tilt angle is determined with reference to the historical panel baseline. If the difference between the relative tilt angle and the historical relative tilt angle does not exceed the second angle threshold, the panel tilt detection result is determined to be passed.

7. The method according to claim 2, characterized in that, The step of using the panel as a reference to perform tilt detection on the pillar based on the road scene image to obtain the pillar tilt detection result includes: Extract the edge lines of the pillars in the road scene image; The tilt angle of the column is determined based on the angle between the panel baseline and the edge line of the column. The column tilt detection result is determined based on the comparison between the column tilt angle and the third angle threshold.

8. A sign defect detection device, characterized in that, include: The acquisition module is used to acquire a road scene image including a target sign, the target sign including a panel and a post; A panel detection module is used to perform validity detection on the panel based on the geometric features of the panel in the road scene image, and obtain the panel validity detection result; The column detection module is used to perform tilt detection on the column based on the road scene image, with the panel as a reference, when the panel validity detection result indicates that the validity detection has passed, and to obtain the column tilt detection result. The determination module is used to determine the defect detection result of the target sign based on the panel validity detection result and the column tilt detection result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the sign defect detection method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the sign defect detection method as described in any one of claims 1 to 7.

Images