A method and system for diagnosing occlusion lighting based on the spatial relationship between roadside trees and streetlights

Abstract

This invention discloses a method and system for diagnosing occlusion lighting based on the spatial relationship between roadside trees and streetlights. The method includes: acquiring a nighttime road inspection image sequence; identifying light fixtures, branches, and road surface areas; tracking the same streetlight across frames to form an observation sequence; extracting the illuminated area; establishing a pairing relationship between light fixtures and illuminated areas and constructing the light path influence area; determining direct occlusion features and light path occlusion features; establishing a normal lighting baseline; calculating the image illumination retention rate of the illuminated area of ​​the road surface and the light fixture brightness retention rate; by fusing the image illumination retention rate, light fixture brightness retention rate, and direct occlusion features and light path occlusion features, distinguishing between tree occlusion, lamp abnormalities, and combined faults, and generating diagnostic results; and outputting diagnostic records after confirming the consistency of multi-frame and multi-angle diagnostic results for the same streetlight. This invention enables the identification of direct occlusion and light path occlusion, quantifies lighting loss, distinguishes different fault causes, and outputs structured diagnostic results.

Description

Technical Field

[0001] This invention relates to computer vision, and more specifically to a method and system for diagnosing occlusion lighting based on the spatial relationship between roadside trees and streetlights. Background Technology

[0002] With the continuous advancement of urban road greening, the branches and leaves of roadside trees are gradually encroaching on the vicinity of streetlight heads and the area illuminated by the lights. This can cause obstruction of the light-emitting surface of the streetlights or a reduction and fragmentation of the illuminated area on the road surface, leading to insufficient road lighting and the formation of dark areas, affecting nighttime traffic safety. This problem is gradual and seasonal, and is particularly prominent after trees grow vigorously, have not been pruned for a long time, or after extreme weather. To address these issues, current road lighting inspections mainly rely on two methods: manual inspection and streetlight on / off status recognition. Manual inspection is inefficient and highly subjective, making it difficult to cover large-scale road networks; on / off status recognition can only determine whether the lights are on, and cannot distinguish between light malfunctions and tree obstruction. In recent years, some visual inspection methods have been introduced into this field. These methods analyze the two-dimensional overlap relationship between streetlight heads and branches and leaves based on single-frame image analysis. While this has improved detection efficiency to some extent, it is still affected by factors such as vehicle movement, ambient light interference, and branch swaying, and lacks stable correlation analysis between streetlight heads, trees, and the road lighting area, resulting in insufficient detection reliability.

[0003] The aforementioned existing technologies still have several significant shortcomings when dealing with dynamic nighttime inspection scenarios. At the target association level, existing visual detection methods are mostly based on single-frame images or local target recognition, making it difficult to continuously associate the same streetlight head, adjacent branches and leaves, and the corresponding illuminated area of ​​the road surface while the vehicle is moving. Furthermore, interference from vehicle lights, advertising lights, building lights, and road surface reflections in the nighttime environment leads to prominent false positives and false negatives. At the spatial relationship analysis level, existing solutions typically only detect the two-dimensional overlap between branches and leaves and the streetlight head, failing to establish spatial mapping relationships between the streetlight head, branches and leaves, the area affected by the light path, and the illuminated area of ​​the road surface. This makes it difficult to identify occlusion caused by branches and leaves directly covering the streetlight head, and also difficult to detect illumination attenuation caused by branches and leaves not covering the streetlight head but encroaching on the light path propagation area. At the quantitative evaluation level, due to differences in streetlight models, installation heights, light distribution methods, observation distances, and road surface reflection conditions, existing technologies struggle to establish lighting benchmarks applicable to different scenarios, making it impossible to objectively and comparablely quantify the actual lighting loss caused by occlusion. At the level of fault differentiation, existing methods are unable to make a joint judgment by comprehensively considering multi-dimensional information such as lamp head brightness, lamp head coverage, degree of light path obstruction, and abnormal road lighting. They cannot accurately distinguish between tree obstruction, lamp malfunction, and compound problems of both, and it is also difficult to form diagnostic results that can directly support maintenance decisions.

[0004] Therefore, it is necessary to design a new method to identify direct occlusion and optical path occlusion, quantify lighting loss, distinguish different causes of failure, and output structured diagnostic results that can be used for trimming, maintenance, and operation and maintenance management. This method can effectively overcome the shortcomings of existing technologies in dynamic inspection environments, such as unstable target association, lack of spatial relationship analysis, lack of quantitative assessment mechanism for lighting loss, and insufficient ability to distinguish the causes of failure. It can also improve the accuracy, reliability, and decision support capabilities of nighttime road lighting inspection. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for diagnosing occlusion lighting based on the spatial relationship between roadside trees and streetlights.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for diagnosing shading lighting based on the spatial relationship between roadside trees and streetlights, comprising: Acquire image sequences for nighttime road inspections; Identify the lamp head region, branch and leaf region, and road surface region in the image sequence, and perform cross-frame tracking on the same street lamp to obtain the observation sequence; Extract the illuminated area of ​​the road surface from the road surface area, establish the pairing relationship between the lamp head area and the illuminated area of ​​the road surface, and construct the light path influence area; The direct occlusion feature is determined based on the overlap relationship between the branch and leaf area and the lamp head area, and the optical path occlusion feature is determined based on the overlap relationship between the branch and leaf area and the optical path influence area. Establish a normal lighting benchmark and determine the image illumination retention rate of the illuminated area of ​​the road surface and the lamp head brightness retention rate of the lamp head area; Based on the direct occlusion features, the optical path occlusion features, the image illumination retention rate, and the lamp head brightness retention rate, by fusing the image illumination retention rate, the lamp head brightness retention rate, the direct occlusion features, and the optical path occlusion features, tree occlusion, lamp abnormalities, and combined faults of the two are distinguished to form a diagnostic result. The diagnostic results for the same street light are confirmed to be consistent across multiple frames and different observation angles, and diagnostic records are output.

[0007] The further technical solution is as follows: the identification of the lamp head region, branch and leaf region, and road surface region in the image sequence, and the cross-frame tracking of the same street lamp to obtain the observation sequence, includes: A detection model is used to identify the lamp head region in the image sequence, and a semantic segmentation model is used to identify the branch and leaf region and the road surface region; Based on the appearance characteristics, movement trajectory, optical flow continuity and spatial position relationship of the lamp head area, a unique street lamp identifier is assigned to the same street lamp, and an observation sequence is formed from the moment the same street lamp enters the image field of view to the moment it leaves the image field of view.

[0008] The further technical solution is as follows: Extracting the illuminated area of ​​the road surface in the road surface area, establishing a pairing relationship between the lamp head area and the illuminated area of ​​the road surface, and constructing the light path influence area includes: Within the road surface area, adaptive threshold segmentation is used to extract connected regions with brightness higher than the road surface background to obtain the illuminated area of ​​the road surface. Based on the spatial nearest neighbor relationship and relative positional relationship between the lamp head area and each road surface illuminated area, the road surface illuminated area corresponding to the lamp head area is determined; when the lamp head area corresponds to multiple road surface illuminated areas, the multiple road surface illuminated areas are merged to establish a unique pairing relationship; A trapezoidal approximate region is constructed using the edges of the lamp head area and the illuminated area of ​​the road surface that are close to each other, and the trapezoidal approximate region is expanded. When the trapezoidal approximate regions corresponding to adjacent lamp head areas overlap, the ownership of the overlapping part is determined according to the spatial nearest neighbor relationship between the overlapping part and each lamp head area, thus obtaining the light path influence area. When no illuminated area of ​​the road surface is extracted in the current image frame, a predicted illuminated area of ​​the road surface is generated based on the location and shape of the illuminated area of ​​the road surface in the historical observation frames corresponding to the same street light identifier, and the light path influence area is constructed.

[0009] A further technical solution is as follows: determining the direct occlusion feature based on the overlap relationship between the branch and leaf area and the lamp head area includes: The branch and leaf area and the lamp head area are converted into branch and leaf mask and lamp head mask, respectively. The ratio of the overlap area of ​​the branch and leaf mask and the lamp head mask to the area of ​​the lamp head mask is calculated to obtain the lamp head coverage rate. When the lamp head coverage rate is not lower than the lamp head coverage threshold, and there is a local brightness attenuation in the lamp head area corresponding to the boundary space of the branch and leaf mask, it is determined that the branch and leaf area directly occludes the lamp head area, and the lamp head coverage rate and the local brightness attenuation are used as the direct occlusion features.

[0010] A further technical solution is as follows: determining the optical path occlusion characteristics based on the overlap relationship between the branch and leaf region and the optical path influence region includes: The optical path intrusion rate is obtained by calculating the ratio of the overlap area between the branch and leaf mask and the area of ​​the optical path affected region to the area of ​​the optical path affected region. When the lamp head coverage rate is lower than the lamp head coverage threshold, the light path intrusion rate is not lower than the light path intrusion threshold, and the illuminated area of ​​the road surface shows a decrease in area or a decrease in edge continuity relative to the previous observation frame corresponding to the same street lamp sign, it is determined that the branch and leaf area forms a light path occlusion in the light path affected area, and the light path intrusion rate, the area change characteristics of the illuminated area of ​​the road surface, and the edge continuity change characteristics are used as the light path occlusion characteristics.

[0011] Its further technical solution is: the establishment and comparison of the normal lighting reference includes: Streetlights of the same type and type on the same road, and whose lamps are confirmed to be in good working order and without obstruction by branches and leaves, are selected as normal lighting samples to establish benchmarks for the illuminated area, lamp head brightness, and illuminated area shape. When the number of normal lighting samples does not reach the preset number, a preset initial benchmark is adopted; when the number of normal lighting samples reaches the preset number, the normal lighting benchmark is updated according to the statistical results of the normal lighting samples. Based on the observation frames with illuminated or predicted road surface areas, the ratio of the average illuminated area area to the illuminated area reference is calculated to obtain the image illumination retention rate. Based on the observation frames that can identify the lamp head area, the ratio of the average brightness of the lamp head area to the lamp head brightness reference is calculated to obtain the lamp head brightness retention rate.

[0012] The further technical solution is as follows: the diagnostic result, which distinguishes between tree obstruction, abnormal lighting fixtures, and combined faults of both, includes: When there is direct obstruction or optical path obstruction, and the lamp head brightness retention rate is lower than the lamp head brightness threshold, a composite fault result is output. When there is no direct obstruction or optical path obstruction, and the lamp head brightness retention rate is lower than the lamp head brightness threshold, an abnormal lamp result is output. When the lamp head brightness retention rate is not lower than the lamp head brightness threshold and there is direct occlusion, output the direct occlusion lamp head result; when the lamp head brightness retention rate is not lower than the lamp head brightness threshold and there is optical path occlusion, output the optical path occlusion result; when the direct occlusion lamp head result or the optical path occlusion result is output and the image illumination retention rate is lower than the illumination retention threshold, determine that the corresponding tree occlusion caused effective illumination loss; When there is no direct occlusion or optical path occlusion, and the lamp head brightness retention rate is not lower than the lamp head brightness threshold, if the image illumination retention rate is lower than the illumination retention threshold, output the illumination deficiency result caused by non-occlusion; otherwise, output the no-occlusion result. When the lamp head area cannot be identified, or when neither the current image frame nor the historical observation frame can determine the illuminated area of ​​the road surface, an insufficient evidence result is output.

[0013] The further technical solution is as follows: The process of confirming the consistency of diagnostic results for the same streetlight across multiple frames and different observation angles, and outputting diagnostic records, includes: For an observation sequence with the same street light identifier, when a preset number of consecutive observation frames obtain the same diagnostic result, and the same diagnostic result remains consistent at at least two different observation angles, the diagnostic result is confirmed to be valid. Obtain vehicle positioning information and acquisition time corresponding to the observation frame, and determine the street light position based on the tracking trajectory, vehicle positioning information and acquisition time of the same street light; when the branch and leaf areas corresponding to multiple consecutive street lights have spatial continuity in consecutive frames and the cross-frame tracking results are correlated, determine that the branch and leaf areas belong to the same tree canopy, and merge the corresponding multiple consecutive street lights into the same affected road segment; The diagnostic record includes street light identification, street light location, acquisition time, and diagnostic category; when generating branch and leaf masks and lamp head masks, the lamp head coverage rate is recorded; when constructing the light path influence area, the light path intrusion rate is recorded; when determining or predicting the illuminated area of ​​the road surface, the image illumination retention rate is recorded; when identifying the lamp head area, the lamp head brightness retention rate is recorded; and the original road inspection image, occlusion feature calculation results, and retention rate calculation results are associated and saved.

[0014] The further technical solution is as follows: the image sequence is collected by the vehicle during driving, including multiple frames of road inspection images and the collection time, vehicle positioning information and vehicle heading information corresponding to each frame of road inspection images; The road inspection images, vehicle positioning information, and vehicle heading information of each frame are aligned according to the acquisition time, and distortion correction, noise suppression, brightness normalization, and local contrast enhancement are performed on each frame of road inspection images. Image frames that are motion-blurred, overexposed, or obscured by vehicles ahead are removed. The remaining consecutive road inspection images are then registered and fused, and perspective normalization is performed on the road surface area in the fused image.

[0015] This invention also provides an obstruction lighting diagnosis system based on the spatial relationship between roadside trees and streetlights, comprising: Image acquisition unit, used to acquire nighttime road inspection image sequences; The area recognition and street light tracking unit is used to identify the lamp head area, branch and leaf area and road surface area in the nighttime road inspection image sequence, and to perform cross-frame tracking of the same street light to obtain the corresponding observation sequence; The optical path influence area construction unit is used to extract the illuminated area of ​​the road surface in the road surface area, establish the pairing relationship between the lamp head area and the illuminated area of ​​the road surface, and construct the optical path influence area between the two. The occlusion feature determination unit is used to determine direct occlusion features based on the overlap relationship between the branch and leaf area and the lamp head area, and to determine optical path occlusion features based on the overlap relationship between the branch and leaf area and the optical path influence area. A reference establishment unit is used to establish a normal lighting reference, determine the image illumination retention rate of the illuminated area of ​​the road surface relative to the normal lighting reference, and the lamp head brightness retention rate of the lamp head area relative to the normal lighting reference; The fault diagnosis unit is used to fuse the direct occlusion features, the optical path occlusion features, the image illumination retention rate, and the lamp head brightness retention rate. By fusing the image illumination retention rate, the lamp head brightness retention rate, and the direct occlusion features and the optical path occlusion features, it distinguishes between tree occlusion, lamp malfunction, and composite faults where tree occlusion and lamp malfunction coexist, and forms a diagnostic result. The consistency confirmation unit is used to confirm the consistency of the diagnostic results of the same street light in multiple frames and at different observation angles, and output the diagnostic record.

[0016] The beneficial effects of this invention compared to existing technologies are as follows: This invention identifies lamp heads, branches, and road surface areas through nighttime inspection image sequences, tracks the same street light across frames to stabilize target association, extracts the illuminated area of ​​the road surface and establishes a pairing relationship between lamp heads and illuminated areas, constructs the light path influence area to establish spatial mapping; it identifies direct occlusion based on the mask overlap between branches and lamp heads and local brightness attenuation, and identifies light path occlusion based on the overlap between branches and light path influence areas and changes in the shape of the illuminated area; it quantifies lighting loss by calculating the image illumination retention rate and lamp head brightness retention rate using normal lighting benchmarks, and performs fusion discrimination to distinguish tree occlusion, lamp abnormalities, and compound faults; finally, after multi-frame, multi-angle consistency confirmation, it outputs a structured diagnostic record containing location, category, coverage, intrusion rate, retention rate, and associated images, thereby effectively overcoming the shortcomings of existing technologies in dynamic inspection environments, such as unstable target association, lack of spatial relationship analysis, lack of quantitative assessment mechanism for lighting loss, and insufficient ability to distinguish fault causes, thus improving the accuracy, reliability, and decision support capabilities of nighttime road lighting inspection.

[0017] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a flowchart illustrating the occlusion lighting diagnosis method based on the spatial relationship between roadside trees and streetlights provided in an embodiment of the present invention. Figure 2 A schematic diagram illustrating the spatial relationship between the lamp head, branches and leaves, the area affected by the light path, and the illuminated area of ​​the road surface, as well as a scenario of light path obstruction, provided for embodiments of the present invention; Figure 3 A schematic diagram of a direct occlusion feature provided in an embodiment of the present invention; Figure 4 This is a statistical chart of cross-frame tracking and illuminated area sequence of the same street light provided in an embodiment of the present invention; Figure 5 A schematic block diagram of an obstruction lighting diagnosis system based on the spatial relationship between roadside trees and streetlights, provided in an embodiment of the present invention; Figure 6 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0022] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] Please see Figure 1 , Figure 1This is a flowchart illustrating a method for diagnosing occlusion lighting based on the spatial relationship between roadside trees and streetlights, provided in an embodiment of the present invention. This method is applied in a server. Multi-frame road image sequences were acquired through nighttime vehicle patrols. After distortion correction, brightness normalization, and registration fusion preprocessing, a detection model was used to identify the lamp head region, and a semantic segmentation model was used to extract the branch and road surface regions. Based on appearance features, motion trajectories, and optical flow continuity, the same street light was tracked across frames to form an observation sequence. In the road surface region, the illuminated area was extracted using adaptive threshold segmentation. Pairing was established with the lamp head according to spatial nearest neighbor relationships. A trapezoidal approximate region was constructed based on the edges of the lamp head and the illuminated area. After dilation and overlap assignment processing, the optical path influence region was formed. At the same time, the illuminated area of ​​missing frames was predicted using historical observations. The percentage of the overlapping area between the branch and leaf and the lamp head mask was used as the lamp head coverage rate. Direct occlusion was determined by combining local brightness attenuation. The percentage of the overlapping area between the branch and leaf and the optical path influence region was used as the optical path intrusion rate. Optical path occlusion was determined by combining the reduction of the illuminated area area and the decrease in edge continuity. The illuminated area area, lamp head brightness, and morphological benchmarks were established using similar normal lamps. The image illumination retention rate and lamp head brightness retention rate were calculated. The direct occlusion features and optical path were combined. The system integrates occlusion features, image illumination retention rate, and lamp head brightness retention rate for judgment. When occlusion exists and the lamp head brightness retention rate is below a threshold, a composite fault is identified. If there is no occlusion but the lamp head brightness retention rate is below a threshold, the lamp is considered abnormal. If there is occlusion and the lamp head brightness retention rate is normal but the image illumination retention rate is below a threshold, effective tree occlusion is identified. If there is no occlusion and both are normal, there is no occlusion or insufficient non-occluded lighting is output. This achieves accurate differentiation of fault causes and quantitative assessment of lighting loss. Finally, the system confirms the consistency of multi-frame, multi-view diagnostic results for the same streetlight, merges consecutive streetlights affected by the same tree canopy as affected road sections, and outputs a structured diagnostic record including streetlight identification, location, time, diagnostic category, coverage, intrusion rate, retention rate, and associated images. This effectively overcomes the shortcomings of existing technologies in dynamic inspection environments, such as unstable target association, lack of spatial relationship analysis, lack of quantitative assessment mechanism for lighting loss, and insufficient ability to differentiate fault causes, thus improving the accuracy, reliability, and decision support capabilities of nighttime road lighting inspection.

[0025] This embodiment uses a nighttime dynamic vehicle inspection scenario as an example for illustration. The method can be executed by an edge computing device installed on the inspection vehicle, or by a local server or cloud server that is communicatively connected to the inspection vehicle.

[0026] Figure 1 This is a flowchart illustrating the occlusion lighting diagnosis method based on the spatial relationship between roadside trees and streetlights provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110 to S170.

[0027] S110. Obtain image sequences of nighttime road inspections.

[0028] In this embodiment, the image sequence refers to multiple frames of road inspection images continuously acquired along the road to be inspected and sorted according to the acquisition time under conditions of insufficient natural light and streetlights being in operation. Each frame of the road inspection image may include streetlights, roadside trees, road surface, vehicles, road signs, and other road environmental targets.

[0029] Specifically, the inspection vehicle travels along the road to be inspected, using high-sensitivity cameras installed on the windshield, roof, or other locations on the front of the vehicle to capture continuous video images at night. The high-sensitivity camera can be a low-light visible light camera, a wide dynamic range camera, an infrared supplemental lighting camera, or other imaging equipment suitable for nighttime imaging.

[0030] While acquiring video images, the system simultaneously obtains the acquisition time, vehicle location information, vehicle speed, and vehicle heading information for each frame of the road inspection image. The vehicle location information can be obtained through the Global Positioning System (GPS), the BeiDou Navigation Satellite System (BDS), an inertial navigation system, or a combination of these methods.

[0031] The road inspection images, vehicle positioning information, vehicle speed, and vehicle heading information of each frame are aligned according to the acquisition time to form a unified inspection data record. This inspection data record is used for subsequent determination of street light locations, assessment of changes in observation angles, and analysis of the spatiotemporal relationships between consecutive frames.

[0032] Specifically, the image sequence is collected by the vehicle during its journey and includes multiple frames of road inspection images, as well as the collection time, vehicle positioning information, and vehicle heading information corresponding to each frame of road inspection images. The road inspection images, vehicle positioning information, and vehicle heading information of each frame are aligned according to the acquisition time, and distortion correction, noise suppression, brightness normalization, and local contrast enhancement are performed on each frame of road inspection images. Image frames that are motion-blurred, overexposed, or obscured by vehicles ahead are removed. The remaining consecutive road inspection images are then registered and fused, and perspective normalization is performed on the road surface area in the fused image.

[0033] In this embodiment, to improve the quality of nighttime images, one or more of the following preprocessing operations can be performed on each frame of the road inspection images: Distortion correction: Based on the camera lens parameters, barrel distortion or pincushion distortion in the image is corrected to make the spatial relationship between the lamp head, branches and road surface closer to the real imaging relationship. Noise suppression: Use Gaussian filtering, median filtering, bilateral filtering or neural network denoising to reduce random noise generated by high-sensitivity imaging at night; Brightness normalization: Maps the overall brightness of different image frames to a uniform brightness range, reducing the impact of exposure differences on brightness statistics; Local contrast enhancement: Enhances the distinguishability of the light head area, the edges of branches and leaves, and the boundaries of the illuminated areas of the road surface; Motion blur detection: Identifying motion-blurred images based on image gradient, edge sharpness, or the degree of pixel variation between adjacent frames; Invalid frame removal: Removes image frames that are unrecognizable due to motion blur, overexposure, obstruction by vehicles in front, or camera shake, resulting in areas such as the headlights, branches, and road surface being unrecognizable.

[0034] In other implementations, image preprocessing may also include white balance correction, gamma correction, histogram equalization, dark channel enhancement, or low-light image enhancement. For scenarios where the image quality already meets the recognition requirements, only distortion correction and brightness normalization may be performed, without performing all preprocessing operations.

[0035] Registration is performed on the retained consecutive multi-frame road inspection images. Specifically, feature points, lane lines, road edges, or stable background regions can be extracted from each image frame, the transformation relationship between adjacent image frames can be calculated, and multiple frames can be mapped to the same image coordinate system.

[0036] After registration is completed, multiple frames of images are fused. Preferably, median fusion is used to process corresponding pixel positions to suppress interference from road lights, oncoming vehicle high beams, road reflections, and other transient highlights. At the same time, weights can be set according to the sharpness, exposure quality, and occlusion degree of each image frame, and a weighted fusion method can be used to enhance the texture features of the headlight area and the edges of branches and leaves.

[0037] In other implementations, multi-frame fusion can also employ average value fusion, maximum value fusion, sharpness-weighted fusion, exposure-weighted fusion, or deep learning-based multi-frame image enhancement methods. Multi-frame registration can be achieved using feature point matching, optical flow estimation, template matching, or vehicle motion parameter-assisted registration methods.

[0038] Because the distance and viewing angle between the inspection vehicle and the streetlights continuously change during the inspection process, the area and shape of the illuminated road surface area projected by the same streetlight may change in different image frames due to perspective variations. To reduce the impact of this variation on area statistics, perspective normalization is performed on the road surface area in the fused image.

[0039] Specifically, road surface areas at different observation distances can be mapped to a uniform scale based on lane lines, road edges, road surface features, or pre-established image mapping relationships, making the illuminated area of ​​the road surface in different image frames comparable.

[0040] The perspective normalization is not limited to strict camera calibration methods, but can also be achieved by road plane homography transformation based on lane line fitting, scale correction based on depth estimation, distance compensation based on vehicle motion parameters, or road surface scale normalization based on deep learning.

[0041] Through the above data acquisition, image preprocessing, multi-frame fusion and perspective normalization, the impact of vehicle movement, road surface reflection, oncoming lights and exposure changes on subsequent diagnosis during nighttime dynamic inspections can be reduced, providing an image basis for stable identification of lamp heads, branches and leaves and illuminated areas of the road.

[0042] In one specific implementation, lane lines, road edges, road markings, or road texture feature points can be extracted from road inspection images. At least four reference points located on the road plane are selected, and the homography transformation relationship between the current image frame and a view at a uniform scale is calculated. The road surface area is then mapped to the view at the uniform scale. The homography transformation refers to the projection transformation relationship between two planar image regions. For scenarios where complete lane lines are unavailable, the homography transformation relationship can be estimated based on the extended road edge line, the vehicle's driving direction, and the positional changes of the road area in adjacent frames. This processing method makes the illuminated area of ​​the road surface comparable under different observation distances and angles.

[0043] S120. Identify the lamp head region, branch and leaf region and road surface region in the image sequence, and perform cross-frame tracking on the same street lamp to obtain the observation sequence.

[0044] In this embodiment, the lamp head area refers to the image area where the light-emitting component of the street lamp is located in the road inspection image, including the lamp cover, the light source and its adjacent light-emitting outer contour area, and is not limited to including the lamp pole area; The branch and leaf area refers to the image area containing the branches, leaves, and crown branches and leaves of roadside trees in road inspection images; The road surface area refers to the image area in a road inspection image where the road surface is used by vehicles. An observation sequence refers to a data sequence formed by arranging the data of the lamp head area, branch and leaf area, road surface area and related features in each valid observation frame of the same street lamp from the time it enters the image field of view to the time it leaves the image field of view, according to the time of acquisition. Streetlight identification refers to a unique identifier used to distinguish and associate different streetlights in consecutive image frames; A valid observation frame refers to an observation frame that, after being screened for image quality, can be used for lamp head identification, branch and leaf identification, or analysis of illuminated areas of the road surface.

[0045] The detection and semantic segmentation models can be pre-trained using nighttime road inspection image samples. The training samples can be labeled with areas such as light headlights, foliage, road surface, and road background, enabling the model to learn image features under conditions of low nighttime illumination, strong light headlights, road surface reflection, and foliage occlusion. After training, the model is deployed to an onboard computing device or server to perform inference on real-time acquired road inspection images, outputting light headlight detection boxes, foliage masks, and road surface masks. For implementation scenarios without a trained model, brightness threshold segmentation, edge detection, and morphological analysis can also be used to obtain the aforementioned regions.

[0046] In one embodiment, step S120 described above may include steps S121 to S122.

[0047] S121. A detection model is used to identify the lamp head region in the image sequence, and a semantic segmentation model is used to identify the branch and leaf region and the road surface region.

[0048] In this embodiment, a sequence of nighttime road inspection images is input into a detection model to identify the streetlight head region in the images. The detection model can be implemented using a convolutional neural network, a Transformer detection network, or other object detection models, and is used to output detection boxes, masks, or key points for the streetlight head region.

[0049] Simultaneously, the nighttime road inspection image sequence is input into a semantic segmentation model to identify foliage regions and road surface regions. The semantic segmentation model outputs foliage masks and road surface masks according to pixel categories. The foliage mask is used to represent the pixel positions in the foliage region, and the road surface mask is used to represent the pixel positions in the road surface region.

[0050] In other implementations, the lamp head area can also be obtained through brightness threshold segmentation, shape detection, edge detection, or infrared bright spot detection; the branch and leaf area can also be obtained through color features, texture features, instance segmentation models, or 3D point cloud segmentation; and the road surface area can also be obtained through lane line fitting, road boundary detection, or road plane estimation. The lamp head, branches and leaves, and road surface can be identified uniformly by a multi-task model, or they can be identified separately by multiple independent models.

[0051] S122. Based on the appearance characteristics, movement trajectory, optical flow continuity and spatial position relationship of the lamp head area, assign a unique street lamp identifier to the same street lamp and form an observation sequence from when the same street lamp enters the image field of view to when it leaves the image field of view.

[0052] Based on the appearance characteristics of the lamp head area, the inter-frame motion characteristics, the continuity of optical flow, and the spatial positional relationship, a unique street lamp identifier is assigned to the same street lamp.

[0053] Among them, appearance features refer to the shape, size, brightness distribution, color, and texture features of the lamp head area; Inter-frame motion features refer to the changes in position, size, and direction of change of the lamp head region between adjacent image frames. Specifically, they can be represented as the motion trajectory of the lamp head region in consecutive frames. Optical flow continuity refers to the characteristic that the direction and amplitude of pixel movement within the lamp head area remain continuous between consecutive image frames; Spatial positional relationship refers to the relative positional relationship between the lamp head area and roads, lamp poles, street trees and other environmental targets.

[0054] Specifically, candidate positions in the current image frame can be predicted based on the position of the lamp head region in the previous image frame. Then, the overlap ratio, appearance similarity, and motion consistency between the currently detected lamp head region and the candidate positions are calculated, and lamp head regions that meet the association conditions are assigned to the same street light sign.

[0055] When a street lamp first enters the image field of view, a new street lamp identifier is created for it; when the street lamp appears continuously in subsequent image frames, the corresponding lamp head area is assigned to the street lamp identifier; when the street lamp leaves the image field of view or cannot be detected for several consecutive frames, the corresponding observation sequence ends.

[0056] In other implementations, cross-frame tracking can be achieved using Kalman filtering and data association, optical flow tracking, template matching, feature point tracking, or deep learning multi-target tracking models. Alternatively, it can combine vehicle location information and road facility maps to match streetlights in the image with pole numbers in the road asset database.

[0057] By tracking the same street light across frames, a continuous observation sequence of the street light can be formed under different observation distances and angles, avoiding misjudgments caused by relying solely on a single frame image, and providing a data foundation for subsequent continuous frame stability confirmation and multi-view consistency verification.

[0058] S130. Extract the illuminated area of ​​the road surface in the road surface area, establish the pairing relationship between the lamp head area and the illuminated area of ​​the road surface, and construct the light path influence area.

[0059] In this embodiment, the illuminated area of ​​the road surface refers to the image area formed in the road surface area after the light emitted by the street lamp is projected onto the road surface, where the brightness is higher than that of the surrounding road surface background; Pairing relationship refers to the correspondence established between the lamp head area of ​​the same street lamp and the illuminated area of ​​the road surface it forms; The light path influence area refers to the potential light propagation influence range between the lamp head and the illuminated area of ​​the road surface, constructed in a two-dimensional image based on the relative position between the lamp head area and the illuminated area of ​​the road surface. Predicted illuminated road surface area refers to the estimated area generated based on the location and shape of the illuminated road surface area in the same street light historical observation frames when the illuminated road surface area cannot be directly extracted in the current image frame.

[0060] In one embodiment, step S130 described above may include steps S131 to S134.

[0061] S131. Within the road surface area, adaptive threshold segmentation is used to extract connected regions with brightness higher than the road surface background to obtain the illuminated area of ​​the road surface.

[0062] In this embodiment, within the road surface area obtained by semantic segmentation, adaptive threshold segmentation is used to extract connected regions with brightness higher than the road surface background.

[0063] Specifically, the background brightness distribution of the current road surface area can be statistically analyzed, and the segmentation threshold can be determined based on the average background brightness, the range of brightness fluctuations, or the local brightness differences. Alternatively, the segmentation threshold can be determined separately for different local areas to adapt to uneven street lighting, differences in road surface materials, and local reflections.

[0064] After extracting the connected regions, morphological processing can be performed on the connected regions to remove isolated noise points, fill local voids, and eliminate interference areas whose area or shape does not conform to the characteristics of road lighting, thus obtaining the illuminated area of ​​the road surface.

[0065] When the same street light creates multiple adjacent bright spots on the road surface, these bright spots are merged into one illuminated area of ​​the road surface; when multiple bright spots correspond to different street lights, they remain as separate illuminated areas of the road surface.

[0066] In other implementations, the illuminated area of ​​the road surface can be extracted using methods such as a fixed brightness threshold, local contrast detection, brightness clustering, saliency detection, or semantic segmentation. For scenes with significant road surface reflection, instantaneous light spots can be eliminated by combining continuous frame brightness stability.

[0067] S132. Based on the spatial nearest neighbor relationship and relative positional relationship between the lamp head area and each road surface illuminated area, determine the road surface illuminated area corresponding to the lamp head area; when the lamp head area corresponds to multiple road surface illuminated areas, merge the multiple road surface illuminated areas and establish a unique pairing relationship.

[0068] In this embodiment, the spatial nearest neighbor relationship can be determined based on the distance between the center point of the lamp head area and the center point of the illuminated area of ​​the road surface, or it can be determined based on the minimum distance between the edge of the lamp head area and the edge of the illuminated area of ​​the road surface. Relative positional relationship refers to the vertical positional relationship, horizontal offset relationship, and relationship between the illuminated area of ​​the lamp head and the illuminated area of ​​the road surface in the image, as well as the relationship between the direction of the line connecting the two and the direction of road extension.

[0069] In one implementation, the pairing relationship can be further determined by considering the lamp head projection direction. The lamp head projection direction refers to the approximate direction in which the lamp head's light is projected onto the road surface, and can be determined based on the direction of the line connecting the center of the lamp head area and the center of the illuminated area of ​​the road surface, the lamp head installation orientation, the road extension direction, or the light distribution direction of the luminaire.

[0070] In one implementation, the distance between the current lamp head area and the candidate road surface illuminated area is calculated, and it is determined whether the relative position between the two conforms to the general spatial rules of street lighting projection. The road surface illuminated area that is closest and has a matching relative position is determined as the road surface illuminated area corresponding to the current lamp head area.

[0071] When the same lamp head area corresponds to multiple illuminated road surfaces, the multiple illuminated road surfaces are merged to establish a unique pairing relationship between the lamp head area and the merged illuminated road surfaces.

[0072] In other implementations, road plan mapping, depth estimation, binocular vision, lidar point clouds, or road facility maps can be combined to determine the pairing relationship between the lamp head area and the illuminated area of ​​the road surface.

[0073] When multiple lamp head regions and multiple illuminated road surfaces exist simultaneously in an image, a candidate pairing set can be constructed, and the pairing cost between each lamp head region and each illuminated road surface can be calculated. The pairing cost can be determined based on the center point distance, vertical position relationship, horizontal offset, the consistency between the connecting line direction and the road extension direction, and the matching degree between the illuminated road surface area and the lamp head brightness. Following the principle of minimizing the overall pairing cost, a unique illuminated road surface area is assigned to each lamp head region. When the pairing cost of a lamp head region with multiple illuminated road surfaces is close, the illuminated road surface area whose pairing relationship remains stable in consecutive observation frames is preferentially selected.

[0074] S133. Construct a trapezoidal approximate region with the edges of the lamp head area and the illuminated area of ​​the road surface on the side closest to each other, and expand the trapezoidal approximate region; when the trapezoidal approximate regions corresponding to adjacent lamp head areas overlap, determine the ownership of the overlapping part according to the spatial nearest neighbor relationship between the overlapping part and each lamp head area, and obtain the light path influence area.

[0075] Specifically, a trapezoidal approximate region is constructed based on the edge of the lamp head area near the illuminated area of ​​the road surface and the edge of the illuminated area of ​​the road surface near the lamp head area.

[0076] Specifically, the two endpoints of the lower edge of the lamp head area can be connected to the two corresponding endpoints of the upper edge of the illuminated area of ​​the road surface, forming a trapezoidal approximation region. This trapezoidal approximation region is used to characterize the main propagation path of the light emitted from the lamp head as it is projected onto the illuminated area of ​​the road surface in a two-dimensional image.

[0077] The trapezoidal approximation region is expanded to cover the potential occlusion caused by branch swaying, image edge errors, and diffuse illumination from the lamp heads. When the trapezoidal approximation regions corresponding to adjacent lamp head regions overlap, the overlapping part is assigned according to the spatial nearest neighbor relationship between the overlapping part and each lamp head region, thus obtaining the optical path influence region corresponding to each lamp head region.

[0078] It should be noted that the optical path influence region is an approximate region of light propagation established based on two-dimensional image coordinates. It is used to characterize the potential occlusion channel between the lamp head and the illuminated area of ​​the road surface, and is not limited to the projection of the strictly physical optical path in real three-dimensional space. This region can be directly constructed based on the pixel coordinates of the lamp head area and the illuminated area of ​​the road surface, without relying on precise camera calibration parameters.

[0079] In other embodiments, the light path influence area can also be constructed as a triangular area, a fan-shaped area, a rectangular area, a polygonal area, a trumpet-shaped diffusion area, or a projection area based on the light distribution curve of the luminaire; or a three-dimensional light path influence area can be constructed by combining depth information or three-dimensional point cloud, and the three-dimensional area can be projected onto the image plane.

[0080] like Figure 2 As shown, for the same lamp head area and its paired illuminated road surface area, the trapezoidal projection area formed by the lower edge of the lamp head area and the upper edge of the illuminated road surface area is taken as the light path influence area. When the foliage area does not directly cover the lamp head area, but intrudes into the light path influence area, it may still reduce the area of ​​the illuminated road surface area or decrease the edge continuity.

[0081] When the optical path influence areas corresponding to adjacent lamp head regions overlap, the distance between each pixel within the overlapping area and the center point of the adjacent lamp head region can be calculated, and the pixel can be assigned to the optical path influence area corresponding to the closer lamp head region. When the difference between the two distances is less than a preset distance difference, the current pixel's assignment can be determined by combining the region assignment results in the previous observation frame or the continuous extension direction of the branch / leaf region. This method avoids the same branch / leaf being repeatedly included in the optical path influence areas corresponding to multiple lamp heads.

[0082] S134. When no illuminated area of ​​the road surface is extracted in the current image frame, a predicted illuminated area of ​​the road surface is generated based on the location and shape of the illuminated area of ​​the road surface in the historical observation frame corresponding to the same street light identifier, and the light path influence area is constructed.

[0083] In this embodiment, when no illuminated area of ​​the road surface is extracted in the current image frame, the historical observation frame corresponding to the same street light identifier is queried. If an available illuminated area of ​​the road surface exists in the historical observation frame, the historical illuminated area is mapped to the current image frame based on the location, shape, and vehicle movement information of the historical illuminated area, thereby generating a predicted illuminated area of ​​the road surface.

[0084] Specifically, based on the motion trajectory of the same street light in consecutive frames, vehicle positioning information, and image registration relationships, position compensation can be performed on the historical illuminated area of ​​the road surface, so that the predicted illuminated area of ​​the road surface can reflect the expected position and shape under the current observation angle.

[0085] If neither the current image frame nor the historical observation frames can determine the illuminated area of ​​the road surface, then no predicted illuminated area of ​​the road surface will be generated, and the result of insufficient evidence will be output in subsequent diagnosis.

[0086] By establishing a spatial correlation model between the lamp head area, the branch and leaf area, the light path affected area, and the illuminated area of ​​the road surface, this embodiment extends the traditional detection method of only judging whether the lamp head is covered to the analysis of the complete lighting link, thereby being able to identify the hidden occlusion problem where the lamp head is visible but the light path has been invaded by branches and leaves.

[0087] The predicted illuminated area of ​​the road surface is not a direct copy of the historical illuminated area. Instead, it is obtained by mapping the position and shape of the historical illuminated area based on the image registration relationship between historical observation frames and the current image frame, vehicle motion information, and the motion trajectory of the same street lamp. When the predicted illuminated area is used in the calculation of image illumination retention rate, it should first undergo perspective normalization processing and be marked as originating from the prediction result in the diagnostic record. When the number of observation frames continuously using the predicted illuminated area exceeds a preset number, or when the spatial relationship between the predicted illuminated area and the road area in the current image is inconsistent, the prediction result will no longer be used in the diagnosis, and an insufficient evidence result will be output.

[0088] S140. Determine the direct occlusion feature based on the overlap relationship between the branch and leaf area and the lamp head area, and determine the optical path occlusion feature based on the overlap relationship between the branch and leaf area and the optical path influence area.

[0089] In this embodiment, the direct occlusion feature refers to the feature used to characterize the branch and leaf area directly covering the lamp head area and affecting the lamp head's light emission performance; Light path occlusion features refer to characteristics used to characterize the fact that branches and leaves do not directly cover the lamp head area, but intrude into the area affected by the light path and affect the road lighting effect; A branch and leaf mask refers to a binary image region used to mark the pixel positions of branches and leaves. A lamp head mask is a binary image region used to mark the pixel positions of the lamp head area.

[0090] In one specific implementation, the lamp head mask can be divided into overlapping sub-regions that overlap with the branch and leaf mask and non-overlapping sub-regions that do not overlap with the branch and leaf mask. The average or median brightness of the overlapping and non-overlapping sub-regions is then statistically analyzed. When the brightness of the overlapping sub-region is lower than that of the non-overlapping sub-region by a preset level, and the location of the brightness decrease corresponds spatially to the boundary of the branch and leaf mask, local brightness attenuation is determined. When the branch and leaf mask almost covers the entire lamp head mask and a reliable non-overlapping sub-region cannot be obtained, the brightness of the current lamp head region can be compared with the brightness of the lamp head region in the previous valid observation frame of the same street light, and local brightness attenuation can be determined based on the brightness decrease.

[0091] The fragmentation of the illuminated area can be determined comprehensively based on the number of edge breaks, the number of connected regions, the integrity of the outline, and changes in edge length within the illuminated area of ​​the road surface. For example, when the illuminated area corresponding to the same streetlight splits from a single connected region into multiple connected regions, or when the number of edge gaps increases and the integrity of the outline decreases, the fragmentation of the illuminated area is determined to be increased. The fragmentation of the illuminated area can serve as an auxiliary feature for light path occlusion, distinguishing between localized lighting defects caused by branches and leaves intruding into the light path and brightness changes caused by overall exposure variations.

[0092] In one embodiment, step S140 described above may include steps S141 to S142.

[0093] S141. Convert the branch and leaf area and the lamp head area into a branch and leaf mask and a lamp head mask, respectively. Calculate the ratio of the overlap area of ​​the branch and leaf mask and the lamp head mask to the area of ​​the lamp head mask to obtain the lamp head coverage rate.

[0094] S142. When the lamp head coverage rate is not lower than the lamp head coverage threshold, and there is a local brightness attenuation in the lamp head area corresponding to the boundary space of the branch and leaf mask, it is determined that the branch and leaf area directly blocks the lamp head area, and the lamp head coverage rate and the local brightness attenuation are used as the direct blocking features.

[0095] In this embodiment, the lamp head coverage rate can be expressed as: ;in, This represents the lamp head coverage rate in frame t. This represents the leaf mask in frame t. This represents the lamp head mask in frame t. This indicates the area of ​​the lamp holder mask.

[0096] In this embodiment, the lamp head coverage rate refers to the ratio of the overlapping area of ​​the branch and leaf mask to the total area of ​​the lamp head mask, which is used to characterize the degree of coverage of the lamp head area by the branches and leaves.

[0097] Simultaneously, the average brightness, brightness distribution characteristics, and directional brightness attenuation characteristics are statistically analyzed within the lamp head area. The average brightness is used to characterize the overall luminous intensity of the lamp head; the brightness distribution characteristics are used to characterize the brightness differences at different locations within the lamp head area; and the directional brightness attenuation characteristics are used to characterize the localized brightness reduction that occurs within the lamp head area along the direction of foliage shading.

[0098] In this embodiment, local brightness attenuation refers to the phenomenon that the brightness of a local area within the lamp head region corresponding to the boundary space of the branch and leaf mask is lower than the brightness of other parts of the lamp head region, or lower than the brightness of the corresponding position in the previous observation frame of the same street light.

[0099] When the lamp head coverage rate is not lower than the lamp head coverage threshold, and there is a local brightness attenuation in the lamp head area corresponding to the boundary space of the branch and leaf mask, it is determined that the branch and leaf area directly occludes the lamp head area, and the lamp head coverage rate and local brightness attenuation are used as direct occlusion features.

[0100] like Figure 3 As shown, when the leaf mask overlaps with the lamp head mask, and a brightness decrease occurs near the overlap position consistent with the leaf boundary, it indicates that the leaves not only overlap with the lamp head in the two-dimensional image, but also have a real impact on the lamp head's light emission performance. Therefore, it can be determined that the lamp head is directly blocked.

[0101] In other embodiments, the lamp head coverage can also be expressed as the ratio of the overlapping area to the area of ​​the leaf mask, the overlap ratio, the shading width, or the boundary overlap length; the local brightness attenuation can also be determined based on the brightness difference between the overlapping area and the non-overlapping area, the lamp head brightness gradient, or the brightness change of the corresponding area in the previous and next frames.

[0102] In one embodiment, step S140 above further includes steps S143 to S144.

[0103] S143. Calculate the ratio of the overlap area between the branch and leaf mask and the area affected by the optical path to the area affected by the optical path, and obtain the optical path intrusion rate.

[0104] S144. When the lamp head coverage rate is lower than the lamp head coverage threshold, the light path intrusion rate is not lower than the light path intrusion threshold, and the illuminated area of ​​the road surface decreases in area or edge continuity decreases relative to the previous observation frame corresponding to the same street lamp sign, it is determined that the branch and leaf area forms a light path occlusion in the light path affected area, and the light path intrusion rate, the area change characteristics of the illuminated area of ​​the road surface, and the edge continuity change characteristics are used as the light path occlusion characteristics.

[0105] In this embodiment, the optical path intrusion rate can be expressed as: ;in, This represents the optical path intrusion rate in frame t. This represents the area affected by the optical path in frame t. This indicates the area affected by the optical path.

[0106] In this embodiment, the light path intrusion rate refers to the proportion of the overlapping area of ​​the branch and leaf mask and the light path influence area to the total area of ​​the light path influence area, which is used to characterize the degree to which the branches and leaves intrude into the potential light propagation channel between the lamp head and the illuminated area of ​​the road surface.

[0107] At the same time, we analyzed whether the illuminated area of ​​the road surface decreased in size or edge continuity compared to the previous observation frame corresponding to the same street light sign.

[0108] Among them: Area reduction means that the illuminated area of ​​the road surface after perspective normalization is smaller than the corresponding area in the previous valid observation frame; Decreased edge continuity refers to an increase in the number of edge breaks, an increase in the degree of edge contour fragmentation, or a decrease in edge closure in the illuminated area of ​​the road surface. Changes in edge continuity can be determined by the number of edge breaks, the number of connected regions, changes in edge length, or contour integrity.

[0109] When the lamp head coverage rate is lower than the lamp head coverage threshold, the light path intrusion rate is not lower than the light path intrusion threshold, and the illuminated area of ​​the road surface decreases in area or the edge continuity decreases compared to the previous observation frame, it is determined that the branch and leaf area forms a light path occlusion in the area affected by the light path, and the light path intrusion rate, the area change characteristics of the illuminated area of ​​the road surface, and the edge continuity change characteristics are used as light path occlusion characteristics.

[0110] like Figure 2 As shown, although the foliage area does not directly cover the lamp head area, it has encroached on the area affected by the light path, resulting in a reduction in the illuminated area of ​​the road surface or fragmented edges. This invention can identify such hidden obstruction problems that are difficult to detect using traditional lamp head coverage detection methods.

[0111] In other embodiments, the light path intrusion rate can also be represented by the depth, width, perimeter ratio of branches and leaves intruding into the area affected by the light path, or the maximum continuous intrusion length; anomalies in the illuminated area of ​​the road surface can also be represented by changes in outline integrity, changes in brightness distribution, or changes in morphological similarity.

[0112] By simultaneously calculating both direct occlusion features and optical path occlusion features, this embodiment can not only detect the explicit problem of branches and leaves directly covering the lamp head, but also the implicit problem of branches and leaves located between the lamp head and the illuminated area of ​​the road surface but not covering the lamp head, thereby improving the ability to detect potential road lighting hazards.

[0113] S150. Establish a normal lighting benchmark and determine the image illumination retention rate of the illuminated area of ​​the road surface and the lamp head brightness retention rate of the lamp head area.

[0114] In this embodiment, the normal lighting reference refers to reference data used to characterize the standard lighting state of the same street light or similar street lights under current road conditions when there are no branches or leaves obstructing the light and the lights are working normally.

[0115] The normal lighting reference includes at least: The illuminated area benchmark refers to the statistical value of the illuminated area of ​​the road surface in a normal lighting sample. The lamp head luminance benchmark refers to the statistical value of the luminance of the lamp head area in a normal lighting sample; The morphological benchmark of the illuminated area refers to the statistical values ​​of the outline, edge continuity, length-to-width ratio, and integrity of the illuminated area of ​​the road surface in a normal lighting sample.

[0116] The establishment and comparison of the normal lighting reference includes: Streetlights of the same type and type on the same road, and whose lamps are confirmed to be in good working order and without obstruction by branches and leaves, are selected as normal lighting samples to establish benchmarks for the illuminated area, lamp head brightness, and illuminated area shape. When the number of normal lighting samples does not reach the preset number, a preset initial benchmark is adopted; when the number of normal lighting samples reaches the preset number, the normal lighting benchmark is updated according to the statistical results of the normal lighting samples. Based on the observation frames with illuminated or predicted road surface areas, the ratio of the average illuminated area area to the illuminated area reference is calculated to obtain the image illumination retention rate. Based on the observation frames that can identify the lamp head area, the ratio of the average brightness of the lamp head area to the lamp head brightness reference is calculated to obtain the lamp head brightness retention rate.

[0117] In this embodiment, streetlights of the same type and along the same road, whose operation has been manually confirmed and who are free from obstruction by branches and leaves, are selected as normal lighting samples. These streetlights of the same type can be categorized according to lamp head shape, light source type, installation method, or lighting pattern; it is not required to obtain the complete model number, installation height, and light distribution parameters of the lamps in advance.

[0118] For the first batch of streetlights on the same road segment, since sufficient normal lighting samples have not yet been accumulated, a preset initial benchmark can be used. The preset initial benchmark refers to initial reference data established based on historical inspection data, experience data from similar roads, or manually set standard lighting data.

[0119] When the number of normal lighting samples reaches a preset number, the normal lighting baseline is updated based on the statistical results of the normal lighting samples. The update can use the mean, median, quantiles, or statistical values ​​after removing outliers to reduce the impact of transient glare, road surface reflections, and occasional shading on the baseline.

[0120] In other implementations, the normal lighting benchmark can also be established based on historical unobstructed observation data of the same street light, normal lighting samples of adjacent road sections, luminaire model parameters, installation height, light distribution curve, or artificial illuminance measurement results; it can be established using offline modeling or updated online during inspection.

[0121] For the observation sequence corresponding to the same street light sign, the observation frames with the illuminated area of ​​the road surface or the predicted illuminated area of ​​the road surface are extracted, and the area of ​​the illuminated area of ​​the road surface after perspective normalization in each observation frame is counted to obtain the average area of ​​the illuminated area.

[0122] The image illumination retention rate is obtained by calculating the ratio of the average illuminated area to the baseline illuminated area. Where R represents the image illumination retention rate, This represents the average illuminated area of ​​the streetlight in the valid observation frames. This refers to the illuminated area reference in normal lighting standards.

[0123] In this embodiment, the image illumination retention rate refers to the degree to which the illuminated area of ​​the road surface under the current streetlight is retained relative to the normal lighting reference. This indicator can be expressed as a ratio or as a percentage; its specific form does not affect the implementation of this invention.

[0124] like Figure 4 As shown, by statistically analyzing the illuminated area of ​​the same streetlight in a continuous observation sequence, the influence of random changes in a single frame image can be eliminated, allowing the image illumination retention rate to stably reflect the degree of illumination loss of the current streetlight relative to its normal illumination state. In the figure, A... This represents the average value of the normalized area.

[0125] For the observation sequence corresponding to the same street light sign, extract the observation frames that can identify the light head area, calculate the average brightness of the light head area, and obtain the average brightness of the current street light's light head.

[0126] The lamp head brightness retention rate is obtained by calculating the ratio of the average brightness of the lamp head to the lamp head brightness reference. Where B represents the lamp head brightness retention rate. This represents the average brightness of the lamp head area within the valid observation frame of the current street light. This refers to the lamp holder brightness reference in normal lighting standards.

[0127] In this embodiment, the lamp head brightness retention rate refers to the degree to which the brightness of the current street lamp head area is retained relative to the normal lighting reference.

[0128] When the image illumination retention rate decreases while the lamp head brightness retention rate remains normal, it indicates that there is no obvious abnormality in the lamp head's own light emission state, and insufficient lighting is more likely caused by tree obstruction. When both the image illumination retention rate and the lamp head brightness retention rate decrease, it indicates that there may be a lamp malfunction, or that the lamp malfunction is accompanied by tree obstruction.

[0129] By establishing normal lighting benchmarks for different roads, lamp types, and installation conditions, and using image illumination retention rate and lamp head brightness retention rate for quantitative evaluation, this embodiment can reduce the impact of observation distance, lamp differences, and environmental brightness differences on diagnostic results, making diagnostic results comparable between different road sections and different inspection batches.

[0130] The samples used to establish or update normal lighting benchmarks should exclude observation frames with direct obstruction, light path intrusion, transient glare, severe road surface reflection, abnormal lamp head brightness, or abnormal image quality. When statistically analyzing the illuminated area benchmark, lamp head brightness benchmark, and illuminated area shape benchmark, the median, the mean after removing outliers, or a set quantile can be used as the benchmark value. As the number of inspection samples increases, the normal lighting benchmark can be updated chronologically to adapt to seasonal changes, lamp aging, and changes in the road environment.

[0131] S160. Based on the direct occlusion features, the optical path occlusion features, the image illumination retention rate, and the lamp head brightness retention rate, by fusing the image illumination retention rate, the lamp head brightness retention rate, the direct occlusion features, and the optical path occlusion features, tree occlusion, lamp malfunction, and a combination of both faults are distinguished to form a diagnostic result.

[0132] In this embodiment, tree obstruction refers to the state in which branches and leaves directly cover the lamp head area or invade the area affected by the light path and affect the road lighting; A lamp malfunction refers to a situation where the lamp holder's own luminous brightness is lower than the normal lighting standard and no obstruction by branches or leaves is detected. A combined fault refers to a situation where both foliage obstruction and lighting malfunctions coexist. The diagnostic result refers to the output result after classifying the current lighting status of the same street light.

[0133] The system performs fusion discrimination based on direct occlusion features, optical path occlusion features, image illumination retention rate, and lamp head brightness retention rate to distinguish tree occlusion, lamp abnormalities, and combined faults of the two, thus forming a diagnostic result.

[0134] Specifically, when there is direct occlusion or optical path occlusion, and the lamp head brightness retention rate is lower than the lamp head brightness threshold, it indicates that the evidence of branch and leaf occlusion and the lamp head brightness abnormality exist at the same time, and a composite fault result is output. When there is no direct obstruction or optical path obstruction, and the lamp head brightness retention rate is lower than the lamp head brightness threshold, it indicates that the lighting abnormality mainly originates from the abnormal light emission state of the lamp head itself, resulting in an abnormal output lamp result. When the lamp head brightness retention rate is not lower than the lamp head brightness threshold and there is direct occlusion, output the direct occlusion lamp head result; when the lamp head brightness retention rate is not lower than the lamp head brightness threshold and there is optical path occlusion, output the optical path occlusion result; when the direct occlusion lamp head result or the optical path occlusion result is output and the image illumination retention rate is lower than the illumination retention threshold, determine that the corresponding tree occlusion caused effective illumination loss; When there is no direct occlusion or optical path occlusion, and the lamp head brightness retention rate is not lower than the lamp head brightness threshold, only the corresponding occlusion result can be output, without considering that it has caused effective lighting loss. If the image lighting retention rate is lower than the lighting retention threshold, the lighting deficiency result caused by non-occlusion reasons is output; otherwise, the no-occlusion result is output. When the lamp head area cannot be identified, or when neither the current image frame nor the historical observation frame can determine the illuminated area of ​​the road surface, an insufficient evidence result is output.

[0135] It should be noted that, in another, more stringent implementation, the image illumination retention rate being lower than the illumination retention threshold can also be used as an additional condition for determining that the optical path occlusion has been established; while in this embodiment, the formation of optical path occlusion is mainly determined based on the optical path intrusion rate and the change in the illuminated area of ​​the road surface, and the image illumination retention rate is used to further determine whether the occlusion has caused effective lighting loss.

[0136] When there is no direct obstruction or optical path obstruction, and the lamp head brightness retention rate is not lower than the lamp head brightness threshold: If the image illumination retention rate is lower than the illumination retention threshold, it indicates that the lamp head brightness is normal and no branches or leaves are found to block the light, but the road illumination is still insufficient. The output result is insufficient illumination due to reasons other than blocking. If the image illumination retention rate is not lower than the illumination retention threshold, it indicates that the current street light illumination status is normal and the output result is unobstructed.

[0137] When the light head area cannot be identified, or when neither the current image frame nor the historical observation frames can determine the illuminated area of ​​the road surface, it indicates a lack of key image evidence required to complete the occlusion diagnosis, and the result of insufficient evidence is output.

[0138] When the number of valid observation frames is insufficient, the image quality cannot meet the analysis requirements, or a stable conclusion cannot be formed according to the above diagnostic rules, the result of insufficient evidence can also be output.

[0139] In this embodiment, the lamp head coverage threshold refers to the lower limit of the lamp head coverage rate used to determine whether the foliage area effectively covers the lamp head area; the light path intrusion threshold refers to the lower limit of the light path intrusion rate used to determine whether the foliage area effectively intrudes into the light path affected area; the illumination retention threshold refers to the lower limit of the image illumination retention rate used to determine whether the road lighting has suffered effective loss; and the lamp head brightness threshold refers to the lower limit of the lamp head brightness retention rate used to determine whether the lamp head's own light emission state is abnormal.

[0140] The system can also output an insufficient evidence result when the road surface area cannot be identified, the number of valid observation frames is less than the preset number, a pairing relationship between the lamp head area and the illuminated road surface area cannot be established in multiple consecutive frames, or the predicted usage ratio of the illuminated road surface area exceeds the preset ratio. For insufficient evidence results, the system can mark the corresponding data missing type to determine whether additional lamp head images, foliage images, or road lighting images need to be collected.

[0141] The system performs fusion discrimination based on lamp head coverage C_cov, optical path intrusion rate S_path, image illumination retention rate R, lamp head brightness retention rate B, and the fragmentation of the illuminated area, and outputs diagnostic results such as direct lamp head occlusion, optical path occlusion, suspected lamp abnormality, composite problem, no effective occlusion, insufficient illumination due to non-occlusion reasons, or insufficient evidence.

[0142] In this embodiment, the fragmentation degree of the illuminated area refers to an index used to characterize the integrity and morphological fragmentation of the illuminated area of ​​the road surface. The fragmentation degree of the illuminated area can be determined based on the number of edge fractures, edge gaps, connected regions, outline integrity, or edge length changes in the illuminated area of ​​the road surface. When branches and leaves intrude into the area affected by the light path, the illuminated area of ​​the road surface may experience edge fractures, local defects, or an increase in connected regions, at which point the fragmentation degree of the illuminated area increases.

[0143] The specific diagnostic rules are as follows: When the lamp head coverage rate C_cov is not lower than the lamp head coverage threshold T_cov, and there is local brightness attenuation in the lamp head area corresponding to the boundary space of the branch and leaf mask, it is determined that the branch and leaf area directly occludes the lamp head area, and the direct occlusion result is output; when the image illumination retention rate R is further lower than the illumination retention threshold T_light, it is determined that the direct occlusion has caused effective illumination loss.

[0144] When the lamp head coverage rate C_cov is lower than the lamp head coverage threshold T_cov, the light path intrusion rate S_path is not lower than the light path intrusion threshold T_path, and the image illumination retention rate R is lower than the illumination retention threshold T_light, it is determined that the branch and leaf area has formed a light path occlusion on the light path influence area, and it is determined that the light path occlusion has caused effective illumination loss.

[0145] In another implementation, optical path occlusion can be determined first based on the optical path intrusion rate and the changes in the area of ​​the illuminated road surface, changes in edge continuity, or fragmentation of the illuminated area. Then, an image illumination retention rate lower than the illumination retention threshold can be used as a condition to confirm effective illumination loss. In this way, situations where optical path intrusion does not cause illumination loss and situations where optical path occlusion has caused illumination loss can be distinguished and output.

[0146] When the lamp head coverage rate C_cov is lower than the lamp head coverage threshold T_cov, the optical path intrusion rate S_path is lower than the optical path intrusion threshold T_path, and the lamp head brightness retention rate B is lower than the lamp head brightness threshold T_lamp, it is determined that the current street light has a lamp malfunction, and this result can be marked as a suspected lamp malfunction result for subsequent manual review or maintenance confirmation.

[0147] When the lamp head brightness retention rate B is lower than the lamp head brightness threshold T_lamp, and there is evidence of direct occlusion or optical path occlusion, it is determined that the current street light is simultaneously obstructed by trees and the lamp is abnormal, and the result of the composite problem is output.

[0148] In actual implementation, the judgment of compound problems can be performed first. That is, first determine whether the brightness retention rate of the lamp head is lower than the brightness threshold of the lamp head, and then determine whether there is direct occlusion or optical path occlusion, so as to avoid the same street light outputting tree occlusion results and compound problem results at the same time.

[0149] When the lamp head coverage rate C_cov is lower than the lamp head coverage threshold T_cov, the optical path intrusion rate S_path is lower than the optical path intrusion threshold T_path, and the image lighting retention rate R is not lower than the lighting retention threshold T_light, the current street light is determined to be unobstructed, or no lighting impact reaching the preset level is found.

[0150] When the lamp head coverage rate C_cov is lower than the lamp head coverage threshold T_cov, the optical path intrusion rate S_path is lower than the optical path intrusion threshold T_path, the lamp head brightness retention rate B is not lower than the lamp head brightness threshold T_lamp, but the image lighting retention rate R is lower than the lighting retention threshold T_light, it is determined that the current street light has insufficient lighting due to non-obstruction reasons, and a manual review prompt is generated.

[0151] The aforementioned insufficient lighting due to non-obstruction reasons can be used to characterize situations where the lamp head brightness is normal, no branches or leaves are found to obstruct the light, but the illuminated area of ​​the road surface is still lower than the normal lighting benchmark. This situation may be caused by changes in road surface reflection conditions, changes in lamp light distribution, changes in lamp head installation angle, or other non-branch / leaf obstruction factors.

[0152] When the lamp head area cannot be identified, the illuminated area of ​​the road surface cannot be determined in the current image frame and the historical observation frames, or the number of effective observation frames is insufficient or the image quality cannot meet the analysis requirements, the result of insufficient evidence will be output.

[0153] The aforementioned lamp head coverage threshold T_cov, light path intrusion threshold T_path, lighting retention threshold T_light, and lamp head brightness threshold T_lamp can be calibrated separately according to different road environments and street light types.

[0154] Specifically, multiple streetlights under the same road conditions, which were manually confirmed to be free of branches and leaves and to be working properly, were selected as calibration samples. The mean μ and standard deviation σ of the lamp head coverage rate C_cov, light path intrusion rate S_path, image illumination retention rate R, and lamp head brightness retention rate B of the calibration samples were statistically analyzed, and each threshold was determined based on the mean and standard deviation.

[0155] For example, the thresholds can be set as follows: Where T represents the corresponding threshold, μ represents the mean of the corresponding feature in the normal calibration sample, σ represents the standard deviation of the corresponding feature in the normal calibration sample, and k is the standard deviation coefficient, which can be 2 to 3.

[0156] For indicators such as image illumination retention rate and lamp head brightness retention rate, where higher values ​​indicate better illumination conditions, the threshold can be determined using the mean minus the standard deviation method described above. For indicators such as lamp head coverage rate and light path intrusion rate, where higher values ​​indicate a higher degree of occlusion, the threshold can also be determined using the mean plus the standard deviation method. The specific threshold direction is determined based on the physical meaning of the corresponding feature.

[0157] In other implementations, fixed empirical thresholds can be set directly based on the requirements of road lighting operation and maintenance management. For example, the lighting retention threshold T_light can be set to 0.7, meaning that when the current street light's image lighting retention rate is less than 0.7 times the normal lighting benchmark, it is considered that there has been effective lighting loss.

[0158] Different roads, different types of lights, different installation conditions, and different observation scenarios can be calibrated with independent thresholds to improve the matching degree between the thresholds and the current road conditions.

[0159] The above diagnostic process can be implemented using rule-based decision-making or a classification model.

[0160] In one implementation, the lamp head coverage, light path intrusion rate, image illumination retention rate, lamp head brightness retention rate, illuminated area fragmentation, lamp head brightness distribution characteristics, and branch and leaf spatial location characteristics are input into a classification model. The classification model outputs the confidence scores of diagnostic categories such as direct lamp head occlusion, light path occlusion, lamp malfunction, composite problem, no occlusion, insufficient lighting due to non-occlusion reasons, or insufficient evidence. The category with the highest confidence score is taken as the diagnostic result.

[0161] The classification model can be implemented using decision trees, support vector machines, gradient boosting trees, random forests, Bayesian classifiers, or neural network classification models.

[0162] In other implementations, a combination of rule-based judgment and classification models can be used: first, rule-based judgment is used to determine whether the evidence is sufficient, and then the confidence level of each diagnostic category is output through the classification model; or candidate diagnostic categories are obtained first through the classification model, and then the lamp head coverage threshold, light path intrusion threshold, illumination retention threshold and lamp head brightness threshold are used for verification to improve the interpretability of the diagnostic results.

[0163] Through the above-mentioned integrated diagnostic method, the system can not only determine whether there is a lighting abnormality in the current street light, but also further distinguish whether the lighting abnormality is caused by branches and leaves directly covering the lamp head, branches and leaves intruding into the light path, abnormality of the lamp itself, or a combination of multiple problems, thus providing a clearer basis for subsequent maintenance dispatch.

[0164] By jointly analyzing the spatial relationship between branches and leaves, changes in lamp head brightness, and road lighting effects, this embodiment can not only detect abnormal lighting phenomena, but also distinguish the specific causes of the abnormal lighting, and generate maintenance tasks for tree pruning, lamp repair, and manual verification, thereby reducing unnecessary on-site inspections.

[0165] S170. Verify the consistency of diagnostic results for the same street light across multiple frames and different observation angles, and output the diagnostic record.

[0166] The different observation angles can be determined based on changes in the vehicle's position relative to the streetlight, changes in the vehicle's heading, or changes in the position of the same streetlight within the image. Different observation angles are considered to have been obtained when the relative observation direction between the vehicle and the streetlight changes by more than a preset angle, or when the position of the streetlight's lamp head area in the image moves from one preset image area to another. This method avoids misinterpreting image frames with varying observation distances but essentially the same observation direction as having different observation angles.

[0167] In this embodiment, the diagnostic record refers to a structured record used to store the diagnostic object, diagnostic location, diagnostic category, key characteristic values, and evidence data.

[0168] To eliminate the momentary interference from oncoming vehicle high beams, short-term swaying of leaves, vehicle bumps, and misjudgments caused by accidental overlap of branches and leaves with the light head, the stability of the observation sequence corresponding to the same street light sign was confirmed by continuous frames.

[0169] For observation sequences with the same street light identifier, the diagnostic result is considered valid if at least N consecutive valid observation frames yield the same diagnostic result. Preferably, N is between 5 and 10.

[0170] To improve diagnostic reliability, effective observation frames must cover at least two different observation angles to ensure that occlusion evidence remains stable across different viewpoints. Different observation angles can be determined based on changes in vehicle position, vehicle heading, or the position of the same streetlight within the image.

[0171] When branches and leaves overlap with lamp heads in two dimensions in a single frame, the verification is performed by comparing the changing trends of lamp head coverage C_cov, optical path intrusion rate S_path, and image illumination retention rate R in consecutive frames.

[0172] If the overlap between branches and leaves and the lamp head is only caused by the observation angle, the lamp head coverage and light path intrusion rate will decrease rapidly as the vehicle position changes, and the image illumination retention rate will also return to normal. If the branches and leaves actually block the lamp head or light path, the lamp head coverage, light path intrusion rate and image illumination retention rate will still maintain a relatively stable trend under different observation angles.

[0173] In other implementations, the current diagnostic results can be reviewed by combining the continuous observation time window, the voting results of the diagnostic results, the characteristic change amplitude of adjacent observation frames, and the historical diagnostic results of the same street light in different inspection batches.

[0174] After confirmation of consecutive frames, the vehicle positioning information and acquisition time corresponding to the same street light observation sequence are obtained. The street light position is determined based on the tracking trajectory of the same street light, vehicle positioning information and acquisition time, and the coordinates and corresponding time information of the diagnostic event are recorded.

[0175] When the branch and leaf areas corresponding to multiple consecutive streetlights have spatial continuity in consecutive frames, and the cross-frame tracking results show that the branch and leaf areas are related to each other, it is determined that the branch and leaf areas belong to the same tree canopy, and multiple consecutive streetlights affected by the same tree canopy are merged into the same affected road segment.

[0176] For the affected road sections after merging, the length of the affected road sections can be calculated based on the location information of the starting and ending streetlights, or it can be determined based on the range of streetlights with continuous abnormalities in the vehicle positioning trajectory.

[0177] By using the above-mentioned continuous frame confirmation and multi-view consistency verification, we can avoid misjudging accidental occlusion, instantaneous strong light and image jitter in a single frame image as real lighting problems, thereby improving the stability of diagnostic results in dynamic inspection scenarios.

[0178] In one embodiment, step S170 described above may include steps S171 to S173.

[0179] S171. For an observation sequence with the same street light identifier, when a preset number of consecutive observation frames obtain the same diagnostic result, and the same diagnostic result remains consistent at at least two different observation angles, the diagnostic result is confirmed to be valid.

[0180] In this embodiment, an observation frame refers to a single frame of road inspection image within the same streetlight observation sequence. The number of consecutive preset frames can be set according to the inspection speed, video frame rate, and road environment, preferably 5 to 10 frames.

[0181] Multi-view consistency verification is used to eliminate the following interferences: The instantaneous brightness change caused by the high beams of oncoming vehicles; The temporary shade caused by leaves swaying in the wind; Image jitter caused by vehicle bumps; The branches and leaves and the lamp base are accidentally overlapped in two dimensions due to the viewing angle; The instantaneous changes in light spots caused by road surface reflections or water accumulation.

[0182] When branches and foliage overlap with the lamp head in a single frame, the trends of lamp head coverage, optical path intrusion rate, and image illumination retention rate are further compared in consecutive frames. True occlusion typically remains stable under different viewing angles, while accidental overlap caused by perspective disappears as the vehicle's position changes.

[0183] In other implementations, diagnostic confirmation can also be achieved through continuous time window confirmation, diagnostic result voting confirmation, confirmation of multiple inspection results of the same street light, or segment-level confirmation by combining the spatial distribution of adjacent street lights.

[0184] S172. Obtain the vehicle positioning information and acquisition time corresponding to the observation frame, and determine the street light position based on the tracking trajectory, vehicle positioning information and acquisition time of the same street light; when the branch and leaf areas corresponding to multiple consecutive street lights have spatial continuity in consecutive frames and the cross-frame tracking results are correlated, determine that the branch and leaf areas belong to the same canopy, and merge the corresponding multiple consecutive street lights into the same affected road segment.

[0185] In this embodiment, the tracking trajectory refers to the sequence of position changes formed by recording the position of the lamp head area of ​​the same street lamp in consecutive observation frames according to the acquisition time when performing cross-frame tracking of the same street lamp in step S120. Cross-frame tracking result refers to the association result obtained in step S120 by associating the lamp head regions in each image frame with the same street lamp identifier based on the appearance features, inter-frame motion features, optical flow continuity and spatial positional relationship of the lamp head regions in different image frames.

[0186] In one specific implementation, the orientation and distance of the streetlight relative to the vehicle can be calculated based on vehicle positioning information, vehicle heading information, camera installation location, and the position of the streetlight area in the image. The streetlight area is then mapped to the road coordinate system to obtain the streetlight position. When precise image ranging is not possible, the vehicle positioning point corresponding to the closest position to the road center in the image can be used as the approximate streetlight position, or the streetlight position can be determined by matching the image recognition results with the position of the light pole in the road lighting asset map.

[0187] When determining whether the foliage regions corresponding to multiple consecutive streetlights belong to the same tree canopy, the foliage regions in adjacent image frames can be registered to determine whether there is spatial overlap, edge continuity, or trunk connection between the foliage masks. If the foliage masks maintain spatial continuity in consecutive frames, and the center distance between the foliage masks corresponding to adjacent streetlights is less than a preset distance, or if the foliage masks have continuous connected areas after registration, then the foliage regions are determined to belong to the same tree canopy.

[0188] The system can also classify alarms based on the degree of decline in image illumination retention rate, lamp head coverage, optical path intrusion rate, presence of compound faults, and length of affected road sections. For example, a low-level alert is generated for scenarios where only branches and leaves intrude but do not cause effective lighting loss; a pruning alarm is generated for scenarios that have caused effective lighting loss; a maintenance alarm is generated for scenarios with lamp malfunctions or compound faults; and a road section-level centralized processing alarm is generated for scenarios where multiple streetlights are affected by the same tree canopy. By classifying alarms, maintenance personnel can prioritize addressing streetlights and road sections that have a greater impact on road lighting.

[0189] Specifically, step S172 is as follows: obtain the vehicle positioning information and acquisition time corresponding to the observation frame, determine the street lamp position based on the tracking trajectory formed in step S120, the vehicle positioning information and the acquisition time; when the branch and leaf areas corresponding to multiple consecutive street lamps have spatial continuity in consecutive frames, and the cross-frame tracking results obtained in step S120 indicate that the branch and leaf areas are related to each other, determine that the branch and leaf areas belong to the same canopy, and merge the corresponding multiple consecutive street lamps into the same affected road segment.

[0190] When the branch and leaf areas corresponding to multiple consecutive streetlights have spatial continuity in consecutive frames and the cross-frame tracking results are correlated, it is determined that the branch and leaf areas belong to the same tree canopy, and the corresponding multiple consecutive streetlights are merged into the same affected road segment.

[0191] For the affected road sections after merging, the length of the affected road sections can be calculated based on the location information of the starting and ending streetlights, or it can be determined based on the range of streetlights with continuous abnormalities in the vehicle positioning trajectory.

[0192] S173. The diagnostic record includes street light identification, street light location, acquisition time, and diagnostic category; when generating branch and leaf masks and lamp head masks, the lamp head coverage rate is recorded; when constructing the light path influence area, the light path intrusion rate is recorded; when determining or predicting the illuminated area of ​​the road surface, the image illumination retention rate is recorded; when identifying the lamp head area, the lamp head brightness retention rate is recorded; and the original road inspection image, occlusion feature calculation results, and retention rate calculation results are associated and saved.

[0193] In this embodiment, the diagnostic record includes street light identification, street light location, collection time, and diagnostic category.

[0194] When generating branch and leaf masks and lamp head masks, the diagnostic record also includes lamp head coverage; when constructing the light path influence area, the diagnostic record also includes light path intrusion rate; when determining or predicting the illuminated area of ​​the road surface, the diagnostic record also includes image illumination retention rate; when identifying the lamp head area, the diagnostic record also includes lamp head brightness retention rate.

[0195] In this embodiment, the original road inspection image refers to the acquired image that has not been labeled with diagnostic results or the acquired image that has undergone basic preprocessing; The occlusion feature calculation results refer to the lamp head coverage rate, light path intrusion rate, foliage masking, lamp head masking, light path influence area, and related intermediate analysis data; The retention rate calculation results refer to the image illumination retention rate, lamp head brightness retention rate, statistical value of illuminated area, statistical value of lamp head brightness, and the corresponding normal lighting benchmark.

[0196] The system links and saves the original road inspection images, occlusion feature calculation results, and retention rate calculation results as evidence for subsequent manual review, maintenance dispatch, and responsibility tracing.

[0197] When the diagnostic results meet the preset alarm conditions, the system triggers an alarm. For example, for direct obstruction, light path obstruction, or combined faults that have caused effective lighting loss, tasks such as tree pruning, lamp repair, or manual verification are generated; for results with insufficient evidence, tasks such as re-photographing or manual verification can be generated.

[0198] The system can also generate structured reports based on diagnostic records. These reports include a list of affected streetlights, affected road sections, diagnostic categories, degree of lighting loss, recommended treatment methods, and an index of evidence images.

[0199] By using continuous frame confirmation, multi-view consistency verification, and spatial range merging, this embodiment can effectively suppress transient interference in nighttime dynamic inspections and expand the abnormal results of a single street light into traceable, locatable, and statistically significant road segment-level diagnostic results, thereby improving the engineering application value of the inspection results.

[0200] When the diagnostic results after consecutive frame confirmation meet the preset alarm conditions, the system triggers an alarm and generates the corresponding diagnostic record.

[0201] The preset alarm conditions may include one or more of the following: The lamp holder is directly blocked, resulting in a loss of effective lighting. The light path is blocked, resulting in a loss of effective lighting. Output of lighting fixture malfunction or suspected lighting fixture malfunction; Output the results of the compound problem; Outputs the result of insufficient lighting due to non-obstruction reasons; The system repeatedly outputs results indicating insufficient evidence.

[0202] The diagnostic record includes the pole number, street light identification, location information, acquisition time, diagnostic category, lamp head coverage C_cov, optical path intrusion rate S_path, image illumination retention rate R, and lamp head brightness retention rate B.

[0203] In this embodiment, the lamp post number can be a street light number pre-registered in the road lighting asset management system; when the pre-registered lamp post number cannot be obtained, the street light identifier assigned by the system to the same street light can be used as a temporary number.

[0204] The system simultaneously saves the corresponding original road inspection images, lamp head detection results, branch and leaf segmentation results, light path influence area analysis results, and pavement illuminated area analysis results, which serve as evidence for subsequent manual review, maintenance dispatch, and responsibility tracing.

[0205] The lamp head detection results include the lamp head area location, lamp head mask, lamp head brightness statistics, and lamp head brightness retention rate; the branch and leaf segmentation results include the branch and leaf area location, branch and leaf mask, and branch and leaf boundary; the optical path influence area analysis results include the optical path influence area, the overlap relationship between the branch and leaf mask and the optical path influence area, and the optical path intrusion rate; the road surface illuminated area analysis results include the road surface illuminated area, the predicted road surface illuminated area, the area change of the illuminated area, the edge continuity change, and the fragmentation degree of the illuminated area.

[0206] The system generates a structured report based on the diagnostic records. The structured report may include a list of affected streetlights, affected road sections, streetlight locations, diagnostic categories, degree of lighting loss, length of affected road sections, recommended handling methods, alarm times, and an index of evidence images.

[0207] Through the above alarm triggering and result output methods, this embodiment can not only provide a diagnostic conclusion on whether the street light is obstructed, but also save complete image evidence and intermediate analysis results, making the diagnostic results traceable, and can directly connect to subsequent operation and maintenance processes such as tree pruning, light fixture repair and manual review.

[0208] This embodiment acquires nighttime road inspection image sequences, identifies the lamp head area, branch and leaf area, and road surface area, establishes a pairing relationship between the lamp head area and the illuminated area of ​​the road surface, and constructs the light path influence area between the lamp head and the illuminated area of ​​the road surface, thus realizing the collaborative analysis of the complete lighting link of "lamp head - branch and leaf - light path influence area - illuminated area of ​​the road surface".

[0209] Based on this, this embodiment calculates direct occlusion features and optical path occlusion features respectively, and establishes a normal lighting benchmark. By analyzing the relationship between the changes in image illumination retention rate and lamp head brightness retention rate, it distinguishes between tree occlusion, lamp abnormalities, and compound faults. At the same time, by combining continuous frame confirmation, multi-view consistency verification, and spatiotemporal continuity constraints, it improves the diagnostic reliability in nighttime dynamic inspection environments.

[0210] Compared with solutions that only identify the on / off status of streetlights, only analyze the two-dimensional overlap relationship between lamp heads and branches, or rely on manual inspections and illuminance sampling, this embodiment can upgrade occlusion identification to lighting effect diagnosis, and has significant improvements in detection accuracy, ability to detect hidden occlusions, quantification of lighting loss, differentiation of fault causes, and support for operation and maintenance decisions.

[0211] Therefore, compared with existing technologies that only identify the on / off status of streetlights, perform two-dimensional overlap analysis of lamp heads and branches based on single-frame images, or rely on manual inspections and illuminance sampling, the "Intelligent Diagnosis Method for Streetlight Tree Occlusion Based on Correlation Analysis of Lamp Head-Branch-Light Path Influence Area-Illuminated Area of ​​Road Surface" proposed and implemented in this embodiment has achieved significant progress and beneficial effects in the following aspects through the expansion of diagnostic objects, spatial relationship modeling, and quantification of lighting effects.

[0212] This embodiment's method is the first to achieve diagnostic analysis of the entire lighting chain, significantly improving the accuracy of occlusion identification. Existing technologies typically only focus on whether the lamp head itself is covered by branches and leaves, failing to determine whether the occlusion actually affects road lighting. This embodiment's method innovatively treats the lamp head, branches and leaves, the area affected by the light path, and the illuminated area of ​​the road surface as a unified analysis object, establishing a complete lighting chain association model. By analyzing actual changes in lighting effects, it determines the degree of occlusion impact, avoiding false alarms based solely on spatial proximity, thereby significantly improving the accuracy and reliability of diagnostic results.

[0213] The method in this embodiment can identify light path obstruction problems that are difficult to detect using traditional methods. Most existing visual inspection schemes rely on whether the light head is covered as the criterion, lacking the ability to detect situations where the light head is visible but branches and leaves have entered the light path propagation area. This embodiment proposes a light path influence area modeling and intrusion analysis mechanism, which can simultaneously identify both direct obstruction ("branches and leaves covering the light head") and light path obstruction ("branches and leaves intruding into the light path"), thereby discovering a large number of hidden lighting problems that are difficult to identify using traditional methods and improving the ability to detect potential road lighting hazards.

[0214] The method in this embodiment achieves a quantitative assessment of the impact of occlusion, improving the comparability and decision-making value of the results. Unlike traditional manual inspections that rely on experience-based judgment and illuminance sampling that depends on a limited number of sampling points, the method in this embodiment establishes a normal lighting benchmark model and quantifies lighting loss through indicators such as image illumination retention rate and degree of occlusion. This method can eliminate differences caused by different lamp types, installation heights, observation distances, and environmental conditions, ensuring that the detection results across different road sections, times, and inspection batches have a unified standard and comparability, providing an objective basis for pruning decisions and operation and maintenance management.

[0215] The method in this embodiment can distinguish different causes of faults, enabling precise maintenance dispatch. Existing technologies can usually only detect abnormal lighting phenomena, but it is difficult to determine their specific causes. The method in this embodiment combines the analysis of lamp head brightness characteristics, branch and leaf spatial relationships, light path obstruction degree, and the state of the illuminated area of ​​the road surface, which can effectively distinguish tree obstruction, lamp malfunction, and combined problems of both, enabling problem classification diagnosis and precise dispatch, improving operation and maintenance efficiency and reducing manual review costs.

[0216] The method in this embodiment possesses high reliability and strong anti-interference capabilities for dynamic inspection scenarios. During nighttime dynamic inspections, factors such as vehicle movement, high beam illumination, road surface reflection, and swaying leaves can easily lead to misjudgments. Compared to single-frame detection or simple temporal analysis methods, the method in this embodiment effectively suppresses transient interference and environmental noise through mechanisms such as cross-frame tracking of the same streetlight, multi-view consistency verification, continuous frame confirmation, and spatiotemporal range constraints, maintaining stable and reliable detection performance even in complex nighttime scenarios.

[0217] The method described in this embodiment has low implementation costs and is suitable for large-scale application. This method requires no sensor installation or hardware modification to existing streetlights; large-scale detection can be completed using only general-purpose imaging equipment mounted on inspection vehicles. Compared to solutions requiring dedicated measuring equipment, fixed monitoring terminals, or manual on-site verification, this method offers advantages such as low deployment costs, wide coverage, and high implementation efficiency. It can be easily integrated into smart city management, road lighting operation and maintenance, and landscaping maintenance management platforms to achieve routine inspection and intelligent management of urban streetlight obstruction issues.

[0218] In summary, the method in this embodiment is not a simple optimization of existing tree occlusion detection or lighting inspection technologies. Instead, it elevates the traditional "occlusion identification" problem to a "lighting effect diagnosis" problem by establishing a correlation analysis framework between the lamp head, branches and leaves, the area affected by the light path, and the illuminated area of ​​the road surface. This results in significant improvements in multiple dimensions, including detection principle, diagnostic capability, quantitative evaluation, fault differentiation, and engineering application, demonstrating outstanding practical value and promising prospects for widespread adoption.

[0219] The aforementioned occlusion lighting diagnosis method based on the spatial relationship between roadside trees and streetlights identifies lamp heads, branches, and road surface areas through nighttime inspection image sequences. It tracks the same streetlight across frames to stabilize target association, then extracts the illuminated area of ​​the road surface and establishes a pairing relationship between lamp heads and illuminated areas, constructing the light path influence area to establish spatial mapping. Direct occlusion is identified based on the mask overlap between branches and lamp heads and local brightness attenuation, while light path occlusion is identified based on the overlap between branches and light path influence areas and changes in the morphology of illuminated areas. The method quantifies lighting loss by calculating the image illumination retention rate and lamp head brightness retention rate using normal lighting benchmarks, and performs fusion discrimination to distinguish between tree occlusion, lamp abnormalities, and combined faults. Finally, after multi-frame, multi-angle consistency confirmation, it outputs a structured diagnostic record containing location, category, coverage, intrusion rate, retention rate, and associated images. This effectively overcomes the shortcomings of existing technologies in dynamic inspection environments, such as unstable target association, lack of spatial relationship analysis, lack of quantitative assessment mechanism for lighting loss, and insufficient ability to distinguish fault causes, thereby improving the accuracy, reliability, and decision support capabilities of nighttime road lighting inspection.

[0220] Figure 5 This is a schematic block diagram of an obstruction lighting diagnosis system 300 based on the spatial relationship between roadside trees and streetlights, provided in an embodiment of the present invention. Figure 5 As shown, corresponding to the above-described method for diagnosing occlusion lighting based on the spatial relationship between roadside trees and streetlights, the present invention also provides an occlusion lighting diagnosis system 300 based on the spatial relationship between roadside trees and streetlights. This occlusion lighting diagnosis system 300 includes a unit for executing the above-described method for diagnosing occlusion lighting based on the spatial relationship between roadside trees and streetlights, and the system can be configured in a server. Specifically, please refer to... Figure 5 The occlusion lighting diagnosis system 300 based on the spatial relationship between roadside trees and streetlights includes an image acquisition unit 301, an area recognition and streetlight tracking unit 302, an optical path influence area construction unit 303, an occlusion feature determination unit 304, a benchmark establishment unit 305, a fault diagnosis unit 306, and a consistency confirmation unit 307.

[0221] Image acquisition unit 301 is used to acquire a sequence of nighttime road inspection images; region recognition and street light tracking unit 302 is used to identify the lamp head region, branch and leaf region, and road surface region in the nighttime road inspection image sequence, and perform cross-frame tracking on the same street light to obtain the corresponding observation sequence; optical path influence region construction unit 303 is used to extract the illuminated area of ​​the road surface in the road surface region, establish a pairing relationship between the lamp head region and the illuminated area of ​​the road surface, and construct the optical path influence region between the two; occlusion feature determination unit 304 is used to determine direct occlusion features based on the overlap relationship between the branch and leaf region and the lamp head region, and determine optical path occlusion features based on the overlap relationship between the branch and leaf region and the optical path influence region; reference establishment unit 305 is used to establish a normal lighting reference, determine the image illumination retention rate of the illuminated area of ​​the road surface relative to the normal lighting reference, and the lamp head brightness retention rate of the lamp head area relative to the normal lighting reference; Fault diagnosis unit 306 is used to fuse the direct occlusion feature, the optical path occlusion feature, the image illumination retention rate, and the lamp head brightness retention rate, and by fusing the image illumination retention rate, the lamp head brightness retention rate, and the direct occlusion feature and the optical path occlusion feature, distinguish tree occlusion, lamp abnormality, and composite faults where tree occlusion and lamp abnormality coexist, and form a diagnosis result; Consistency confirmation unit 307 is used to confirm the consistency of the diagnosis results of the same street light under multiple frames and different observation angles, and output a diagnosis record.

[0222] In one embodiment, the area identification and street light tracking unit 302 includes an identification subunit and a tracking subunit.

[0223] The identification subunit is used to identify the lamp head region in the image sequence using a detection model, and to identify the branch and leaf region and the road surface region using a semantic segmentation model; the tracking subunit is used to assign a unique street lamp identifier to the same street lamp based on the appearance features, motion trajectory, optical flow continuity and spatial position relationship of the lamp head region, and to form an observation sequence from when the same street lamp enters the image field of view to when it leaves the image field of view.

[0224] In one embodiment, the optical path influence region construction unit 303 includes: The system comprises the following subunits: a region extraction subunit, used to extract connected regions with brightness higher than the road background using adaptive threshold segmentation within the road surface area, to obtain the illuminated area of ​​the road surface; a pairing subunit, used to determine the illuminated area of ​​the road surface corresponding to the lamp head area based on the spatial nearest neighbor relationship and relative position relationship between the lamp head area and each illuminated area of ​​the road surface; when the lamp head area corresponds to multiple illuminated areas of the road surface, the multiple illuminated areas of the road surface are merged to establish a unique pairing relationship; an overlap determination subunit, used to construct a trapezoidal approximate region based on the edges of the lamp head area and the illuminated area of ​​the road surface that are close to each other, and to perform dilation processing on the trapezoidal approximate region; when the trapezoidal approximate regions corresponding to adjacent lamp head areas overlap, the ownership of the overlapping part is determined according to the spatial nearest neighbor relationship between the overlapping part and each lamp head area, to obtain the optical path influence region; and a region generation subunit, used to generate a predicted illuminated area of ​​the road surface based on the position and shape of the illuminated area of ​​the road surface in the historical observation frame corresponding to the same street lamp identifier when no illuminated area of ​​the road surface is extracted in the current image frame, and to construct the optical path influence region.

[0225] In one embodiment, the occlusion feature determination unit 304 includes: The lamp head coverage determination subunit is used to convert the branch and leaf region and the lamp head region into a branch and leaf mask and a lamp head mask, respectively, and calculate the ratio of the overlap area of ​​the branch and leaf mask to the area of ​​the lamp head mask to obtain the lamp head coverage rate; the direct occlusion determination subunit is used to determine that the branch and leaf region directly occludes the lamp head region when the lamp head coverage rate is not lower than the lamp head coverage threshold and there is a local brightness attenuation in the lamp head region corresponding to the boundary space of the branch and leaf mask, and to use the lamp head coverage rate and the local brightness attenuation as the direct occlusion features. The optical path intrusion rate generation subunit is used to calculate the ratio of the overlap area between the branch and leaf mask and the area affected by the optical path to the area of ​​the area affected by the optical path, thereby obtaining the optical path intrusion rate. The optical path occlusion determination subunit is used to determine that the branch and leaf area forms an optical path occlusion on the area affected by the optical path when the lamp head coverage rate is lower than the lamp head coverage threshold, the optical path intrusion rate is not lower than the optical path intrusion threshold, and the illuminated area of ​​the road surface decreases or the edge continuity decreases relative to the previous observation frame corresponding to the same street lamp sign. The optical path intrusion rate, the area change characteristics of the illuminated area of ​​the road surface, and the edge continuity change characteristics are used as the optical path occlusion characteristics.

[0226] In one embodiment, the reference establishment unit 305 is further configured to select streetlights of the same type and type on the same road, and whose lamps are confirmed to be normal and unobstructed by branches and leaves, as normal lighting samples, and establish a reference for the illuminated area area, a reference for the lamp head brightness, and a reference for the illuminated area shape; when the number of normal lighting samples does not reach a preset number, a preset initial reference is adopted; when the number of normal lighting samples reaches a preset number, the normal lighting reference is updated according to the statistical results of the normal lighting samples; based on observation frames with illuminated road surfaces or predicted illuminated road surfaces, the ratio of the average illuminated area area to the illuminated area reference is calculated to obtain the image illumination retention rate; based on observation frames that can identify the lamp head area, the ratio of the average brightness of the lamp head area to the lamp head brightness reference is calculated to obtain the lamp head brightness retention rate.

[0227] In one embodiment, the fault diagnosis unit 306 is configured to: output a composite fault result when there is direct obstruction or optical path obstruction and the lamp head brightness retention rate is lower than the lamp head brightness threshold; output a luminaire abnormality result when there is no direct obstruction or optical path obstruction and the lamp head brightness retention rate is lower than the lamp head brightness threshold; output a direct obstruction lamp head result when the lamp head brightness retention rate is not lower than the lamp head brightness threshold and there is direct obstruction; and output an optical path obstruction result when the lamp head brightness retention rate is not lower than the lamp head brightness threshold and there is optical path obstruction. When the result of direct occlusion of the lamp head or the result of optical path occlusion is output and the image illumination retention rate is lower than the illumination retention threshold, the corresponding tree occlusion is determined to cause effective illumination loss; when there is no direct occlusion or optical path occlusion, and the lamp head brightness retention rate is not lower than the lamp head brightness threshold, if the image illumination retention rate is lower than the illumination retention threshold, the result of insufficient illumination caused by non-occlusion is output, otherwise the result of no occlusion is output; when the lamp head area cannot be identified, or the illuminated area of ​​the road surface cannot be determined in the current image frame and the historical observation frame, the result of insufficient evidence is output.

[0228] In one embodiment, the consistency confirmation unit 307 is used to: for an observation sequence with the same street light identifier, when a preset number of consecutive observation frames obtain the same diagnostic result, and the same diagnostic result remains consistent at at least two different observation angles, confirm that the diagnostic result is valid; acquire vehicle positioning information and acquisition time corresponding to the observation frame, and determine the street light position based on the tracking trajectory, vehicle positioning information, and acquisition time of the same street light; when the branch and leaf areas corresponding to multiple consecutive street lights have spatial continuity in consecutive frames, and the cross-frame tracking results are correlated, determine that the branch and leaf areas belong to the same canopy, and merge the corresponding multiple consecutive street lights into the same affected road segment; the diagnostic record includes street light identifier, street light position, acquisition time, and diagnostic category; when generating branch and leaf masks and lamp head masks, record the lamp head coverage rate; when constructing the light path influence area, record the light path intrusion rate; when determining or predicting the illuminated area of ​​the road surface, record the image illumination retention rate; when identifying the lamp head area, record the lamp head brightness retention rate; and associate and save the original road inspection image, occlusion feature calculation result, and retention rate calculation result.

[0229] In one embodiment, the image acquisition unit 301 is used to acquire the image sequence by the vehicle during driving, including multiple frames of road inspection images and the acquisition time, vehicle positioning information and vehicle heading information corresponding to each frame of road inspection images; align each frame of road inspection images, vehicle positioning information and vehicle heading information according to the acquisition time, and perform distortion correction, noise suppression, brightness normalization and local contrast enhancement on each frame of road inspection images; remove image frames with motion blur, overexposure or obscured by vehicles in front, register and fuse the retained consecutive multiple frames of road inspection images, and perform perspective normalization on the road surface area in the fused image.

[0230] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned occlusion lighting diagnosis system 300 based on the spatial relationship between roadside trees and streetlights and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0231] The aforementioned occlusion lighting diagnostic system 300 based on the spatial relationship between roadside trees and streetlights can be implemented as a computer program, which can be used in various ways, such as... Figure 6 It runs on the computer device shown.

[0232] Please see Figure 6 , Figure 6 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0233] See Figure 6 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0234] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a method for diagnosing occlusion lighting based on the spatial relationship between roadside trees and streetlights.

[0235] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0236] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a method for diagnosing occlusion lighting based on the spatial relationship between roadside trees and streetlights.

[0237] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0238] The processor 502 is used to run the computer program 5032 stored in the memory to implement all the steps of the occlusion lighting diagnosis method based on the spatial relationship between roadside trees and streetlights.

[0239] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0240] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0241] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform all the steps of the occlusion lighting diagnosis method based on the spatial relationship between roadside trees and streetlights.

[0242] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0243] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0244] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0245] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the system of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0246] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0247] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for diagnosing occlusion lighting based on the spatial relationship between roadside trees and streetlights, characterized in that, include: Acquire image sequences for nighttime road inspections; Identify the lamp head region, branch and leaf region, and road surface region in the image sequence, and perform cross-frame tracking on the same street lamp to obtain the observation sequence; Extract the illuminated area of ​​the road surface from the road surface area, establish the pairing relationship between the lamp head area and the illuminated area of ​​the road surface, and construct the light path influence area; The direct occlusion feature is determined based on the overlap relationship between the branch and leaf area and the lamp head area, and the optical path occlusion feature is determined based on the overlap relationship between the branch and leaf area and the optical path influence area. Establish a normal lighting benchmark and determine the image illumination retention rate of the illuminated area of ​​the road surface and the lamp head brightness retention rate of the lamp head area; Based on the direct occlusion features, the optical path occlusion features, the image illumination retention rate, and the lamp head brightness retention rate, by fusing the image illumination retention rate, the lamp head brightness retention rate, the direct occlusion features, and the optical path occlusion features, tree occlusion, lamp abnormalities, and combined faults of the two are distinguished to form a diagnostic result. The diagnostic results for the same street light are confirmed to be consistent across multiple frames and different observation angles, and diagnostic records are output.

2. The occlusion lighting diagnosis method based on the spatial relationship between roadside trees and streetlights according to claim 1, characterized in that, The process of identifying the lamp head region, branch and leaf region, and road surface region in the image sequence, and performing cross-frame tracking on the same street lamp to obtain the observation sequence, includes: A detection model is used to identify the lamp head region in the image sequence, and a semantic segmentation model is used to identify the branch and leaf region and the road surface region; Based on the appearance characteristics, movement trajectory, optical flow continuity and spatial position relationship of the lamp head area, a unique street lamp identifier is assigned to the same street lamp, and an observation sequence is formed from the moment the same street lamp enters the image field of view to the moment it leaves the image field of view.

3. The occlusion lighting diagnosis method based on the spatial relationship between roadside trees and streetlights according to claim 2, characterized in that, The process of extracting the illuminated area of ​​the road surface in the road surface area, establishing a pairing relationship between the lamp head area and the illuminated area of ​​the road surface, and constructing the light path influence area includes: Within the road surface area, adaptive threshold segmentation is used to extract connected regions with brightness higher than the road surface background to obtain the illuminated area of ​​the road surface. Based on the spatial nearest neighbor relationship and relative positional relationship between the lamp head area and each road surface illuminated area, the road surface illuminated area corresponding to the lamp head area is determined; when the lamp head area corresponds to multiple road surface illuminated areas, the multiple road surface illuminated areas are merged to establish a unique pairing relationship; A trapezoidal approximate region is constructed using the edges of the lamp head area and the illuminated area of ​​the road surface that are close to each other, and the trapezoidal approximate region is expanded. When the trapezoidal approximate regions corresponding to adjacent lamp head areas overlap, the ownership of the overlapping part is determined according to the spatial nearest neighbor relationship between the overlapping part and each lamp head area, thus obtaining the light path influence area. When no illuminated area of ​​the road surface is extracted in the current image frame, a predicted illuminated area of ​​the road surface is generated based on the location and shape of the illuminated area of ​​the road surface in the historical observation frames corresponding to the same street light identifier, and the light path influence area is constructed.

4. The occlusion lighting diagnosis method based on the spatial relationship between roadside trees and streetlights according to claim 3, characterized in that, The step of determining the direct occlusion feature based on the overlap between the branch and leaf area and the lamp head area includes: The branch and leaf area and the lamp head area are converted into branch and leaf mask and lamp head mask, respectively. The ratio of the overlap area of ​​the branch and leaf mask and the lamp head mask to the area of ​​the lamp head mask is calculated to obtain the lamp head coverage rate. When the lamp head coverage rate is not lower than the lamp head coverage threshold, and there is a local brightness attenuation in the lamp head area corresponding to the boundary space of the branch and leaf mask, it is determined that the branch and leaf area directly occludes the lamp head area, and the lamp head coverage rate and the local brightness attenuation are used as the direct occlusion features.

5. The occlusion lighting diagnosis method based on the spatial relationship between roadside trees and streetlights according to claim 4, characterized in that, Determining the optical path occlusion features based on the overlap relationship between the branch and leaf region and the optical path influence region includes: The optical path intrusion rate is obtained by calculating the ratio of the overlap area between the branch and leaf mask and the area of ​​the optical path affected region to the area of ​​the optical path affected region. When the lamp head coverage rate is lower than the lamp head coverage threshold, the light path intrusion rate is not lower than the light path intrusion threshold, and the illuminated area of ​​the road surface shows a decrease in area or a decrease in edge continuity relative to the previous observation frame corresponding to the same street lamp sign, it is determined that the branch and leaf area forms a light path occlusion in the light path affected area, and the light path intrusion rate, the area change characteristics of the illuminated area of ​​the road surface, and the edge continuity change characteristics are used as the light path occlusion characteristics.

6. The occlusion lighting diagnosis method based on the spatial relationship between roadside trees and streetlights according to claim 5, characterized in that, The establishment and comparison of the normal lighting reference includes: Streetlights of the same type and type on the same road, and whose lamps are confirmed to be in good working order and without obstruction by branches and leaves, are selected as normal lighting samples to establish benchmarks for the illuminated area, lamp head brightness, and illuminated area shape. When the number of normal lighting samples does not reach the preset number, a preset initial benchmark is adopted; when the number of normal lighting samples reaches the preset number, the normal lighting benchmark is updated according to the statistical results of the normal lighting samples. Based on the observation frames with illuminated or predicted road surface areas, the ratio of the average illuminated area area to the illuminated area reference is calculated to obtain the image illumination retention rate. Based on the observation frames that can identify the lamp head area, the ratio of the average brightness of the lamp head area to the lamp head brightness reference is calculated to obtain the lamp head brightness retention rate.

7. The method for diagnosing occlusion lighting based on the spatial relationship between roadside trees and streetlights according to claim 6, characterized in that, The diagnostic results, which differentiate between tree obstruction, malfunctioning lighting fixtures, and combined faults of both, include: When there is direct obstruction or optical path obstruction, and the lamp head brightness retention rate is lower than the lamp head brightness threshold, a composite fault result is output. When there is no direct obstruction or optical path obstruction, and the lamp head brightness retention rate is lower than the lamp head brightness threshold, an abnormal lamp result is output. When the lamp head brightness retention rate is not lower than the lamp head brightness threshold and there is direct occlusion, output the direct occlusion lamp head result; when the lamp head brightness retention rate is not lower than the lamp head brightness threshold and there is optical path occlusion, output the optical path occlusion result; when the direct occlusion lamp head result or the optical path occlusion result is output and the image illumination retention rate is lower than the illumination retention threshold, determine that the corresponding tree occlusion caused effective illumination loss; When there is no direct occlusion or optical path occlusion, and the lamp head brightness retention rate is not lower than the lamp head brightness threshold, if the image illumination retention rate is lower than the illumination retention threshold, output the illumination deficiency result caused by non-occlusion; otherwise, output the no-occlusion result. When the lamp head area cannot be identified, or when neither the current image frame nor the historical observation frame can determine the illuminated area of ​​the road surface, an insufficient evidence result is output.

8. The method for diagnosing occlusion lighting based on the spatial relationship between roadside trees and streetlights according to claim 7, characterized in that, The process of verifying the consistency of diagnostic results for the same streetlight across multiple frames and different observation angles, and outputting diagnostic records, includes: For an observation sequence with the same street light identifier, when a preset number of consecutive observation frames obtain the same diagnostic result, and the same diagnostic result remains consistent at at least two different observation angles, the diagnostic result is confirmed to be valid. Obtain vehicle positioning information and acquisition time corresponding to the observation frame, and determine the street light position based on the tracking trajectory, vehicle positioning information and acquisition time of the same street light; when the branch and leaf areas corresponding to multiple consecutive street lights have spatial continuity in consecutive frames and the cross-frame tracking results are correlated, determine that the branch and leaf areas belong to the same tree canopy, and merge the corresponding multiple consecutive street lights into the same affected road segment; The diagnostic record includes street light identification, street light location, acquisition time, and diagnostic category; when generating branch and leaf masks and lamp head masks, the lamp head coverage rate is recorded; when constructing the light path influence area, the light path intrusion rate is recorded; when determining or predicting the illuminated area of ​​the road surface, the image illumination retention rate is recorded; when identifying the lamp head area, the lamp head brightness retention rate is recorded; and the original road inspection image, occlusion feature calculation results, and retention rate calculation results are associated and saved.

9. The method for diagnosing occlusion lighting based on the spatial relationship between roadside trees and streetlights according to claim 1, characterized in that, The image sequence was collected by the vehicle during its journey and includes multiple road inspection images, as well as the collection time, vehicle positioning information, and vehicle heading information corresponding to each road inspection image. The road inspection images, vehicle positioning information, and vehicle heading information of each frame are aligned according to the acquisition time, and distortion correction, noise suppression, brightness normalization, and local contrast enhancement are performed on each frame of road inspection images. Image frames that are motion-blurred, overexposed, or obscured by vehicles ahead are removed. The remaining consecutive road inspection images are then registered and fused, and perspective normalization is performed on the road surface area in the fused image.

10. A shading lighting diagnostic system based on the spatial relationship between roadside trees and streetlights, characterized in that, include: Image acquisition unit, used to acquire nighttime road inspection image sequences; The area recognition and street light tracking unit is used to identify the lamp head area, branch and leaf area and road surface area in the nighttime road inspection image sequence, and to perform cross-frame tracking of the same street light to obtain the corresponding observation sequence; The optical path influence area construction unit is used to extract the illuminated area of ​​the road surface in the road surface area, establish the pairing relationship between the lamp head area and the illuminated area of ​​the road surface, and construct the optical path influence area between the two. The occlusion feature determination unit is used to determine direct occlusion features based on the overlap relationship between the branch and leaf area and the lamp head area, and to determine optical path occlusion features based on the overlap relationship between the branch and leaf area and the optical path influence area. A reference establishment unit is used to establish a normal lighting reference, determine the image illumination retention rate of the illuminated area of ​​the road surface relative to the normal lighting reference, and the lamp head brightness retention rate of the lamp head area relative to the normal lighting reference; The fault diagnosis unit is used to fuse the direct occlusion features, the optical path occlusion features, the image illumination retention rate, and the lamp head brightness retention rate. By fusing the image illumination retention rate, the lamp head brightness retention rate, and the direct occlusion features and the optical path occlusion features, it distinguishes between tree occlusion, lamp malfunction, and composite faults where tree occlusion and lamp malfunction coexist, and forms a diagnostic result. The consistency confirmation unit is used to confirm the consistency of the diagnostic results of the same street light in multiple frames and at different observation angles, and output the diagnostic record.

Images