A block hidden danger identification method and system based on static background reconstruction and a medium

Abstract

The application relates to the technical field of urban governance risk perception and intelligent video analysis, and discloses a street block hidden danger identification method and system based on static background reconstruction and a medium. The method first acquires video data and spatial attribute data of a target street block point, extracts and quality screens video frames; subsequently, generates a street static background image after removing pedestrians, vehicles and other instantaneous dynamic occlusions through local feature matching, robust spatial alignment and time sequence multi-frame background fusion; identifies static hidden dangers such as flying wire charging, fire passage occupation, illegal overlaying, outer facade damage and shop outdoor stacking based on the static background image and the point spatial attribute, and generates a risk label, a risk level, a confidence, an evidence abstract and an evidence backtracking path. The application can improve the stability and reviewability of static hidden danger identification in high-occlusion street and alley scenes, and is suitable for governance risk screening in urban villages, old street blocks and street-side shops and the like.

Description

Technical Field

[0001] This invention belongs to the fields of urban governance risk perception, intelligent video analysis, image registration, video background reconstruction and computer vision technology, and specifically relates to a method, system and medium for identifying street hazards based on static background reconstruction. Background Technology

[0002] With the advancement of urban renewal, urban health checks, and the construction of urban operation management service platforms, the targets of urban governance have gradually expanded from open public spaces such as main and secondary roads, squares, and parks to high-density built environments such as urban villages, old neighborhoods, back streets and alleys, street-front shops, storefront back areas, areas adjacent to small factories, and the periphery of enclosed courtyards. These areas typically feature small building spacing, high pedestrian and vehicular traffic, complex spatial interfaces, dense business activities, and aging public facilities, making them prone to static governance hazards such as illegal charging cables, blocked fire lanes, unauthorized additions, damaged facades, outdoor storage, damaged public facilities, road or alleyway clutter, and external signs of suspected "three-in-one" premises (combined residential, commercial, and industrial uses). These hazards are usually attached to building facades, storefront front areas, alleyway boundaries, road cross-sections, pipeline facilities, and the edges of public spaces, exhibiting characteristics such as persistence, slow change, partial obstruction, and a strong need for on-site verification.

[0003] Current methods for identifying urban governance risks mainly include manual patrols, reports from grid workers, public complaints, special campaigns, and intelligent video recognition. Manual patrols and reports from grid workers offer strong on-site judgment capabilities, but the frequency of patrols, patrol routes, and personnel experience can affect detection efficiency, making it difficult to continuously, stably, and cost-effectively screen for static hazards in high-density neighborhoods. Public complaints are typically delayed and cannot cover potential hazards that have not yet caused obvious disturbances or safety incidents. Special campaigns can identify problems concentrated in specific time periods, but are not suitable for routine, granular, and multi-location risk front-end detection.

[0004] In recent years, urban operation management service platforms, smart city management systems, and unified management platforms have gradually introduced technologies such as video surveillance, mobile inspection, artificial intelligence recognition, and automatic dispatching to identify urban management issues such as street vending, exposed garbage, road damage, illegal parking, and abnormal manhole covers. These technologies can improve the efficiency of discovering and dispatching urban governance issues. However, their video intelligent recognition typically focuses on directly visible targets in single frames or short clips, emphasizing the detection of dynamic events, target categories, or local anomalies. For static hazards such as overhead wires, illegal additions, damaged facades, piles of goods in front of doors, and long-term obstruction of fire lanes, if pedestrians, vehicles, stalls, temporary obstructions, or high-frequency moving targets are present in the image, single-frame recognition is easily affected by occlusion interference, leading to missed detections, false detections, or unstable location of hazard areas.

[0005] In the field of computer vision, background subtraction, background modeling, and background initialization are fundamental techniques in video surveillance, moving object detection, object tracking, and foreground extraction. Existing background modeling methods typically separate foreground moving objects from background regions in video sequences using pixel statistics, Gaussian mixture models, low-rank decomposition, optical flow, deep learning, or spatiotemporal models. These methods primarily serve moving object detection, i.e., extracting foreground objects from the background, or providing foreground masks for subsequent object tracking. However, the goal of static hazard identification in urban blocks is not simply to extract moving foregrounds, but to obtain a static block background image that is as complete and clear as possible, allowing for manual review and semantic recognition, and to identify static hazards attached to building facades, roadside facilities, store boundaries, passageway entrances, and public space edges from this background image. Therefore, there are still differences in objectives and application gaps between ordinary background subtraction techniques and static hazard identification in urban blocks.

[0006] For fixed or surveillance-like videos, street scenes present challenges such as slight camera shake, changes in shooting angle, lighting variations, shadow changes, close-range vehicle obstruction, dense pedestrian traffic, localized image blurring, and difficulties in matching low-texture areas. Directly performing pixel-level averaging or median fusion on unaligned multi-frame images can easily result in ghosting, edge misalignment, or blurred background details, affecting subsequent judgments of overhead wires, facade damage, object boundaries, and passageway conditions. Conversely, relying solely on single-frame image recognition makes it difficult to reconstruct static elements such as roller shutters, fire lanes, street railings, road boundaries, and building facades obscured by people and vehicles. Therefore, it is necessary to perform candidate frame screening and robust spatial alignment before background fusion to improve the integrity, clarity, and geometric consistency of the static background image.

[0007] Furthermore, existing intelligent video recognition systems typically focus on the target identification result itself, lacking an organizational method for organizing the evidence chain of static hazards. For urban governance, simply outputting "abnormality exists" or "suspected hazard" is insufficient to support subsequent review and handling. It is also necessary to retain structured information such as location number, collection time, spatial coordinates, original video path, candidate frame path, background image path, hazard area, risk level, confidence level, and evidence summary. Especially in street governance scenarios, hazards may be long-term and changeable, such as newly added piles of objects, newly added overhead wires, newly added covered structures, long-term occupied areas, or gradual damage to facades. Without a comparison mechanism of static background images of the same location at multiple time periods, it is difficult to distinguish between temporary obstructions, short-term activities, and persistent static hazards.

[0008] Existing multimodal visual models and image semantic recognition models possess strong scene understanding capabilities. However, directly inputting continuous video or a large number of unfiltered raw frames leads to high data transmission and model computation costs. Furthermore, these models are susceptible to interference from dynamic foregrounds and redundant frames, making it difficult to focus on the static structure and potential hazards of the street. A more suitable approach for identifying potential hazards in street areas is to first perform candidate frame filtering, image alignment, and background reconstruction locally to generate a low-redundancy, minimally occluded, and geometrically consistent static background image of the street. This static background image is then combined with location spatial attributes, structured prompts, or local rules to identify static hazards and generate verifiable evidence.

[0009] In summary, current technologies lack a method for identifying static hazards in heavily obscured street videos. This method should be capable of selecting suitable candidate frames for background reconstruction from fixed or surveillance-like videos, eliminating minor camera shake and frame misalignment through local feature matching and robust spatial alignment, and then reconstructing a static street background image by removing transient dynamic foreground elements such as pedestrians and vehicles through multi-frame background fusion. Furthermore, it should combine spatial attribute data and hazard rules to identify static hazards such as illegal charging cables, obstruction of fire lanes, unauthorized additions, damaged facades, and outdoor storage. The final output should be structured hazard evidence data including the background image, hazard category, hazard area, risk level, confidence level, evidence summary, and original video playback path. Developing such a method is of significant technical importance for improving the automated screening capabilities for static hazards in high-density street areas, reducing manual inspection costs, increasing the reusability of video data, and enhancing the verifiability of governance evidence. Summary of the Invention

[0010] To address the challenges of existing urban video intelligent recognition methods in reliably identifying static hazards in heavily obscured street scenes, the susceptibility of ordinary single-frame recognition to instantaneous dynamic foreground interference from pedestrians, vehicles, stalls, and temporary obstructions, the difficulty of directly using traditional background modeling methods to generate street hazard evidence, and the lack of a structured conversion mechanism in existing urban governance systems from raw video to static background images, hazard labels, risk levels, and evidence backtracking paths, this invention proposes a street hazard identification method, system, and medium based on static background reconstruction.

[0011] The core idea of ​​this invention is as follows: For urban villages, old neighborhoods, street-front shops, back alleys, courtyard perimeters, and store backyard perimeters, where human and vehicle obstructions are frequent and spatial interfaces are complex, this invention does not rely directly on single-frame images for hazard assessment. Instead, it first selects suitable candidate frames for background reconstruction from fixed or surveillance-like videos, then eliminates slight camera jitter and inter-frame misalignment through local feature matching and robust spatial alignment, and generates a static background image of the street after removing instantaneous dynamic foreground elements such as pedestrians and vehicles through multi-frame background fusion. Based on this static background image and spatial attributes of the location, it identifies static hazards such as illegal charging wires, fire lane obstruction, illegal additions, facade damage, outdoor storage, and damage to public facilities, and outputs structured hazard evidence data that is verifiable, traceable, and system-interoperable.

[0012] To achieve the above objectives, the present invention adopts the following technical solution.

[0013] A method for identifying potential hazards in urban areas based on static background reconstruction includes the following steps: Step 1: Set up observation points in the target street area, and use camera equipment to collect video data and obtain spatial attribute data at the observation points. The spatial attribute data includes point number, latitude and longitude coordinates, road name, street type, collection time, camera position type, nearby points of interest, road network relationship, and the governance grid to which the target street point belongs. Step 2: Extract video frame sequences from the video data, perform frame extraction and quality screening on the video frame sequences to obtain a candidate frame set; Step 3: Normalize the size of the candidate frames in the candidate frame set, select one frame in the candidate frame set as the reference frame, perform feature matching between the candidate frame and the reference frame to obtain the matching point set; use the random sampling consensus algorithm to estimate the affine transformation matrix of the matching points in the matching point set, and map the candidate frame to the coordinate system of the reference frame to obtain the aligned frame. Step 4: Repeat steps 3-4 for all candidate frames in the candidate frame set to obtain the aligned frame set; Step 5: Perform multi-frame background fusion on the aligned frame set to obtain a static background image of the street block, and calculate the confidence score of the static background image of the street block; Step 6: Perform static hazard identification based on the static background image of the street block and the spatial attribute data to obtain the static hazard identification result; wherein, the static hazard identification result includes at least one of candidate hazard label, hazard area, risk level, model identification confidence score and evidence summary; Step 7: Repeat steps 1-6 in multiple time windows to obtain static background images of the same target street location and compare them to calculate the confidence level of hazard persistence. Step 8: Calculate the hazard confidence score by weighting the model identification confidence score, the street static background image confidence score, the spatial scene matching confidence score, and the multi-time period hazard persistence confidence score, and determine the risk level based on the hazard confidence score and hazard type weight; The target street locations mentioned in step 1 include at least one of the following: street entrances, street-front shops, old street roads, alleyways in urban villages, courtyard perimeters, store backyard perimeters, areas adjacent to public facilities, or road sections in mixed residential and commercial areas; the video data includes fixed camera videos, mobile inspection videos, surveillance-like video collection videos, or other video files that can reflect the spatial status of the street. The location number, latitude and longitude coordinates, or acquisition time are obtained from the video file name parsing results, acquisition device positioning data, manual annotation data, or geographic information matching results.

[0014] Furthermore, the step 2 of extracting frames and screening quality in the video frame sequence includes: extracting initial frames according to a preset frame interval or a preset time interval to obtain an initial frame set, performing quality screening on the initial frame set to remove low-value frames, and forming the candidate frame set with the retained initial frames. The low-value frames include blurred frames, overexposed frames, underexposed frames, frames with an occlusion ratio exceeding a threshold, or frames with a similarity to the retained frames exceeding a threshold. The blurred frames are those whose scores, identified by Laplacian variance, are below the first threshold. Overexposed and underexposed frames are those whose grayscale mean is below the second threshold or above the third threshold. The inter-frame similarity is obtained through structural similarity index, image histogram correlation, perceptual hash distance, and / or feature descriptor similarity. The occlusion ratio is the proportion of the foreground region or the detection result of the significantly occluded region.

[0015] Further, the feature matching between the candidate frame and the reference frame in step 3 includes: using a ratio test to eliminate ambiguous matching points; the matcher can be a FLANN approximate nearest neighbor matcher or a brute-force matcher; for the same candidate descriptor, let the nearest neighbor matching distance be d1 and the second nearest neighbor matching distance be d2, when... When a match is selected, a matching point is retained; where ρ is the ratio test threshold; all others are discarded. The affine transformation matrix is ​​estimated using a random sampling consensus algorithm. Candidate frames are then mapped to the reference frame coordinate system to obtain aligned frames. This process involves: randomly selecting at least three pairs of matching points from the set of matching point pairs; estimating the candidate affine transformation matrix; calculating the reprojection error of all matching point pairs; marking the matching point pair as an interior point if the reprojection error is less than a threshold; otherwise, marking it as an exterior point. This process of estimating the candidate affine transformation matrix is ​​repeated multiple times. The candidate affine transformation matrix with the most interior points or the highest proportion of interior points is selected, and the least squares solution is performed again using all interior points to obtain the final affine transformation matrix. Finally, the candidate frames are mapped to the reference frame coordinate system using the affine transformation matrix to obtain aligned frames.

[0016] Further, step 5, which involves performing multi-frame background fusion on the aligned frame set to obtain a static street background image, includes: performing dynamic foreground segmentation on the aligned frame set to obtain a dynamic foreground mask; based on the dynamic foreground mask, reducing the weight of pixels in the dynamic foreground region in the background fusion, or removing pixels in the dynamic foreground region from the pixel set participating in the fusion; and performing fusion calculation on the pixel values ​​at the same pixel position in the pixel set participating in the fusion to obtain the static street background image. The fusion calculation method is at least one of median fusion, quantile fusion, mean fusion, weighted fusion, or background modeling fusion. Calculate the confidence level of static background images of a street block include: in, Rate the image alignment quality. Rate the background clarity. Rate the completeness of background coverage. , , Let be the weighting coefficient, and satisfy: The image alignment quality score can be determined based on the proportion of matched inliers, the average reprojection error, or the consistency of aligned edges: in, The proportion of RANSAC interior points. This represents the average reprojection error. These are the weighting coefficients; Furthermore, the blurred frame The score can be based on the Laplace variance score. Gradient strength scoring or edge density score Confirmed, the calculation formula is as follows:

[0017] Where η1, η2, and η3 are weighting coefficients, and η1+η2+η3=1.

[0018] In the specific calculation, the static background image B of the street is first converted into a grayscale image B. gray The grayscale value ranges from 0 to 255.

[0019] Furthermore, Laplace variance score The calculation formula is:

[0020] in, Represents the Laplacian response of a grayscale background image. This represents the variance of the Laplace response. The Laplacian variance normalization threshold is used; a larger Laplacian variance indicates clearer image edges and details. Greater than or equal to hour, Take 1; Gradient strength score for: Where G represents the gradient magnitude map of the grayscale background image, which can be calculated by the Sobel operator:

[0021] Gx and Gy are the gradients in the horizontal and vertical directions, respectively, and Mean(G) represents the mean gradient magnitude. This is the gradient strength normalization threshold; Edge density score The calculation formula is: Where Redge is the edge pixel ratio, obtained through Canny edge detection, Sobel edge thresholding, or other edge detection methods: Redge = Nedge / Nvalid; Where Nedge is the number of edge pixels and Nvalid is the number of valid background pixels. This is the threshold for edge density normalization.

[0022] The background coverage integrity score It is used to characterize whether each pixel position in a static background image of a street has sufficient effective background observation, and whether dynamic foreground pixels still contribute to the background fusion result.

[0023] Specifically, the background coverage integrity score It can be determined based on the dynamic foreground residual ratio and the background hole ratio: Furthermore, the dynamic foreground residual ratio The calculation process is as follows: Let the set of aligned frames be ,in Let t be the aligned frame of frame t, and T be the number of aligned frames participating in background reconstruction; let t be the aligned frame of frame t. Generate dynamic foreground mask ,in, This indicates that the pixel location belongs to the dynamic foreground region; This indicates that the pixel location belongs to the static background candidate region; let... Align the effective pixel markers at pixel position (x, y) for frame t. This indicates the effective pixel area of ​​the aligned frame after spatial alignment at this location. This indicates that the location is a region with no valid pixels, fill values, or outside the image boundary after spatial transformation.

[0024] Furthermore, based on the dynamic foreground mask, the weights are fused. It can be set to:

[0025] in, As background candidate region weights, Foreground region weights are dynamic, and Less than .

[0026] Preferably, the Setting it to 0 directly removes pixel contributions from dynamic foreground regions during the fusion calculation. Further, for pixel location (x, y), the frequency of dynamic foreground occurrence is calculated. :

[0027] Where T is the number of aligned frames participating in background fusion; if Greater than or equal to the foreground residue threshold τ fg Then the foreground residual mask They believe that there is a high risk of dynamic foreground residue at this location; if Less than the foreground residue determination threshold τ fg Then the foreground residual mask They believe that there is a low risk of dynamic foreground residue at this location; Dynamic foreground residual ratio Calculate according to the following formula: in, for The number of pixels, where Nvalid is the total number of valid pixels in the static background image of the block; The calculation process is as follows: For pixel position (x, y), the effective background observation for frame t is calculated as follows:

[0028] Summing over all T frames yields the number of valid background observations for that pixel location:

[0029] if Less than the minimum number of background observations n min Then the background hole mask The pixel is considered to be a background hole pixel; if Greater than or equal to the minimum number of background observations n min Then the background hole mask The pixel is considered not to be a background hole pixel. Background Hole Ratio Calculate according to the following formula: in, N is the number of background hole pixels, and Nvalid is the total number of valid pixels in the target background image coordinate system.

[0030] Furthermore, dynamic foreground mask Generates based on the temporal differences of the same pixel position in the aligned frame set.

[0031] Specifically, after completing the spatial alignment of candidate frames, the aligned frame set is obtained: ,in Let t represent the aligned frame, and T represent the number of aligned frames participating in background reconstruction. First, the temporal median of the pixel values ​​at the same pixel position (x, y) of multiple aligned frames is calculated to obtain the initial median background image. :

[0032] Then, the color difference between the t-th aligned frame and the initial median background image at pixel position (x, y) is calculated. : When using RGB images The average or maximum value of the absolute differences between the three color channels can be used; when using a grayscale image... This represents the grayscale difference.

[0033] like Greater than the dynamic foreground determination threshold ,but Mark this pixel as a dynamic foreground pixel; otherwise... Marked as background candidate pixels; Furthermore, if the image grayscale values ​​or color channel values ​​are normalized to the range of 0-255, then A value of 25 can be chosen; alternatively, it can be adaptively determined based on the difference map of the same aligned frame, for example... .

[0034] Furthermore, to improve mask stability, opening and closing operations of 3×3 or 5×5 structuring elements are performed on the initial dynamic foreground mask to remove isolated noise and fill small holes; Furthermore, connected components with an area smaller than a preset area threshold, such as connected components with an area less than 50 pixels, are removed to obtain the dynamic foreground mask for frame t. .

[0035] The meaning of this method is: if a pixel location remains stable across most aligned frames, then the temporal median represents the static background at that location; if a pedestrian, vehicle, or cyclist momentarily occludes that location in a frame, the pixel value of that frame will significantly deviate from the median background value and is therefore marked as dynamic foreground. During subsequent background fusion, the median value is then... Pixels that are deweighted or removed are considered to have reduced weight. The pixels participate in median fusion, quantile fusion, or weighted fusion to generate a static background image of the street after removing the instantaneous dynamic foreground.

[0036] When the confidence level of the static background image of the block is lower than the preset quality threshold, the candidate frame set, time window, reference frame or background fusion parameters are readjusted, and steps 1 to 5 are repeated to generate the static background image of the block. The background reconstruction parameters include at least one of the following: frame extraction interval, time window size or background fusion parameters.

[0037] Furthermore, the comparison in step 7 includes: acquiring multiple static background images of the same target street location generated in different time windows, and calculating the background difference value between the static background images of the street in different time windows: Among them, D t B represents the background difference value at the pixel location. t and B (t-1) Background images generated for adjacent time windows; Threshold segmentation, morphological processing, and connected component analysis are performed on the difference map to obtain candidate regions of change. Perform static hazard identification on the candidate change areas according to step 6; The confidence level for the persistence of hidden dangers over multiple time periods is: ; n p This indicates the number of times the same candidate hazard area is identified within K time windows.

[0038] Furthermore, the categories of potential hazards include at least one of the following: unauthorized private wiring for charging, obstruction of fire lanes, illegal addition of structures, damage to building facades, illegal stacking outside shops, damage to public facilities, stacking of objects on roads or alleys, abnormal closure of the back area of ​​shops, mixed production, warehousing and residential areas, abnormal spatial boundaries, or abnormal street environment quality. The risk level is determined by calculating the overall hazard confidence level. Based on the preset threshold range where the comprehensive hazard confidence level falls, a corresponding risk level is generated, calculated using the following formula:

[0039] in, To identify confidence levels for the model, Confidence level of static background images of the street block; Match confidence levels to spatial scenes. For the persistence confidence level of potential hazards; ;

[0040] in: The neighborhood type is matched and scored, and the scenarios with high matching scores for candidate hazard tags are selected. , middle match Low matching , Scoring is performed on surrounding points of interest (POIs). POIs within a 100-meter radius of the target location are retrieved, and calculations are made based on POI type and distance decay. ;in For surrounding points of interest, Weights for interest point types, The distance from the target point to the point of interest. For distance scoring, the shortest distance from the target point to the static risk-related area is defined as . If the maximum effective distance threshold is set to 100 meters, then: ; Based on the comprehensive hazard confidence level Weight of hazard type Determine the risk level G, and calculate it using the grade function Grade: .

[0041] Furthermore, in step 6, the static hazard identification is performed using an image recognition model, a multimodal semantic recognition model, or a local rule model to obtain the static hazard identification result. Step 8 further includes: the image recognition model is a target detection model, image classification model, semantic segmentation model, visual language model, or multimodal large model; when the interface of the image recognition model or multimodal semantic recognition model is unavailable, the call times out, the output format is abnormal, or the confidence level is lower than a preset threshold, the local rule model is called to generate basic static hidden danger identification results; the local rule model identifies based on image edges, linear targets, region occupancy, texture anomalies, color anomalies, geometric shapes, and spatial positional relationships.

[0042] When using a multimodal semantic recognition model to identify static hidden dangers The model recognition confidence score returned by the multimodal semantic recognition model is directly used; when using a local rule model to identify static hidden dangers... By candidate hazard label calculate: Targeting candidate hazard labels Pre-configure rule sets: Each rule Configure corresponding weights ,and If the first If the rule is satisfied, then If not satisfied, then Based on this calculation: .

[0043] Furthermore, the method also includes: performing personal identity information desensitization processing on the static background image of the street block to obtain a desensitized evidence image; and structurally encapsulating and persistently storing the static hazard identification result, the spatial attribute data, and the desensitized evidence image.

[0044] The present invention also provides an electronic device, including at least one processor; and a memory communicatively connected to said at least one processor; The memory may contain a computer program that can be executed by the at least one processor, such that the at least one processor can execute the computer program to implement the above-mentioned method for identifying street hazards based on static background reconstruction.

[0045] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method for identifying street hazards based on static background reconstruction.

[0046] Compared with the prior art, the present invention has at least the following beneficial effects: 1. This invention does not rely directly on single-frame images to identify static hazards in the street. Instead, it first generates a static background image of the street after removing the instantaneous dynamic foreground by candidate frame screening, robust spatial alignment and multi-frame background fusion. This can reduce the interference of short-term occlusion by pedestrians, vehicles and other objects on the identification of overhead wires, fire lanes, building facades, storefronts and public facilities.

[0047] 2. This invention estimates spatial transformation parameters through local feature matching and random sampling consistency algorithms, which can improve the geometric consistency between multiple frames of images and reduce ghosting and edge misalignment in background fusion when there are slight jitter, viewpoint shifts and partial occlusions in fixed or similar surveillance videos.

[0048] 3. This invention reconstructs static background images of blocks through median fusion, quantile fusion, weighted fusion, or foreground mask-assisted fusion, so that relatively stable elements such as building facades, roads, pipelines, shop signs, covered structures, fire lane boundaries, and spatial boundaries are presented more completely, thereby improving the image quality of static hazard identification.

[0049] 4. The present invention sets up a background quality evaluation mechanism, which evaluates the usability of the background image by jointly evaluating the image alignment quality, background clarity and background coverage integrity, and adjusts the candidate frames or fusion parameters when the background quality is insufficient, which can improve the stability and verifiability of the background reconstruction results.

[0050] 5. By comparing static background images over multiple time periods, this invention can identify newly added piles of materials, newly added overhead wires, newly added covered structures, long-term occupied areas, and areas with changes in the facade, which helps to distinguish between temporary obstructions, short-term activities, and persistent static hazards.

[0051] 6. This invention combines static background images of the street, spatial attribute data, and hazard rules to transform ordinary image recognition results into static hazard identification results with governance implications, such as illegal charging of electrical wires, obstruction of fire lanes, illegal additions, damage to building facades, and stacking outside shops, thereby improving the correspondence between the identification results and grassroots governance tasks.

[0052] 7. The present invention outputs structured hazard evidence data including location number, collection time, hazard category, hazard area, risk level, confidence level, evidence summary, background image path and original video backtracking path, which can support manual review, platform integration, spatial display and follow-up handling tracking.

[0053] 8. This invention can desensitize personal identification information such as faces and license plates before output, while retaining the spatial structure and evidence content required for street hazard identification, thereby reducing the risk of unnecessary exposure of personal information. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating the overall process of identifying potential hazards in urban areas based on static background reconstruction, as described in this invention. Figure 2 This is a schematic diagram of local feature matching and robust spatial alignment according to the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to specific embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this invention. All equivalent substitutions, parameter adjustments, module combinations, algorithm replacements, or application scenario extensions made to this invention within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0056] This embodiment provides a method, system, and medium for identifying potential hazards in urban blocks based on static background reconstruction. This method is primarily aimed at high-obscuration streetscapes such as urban villages, old neighborhoods, street-front shops, back alleys, courtyard perimeters, and storefront back perimeters. These high-obscuration streetscapes refer to urban built environments where street space is narrow, pedestrian and vehicular traffic is high, building facades and public facilities are easily obscured by short-term dynamic foreground elements, and traditional single-frame video recognition is insufficient to reliably determine static hazards.

[0057] Static hazards in this embodiment include, but are not limited to, unauthorized charging via extension cords, obstruction of fire lanes, illegal additions, damage to building facades, storage outside shops, damage to public facilities, storage of objects on roads or alleys, abnormal closure of the back area of ​​shops, external signs of suspected "three-in-one" premises, abnormal spatial boundaries, and abnormal street environment quality.

[0058] Example 1: A method for identifying potential hazards in urban areas based on static background reconstruction The overall process of the method in this embodiment can be found in [reference]. Figure 1 The process includes, in sequence: acquisition of video of target street locations, extraction and screening of candidate frames, local feature matching, robust spatial alignment, multi-frame background fusion, background quality evaluation, static hazard identification, multi-time period background comparison, and structured evidence output.

[0059] Step S1: Collect video data for the target street locations and simultaneously acquire the spatial attribute data corresponding to the locations; First, acquire video data and spatial attribute data of the target street locations. The target street locations include at least one of the following: street entrances, street-front shops, old street roads, alleyways in urban villages, courtyard perimeters, storefront perimeters, areas adjacent to public facilities, or road sections in mixed residential and commercial areas.

[0060] In one specific implementation, the Lujiang urban village cluster and its surrounding blocks in Haizhu District, Guangzhou City, were selected as the data collection targets. This area has dense streets and alleys, small building spacing, and a mixed distribution of residential, commercial, and small-scale business activities. Some courtyards, store backyards, and alleyways are difficult to observe directly through routine patrols, making it suitable as a typical scenario for screening potential hazards in street governance.

[0061] The video acquisition device can be a fixed camera, a mobile inspection device, a surveillance-like acquisition device, or an action camera with video acquisition capabilities. Preferably, a GoPro Hero9 high-definition action camera is used, positioned at a fixed location with a horizontal viewing angle to simulate the perception process of street space status by public surveillance video. The video acquisition parameters are 1080P resolution, 24 frames per second, and GPS positioning is enabled to obtain the device location information corresponding to the video.

[0062] In one specific embodiment, sampling was completed at 19 street observation points, resulting in 136 MP4 audio and video samples with a total duration of approximately 390 minutes. The samples covered various scenarios, including commercial streets, residential areas, urban villages, alleyway entrances, and areas near factories. The device location information, collection time, and location number information attached to the video files were used for subsequent evidence organization and risk scenario verification.

[0063] Based on the video file name, the location information of the acquisition device, and the matching results of manual annotation tables or geographic information, the location number, latitude and longitude coordinates, and acquisition time of each video sample are obtained and determined, and a location evidence unit is established. This location evidence unit is used for the unified management and storage of data carriers related to hazard identification, including: original video, candidate frames, aligned frames, static background images of the street, hazard identification results, evidence summaries, and the original video backtracking path, thereby providing a traceable data foundation for subsequent hazard identification and evidence presentation.

[0064] Step S2: Extract video frame sequences from the video data, perform frame extraction processing and candidate frame quality screening on the video frame sequences, and obtain a set of candidate frames for background reconstruction; (1) To reduce redundant input and computational overhead, the video data from each point evidence unit obtained in step S1 is processed according to a preset frame interval. Or preset time interval Perform frame extraction to obtain the initial frame set:

[0065] Among them, I iLet represent the i-th frame, and N be the total number of video frames. The frame sampling frequency can be adaptively adjusted based on the motion characteristics of the point, the channel width, and the occlusion situation. For points with fast-moving targets, narrow channels, or frequent occlusion, the frame sampling frequency can be appropriately increased; for points under continuous monitoring for a long time or with low target speed, the frame sampling frequency can be appropriately decreased.

[0066] (2) Perform quality screening on the initial frame set F0 to remove low-value frames.

[0067] Low-value frames include blurred frames, overexposed frames, underexposed frames, frames with an occlusion ratio exceeding a threshold, or frames with a similarity to retained frames exceeding a threshold. Among them, blurred frames can be identified using Laplacian variance, overexposed and underexposed frames can be identified using grayscale mean, and inter-frame similarity can be evaluated using structural similarity index, image histogram correlation, perceptual hash distance, or feature descriptor similarity.

[0068] Specifically, the formula for calculating the blurred frame score is as follows: in, Image I i The Laplace response, where Var represents the variance, indicates that a blurred frame is considered blurred if its score is below the sharpness threshold.

[0069] The exposure score calculation formula is as follows: Among them, G i (x,y) represents the image I i The grayscale value at the pixel location, W and H are the image width and height, respectively. If the brightness score is lower than the lower limit of brightness or higher than the upper limit of brightness, the frame is determined to be an underexposed frame or an overexposed frame.

[0070] The occlusion ratio is the percentage of foreground areas such as people / vehicles or the detection results of significantly occluded areas.

[0071] After screening, candidate frames are assigned to the corresponding evidence units based on location number, collection date, and time window. The number of candidate frames is preferably no less than 5, and more preferably 10 to 60. When the video duration is short or street occlusion is extremely severe, some screening criteria can be relaxed to ensure sufficient frames for background fusion; when the video duration is long, candidate frames covering different levels of pedestrian and vehicle occlusion can be prioritized.

[0072] Step S3: Perform local feature point extraction and feature matching on the candidate frames; To eliminate slight jitter, camera position shifts, and inter-frame misalignment in fixed or surveillance videos, this embodiment performs image registration before background fusion and performs local feature point extraction and feature matching on candidate frames. Specifically, this includes: Candidate frames are size-normalized. In one implementation, candidate frames are scaled to a maximum side size of 960 pixels to reduce the computational cost of feature extraction and matching while preserving details such as street facades, pipelines, debris, and passageway boundaries.

[0073] Extract local feature points and their descriptors from the scaled image, using the t-th frame image as I. t Its feature point set is: Where p represents a local feature point in frame t. In a preferred embodiment, the local feature point extraction uses the SIFT algorithm to extract feature points and generate a 128-dimensional descriptor. SIFT features are robust to scale changes, rotation, illumination changes, and a certain degree of affine transformation, making them suitable for stable region matching between adjacent frames in street video.

[0074] Select one frame from the candidate frame set as the reference frame, and perform feature matching between the candidate frame and the reference frame, or perform feature matching between adjacent candidate frames in chronological order. A ratio test is used to eliminate ambiguous matches. The matcher can be a FLANN approximate nearest neighbor matcher or a brute-force matcher. For the same candidate descriptor, let the nearest neighbor matching distance be d1 and the second nearest neighbor matching distance be d2. When... When the ratio of d1 to d2 is close to 1, the matching points are retained; where ρ is the ratio test threshold. ρ is preferably between 0.6 and 0.8, and more preferably 0.75. When the ratio of d1 to d2 is close to 1, it indicates that the nearest neighbor and the second nearest neighbor are close in distance, the matching result is ambiguous, and they should be discarded.

[0075] Step S4: Use the random sampling consensus algorithm to estimate the affine transformation matrix for the retained matching points, map the candidate frames to the reference frame coordinate system, and obtain the aligned frames; Because street videos may contain dynamic foreground elements such as pedestrians, vehicles, and temporary stalls, the initial set of matching point pairs may include incorrectly matched points and matching points located on dynamic targets. Directly solving for the spatial transformation parameters based on all matching points can easily lead to alignment bias. Therefore, this embodiment uses a random sample consensus algorithm to estimate the affine transformation matrix.

[0076] In one implementation, the spatial transformation employs an affine transformation model. An affine transformation describes the rotation, scaling, shearing, and translation relationships between the candidate frame and the reference frame. Let the coordinates of the matching point in the candidate frame be ( , The coordinates of the corresponding point in the reference frame are (x... i ',y i The affine transformation matrix A is expressed as:

[0077] in, , , , Used to describe rotation, scaling, and shearing relationships. and Used to describe translation relationships. For a matching point in a candidate frame, its predicted coordinates after affine transformation are:

[0078]

[0079] If the coordinates of the corresponding matching point in the reference frame are the same as the corresponding coordinates, then the reprojection error is:

[0080] To reduce the impact of mismatches, moving foreground matching points, and noise on the estimation of spatial transformation parameters, a random sample consensus (RANSAC) algorithm is used for robust estimation of the affine transformation matrix. Specifically, this involves: randomly selecting at least three pairs of matching points from the set of matching point pairs and estimating candidate affine transformation matrices. Then, the reprojection error of all matching point pairs is calculated. If the reprojection error is less than a threshold, the matching point pair is marked as an inlier; otherwise, it is marked as an outlier. The reprojection error threshold is preferably set to 2 to 5 pixels. After multiple RANSAC iterations, the candidate affine transformation matrix with the most inliers or the highest proportion of inliers is selected, and the least squares solution is performed again using all inliers to obtain the final affine transformation matrix. Using the affine transformation matrix, the candidate frames are mapped to the reference frame coordinate system to obtain aligned frames. The above registration process is repeated for all candidate frames in the candidate frame set to obtain the aligned frame set. Where I' represents the spatially aligned image, and T is the number of aligned frames that participate in background fusion.

[0081] Step S5: For T-frame alignment frames within the same time window at the same point, perform median fusion at each pixel position and color channel to generate a static background image B of the street block. For pixel position (x, y) and color channel c, the pixel value of the static background image B of the street block is: in, This represents the spatially aligned pixel value of frame t. Through the above processing, the occlusion effect of transient dynamic foregrounds such as pedestrians and vehicles on relatively stable backgrounds such as building facades, roads, pipelines, shop signs, covered structures, roller shutters, fire lanes, roadside guardrails, and spatial boundaries can be reduced.

[0082] Median fusion, or median operation, is used to suppress dynamic foreground elements that appear only in a few frames, thus restoring a stable background. This makes consistently appearing street facades, road surfaces, and public facilities the main content of the background image. In other implementations, quantile fusion, temporal mode fusion, weighted fusion, or background modeling methods can also be used to generate the background image.

[0083] When using weighted fusion, the static background image B of the street block can be represented as:

[0084] in, This represents the fusion weight, which can be determined based on image sharpness, alignment quality, foreground mask, illumination stability, or temporal distance. If a pixel in a given frame has a high probability of dynamic foreground, the weight corresponding to that pixel is reduced.

[0085] Furthermore, before performing multi-frame background fusion, a dynamic foreground suppression step can be performed. This dynamic foreground suppression step includes generating a dynamic foreground mask based on inter-frame difference, optical flow, target detection results, pixel temporal variance, or semantic segmentation results. .like If , it indicates that the location belongs to the dynamic foreground region; if If the value is 0, it means that the location belongs to the background candidate region, and the fusion weight can be set as follows:

[0086] in, As background candidate region weights, Foreground region weights are dynamic, and Less than Preferably, It can be set to 0 to remove pixels from the dynamic foreground region from the merged pixel set.

[0087] After fusion, a static background image of the block is output and saved to the point evidence unit. The number of candidate frames participating in the fusion, the time range, alignment parameters, fusion method, and background image path are also recorded. The static background image of the block can be saved as PNG, JPEG, TIFF, or other image formats.

[0088] Step S6: Calculate the confidence level of the static background image of the street block; To ensure the reliability of hazard identification based on static street background images, this embodiment calculates the confidence level of the generated static street background image. and confidence in static background images of the street. When the value is less than the threshold, parameter rollback and regeneration are triggered. in, Rate the image alignment quality. Scoring for blurry frames, Rate the completeness of background coverage. , , Let be the weighting coefficient, and satisfy: α, β, and γ can each be one-third, or adjusted according to the application scenario. In one implementation, α = 0.40, β = 0.30, and γ = 0.30. The image alignment quality score can be determined based on the proportion of matched inliers, the average reprojection error, or the consistency of aligned edges. in, The proportion of RANSAC interior points. This represents the average reprojection error. These are the weighting coefficients.

[0089] The blurred frame The score can be based on the Laplace variance score. Gradient strength scoring or edge density score Confirmed, the calculation formula is as follows: Where η1, η2, and η3 are weighting coefficients, and η1+η2+η3=1.

[0090] In one specific embodiment, η1=0.5, η2=0.3, and η3=0.2 are selected.

[0091] In the specific calculation, the static background image B of the street block is first converted into a grayscale image. The grayscale value ranges from 0 to 255.

[0092] Laplace variance score The calculation formula is: in, Represents the Laplacian response of a grayscale background image. This represents the variance of the Laplace response. The Laplacian variance normalization threshold is used; a larger Laplacian variance indicates clearer image edges and details. Greater than or equal to hour, Take 1.

[0093] Gradient strength score for: Where G represents the gradient magnitude map of the grayscale background image, which can be calculated by the Sobel operator:

[0094] Gx and Gy are the gradients in the horizontal and vertical directions, respectively, and Mean(G) represents the mean gradient magnitude. This is the gradient strength normalization threshold; Edge density score The calculation formula is: Where Redge is the edge pixel ratio, obtained through Canny edge detection, Sobel edge thresholding, or other edge detection methods: Redge = Nedge / Nvalid Where Nedge is the number of edge pixels and Nvalid is the number of valid background pixels. This is the threshold for edge density normalization.

[0095] In one specific embodiment, take .

[0096] The background coverage integrity score The ratio of dynamic foreground residue to background voids can be determined. The higher the value, the more pixel locations in the static background image of the block have sufficient background observations, and the smaller the residual contribution of dynamic foreground pixels to the background fusion result; The lower the value, the more background areas are missing due to occlusion by dynamic targets such as pedestrians and vehicles, or the dynamic foreground pixels still have a significant impact on the background fusion result. in, The dynamic foreground residue ratio, The proportion of the background void. The proportion of the dynamic foreground residue. The calculation process is as follows: Let the set of aligned frames be ,in Let t be the aligned frame of frame t, and T be the number of aligned frames participating in background reconstruction; let t be the aligned frame of frame t. Generate dynamic foreground mask ,in, This indicates that the pixel location belongs to the dynamic foreground region; This indicates that the pixel location belongs to the static background candidate region; let... Align the effective pixel markers at pixel position (x, y) for frame t. This indicates the effective pixel area of ​​the aligned frame after spatial alignment at this location. This indicates that the location is a region with no valid pixels, fill values, or outside the image boundary after spatial transformation.

[0097] Based on the dynamic foreground mask, the fusion weights are... It can be set to:

[0098] in, As background candidate region weights, Foreground region weights are dynamic, and Less than .

[0099] Preferably, the Setting it to 0 directly removes pixel contributions from dynamic foreground regions during the fusion calculation. Further, for pixel location (x, y), the frequency of dynamic foreground occurrence is calculated. :

[0100] Where T is the number of aligned frames participating in background fusion; if Greater than or equal to the foreground residue threshold τ fg Then the foreground residual mask They believe that there is a high risk of dynamic foreground residue at this location; if Less than the foreground residue determination threshold τ fg Then the foreground residual mask They believe that there is a low risk of dynamic foreground residue at this location.

[0101] In one specific embodiment, τ fg 0.30 is acceptable.

[0102] Dynamic foreground residual ratio Calculate according to the following formula: in, for The number of pixels, The total number of valid pixels in the static background image of the street block; The calculation process is as follows: For pixel position (x, y), the effective background observation for frame t is calculated as follows:

[0103] Summing over all T frames yields the number of valid background observations for that pixel location:

[0104] if Less than the minimum number of background observations n min Then the background hole mask The pixel is considered to be a background hole pixel; if Greater than or equal to the minimum number of background observations n min Then the background hole mask It is believed that the pixel is not a background hole pixel.

[0105] Background Hole Ratio Calculate according to the following formula: in, The number of pixels in the background holes. This represents the total number of valid pixels in the target background image coordinate system.

[0106] Dynamic foreground mask Generates based on the temporal differences of the same pixel position in the aligned frame set.

[0107] Specifically, after completing the spatial alignment of candidate frames, the aligned frame set is obtained: ,in Let t represent the aligned frame, and T represent the number of aligned frames participating in background reconstruction. First, the temporal median of the pixel values ​​at the same pixel position (x, y) of multiple aligned frames is calculated to obtain the initial median background image. :

[0108] Then, the color difference between the t-th aligned frame and the initial median background image at pixel position (x, y) is calculated. : When using RGB images The average or maximum value of the absolute differences between the three color channels can be used; when using a grayscale image... This represents the grayscale difference.

[0109] like Greater than the dynamic foreground determination threshold ,but Mark this pixel as a dynamic foreground pixel; otherwise... Marked as background candidate pixels; If the image grayscale values ​​or color channel values ​​are normalized to the range of 0-255, then A value of 25 can be chosen; alternatively, it can be adaptively determined based on the difference map of the same aligned frame, for example... .

[0110] To improve mask stability, opening and closing operations of 3×3 or 5×5 structuring elements are performed on the initial dynamic foreground mask to remove isolated noise and fill small holes. Connected components with an area smaller than a preset area threshold, such as those with an area less than 50 pixels, are removed to obtain the dynamic foreground mask for frame t. .

[0111] When the confidence level of the static background image of the block is lower than the preset quality threshold, the candidate frame set, time window, reference frame or background fusion parameters are readjusted, and the static background image of the block is regenerated.

[0112] The time window refers to the time range within which candidate frames are selected from videos of the same target street location.

[0113] Step S7: Perform static hazard identification based on the static background image and spatial attribute data of the block, and generate static hazard identification results; The static hazard identification results include at least one of the following: candidate hazard label, hazard area, risk level, confidence level, and evidence summary. The hazard categories include at least one of the following: illegal charging cables, obstruction of fire lanes, illegal additions, damage to building facades, items piled up outside shops, damage to public facilities, items piled up on roads or alleys, abnormal closure of the back area of ​​shops, external signs of suspected "three-in-one" premises, abnormal spatial boundaries, or abnormal street environment quality.

[0114] In one implementation, a static background image of the street block, spatial attribute data, and structured prompt information are input into an image recognition model or a multimodal semantic recognition model to generate candidate hazard labels, hazard areas, and evidence summaries. The spatial attribute data includes location number, street type, road name, data collection time, camera location type, nearby points of interest, and governance grid. The structured prompt information may include the hazard category to be identified, output field requirements, risk level rules, and evidence summary format. The image recognition model can be a target detection model, image classification model, semantic segmentation model, visual language model, or a multimodal large-scale model. This invention does not limit the specific model name or cloud platform.

[0115] In another implementation, when the image recognition model or multimodal semantic recognition model interface is unavailable, times out, has an abnormal output format, or has a confidence level below a preset threshold, a local rule model is invoked to generate basic static hazard identification results. The local rule model can identify hazards based on image edges, linear targets, area occupancy, texture anomalies, color anomalies, geometric shapes, and spatial relationships. For example, for illegal charging cables, it can detect linear targets extending from building facades, windows, shop fronts, or utility poles to vehicle parking areas; for obstructed fire lanes, it can detect piles of objects, vehicles, or fixed obstructions in the lane area; for illegal additions, it can detect non-original structures protruding from the building top or facade; for damage to the exterior facade, it can detect abnormal textures and shapes in the walls, shop signs, railings, or road surfaces.

[0116] To improve the correspondence between the output of image recognition models or multimodal semantic recognition models and governance scenarios, this invention verifies candidate hazard labels based on preset hazard rules and spatial scene constraints. For example, when the street type is an urban village alley or an old street road, and overhead wires, piled-up objects, or passageway obstruction are identified in the background image, the candidate hazard label is mapped as a fire safety hazard; when the street type is a street-front shop section, and outdoor piles, outdoor facilities, or obstruction of the store front area are identified, the candidate hazard label is mapped as a spatial order hazard; when the nearby point of interest is a mixed residential and commercial area, and signs of residential use, piled-up objects, and abnormal electrical wires are identified inside the shops, the candidate hazard label is mapped as signs of a suspected "three-in-one" premises.

[0117] Step S8: Comparison of static backgrounds of blocks across multiple time periods; Static hazards in urban areas typically exhibit characteristics of being persistent or slowly changing. To differentiate between temporary obstructions, short-term activities, and persistent static hazards, this embodiment further compares static background images of the same location across multiple time periods, specifically including: For the same target street location, static background images of the street are constructed according to different time windows. The time window can be a daytime period such as morning, noon, evening, or night, or it can be a daily, weekly, monthly, or periodic inspection period. Let B be the background images generated for a certain location in adjacent time windows. t and B (t-1) Calculate the background difference map: Subsequently, threshold segmentation, morphological processing, and connected component analysis are performed on the difference map to obtain candidate change regions. Static hazard identification is then performed on the candidate change regions according to step S7. The candidate change regions can be further input into an image recognition model or a local rule model for semantic verification to determine whether they correspond to newly added materials, newly added overhead wires, newly added covered structures, long-term occupied areas, or areas with changes to the facade.

[0118] To distinguish between temporary changes and persistent hazards, the persistence confidence level of the hazard can be calculated. Let n be the number of times a candidate hazard area is identified in K time windows. p ,but: When the persistence confidence level is higher than the persistence threshold, the candidate hidden danger is judged as a persistent static hidden danger; when the persistence confidence level is lower than the persistence threshold, it is marked as a temporary change or a change to be reviewed.

[0119] Step S9: Determine the confidence level and risk level of the hidden danger.

[0120] The hazard confidence score is calculated based on the model identification confidence score, the street static background image confidence score, the spatial scene matching confidence score, and the multi-time period hazard persistence confidence score.

[0121] in, To identify confidence levels for the model, Confidence level of static background images of the street block Match confidence levels to spatial scenes. Let λ1, λ2, λ3, and λ4 be the weighting coefficients for the persistence confidence of the hazard, and satisfy the following: , = 0.35, = 0.35, = 0.15, = 0.15; When using a multimodal semantic recognition model to identify static hidden dangers The model recognition confidence score returned by the multimodal semantic recognition model is directly used; when using a local rule model to identify static hidden dangers... It can be calculated from the number of rule matches, the area of ​​the hidden danger zone, the number of times it occurs continuously, or the satisfaction of spatial constraints.

[0122] By candidate hazard label Calculate the confidence score of local rule validation : Targeting candidate hazard labels Pre-configure rule sets: Each rule Configure corresponding weights ,and If the first If the rule is satisfied, then If not satisfied, then Based on this calculation: ; For the static hazard of blocked fire lanes, the rules may include: the candidate area is located within a road, alley, or passageway area ( The area of ​​the candidate region is not less than the preset area threshold. ), matching candidate region shape with piled-up objects, vehicles, or obstacles ( Candidate regions appear continuously in the background across multiple frames or time periods. The location type is related to fire exits, alleyways, or the perimeter of the store's back area. ); For static hazards such as unauthorized additions or temporary structures, the rules may include: detecting new structures on the roof, facade, or boundary ( ), The form of the structure should match the shed, cover, enclosure or temporary construction. ), and there is a spatial conflict with the original building boundary or road red line ( ), continuous changes exist in the background comparison across multiple time periods ( The locations are situated in urban villages, old neighborhoods, or on the outskirts of courtyards. ).

[0123] The confidence level of the spatial scene matching is: in: The neighborhood type is matched and scored, and the scenarios with high matching scores for candidate hazard tags are selected. , middle match Low matching . Scoring is performed on surrounding points of interest (POIs). POIs within a 100-meter radius of the target location are retrieved, and calculations are made based on POI type and distance decay. ;in For surrounding points of interest, Weights for interest point types, The distance (in meters) from the target point to the point of interest. For distance scoring, the shortest distance from the target point to the static risk-related area is defined as . If the maximum effective distance threshold is set to 100 meters, then: .

[0124] Risk levels can be determined based on the confidence level of the hazard and the weight of the hazard type: Where Grade is the graded function, ψ h Based on the weighting of hazard types, hazards with higher priority for handling, such as obstruction of fire lanes, illegal charging of electrical wires, and external signs of suspected "three-in-one" premises, are assigned a weighting factor ψ. h It can be set to a higher value; for general street environment quality abnormalities, ψ h It can be set to a lower value.

[0125] In some specific embodiments, The weighted table can be set to a preset type, or it can be adjusted according to local governance rules, manual review results, or disposal priorities. In the example: Fire lane obstruction: ; Flying wire charging: ; External signs of a suspected "three-in-one" venue: ; Unauthorized or suspected additions to structures: ; The store's back area is unusually enclosed, and the spatial boundaries are abnormal: ; Items piled up outside the store, or piled up on roads or in alleys: ; Damaged public facilities and facade: ; The environmental quality of the general neighborhood is abnormal: .

[0126] Step S10: Generation of structured hazard evidence data.

[0127] Structured hazard evidence data is generated based on the static hazard identification results, and the static background image of the block, candidate frames, aligned frames, hazard areas, original video playback paths, and spatial attribute data are saved. The structured hazard evidence data includes alarm number, location number, collection time, spatial coordinates, block type, hazard category, hazard area coordinates, risk level, hazard confidence level, evidence summary, block static background image path, original video path, candidate frame path, aligned frame path, suggested responsible party, suggested review action, and manual review status.

[0128] Structured hazard evidence data can be output as at least one of the following formats: JSON file, PDF file, GeoJSON file, Shapefile point feature file, database record, or message interface data. JSON files are used for system integration and batch processing, PDF files are used for manual reading and archiving, and GeoJSON or Shapefile point feature files are used for spatial display and overlay analysis within geographic information systems.

[0129] Step S11: Privacy protection processing.

[0130] Before generating evidence materials for external display or circulation, privacy protection processing can be applied to faces, license plates, or other personally identifiable information in candidate frames, aligned frames, static background images of blocks, or evidence images. Privacy protection processing includes at least one of blurring, occlusion, pixelation, hashing, or de-identification storage. Privacy protection processing does not alter the static structure of the block, the state of passageways, building facades, clutter, and public facilities required for hazard identification.

[0131] Example 2: Static Background Reconstruction and Hazard Identification Case Study of the Middle Section of Lujiang South Avenue This embodiment uses a sampling point in the middle section of Lujiang South Street, Haizhu District, Guangzhou City as an example to illustrate the application process of the method of the present invention. This sampling point is located in a street within an urban village, densely packed with shops on both sides, with high pedestrian and vehicle traffic on weekdays and relatively narrow street space. The sampling time was around 7:00 AM. The original video frame sequence contained a large number of pedestrians, two-wheeled vehicles, three-wheeled vehicles, and temporarily parked vehicles. These instantaneous dynamic elements would obscure building facades, shop shutters along the street, roadside metal guardrails, fire lane boundaries, and static road features, making it difficult to reliably use a single frame image for static hazard identification.

[0132] First, the video file for the location is read and parsed to obtain the location number, acquisition time, and spatial coordinates. Then, initial frames are extracted from the video frame sequence at preset time intervals, and blurry frames, overexposed frames, and frames with high repetition are removed to obtain a candidate frame set.

[0133] Next, the candidate frames are scaled to a maximum side size of 960 pixels, and feature points and 128-dimensional descriptors are extracted using the SIFT method. The descriptors between the candidate frames and the reference frame are matched using a FLANN approximate nearest neighbor matcher, and ambiguous matching points are eliminated using the Lowe ratio test. For the remaining matching point pairs, the affine transformation matrix is ​​estimated using RANSAC, the reprojection error is calculated, and interior points are filtered to obtain the robust affine transformation matrix. This affine transformation matrix is ​​then used to align the candidate frames to the reference coordinate system, forming an aligned frame set.

[0134] After alignment, median fusion is performed on the multi-frame aligned images to generate a static background image of the street. Since pedestrians and vehicles only appear in a few frames, while the roller shutters of street shops, roadside metal guardrails, asphalt pavement, building facades and spatial boundaries remain relatively stable in multiple frames, median fusion can effectively suppress instantaneous dynamic foreground and restore the static street elements that are obscured by vehicles and pedestrians.

[0135] After generating a static background image of the street, a background quality score is calculated. If the proportion of aligned inner points is high, the average reprojection error is low, and the background clarity and coverage integrity meet the threshold requirements, the background image is sent to the static hazard identification module. The static hazard identification module identifies risks such as illegal charging wires, blocked fire lanes, unauthorized additions, damaged facades, and external signs of suspected "three-in-one" premises based on the background image and the spatial attributes of the locations.

[0136] In a specific output example, a medium-risk alarm is generated for "fire lane obstruction," with a confidence level of 0.71. The evidence summary is: "The static background image shows that there are continuously piled-up items in front of the shops along the street, obstructing the passageway space. It is recommended to check the fire lane access status." This case illustrates that the present invention, through image registration and multi-frame background fusion, can restore the static physical background of the street obscured by people and vehicles in the original video to a relatively clean background image, thereby improving the stability of identifying static management hazards and the verifiability of evidence.

[0137] Example 3: Case Study of Comparison of Static Background Images Across Multiple Time Periods In another embodiment, video is captured periodically or intermittently at the same street location, and static background images of the street are generated at different time windows. For example, the system can generate background images in the morning, at noon, and at night, or it can generate background images daily or weekly.

[0138] Suppose a background image B1 is generated for a certain location on day 1, and a background image B7 is generated on day 7. Spatial registration is performed on B1 and B7, and a background difference map is calculated. Subsequently, threshold segmentation and connected component analysis are performed on the difference map to obtain candidate change regions. If the candidate change region is located on the outer side of the building facade, and the semantic recognition result is a newly added shed, covered panel, or protruding structure, an alarm "Newly added covered structure pending verification" is generated; if the candidate change region is located in front of the store or at the boundary of the alley, and the recognition result is a newly added box, bagged goods, or shelf, an alarm "Newly added outdoor stacked items pending verification" is generated; if the candidate change region shows a newly added linear target extending into the vehicle parking area, an alarm "New flying wire pending verification" is generated.

[0139] If the same potential hazard area is identified in multiple time windows, the persistence confidence level is calculated. When the persistence confidence level exceeds the persistence threshold, the hazard is classified as a persistent static hazard, and its risk level or review priority is increased.

[0140] Example 4: Electronic devices and computer-readable storage media This invention can also be implemented using an electronic device. The electronic device includes a processor, a memory, an input / output interface, and a communication interface. The memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned method for identifying street hazards based on static background reconstruction.

[0141] In one implementation, it can be deployed on a computer with an 8-core, 16-thread or higher CPU and at least 32GB of RAM. An optional GPU with at least 8GB of video memory can be used to accelerate image recognition, object detection, or multimodal model inference. It supports Windows or Linux operating systems and can be developed using Python, employing libraries such as OpenCV, FFmpeg, NumPy, PyTorch, and GDAL / OGR. OpenCV can be used for video frame reading, SIFT feature detection, FLANN matching, RANSAC estimation, and affine transformations; NumPy can be used for matrix computation and background fusion; PyTorch can be used for deep model inference; and GDAL / OGR can be used for geographic data reading and writing and spatial feature output.

[0142] This invention can also be implemented as a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the aforementioned method for identifying street hazards based on static background reconstruction. The computer-readable storage medium includes, but is not limited to, hard disks, solid-state drives, flash memory, read-only memory, random access memory, optical disks, portable storage devices, or cloud storage media.

[0143] Other alternative implementation methods It should be noted that this invention is not limited to the specific algorithms and parameters described above. SIFT in local feature extraction can be replaced with ORB, AKAZE, SURF, or deep local features; FLANN matching can be replaced with brute-force matching or learned matching; RANSAC affine estimation can be replaced with homography matrix estimation, similarity transformation estimation, optical flow registration, or electronic image stabilization methods; median fusion can be replaced with quantile fusion, weighted fusion, low-rank background modeling, or neural network-based background estimation.

[0144] Image recognition models or multimodal semantic recognition models are not limited to specific cloud platforms or specific models. As long as they can receive static background images of blocks, spatial attribute data, or structured prompt information, and output static hazard categories, hazard areas, confidence levels, or evidence summaries, they can serve as the hazard identification model of this invention. When the model is unavailable, the local rule engine can independently complete basic static hazard identification.

[0145] The above embodiments collectively demonstrate that the present invention can generate static background images of streets that can be used for hazard identification from highly obscured street videos through candidate frame screening, local feature matching, robust spatial alignment, and multi-frame background fusion. The background image is then combined with spatial attribute data and hazard rules to form structured hazard evidence data that is verifiable, traceable, and spatially expressible.

Claims

1. A method for identifying potential street hazards based on static background reconstruction, characterized in that, include: Step 1: Set up observation points in the target street area, collect video data and obtain spatial attribute data at the observation points using camera equipment. The spatial attribute data includes point number, latitude and longitude coordinates, road name, street type, collection time, camera position type, nearby points of interest, road network relationship and the governance grid to which the target street point belongs. Step 2: Extract video frame sequences from the video data, perform frame extraction and quality screening on the video frame sequences to obtain a candidate frame set; Step 3: Normalize the size of the candidate frames in the candidate frame set, select one frame in the candidate frame set as the reference frame, perform feature matching between the candidate frame and the reference frame to obtain the matching point set; use the random sampling consensus algorithm to estimate the affine transformation matrix of the matching points in the matching point set, and map the candidate frame to the coordinate system of the reference frame to obtain the aligned frame. Step 4: Repeat steps 3-4 for all candidate frames in the candidate frame set to obtain the aligned frame set; Step 5: Perform multi-frame background fusion on the aligned frame set to obtain a static background image of the street block, and calculate the confidence score of the static background image of the street block; Step 6: Perform static hazard identification based on the static background image of the street block and the spatial attribute data to obtain the static hazard identification result; wherein, the static hazard identification result includes at least one of candidate hazard label, hazard area, risk level, model identification confidence score and evidence summary; Step 7: Repeat steps 1-6 in multiple time windows to obtain static background images of the same target street location and compare them to calculate the confidence level of hazard persistence. Step 8: Calculate the hazard confidence score by weighting the model identification confidence score, the street static background image confidence score, the spatial scene matching confidence score, and the multi-time period hazard persistence confidence score, and determine the risk level based on the hazard confidence score and hazard type weight.

2. The method according to claim 1, characterized in that, The target block mentioned in step 1 includes at least one of the following: street entrance, street-front shops, old street roads, alleyways in urban villages, courtyard perimeter, storefront perimeter, areas adjacent to public facilities, or road sections in mixed residential and commercial areas; the video data includes fixed camera videos, mobile inspection videos, surveillance-like videos, or other video files that can reflect the spatial status of the block. The location number, latitude and longitude coordinates, or acquisition time are obtained from the video file name parsing results, acquisition device positioning data, manually labeled data, or geographic information matching results.

3. The method according to claim 1, characterized in that, Step 2, which involves extracting frames and filtering quality from the video frame sequence, includes: extracting initial frames according to a preset frame interval or a preset time interval to obtain an initial frame set; performing quality filtering on the initial frame set to remove low-value frames; and forming the candidate frame set with the retained initial frames. The low-value frames include blurred frames, overexposed frames, underexposed frames, frames with an occlusion ratio exceeding a threshold, or frames with a similarity to the retained frames exceeding a threshold. The blurred frames are those whose scores, identified by Laplacian variance, are below the first threshold. Overexposed and underexposed frames are those whose grayscale mean is below the second threshold or above the third threshold. The inter-frame similarity is obtained through structural similarity index, image histogram correlation, perceptual hash distance, and / or feature descriptor similarity. The occlusion ratio is the proportion of the foreground region or the detection result of the significantly occluded region.

4. The method according to claim 1, characterized in that, Step 3, which involves feature matching between candidate frames and the reference frame, includes: using a ratio test to eliminate ambiguous matching points; employing a FLANN approximate nearest neighbor matcher or a brute-force matcher; and for the same candidate descriptor, assuming the nearest neighbor matching distance is d1 and the second nearest neighbor matching distance is d2, when... When a match is selected, a matching point is retained; where ρ is the ratio test threshold; all others are discarded. The affine transformation matrix is ​​estimated using a random sampling consensus algorithm. Candidate frames are then mapped to the reference frame coordinate system to obtain aligned frames. This process involves: randomly selecting at least three pairs of matching points from the set of matching point pairs; estimating the candidate affine transformation matrix; calculating the reprojection error of all matching point pairs; marking the matching point pair as an interior point if the reprojection error is less than a threshold; otherwise, marking it as an exterior point. This process of estimating the candidate affine transformation matrix is ​​repeated multiple times. The candidate affine transformation matrix with the most interior points or the highest proportion of interior points is selected, and the least squares solution is performed again using all interior points to obtain the final affine transformation matrix. Finally, the candidate frames are mapped to the reference frame coordinate system using the affine transformation matrix to obtain aligned frames.

5. The method according to claim 1, characterized in that, Step 5, which involves performing multi-frame background fusion on the aligned frame set to obtain a static street background image, includes: performing dynamic foreground segmentation on the aligned frame set to obtain a dynamic foreground mask; based on the dynamic foreground mask, reducing the weight of pixels in the dynamic foreground region in the background fusion, or removing pixels in the dynamic foreground region from the pixel set participating in the fusion; and performing fusion calculation on the pixel values ​​at the same pixel position in the pixel set participating in the fusion to obtain the static street background image. The fusion calculation method is at least one of median fusion, quantile fusion, mean fusion, weighted fusion, or background modeling fusion. Confidence of static background images of street blocks The calculation formula is as follows: in, Rate the image alignment quality. Scoring for blurry frames, Rate the completeness of background coverage. , , Let be the weighting coefficients, and satisfy: The image alignment quality score can be determined based on the proportion of matched inliers, the average reprojection error, or the consistency of aligned edges: in, The proportion of RANSAC in-points. This represents the average reprojection error. These are the weighting coefficients; The blurred frame The score is based on the Laplace variance score. Gradient strength scoring or edge density score Confirmed, the calculation formula is as follows: Where η1, η2, and η3 are weighting coefficients, and η1 + η2 + η3 = 1; Laplace variance score The calculation formula is: in, Represents the Laplacian response of a grayscale background image. This represents the variance of the Laplace response. The threshold for Laplace variance normalization; Gradient strength score for: Where G represents the gradient magnitude map of the grayscale background image, which can be calculated by the Sobel operator: Gx and Gy are the gradients in the horizontal and vertical directions, respectively, and Mean(G) represents the mean magnitude of the gradient. This is the gradient strength normalization threshold; Edge density score The calculation formula is: Where Redge is the edge pixel ratio, obtained through Canny edge detection, Sobel edge thresholding, or other edge detection methods: Redge = Nedge / Nvalid Where Nedge is the number of edge pixels and Nvalid is the number of valid background pixels. The threshold for edge density normalization; The background coverage integrity score can be determined based on the dynamic foreground retention ratio and the background hole ratio: Among them, the dynamic foreground residual ratio The calculation process is as follows: Align frame I with frame t t Generate dynamic foreground mask ,in, This indicates that the pixel location belongs to the dynamic foreground region; This indicates that the pixel location belongs to the static background candidate region; For pixel position (x, y), calculate its dynamic foreground occurrence frequency. : Where T is the number of aligned frames participating in background fusion; if Greater than or equal to the foreground residue threshold τ fg Then the foreground residual mask They believe that there is a high risk of dynamic foreground residue at this location; if Less than the foreground residue determination threshold τ fg Then the foreground residual mask They believe that there is a low risk of dynamic foreground residue at this location; Dynamic foreground residual ratio Calculate according to the following formula: in, for The number of pixels, where Nvalid is the total number of valid pixels in the static background image of the block; The calculation process is as follows: set up Align the effective pixel markers at pixel position (x, y) for frame t. This indicates that the aligned frame has a valid pixel at that position. This indicates that there are no valid pixels at this location; For pixel position (x, y), the effective background observation of the T-th frame is calculated as follows: Summing over all T frames yields the number of valid background observations for that pixel location: if Less than the minimum number of background observations n min Then the background hole mask The pixel is considered to be a background hole pixel; if Greater than or equal to the minimum number of background observations n min Then the background hole mask The pixel is considered not to be a background hole pixel. Background Hole Ratio Calculate according to the following formula: in, Nvalid is the number of background holes in pixels, and Ntarget is the total number of valid pixels in the target background image coordinate system. When the confidence level of the static background image of the block is lower than the preset quality threshold, the candidate frame set, time window, reference frame or background fusion parameters are readjusted, and steps 1 to 5 are repeated to generate the static background image of the block. The background reconstruction parameters include at least one of the following: frame extraction interval, time window size or background fusion parameters.

6. The method according to claim 5, characterized in that, Step 7 involves the following comparison: acquiring multiple static background images of the same target street location generated at different time windows, and calculating the background difference value between the static background images of the street in different time windows. Among them, D t B represents the background difference value at the pixel location. t and B (t-1) Background images generated for adjacent time windows; Threshold segmentation, morphological processing, and connected component analysis are performed on the difference map to obtain candidate regions of change. Perform static hazard identification on the candidate change areas according to step 6; The confidence level for the persistence of hidden dangers over multiple time periods is: ; n p This indicates the number of times the same candidate hazard area is identified within K time windows.

7. The method according to claim 6, characterized in that, The categories of hazards include at least one of the following: unauthorized private wiring for charging, obstruction of fire lanes, illegal addition of structures, damage to building facades, illegal stacking outside shops, damage to public facilities, stacking of objects on roads or alleys, abnormal closure of the back area of ​​shops, mixed production, storage and living areas, abnormal spatial boundaries, or abnormal street environment quality. The risk level is determined by calculating the overall hazard confidence level. Based on the preset threshold range where the comprehensive hazard confidence level falls, a corresponding risk level is generated, calculated using the following formula: in, To identify confidence levels for the model, Confidence level of static background images of the street block; Match confidence levels to spatial scenes. For the persistence confidence level of potential hazards; ; in: The neighborhood type is matched and scored, and the scenarios with high matching scores for candidate hazard tags are selected. , middle matching Low matching , Scoring is performed on surrounding points of interest (POIs). POIs within a 100-meter radius of the target location are retrieved, and calculations are made based on POI type and distance decay. ;in For surrounding points of interest, Weights for interest point types, The distance from the target point to the point of interest. For distance scoring, the shortest distance from the target point to the static risk-related area is defined as . If the maximum effective distance threshold is set to 100 meters, then: ; Based on the aforementioned comprehensive hazard confidence level Weight of hazard type Determine the risk level G, and calculate it using the grade function Grade: .

8. The method according to claim 7, characterized in that, In step 6, static hazard identification is performed using an image recognition model, a multimodal semantic recognition model, or a local rule model to obtain the static hazard identification result. Step 8 further includes: the image recognition model is a target detection model, image classification model, semantic segmentation model, visual language model, or multimodal large model; when the interface of the image recognition model or multimodal semantic recognition model is unavailable, the call times out, the output format is abnormal, or the confidence level is lower than a preset threshold, the local rule model is called to generate basic static hidden danger identification results; the local rule model identifies based on image edges, linear targets, region occupancy, texture anomalies, color anomalies, geometric shapes, and spatial positional relationships; When using a multimodal semantic recognition model to identify static hidden dangers The model recognition confidence score returned by the multimodal semantic recognition model is directly used; when using a local rule model to identify static hidden dangers... By candidate hazard label calculate: Targeting candidate hazard labels Pre-configure rule sets: Each rule Configure corresponding weights ,and If the first If the rule is satisfied, then If not satisfied, then Based on this calculation: .

9. The method according to claim 1, characterized in that, The method further includes: performing personal identity information desensitization processing on the static background image of the street block to obtain a desensitized evidence image; and structurally encapsulating and persistently storing the static hazard identification result, the spatial attribute data, and the desensitized evidence image.

10. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the street hazard identification method based on static background reconstruction as described in any one of claims 1 to 9.

Images