Method and device for judging illegal events based on spatio-temporal environment perception and visual semantic mapping

Abstract

The application discloses a kind of method and device for judging illegal event based on space-time environment perception and visual semantic mapping, the method obtains the multi-source perception data of city monitoring area, and determines the exception state of target object according to target object identity auxiliary information and license record data;For target object not determined as exception object, the mapping relationship between video image coordinates and city space coordinates is established, and the target object is mapped into city space coordinates;City space semantic layer is constructed, visual semantic mapping result is generated based on target object visual semantic feature, space unit semantic feature and spatial relationship;Event context features are generated by combining time, environment, rules and license record information, and input into rule-enhanced space-time semantic illegal judgment model, to determine the violation category, violation level and violation confidence.The application can improve the accuracy, real-time performance and interpretability of city illegal event judgment.

Description

Technical Field

[0001] This invention relates to the technical field of urban management, specifically to a method and apparatus for identifying violations based on spatiotemporal environmental perception and visual semantic mapping. Background Technology

[0002] With the continuous improvement of urban governance, video surveillance equipment, inspection equipment, and urban management platforms are being deployed more and more in urban roads, commercial streets, community entrances and exits, areas around schools and hospitals, parks and squares, parking areas, and fire lanes. These platforms are used to identify and handle urban violations such as illegal parking of motor vehicles, disorderly parking of non-motor vehicles, street vending, businesses operating outside their premises, illegal dumping of garbage, illegal construction barriers, obstruction of fire lanes, obstruction of blind paths, and obstruction of bus stops.

[0003] In existing technologies, urban violation identification typically relies on video image recognition models to detect, classify, track, and identify the behavior of target objects in surveillance footage. However, urban violations are not solely determined by the target object or its behavior, but are closely related to the specific urban spatial semantics in which the target object is located. Furthermore, urban violations often have significant spatiotemporal and rule-based constraints. Existing technologies, if employing only fixed rules or a single model for violation identification, struggle to simultaneously consider the target object's identity, space occupancy ratio, behavior duration, environmental conditions, permit / registration information, exception rules, and dynamic control rules. Especially when dynamic rules and urban spatial layers are maintained separately, issues may arise where the rule base has been updated but the spatial layer has not, leading to inconsistent violation identification criteria and affecting the stability and reliability of the system's results.

[0004] Therefore, there is a need to provide a method for identifying urban violations that integrates video visual recognition, urban spatial semantic mapping, spatiotemporal environment perception, rule-enhanced discrimination, rapid exception screening, and dynamic layer consistency verification, in order to improve the accuracy, adaptability, and interpretability of urban violation identification and discrimination. Summary of the Invention

[0005] The purpose of this invention is to provide a method for identifying urban violations based on spatiotemporal environment perception and visual semantic mapping, which solves the problems that existing urban violation identification methods mainly rely on target detection and behavior recognition results in video images, and that violation identification methods mostly use fixed rules or single models for identification.

[0006] The first aspect of this invention provides a method for identifying urban violation events based on spatiotemporal environmental perception and visual semantic mapping, the method comprising:

[0007] Acquire multi-source sensing data of the urban monitoring area, including video image data, camera equipment parameters, shooting time information, urban spatial data, environmental status data, permit and filing data, and urban management rule data; Visual target recognition and behavioral feature extraction are performed on the video image data to obtain the visual recognition result of the target object. The visual recognition result includes at least the target object category, target object location, target object occupied area, target object trajectory, target object behavioral state, and target object identity auxiliary information. Based on the target object's identity auxiliary information and the license filing data, determine whether the target object belongs to the exception object; If the target object is not an exception object, then a mapping relationship between the video image coordinates and the urban spatial coordinate system is established based on the camera equipment parameters and the urban spatial data, and the target object is mapped to the urban spatial coordinate system based on the mapping relationship; A city spatial semantic layer is constructed, and a visual semantic mapping result is generated based on the visual semantic feature vector of the target object, the spatial semantic feature vector of the spatial unit, and the edge attribute vector between the target object and the spatial unit. By integrating the visual semantic mapping results, the target object's behavioral state, the shooting time information, the environmental state data, the permit and filing data, and the urban management rule data, event context features of candidate urban events are generated. The event context features are input into a rule-enhanced spatiotemporal semantic violation discrimination model to determine the violation category, violation level, and violation confidence of the candidate city events. Based on the violation category, the violation level, and the violation confidence level, a violation event evidence chain is generated and a judgment result is output.

[0008] In a second aspect, the present invention provides an urban violation incident identification device based on spatiotemporal environment perception and visual semantic mapping, the device comprising: The data acquisition module is used to acquire multi-source sensing data of the urban monitoring area; the multi-source sensing data includes video image data, camera equipment parameters, shooting time information, urban spatial data, environmental status data, permit and filing data, and urban management rule data; The visual recognition module is used to perform visual target recognition and behavioral feature extraction on the video image data to obtain the visual recognition result of the target object. The visual recognition result includes at least the target object category, target object location, target object occupied area, target object trajectory, target object behavioral state and target object identity auxiliary information. The rapid exception screening module is used to determine whether the target object belongs to the exception object based on the target object's identity auxiliary information and the license filing data; The spatial mapping module is used to establish a mapping relationship between video image coordinates and urban spatial coordinate system based on the camera device parameters and urban spatial data if the target object is not an exception object, and to map the target object into the urban spatial coordinate system based on the mapping relationship. The semantic mapping module is used to generate a visual semantic mapping result based on the visual semantic feature vector of the target object, the spatial semantic feature vector of the spatial unit, and the edge attribute vector between the target object and the spatial unit. The context generation module is used to fuse the visual semantic mapping results, the target object's behavioral state, the shooting time information, the environmental state data, the permit and filing data, and the urban management rule data to generate event context features of candidate city events; The violation detection module is used to input the event context features into a rule-enhanced spatiotemporal semantic violation detection model to determine the violation category, violation level, and violation confidence of the candidate city event; The evidence chain generation module is used to generate an evidence chain for the violation event and output the judgment result based on the violation category, the violation level and the violation confidence level.

[0009] Compared with existing technologies, this invention acquires multi-source sensing data from urban monitoring areas, performs visual target recognition and behavioral feature extraction on video images, and establishes a mapping relationship between video image coordinates and urban spatial coordinates based on camera equipment parameters and urban spatial data, enabling target objects in video images to be transformed into real urban spatial coordinates. Furthermore, this invention constructs an urban spatial semantic layer and builds an edge attribute vector relationship graph between target objects and spatial units based on the visual semantic features of target objects, the spatial semantic features of spatial units, and the spatial relationships between them. Then, it uses a spatial relationship graph network to generate visual semantic mapping results, thereby determining the primary spatial semantics, auxiliary spatial semantics, spatial occupancy ratio, spatial relationship confidence, and potential violation prior labels corresponding to the target object. Thus, the system can not only identify target objects appearing in images but also determine the relationship between the target object and urban management spaces such as bus stops, fire lanes, tactile paving, sidewalks, no-parking zones, and permitted outdoor parking areas, improving the accuracy of spatial semantic recognition of urban violations and reducing false alarms and missed alarms caused by unclear spatial semantic judgments.

[0010] This invention also comprehensively judges candidate city events through rapid exception screening, event context feature generation, temporal behavior modeling, rule-enhanced discrimination, attention fusion model, and consistency verification between dynamic layers and rule base versions. For target objects with emergency support attributes, law enforcement attributes, or permit registration attributes, exception records can be generated in advance to avoid them entering the high-cost violation judgment process and reduce computational resource consumption. For candidate city events entering the subsequent judgment process, visual behavioral features, spatial semantic features, temporal state features, environmental state features, rule semantic features, historical event features, and temporal dynamic features can be comprehensively considered to determine the violation category, violation level, and violation confidence. Simultaneously, by binding the dynamic state semantic layer to the rule version number and performing consistency verification before judgment, judgment errors caused by inconsistencies between dynamic rules and spatial layer data can be avoided. Furthermore, this invention generates a chain of evidence for violation events that includes event type, occurrence time, occurrence location, target trajectory, spatial semantic region, rule version number, dynamic layer version number, duration, occupancy ratio, violation confidence level, and determination reason. This makes the determination results highly interpretable and verifiable, facilitating dispatching by the urban management platform, handling by law enforcement terminals, and manual review.

[0011] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art are briefly described below. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating the urban violation event identification method based on spatiotemporal environment perception and visual semantic mapping provided in Embodiment 1 of the present invention. Figure 2 This is a flowchart illustrating the visual semantic mapping process provided in Embodiment 1 of the present invention; Figure 3 A flowchart illustrating the rule-enhanced spatiotemporal violation event discrimination process provided in Embodiment 1 of the present invention; Figure 4 This is a flowchart illustrating the synchronization process between the dynamic layer and the rule base provided in Embodiment 2 of the present invention; Figure 5This is a schematic diagram of the urban violation incident discrimination device based on spatiotemporal environment perception and visual semantic mapping provided in Embodiment 3 of the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0014] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention and not the entire structure.

[0015] Example 1 Figure 1 This is a flowchart illustrating the urban violation incident identification method based on spatiotemporal environment perception and visual semantic mapping provided in Embodiment 1 of the present invention. This embodiment is applicable to the automatic identification and judgment of violation incidents in urban governance scenarios such as urban roads, commercial streets, community entrances and exits, school perimeters, hospital perimeters, parks and squares, fire lanes, parking areas, and construction areas. This method can be executed by an urban violation incident identification device based on spatiotemporal environment perception and visual semantic mapping. This device can be implemented using software and / or hardware and can be configured in a server, edge computing device, urban management platform, or video analysis terminal.

[0016] like Figure 1 As shown, the method includes the following steps.

[0017] S100: Acquire multi-source sensing data of the urban monitoring area.

[0018] In this embodiment, multi-source sensing data of the urban monitoring area can be acquired through the data acquisition module 100.

[0019] Urban monitoring areas can include road sections, intersections, bus stops, school gates, hospital entrances, commercial streets, community entrances and exits, parks and squares, parking areas, fire lanes, construction areas, and garbage disposal points.

[0020] Multi-source sensing data includes video image data, camera equipment parameters, shooting time information, urban spatial data, environmental status data, permit and filing data, and urban management rule data.

[0021] Video image data can be collected by fixed cameras, road monitoring equipment, urban management patrol vehicles, drones, law enforcement recorders, mobile patrol terminals, or community video surveillance equipment.

[0022] Camera equipment parameters may include camera equipment number, installation location, installation height, shooting direction, pitch angle, focal length, field of view, internal parameters, external parameters, shooting range, and monitoring area number, etc.

[0023] Urban spatial data can include road boundaries, pedestrian boundaries, motor vehicle lanes, non-motor vehicle lanes, bus stops, parking spaces, pedestrian crossings, fire lanes, tactile paving for the visually impaired, green belts, building sets, shop facades, plazas, park entrances, areas surrounding schools, hospital emergency access routes, temporary construction areas, etc.

[0024] Environmental status data can include weather conditions, visibility, road congestion, pedestrian density, vehicle density, temporary event status, construction status, and holiday status.

[0025] Permit and registration data can include emergency vehicle registration data, law enforcement vehicle registration data, temporary construction permit data, outdoor business permit data, temporary loading and unloading permit data, major event registration data, and temporary traffic control data, etc.

[0026] Urban management rules data can include rules on no-parking, rules on street vending, rules on businesses operating outside their premises, rules on non-motorized vehicle parking, rules on construction site enclosures, rules on garbage disposal, rules on the passage of construction waste trucks, rules on the occupation of fire lanes, rules on the protection of tactile paving, rules on no-parking at bus stops, and temporary control rules, etc.

[0027] S200: Perform visual target recognition and behavioral feature extraction on video images.

[0028] In this embodiment, the visual recognition module 200 can be used to perform visual target recognition and behavioral feature extraction on video images.

[0029] The video image data is processed by image enhancement, distortion correction, brightness normalization and noise suppression, and the processed video image is input into the visual recognition model to obtain the visual recognition result of the target object.

[0030] Target objects may include motor vehicles, non-motor vehicles, shared bicycles, pedestrians, stalls, tables and chairs, goods, garbage bags, billboards, construction materials, construction fences, sources of smoke and fire, road obstacles, dump trucks, food delivery vehicles, express delivery vehicles, law enforcement vehicles, fire trucks, ambulances, police vehicles, etc.

[0031] The visual recognition results include target object category, target object detection box, target object segmentation mask, target object confidence, target object pose, target object motion trajectory, target object behavior state, and target object identity auxiliary information.

[0032] The auxiliary information for the target object's identity may include vehicle license plate recognition results, vehicle appearance category, vehicle color, vehicle-specific markings, vehicle body text markings, law enforcement markings, emergency markings, registration number, and matching results with permit registration data.

[0033] In one embodiment, the visual recognition module 200 includes a target detection sub-model 210, an instance segmentation sub-model 220, a target tracking sub-model 230, a behavior recognition sub-model 240, and an identity recognition sub-model 250.

[0034] The target detection sub-model 210 is used to determine the category and detection box position of the target object in the video image, and to determine the position of the target object based on the position of the detection box. The instance segmentation sub-model 220 is used to determine the actual area occupied by the target object in the frame. The target tracking sub-model 230 is used to assign a target identifier to the same target object in consecutive video frames and generate the target object trajectory. The behavior recognition sub-model 240 is used to determine whether the target object is engaged in behaviors such as staying, moving, placing, gathering, dumping, construction, operation, occupying the road, loading and unloading, going against the flow, or stacking, based on the target object status in consecutive frames. The identity recognition sub-model 250 is used to identify whether the target object belongs to an emergency target, law enforcement target, registered target, or licensed target.

[0035] For example, in road surveillance videos, the visual recognition module 200 can identify vehicle targets and determine the vehicle's speed, direction, dwell time, trajectory, and license plate number in consecutive video frames; in commercial street videos, the visual recognition module 200 can identify objects such as tables, chairs, stalls, and goods, and determine whether they are continuously placed in public passage areas.

[0036] S250. Perform rapid exception screening on the target object.

[0037] In this embodiment, the target object can be quickly screened for exceptions using the quick exception screening module 260.

[0038] After obtaining the visual recognition results of the target object, a rapid exception screening is performed on the target object based on the target object's identity auxiliary information and license filing data.

[0039] Specifically, if the target object is identified as a fire truck, ambulance, police car, law enforcement vehicle, emergency rescue vehicle, or other vehicle with emergency support attributes, and the vehicle license plate, vehicle exterior markings, or registration number of the target object matches the permit registration data, then the target object is marked as an exception object.

[0040] If the area corresponding to the target object has a valid temporary construction permit, outdoor business permit, temporary loading and unloading permit, or major event filing information, and the time, location, and object type of the target object all meet the corresponding permit conditions, then the target object will also be marked as an exception object.

[0041] For exception objects, the rapid exception screening module 260 generates an exception record and stores it in the event database, without proceeding to the high-overhead violation judgment process corresponding to S300 to S700. For target objects that cannot be determined to be exceptions or that are determined not to be exceptions, the processing flow continues to S300 to S700.

[0042] This step allows for the early exclusion of objects that should not be classified as violations, reducing the computational burden on subsequent spatial mapping, semantic reasoning, and violation classification models, and minimizing the interference of exceptional samples on the training process of the violation discrimination model.

[0043] In one implementation, S250 can be implemented using a branching decision process. If the target object is determined to be a fast exception object, an exception record is generated directly; if the target object is not determined to be a fast exception object, or the exception status of the target object cannot be determined, then S300 continues to be executed.

[0044] S300. Establish the mapping relationship between video image coordinates and urban spatial coordinates.

[0045] In this embodiment, the spatial mapping module 300 can be used to establish a mapping relationship between video image coordinates and urban spatial coordinates.

[0046] Based on camera equipment parameters, urban spatial data, and spatial reference points, a mapping relationship is established between the video image coordinate system and the preset urban spatial coordinate system.

[0047] Specifically, road boundary points, pedestrian crossing corner points, parking space corner points, curb boundary points, building corner points, traffic sign locations, and manhole cover center points within the monitoring area are selected as spatial reference points. The pixel positions of these spatial reference points in the video image coordinate system and their actual positions in the urban spatial coordinate system are determined.

[0048] Based on the pixel positions of spatial reference points and their actual positions, calculate the transformation parameters from image coordinates to urban spatial coordinates. These transformation parameters may include homography matrix, perspective transformation parameters, camera projection matrix, or local spatial registration parameters.

[0049] In one implementation, the spatial mapping module 300 maps the target object from the video image coordinate system to the urban spatial coordinate system based on the center point of the bottom edge of the target object detection box, the boundary point of the target object segmentation mask, or the ground contour point of the target object, thereby obtaining the center position, occupied area, movement direction, movement speed, and trajectory sequence of the target object in the real urban space.

[0050] This step can change the meaning of "the target vehicle is located in the lower left corner of the screen" in the video image to "the target vehicle is located within the boundary of a bus stop area on a certain road" or "the target vehicle is occupying the boundary area of ​​a fire lane".

[0051] S400: Construct a semantic layer of urban space and generate visual semantic mapping results.

[0052] In this embodiment, a city spatial semantic layer can be constructed and a visual semantic mapping result can be generated through the semantic mapping module 400.

[0053] This step transforms the target object from a purely visually recognizable object into a spatial semantic object with urban management implications. It's important to note that S400 does not output a final violation conclusion, nor is it a final violation determination step. Instead, it outputs spatial semantic features, spatial relationship confidence scores, and potential violation prior labels for use by the subsequent violation determination module 600. Whether a violation occurs, what type of violation it is, the violation level, and the violation confidence score are all determined by the rule-enhanced spatiotemporal semantic violation determination model in the violation determination module 600.

[0054] like Figure 2 As shown, S400 may include the following steps.

[0055] S410. Construct a multi-level urban spatial semantic layer based on the rule base.

[0056] In this embodiment, a multi-level urban spatial semantic layer can be constructed based on a rule base using the urban spatial semantic layer construction unit 410.

[0057] Based on urban geographic information data, road facility data, urban component data, management area data, and urban violation event rule base, a multi-level urban spatial semantic layer is constructed.

[0058] The urban spatial semantic layer includes a road traffic semantic layer 411, a public space semantic layer 412, an urban component semantic layer 413, a management rule semantic layer 414, and a dynamic status semantic layer 415.

[0059] The road traffic semantic layer 411 is used to represent road traffic spaces such as motor vehicle lanes, non-motor vehicle lanes, sidewalks, pedestrian crossings, bus stops, parking spaces, intersection areas, temporary parking areas, and no-parking areas.

[0060] The public space semantic layer 412 is used to represent public activity spaces such as squares, park entrances, subway entrances, school gates, hospital entrances, community entrances and exits, and commercial streets.

[0061] The urban component semantic layer 413 is used to represent urban components such as fire hydrants, fire lanes, manhole covers, trash cans, street light poles, guardrails, tactile paving, traffic signs, parking posts, and charging piles.

[0062] The semantic layer 414 of the management rules is used to represent urban management areas such as strictly managed streets, outdoor vending permit areas, night market permitted areas, construction permit areas, construction waste truck restricted areas, holiday control areas, and key protection areas.

[0063] The dynamic state semantic layer 415 is used to represent the dynamic management status that changes over time, including no parking during peak hours, temporary nighttime operating permits, temporary control of major events, traffic restrictions during rain and snow, and temporary construction closures.

[0064] The dynamic state semantic layer 415 is generated by regional rules, time rules, and exception rules from the city's violation event rule base. Each dynamic state semantic layer 415 carries a rule identifier, rule version number, layer version number, effective time, expiration time, and applicable area identifier.

[0065] When the city's violation rule base is updated, the system automatically generates or updates the corresponding dynamic state semantic layer 415 based on the updated rules, ensuring that the rule version number and the layer version number are consistent. If the rule version number and the layer version number are inconsistent, the use of the corresponding dynamic state semantic layer 415 for violation identification is suspended, and a rule layer consistency check is triggered.

[0066] By using the above method, the problem of data inconsistency caused by "the same temporary management state existing in both the rule base and the dynamic layer" is avoided. The dynamic state semantic layer 415 is used as the spatial projection of the rules, and the rule base is used as the source data for the generation and discrimination of the dynamic layer.

[0067] S420. Generate the visual semantic feature vector of the target object.

[0068] In this embodiment, the visual semantic feature vector of the target object can be generated by the target object visual semantic encoding unit 420.

[0069] The target object recognition results obtained by S200 are used for feature encoding to generate the visual semantic feature vector of the target object.

[0070] Visual semantic feature vectors include target category features, target morphological features, target behavioral features, target temporal features, and target spatial occupancy features.

[0071] The target category feature is used to characterize the target object as a category such as motor vehicle, non-motor vehicle, stall, garbage, construction materials, tables and chairs, billboard, etc.

[0072] Target morphological features are used to characterize the area, aspect ratio, outline shape, pose angle, and occlusion ratio of a target object.

[0073] Target behavior characteristics are used to characterize whether the target object engages in behaviors such as staying, moving, placing, gathering, dumping, construction, operation, loading and unloading.

[0074] Target temporal features are used to characterize the duration, speed, frequency of stay, trajectory direction, and number of times a target object enters or exits a region.

[0075] The target space occupancy feature is used to characterize the target object's occupied area, ground contour, overlap ratio with road boundaries, and distance from urban components.

[0076] In one implementation, the target object visual semantic encoding unit 420 can input the target object category, target object behavior, target object trajectory, and target object occupied area into a multimodal feature encoding model to generate a unified target visual semantic vector. The multimodal feature encoding model can employ one or more combinations of convolutional neural networks, Transformer networks, temporal convolutional networks, or visual language models.

[0077] Let the visual semantic feature vector of the target object be... ,in, Indicates the first One target object.

[0078] S430, Generate spatial semantic feature vectors for urban spatial units.

[0079] In this embodiment, the spatial semantic feature vector of the urban spatial unit can be generated by the spatial unit semantic encoding unit 430.

[0080] The urban monitoring area is divided into multiple spatial units, each corresponding to a spatial semantic node. Spatial units can be divided according to road grids, image projection areas, GIS areal areas, management area boundaries, or fixed-size spatial grids.

[0081] For each spatial unit, its spatial semantic features are extracted to generate a spatial semantic feature vector.

[0082] Spatial semantic features include spatial type features, administrative attribute features, adjacency relationship features, distance relationship features, and temporal rule features.

[0083] Spatial type characteristics are used to indicate whether the spatial unit belongs to a motor vehicle lane, sidewalk, bus stop, fire lane, tactile paving, parking space or green belt.

[0084] Management attributes are used to characterize whether parking is prohibited, temporary parking is allowed, outdoor parking is allowed, business operations are prohibited, or construction is allowed in the space unit.

[0085] The adjacency relationship feature is used to characterize whether the spatial unit is adjacent to an intersection, school, shop, fire-fighting facility or public entrance / exit.

[0086] Distance relationship features are used to characterize the distance of the spatial unit from objects such as pedestrian crossings, fire hydrants, bus stops, road boundaries, and building entrances.

[0087] The time rule feature is used to characterize the management status of the spatial unit in different time periods, and is associated with the corresponding rule identifier, rule version number and layer version number.

[0088] Let the spatial semantic feature vector of a spatial unit be... ,in, Indicates the first Each spatial unit.

[0089] S440. Construct a graph showing the relationship between the edge attribute vectors of the target object and the spatial unit.

[0090] In this embodiment, a relationship graph between the target object and the edge attribute vector of the spatial unit can be constructed using the relationship graph construction unit 440.

[0091] Based on the location, trajectory, and occupied area of ​​the target object in the urban spatial coordinates, an edge attribute vector relationship graph between the target object and the spatial unit is constructed.

[0092] In the graph of the relationship between the target object and the spatial unit edge attribute vector, the target object is treated as an object node, the urban spatial unit is treated as a spatial node, and the spatial relationship between the target object and the spatial unit is treated as an edge.

[0093] The attributes of the edge include the overlap ratio between the target object and the spatial unit, whether the center point of the target object is located within the spatial unit, the distance between the boundary of the target object and the boundary of the spatial unit, whether the trajectory of the target object enters or crosses the spatial unit, whether the target object stays within the spatial unit, and the occupancy or occupation relationship of the target object on urban components within the spatial unit.

[0094] Let the edge attribute vector between target object i and spatial unit j be... It should include at least the overlap ratio, normalized distance, dwell time, number of entries, crossing status, occlusion status, and spatial relationship type encoding.

[0095] For example, when there is a high degree of overlap between the grounding area of ​​a motor vehicle and the bus stop area, and the vehicle speed is lower than the preset speed and the dwell time exceeds the preset time, the relationship graph construction unit 440 forms a relationship edge of "motor vehicle - occupation - bus stop" in the relationship graph.

[0096] S450. Visual semantic mapping is performed using graph attention networks.

[0097] In this embodiment, visual semantic mapping can be performed using a graph attention network through the spatial relationship graph network inference unit 450.

[0098] The target object and spatial unit edge attribute vector relationship graph is input into the spatial relationship graph network reasoning unit 450. Joint reasoning is performed on the object node, spatial node and edge attribute to obtain the mapping relationship between the target object and urban spatial semantics.

[0099] In a preferred embodiment, the spatial relation graph network reasoning unit 450 employs a graph attention network. The graph attention network includes a node initialization layer 451, an edge attribute encoding layer 452, an attention calculation layer 453, a neighborhood aggregation layer 454, and a semantic output layer 455.

[0100] First, the object nodes and spatial nodes are uniformly dimensionally encoded through the node initialization layer 451.

[0101] For the target object node Its visual semantic feature vector The initial representation of the object node is obtained through linear mapping. :

[0102] in, , This indicates a unified hidden dimension.

[0103] For spatial node j, its spatial semantic feature vector is... The initial representation of the spatial nodes is obtained through linear mapping. : .

[0104] Secondly, edge attributes are uniformly encoded using edge attribute encoding layer 452. For edge attributes... The edge attribute representation is obtained by linear mapping. .

[0105] Next, the attention coefficient between the target object node i and its neighboring nodes j is calculated using the attention computation layer 453. Then, the attention coefficients for the set of neighboring nodes of the target object node i are calculated. Normalize all spatial nodes to obtain attention weights. :

[0106] in, This represents the set of spatial nodes that have spatial relationships with the target object node i.

[0107] Subsequently, the neighborhood aggregation layer 454 aggregates the adjacency space node information and edge attribute information based on attention weights to obtain the updated representation of the target object node i. For multi-head attention structures, the outputs of multiple attention heads are concatenated or averaged to obtain the final spatial semantic fusion representation of the target object node. :

[0108] Where H represents the number of attention heads.

[0109] Finally, the final spatial semantic fusion representation of the target object node is obtained through the semantic output layer 455. Input semantics to the output layer to obtain visual semantic mapping results:

[0110] in, This represents the probability distribution of target object i belonging to various spatial semantic categories. and These are the parameters for the semantic output layer.

[0111] The visual semantic mapping results include the main spatial semantics, auxiliary spatial semantics, spatial occupancy ratio, spatial relationship confidence, and potential violation prior labels corresponding to the target object.

[0112] It should be noted that the potential violation prior label only indicates the possibility of a spatial semantic association between the target object and a certain type of violation rule. It serves as one of the input features of the subsequent S500 and S600, but is not used as the final violation category or violation conclusion.

[0113] For example, for a vehicle located at the edge of the road, the spatial relationship graph network reasoning unit 450 can comprehensively consider the spatial relationships between its grounding area and parking spaces, motor vehicle lanes, sidewalks, bus stops, and no-parking zones to determine its primary spatial semantics as "no-parking zone at a bus stop" and its secondary spatial semantics as "edge area of ​​the motor vehicle lane," and output a potential violation prior label "candidate for parking at a bus stop." This potential violation prior label only indicates that the subsequent violation judgment process needs to be entered, and does not indicate that it has been determined to constitute illegal parking at a bus stop.

[0114] S460, Generate visual semantic mapping results.

[0115] In this embodiment, the visual semantic mapping result can be generated through the semantic mapping module 400.

[0116] Based on the output of the spatial relationship graph network reasoning unit 450, a visual semantic mapping result is generated.

[0117] The visual semantic mapping results include the main spatial semantic category to which the target object belongs, the auxiliary spatial semantic category, the relationship between the target object and urban components, the proportion of the target object's occupancy in a specific spatial unit, the state of the target object entering or staying in a specific management area, the confidence level of the target object's spatial semantic mapping, and potential violation prior labels.

[0118] For example, the visual semantic mapping result may include: the target object is a motor vehicle, the main spatial semantic is a no-stopping area at a bus stop, the auxiliary spatial semantic is the edge area of ​​the motor vehicle lane, the overlap ratio with the bus stop area is 0.63, the dwell time is 185 seconds, the spatial semantic mapping confidence is 0.91, and the potential violation prior label is a candidate for parking at a bus stop.

[0119] Through the above visual semantic mapping process, "identifying motor vehicles in an image" can be further improved to "there is a high-confidence spatial occupancy relationship between motor vehicles and the no-stopping semantic area of ​​bus stops", thus providing a reliable spatial semantic basis for subsequent violation judgment.

[0120] S500 integrates time, space, environment, and rule information to generate event context features.

[0121] In this embodiment, the event context features can be generated by the context generation module 500 by fusing time, space, environment and rule information.

[0122] Candidate city events are generated based on the visual recognition results obtained by the visual recognition module 200, the rapid exception screening results obtained by the rapid exception screening module 260, the spatial mapping results obtained by the spatial mapping module 300, and the visual semantic mapping results obtained by the semantic mapping module 400.

[0123] Candidate city events include objects, behaviors, spatial semantics, temporal states, environmental states, rule states, and exception states.

[0124] Specifically, the trajectory, speed, dwell time, behavioral state, and spatial semantic mapping results of the target object in consecutive frames are combined to determine whether a candidate city event is formed.

[0125] For example, when a vehicle target stays in a certain spatial semantic area for more than a preset time, a candidate parking event is generated; when a stall, table, chair, or goods target exists continuously in a sidewalk area, a candidate street vending event is generated; when garbage bags or building materials are continuously piled up in a public space, a candidate illegal dumping event is generated.

[0126] Furthermore, the context generation module 500 integrates the event occurrence time, duration, road level, regional functional attributes, weather conditions, pedestrian and vehicle flow conditions, temporary control status, permit and filing status, and urban management rules to generate event context features.

[0127] Event context features include object features, behavioral features, spatial semantic features, temporal semantic features, environmental state features, rule matching features, historical event features, regional risk features, and exception state features.

[0128] S600. Based on the violation event discrimination model, the candidate city events are discriminated.

[0129] In this embodiment, the violation identification module 600 can identify candidate city events based on the violation event identification model.

[0130] This step is used to classify candidate city events using a rule-enhanced spatiotemporal semantic violation discrimination model. The violation discrimination module 600 is the only module in this invention that outputs the final violation category, violation level, and violation confidence score. The potential violation prior labels, spatial semantic categories, and spatial relationship confidence scores output by the semantic mapping module 400 are used as input features of the violation discrimination module 600, and not directly as the final violation discrimination result.

[0131] like Figure 3 As shown, S600 may include the following steps.

[0132] S610. Establish a rule database for urban violations.

[0133] In this embodiment, a rule base for urban violation events can be established through the rule base construction unit 610.

[0134] Establish a rule base for urban violation incidents. The rule base includes basic rules, regional rules, time rules, exception rules, evidence rules, and layer generation rules.

[0135] Basic rules describe the conditions under which general violations are committed, such as vehicles must not block fire lanes, non-motorized vehicles must not block blind paths, stalls must not occupy sidewalks, and garbage must not be piled up in public passageways.

[0136] Regional rules are used to describe the management requirements of a specific area, such as strict management around schools, strict management at the entrance of hospitals, time-limited outdoor stalls in commercial districts, and permission to operate in night market areas.

[0137] Time rules are used to describe the management status corresponding to different time periods, such as no parking during morning and evening rush hours, temporary loading and unloading allowed at night, and temporary traffic control during holidays.

[0138] The exception rule is used to exclude situations that should not be judged as violations, such as law enforcement vehicles, emergency vehicles, temporary construction permits, short-term loading and unloading permits, and registered outdoor parking permits.

[0139] Evidence rules are used to specify the evidentiary conditions required for a certain type of violation to be established, such as duration thresholds, occupied area thresholds, spatial overlap ratio thresholds, number of keyframes, and trajectory continuity requirements.

[0140] Layer generation rules are used to convert region rules, time rules, and exception rules into dynamic state semantic layers 415. Layer generation rules include rule identifier, applicable region, applicable time, spatial boundaries, priority, effective status, and version number.

[0141] The city violation event rule base serves as a unified data source for rule discrimination and dynamic layer generation. The dynamic state semantic layer 415 in S410 is generated by the layer generation rules in this step to ensure consistency between the rule side and the layer side.

[0142] S620. Construct the spatiotemporal state sequence of candidate city events.

[0143] In this embodiment, the spatiotemporal state sequence of candidate city events can be constructed by the spatiotemporal state sequence construction unit 620.

[0144] For each candidate city event, a spatiotemporal state sequence is constructed in chronological order. The spatiotemporal state sequence includes multiple time slices, each time slice corresponding to the state of the candidate city event at a given moment.

[0145] The state data for each time slice includes target object category, target location, target movement speed, target dwell state, target spatial semantics, target occupancy ratio, target behavior state, environmental state, rule matching state, visual recognition confidence, spatial semantic mapping confidence, and exception state.

[0146] For example, for a vehicle parking event, the following spatiotemporal state sequence can be generated: In the 1st second, the vehicle enters the boundary area of ​​the bus stop and travels at a low speed; in the 15th second, the vehicle is stationary and the spatial overlap ratio increases; in the 60th second, the vehicle remains stationary; in the 180th second, the vehicle remains stationary and meets the condition of continuous occupancy of the bus stop area.

[0147] By constructing a spatiotemporal state sequence, violations can be avoided by directly judging them based on a single frame image, thus improving the stability of event discrimination.

[0148] S630. Use the temporal behavior model to determine the duration of an event.

[0149] In this embodiment, the timing behavior analysis unit 630 can use the timing behavior model to determine the duration of an event.

[0150] Input the spatiotemporal state sequence into the temporal behavior model to determine whether candidate city events have stable, continuous, and identifiable behavioral states.

[0151] Temporal behavior models can employ long short-term memory networks, gated recurrent units, temporal convolutional networks, or time series Transformer models.

[0152] The temporal behavior model is used to determine whether a target object is actually stationary, rather than being misjudged due to screen occlusion, tracking loss, or short pauses; to determine whether the target behavior is continuous, rather than being a momentary passing or temporary action; to determine whether the target space occupancy relationship is stable, rather than being a short-term overlap caused by detection box jitter; and to determine whether the target behavior state undergoes a phase change, such as entering, stopping, loading / unloading, or leaving.

[0153] The temporal behavior analysis unit 630 outputs event persistence score, behavior stability score, trajectory continuity score, and temporal confidence score.

[0154] Let the time-series dynamic feature vector t output by the time-series behavior analysis unit 630 be... dyn This includes event persistence score, behavior stability score, trajectory continuity score, temporal confidence, stage change state, and abnormal interruption state. The temporal dynamic feature vector serves as the input to the temporal dynamic coding branch 657 of the feature fusion unit 650 in S650.

[0155] S640. Use rule enhancement models to make preliminary compliance judgments.

[0156] In this embodiment, the rule semantic encoding unit 640 can be used to perform preliminary compliance judgment using a rule enhancement model.

[0157] The event context features of candidate city events are matched with the city violation event rule base to generate preliminary compliance judgment results.

[0158] Specifically, the rule clauses for candidate city events are determined based on event type, target object category, spatial semantics, occurrence time, duration, regional rules, and exception status.

[0159] For example, for candidate city events involving "motor vehicles - stopping - no-stopping areas at bus stops", the system matches the no-stopping rules at bus stops; for candidate city events involving "tables and chairs - placement - sidewalk areas", the system matches the rules for occupying roads for business purposes; and for candidate city events involving "shared bicycles - parking - tactile paving areas", the system matches the rules for obstructing tactile paving.

[0160] In one implementation, the rule semantic encoding unit 640 uses a rule semantic encoding model to encode the rule matching results and generate a rule semantic vector. The rule semantic vector includes rule type, rule level, applicable area, applicable time, exception conditions, evidence conditions, rule priority, and handling level.

[0161] S650. Construct a spatiotemporal semantic violation detection model based on attention fusion.

[0162] In this embodiment, a spatiotemporal semantic violation detection model based on attention fusion can be constructed through the feature fusion unit 650.

[0163] A spatiotemporal semantic violation detection model is constructed to comprehensively identify candidate city events.

[0164] The spatiotemporal semantic violation discrimination model includes a visual behavior coding branch 651, a spatial semantic coding branch 652, a temporal state coding branch 653, an environmental state coding branch 654, a rule semantic coding branch 655, a historical event coding branch 656, and a temporal dynamic coding branch 657.

[0165] The visual behavior coding branch 651 is used to encode the target object category, behavior state, detection confidence, and behavior confidence.

[0166] The spatial semantic coding branch 652 is used to encode the spatial semantics of the target object, the spatial overlap ratio, the distance relationship, the spatial mapping confidence, and the potential violation prior labels output by the semantic mapping module 400.

[0167] The time status coding branch 653 is used to encode the event occurrence time, duration, holiday status, peak status, and temporary control status.

[0168] The environmental status coding branch 654 is used to code weather, pedestrian density, vehicle density, congestion status, and visibility.

[0169] Rule semantic encoding branch 655 is used to encode the management rules, exception rules, evidence rules, rule priority, and layer version information of the hit.

[0170] Historical event coding branch 656 is used to code the historical violation frequency, target object historical behavior, and historical alarm status at the same location in the area.

[0171] The temporal dynamic coding branch 657 is used to encode the event persistence score, behavior stability score, trajectory continuity score, temporal confidence and stage change status output by the temporal behavior analysis unit 630.

[0172] In a preferred embodiment, the feature fusion unit 650 employs a self-attention mechanism to fuse the seven coding branches.

[0173] Let the outputs of the seven coding branches be respectively ,in, Represents a visual behavior encoding vector. Represents a spatial semantic encoding vector. Represents the time-state encoding vector. Represents the environment state encoding vector. Represents the semantic encoding vector of the rules. Represents the encoded vector of historical events. This represents a time-series dynamic coding vector.

[0174] The outputs of each coding branch are unified to a linear mapping. Dimension, we get:

[0175] in, and These are the learnable parameters corresponding to the k-th encoding branch.

[0176] The branch vectors with seven unified dimensions are concatenated row by row to obtain the feature matrix. : .

[0177] Based on feature matrix Generate query matrix Key matrix Sum matrix :

[0178] in, All of these are learnable parameter matrices.

[0179] Calculate the self-attention fusion result : Softmax is used to normalize the attention weights between different branches.

[0180] Will Perform flattening or pooling to obtain the fused event vector. :

[0181] Here, Pool can be average pooling, max pooling, weighted pooling, or attention pooling. (Fusing event vectors) In other words, it is a comprehensive event representation vector, used to represent the comprehensive state of candidate city events in visual, spatial, temporal, environmental, rule-based, historical, and temporal dynamic dimensions.

[0182] In another implementation, the feature fusion unit 650 may employ a cross-attention mechanism to encode vectors using regular semantics. Using other branch vectors as the query source, and other branch vectors as keys and values, the rule features actively focus on visual behavior, spatial semantics, environmental state, and temporal dynamic features. At this point, the query matrix, key matrix, and value matrix are respectively:

[0183] Rule-guided fusion vector for:

[0184] Then With the rule semantic encoding vector The vectors are concatenated to obtain the final comprehensive event representation vector, which is then input into the subsequent violation classification unit 660 to output the violation category, violation level, and violation confidence.

[0185] Through the above fusion method, the dynamic stability features output by the temporal behavior analysis unit 630 are explicitly incorporated into the feature fusion unit 650, and the potential violation prior labels output by the semantic mapping module 400 are also explicitly used as input features of the spatial semantic coding branch 652, thus forming a complete data dependency chain.

[0186] S660, determine the type, level, and confidence level of the violation.

[0187] In this embodiment, violation category, violation level, and confidence level can be determined by the violation classification unit 660.

[0188] The comprehensive event representation vector is input into the classification layer and the regression layer, and the violation category, violation level and violation confidence of the candidate city event are output.

[0189] Violations include illegal parking of motor vehicles, disorderly parking of non-motor vehicles, street vending, business operations outside designated areas, illegal dumping of garbage, illegal construction site fencing, obstruction of fire lanes, obstruction of tactile paving, obstruction of bus stops, and illegal passage of dump trucks.

[0190] The level of violation can be determined based on the degree of space occupancy, duration, scope of impact, road grade, pedestrian and vehicle flow status, rule level, and temporal stability.

[0191] For example, for the same illegal parking incident, if it occurs on the edge of an ordinary road and lasts for a short period of time, it can be judged as a low-level suspected violation; if it occurs in a fire lane, bus stop, school gate, or hospital emergency lane, and lasts for a long period of time with a high temporal stability score, it can be judged as a high-level violation.

[0192] The violation confidence score is obtained by fusing together the visual recognition confidence score, spatial semantic mapping confidence score, temporal behavior confidence score, rule matching confidence score, model classification confidence score, and exception state confidence score.

[0193] S670, Perform final exception rule review and conflict resolution.

[0194] In this embodiment, the final exception rule review and conflict resolution can be performed by the exception rule review and conflict resolution unit 670.

[0195] Before outputting the final judgment result, the candidate city events undergo a final exception rule review and conflict resolution.

[0196] It should be noted that the rapid exception screening module 260 is used for early rapid screening of obvious exceptions to reduce subsequent computational overhead; the exception rule review and conflict resolution unit 670 is used for final review of candidate city events that were not identified as exceptions in stage S250 but have rule conflicts or exception evidence in the subsequent spatiotemporal semantic discrimination process. Therefore, the rapid exception screening module 260 and the exception rule review and conflict resolution unit 670 do not overlap, and they respectively undertake the functions of early filtering and final review.

[0197] The final exception rule review includes: determining whether the target vehicle is a special vehicle such as an ambulance, fire truck, police car, law enforcement vehicle, or emergency support vehicle; determining whether there are special permits or registration information such as temporary construction permits, outdoor operation permits, temporary loading and unloading permits, or activity registration information in the area; determining whether the current time is a legal time period that allows temporary operation, temporary parking, or temporary loading and unloading; determining whether the target behavior is a passive behavior such as short-term passage, temporary avoidance, or passive stay caused by traffic congestion; and determining whether the layer version number of the dynamic state semantic layer 415 is consistent with the rule version number in the rule base construction unit 610.

[0198] If a candidate city event meets the final exception rule review criteria, the violation confidence level will be lowered, the violation grade will be adjusted, or the event status will be changed from "violation established" to "requires manual review" or "does not constitute a violation." Meeting the final exception rule review criteria for a candidate city event includes: the target vehicle being a special vehicle; the target object's location having special permit registration information; whether the current time is within a legal time period; whether the target behavior is a passive behavior; or the layer version number of the dynamic state semantic layer being inconsistent with the rule version number, etc.

[0199] If there are conflicts among multiple rules, the conflicts will be resolved according to rule priority, spatial priority, time priority, security priority, and version consistency priority.

[0200] For example, when an area is simultaneously a permitted area for outdoor commercial seating and a protected area for fire lanes, fire safety rules take precedence over outdoor seating permit rules. Therefore, if the target object occupies the fire lane, it will still be considered a violation.

[0201] For example, when the dynamic state semantic layer 415 shows that a commercial street belongs to the nighttime outdoor vending permit area, but the corresponding rule in the rule base has expired or the version number is inconsistent, the system does not directly apply the dynamic state semantic layer 415, but adjusts the event status to require manual review and triggers the rule layer consistency check.

[0202] S680, Generate interpretable criteria for violation detection.

[0203] In this embodiment, interpretable violation judgment criteria can be generated by the interpretable criteria generation unit 680.

[0204] Based on the output of the violation event discrimination model, an interpretable basis for violation discrimination is generated.

[0205] The criteria for violation determination include the main rules that are hit, the category of the target object, the state of the target behavior, the semantic region of the target space, the proportion of the target occupancy, the duration of the event, the time-series dynamic score, whether the evidence conditions are met, whether there are exception rules, the dynamic layer version number, the rule version number, the final violation confidence, and the key features that contribute significantly to the determination model.

[0206] In one implementation, the interpretable basis generation unit 680 can generate readable discrimination descriptions based on attention weights, rule matching results, keyframe states, and temporal dynamic scores.

[0207] For example, the criteria could be: the target vehicle remained stationary for 185 seconds within the no-stopping semantic area of ​​the bus stop, with an overlap of 63% with the bus stop area, an event persistence score of 0.94, a behavior stability score of 0.91, the current time period was not within the permitted temporary stopping period, and no valid emergency vehicle or law enforcement vehicle registration information was identified. This meets the criteria for establishing a bus stop occupancy violation event and is therefore judged as a suspected illegal parking incident with a confidence level of 91%.

[0208] S700 generates a chain of evidence for the violation and outputs the judgment result.

[0209] In this embodiment, the evidence chain generation module 700 can generate a violation event evidence chain and output the judgment result.

[0210] When a candidate city event is determined to be a violation event, a corresponding violation event evidence chain is generated, and the judgment result is output.

[0211] The evidence chain for a violation includes the event number, event type, event time, event location, target object category, target object trajectory, keyframe image, spatial semantic region, city management rules, rule version number, dynamic layer version number, duration, occupancy ratio, time-series dynamic score, violation confidence level, cause of identification, and handling recommendations.

[0212] In one implementation, the evidence chain generation module 700 overlays keyframe images, target object detection boxes, target object segmentation regions, spatial semantic boundaries, target object trajectories, and rule matching results to generate a visual evidence map.

[0213] Structured violation logs may include the following: "On a certain day, month, year, hour, and minute, target vehicle A remained stationary for 185 seconds in a bus stop area on a certain road. After spatial mapping of the target location, it fell into the no-stopping semantic area of ​​the bus stop. The spatial semantic mapping confidence score was 0.91, the event persistence score was 0.94, the current time did not fall within the permitted temporary stopping period, and no valid emergency vehicle or law enforcement vehicle registration information was matched. According to the urban road parking management rules, it was determined to be a suspected illegal parking event at a bus stop, with a violation confidence score of 91%." The judgment results can be output to the city operation management platform, the city governance event dispatch system, the law enforcement terminal, the manual review platform, or the data statistical analysis system.

[0214] This embodiment forms a complete processing chain through data acquisition module 100, visual recognition module 200, rapid exception screening module 260, spatial mapping module 300, semantic mapping module 400, context generation module 500, violation judgment module 600, and evidence chain generation module 700. This transforms the judgment of urban violation events from simple target identification to joint judgment based on spatiotemporal semantic rules, thereby improving the accuracy of urban violation event identification, system operating efficiency, and management and handling efficiency.

[0215] Example 2 Figure 4 This is a flowchart illustrating the synchronization process between the dynamic layer and the rule base provided in Embodiment 2 of the present invention. This embodiment is used to explain the data synchronization mechanism between the dynamic state semantic layer 415 and the city violation event rule base. This synchronization process can be executed by the rule layer synchronization module 800.

[0216] like Figure 4 As shown, the process of synchronizing dynamic layers with the rule base includes the following steps.

[0217] S810. Obtain rule data from the city's violation event rule base.

[0218] In this embodiment, rule data in the city violation event rule base can be obtained through the rule data acquisition unit 810.

[0219] Retrieves regional rules, time rules, exception rules, and layer generation rules from the city's violation event rule base. Each rule includes a rule identifier, rule type, applicable area, applicable object, applicable time, effective status, priority, and rule version number.

[0220] S820. Generate a dynamic state semantic layer based on the rule data.

[0221] In this embodiment, a dynamic state semantic layer 415 can be generated by the dynamic layer generation unit 820 according to the rule data.

[0222] Based on the rule identifier, applicable area, applicable time, and effective status, the rule data is converted into a spatially computable dynamic state semantic layer 415.

[0223] For example, the rule "outdoor business is allowed in a certain commercial district from 7 pm to 11 pm every day" is converted into a nighttime temporary business permit layer; the rule "temporary parking is prohibited in a certain section of road during rain or snow" is converted into a rain or snow weather traffic restriction layer.

[0224] S830. Establish the mapping relationship between rule version number and layer version number.

[0225] In this embodiment, the mapping relationship between rule version number and layer version number can be established through version mapping unit 830.

[0226] Configure a layer version number for each dynamic state semantic layer 415, and bind the layer version number to the corresponding rule version number.

[0227] If the rules are updated, the dynamic state semantic layer 415 will be regenerated according to the new rule version number; if the dynamic layer update fails, the original layer will remain in an unavailable state and an exception record will be generated.

[0228] S840, Perform rule layer consistency check.

[0229] In this embodiment, the rule layer consistency verification can be performed by the rule layer consistency verification unit 840.

[0230] Before a candidate city event enters the violation judgment process, verify whether the layer version number of the dynamic state semantic layer 415 to which the event is matched is consistent with the corresponding rule version number.

[0231] If the two are consistent, the dynamic state semantic layer 415 is allowed to participate in the visual semantic mapping and violation judgment process.

[0232] If the two are inconsistent, the dynamic state semantic layer 415 is prohibited from directly participating in the final judgment process, and the corresponding event is marked as requiring manual review.

[0233] In one embodiment, the rule layer consistency verification unit 840 may include a version consistency judgment unit 841, a dynamic layer disabling unit 842, and a manual review marking unit 843.

[0234] Version consistency judgment unit 841 is used to determine whether the rule version number and the layer version number are consistent.

[0235] The dynamic layer disable unit 842 is used to prevent the dynamic state semantic layer 415 from continuing to participate in the final violation judgment when the rule version number and the layer version number are inconsistent.

[0236] The manual review marking unit 843 is used to mark relevant candidate city events as requiring manual review.

[0237] Through this embodiment, the dynamic state semantic layer 415 can become a projection of urban management rules in the spatial dimension, avoiding logical conflicts caused by maintaining the dynamic layer and the rule base separately.

[0238] Example 3 Figure 5 This is a schematic diagram of the urban violation incident discrimination device based on spatiotemporal environment perception and visual semantic mapping provided in Embodiment 3 of the present invention. This embodiment can be applied to perform the methods described in any of the above embodiments.

[0239] like Figure 5 As shown, the device includes a data acquisition module 100, a visual recognition module 200, a rapid exception screening module 260, a spatial mapping module 300, a semantic mapping module 400, a context generation module 500, a violation judgment module 600, an evidence chain generation module 700, and a rule layer synchronization module 800.

[0240] The data acquisition module 100 is used to acquire multi-source sensing data of the urban monitoring area. The multi-source sensing data includes video image data, camera equipment parameters, shooting time information, urban spatial data, environmental status data, permit and filing data, and urban management rule data.

[0241] The visual recognition module 200 is used to perform visual target recognition, identity-assisted recognition, and behavioral feature extraction on video image data to obtain the visual recognition results of the target object.

[0242] The visual recognition module 200 may include a target detection sub-model 210, an instance segmentation sub-model 220, a target tracking sub-model 230, a behavior recognition sub-model 240, and an identity recognition sub-model 250. The target detection sub-model 210 is used to determine the category of the target object and the location of the detection box in the video image; the instance segmentation sub-model 220 is used to determine the actual occupied area of ​​the target object; the target tracking sub-model 230 is used to generate the trajectory of the target object; the behavior recognition sub-model 240 is used to determine the behavioral state of the target object; and the identity recognition sub-model 250 is used to identify whether the target object belongs to an emergency response target, a law enforcement target, a registered target, or a licensed target.

[0243] The rapid exception screening module 260 is used to perform rapid exception screening on target objects based on the target object's identity auxiliary information and license filing data, and generate exception records for target objects that should obviously not enter the violation judgment process.

[0244] The spatial mapping module 300 is used to establish a mapping relationship between video image coordinates and urban spatial coordinates based on camera equipment parameters and urban spatial data, and to map the target object to the urban spatial coordinates.

[0245] The semantic mapping module 400 is used to construct a city spatial semantic layer and generate visual semantic mapping results based on the visual semantic features of the target object, the spatial semantic features of the spatial unit, and the spatial relationship between the target object and the spatial unit.

[0246] The semantic mapping module 400 may include a city space semantic layer construction unit 410, a target object visual semantic encoding unit 420, a spatial unit semantic encoding unit 430, a relationship graph construction unit 440, and a spatial relationship graph network reasoning unit 450.

[0247] The urban space semantic layer construction unit 410 is used to construct the road traffic semantic layer 411, the public space semantic layer 412, the urban component semantic layer 413, the management rule semantic layer 414, and the dynamic state semantic layer 415.

[0248] The spatial relation graph network reasoning unit 450 may further include a node initialization layer 451, an edge attribute encoding layer 452, an attention calculation layer 453, a neighborhood aggregation layer 454, and a semantic output layer 455.

[0249] The context generation module 500 is used to generate event context features of candidate city events based on visual semantic mapping results, target object behavior status, shooting time information, environmental status data, permit filing data, and urban management rule data.

[0250] The violation identification module 600 is used to input event context features into a rule-enhanced spatiotemporal semantic violation identification model to determine the violation category, violation level, and violation confidence of candidate city events.

[0251] The violation detection module 600 may include a rule base construction unit 610, a spatiotemporal state sequence construction unit 620, a temporal behavior analysis unit 630, a rule semantic encoding unit 640, a feature fusion unit 650, a violation classification unit 660, an exception rule review and conflict resolution unit 670, and an interpretable basis generation unit 680.

[0252] The feature fusion unit 650 includes a visual behavior coding branch 651, a spatial semantic coding branch 652, a temporal state coding branch 653, an environmental state coding branch 654, a rule semantic coding branch 655, a historical event coding branch 656, and a temporal dynamic coding branch 657. The temporal dynamic coding branch 657 is used to receive the event persistence score, behavior stability score, trajectory continuity score, and temporal confidence score output by the temporal behavior analysis unit 630.

[0253] The rule layer synchronization module 800 is used to generate a dynamic state semantic layer 415 based on the city violation event rule library, and to perform a consistency check between the rule version number and the layer version number.

[0254] The rule layer synchronization module 800 may include a rule data acquisition unit 810, a dynamic layer generation unit 820, a version mapping unit 830, a rule layer consistency verification unit 840, a version consistency judgment unit 841, a dynamic layer disabling unit 842, and a manual review marking unit 843.

[0255] The evidence chain generation module 700 is used to generate evidence chains for violation events and output the judgment results based on the violation category, violation level and violation confidence level.

[0256] The urban violation event discrimination device based on spatiotemporal environment perception and visual semantic mapping provided in this embodiment can execute the urban violation event discrimination method based on spatiotemporal environment perception and visual semantic mapping provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0257] Example 4 Figure 6 This is a schematic diagram of the electronic device provided in Embodiment 4 of the present invention. The electronic device can be a server, an edge computing device, a video analytics terminal, a city management platform device, or a cloud computing device.

[0258] Electronic devices may include one or more processors, storage devices, communication devices, input / output devices, and buses. The storage devices are used to store one or more programs. When one or more programs are executed by one or more processors, the one or more processors implement the urban violation event discrimination method based on spatiotemporal environment perception and visual semantic mapping as described in any embodiment of the present invention.

[0259] The processor can be used to perform computational tasks such as object detection, instance segmentation, object tracking, identity-assisted recognition, behavior recognition, rapid exception screening, spatial mapping, spatial relationship graph network reasoning, rule matching, temporal behavior analysis, multi-branch attention fusion, violation event classification, rule layer synchronization, and evidence chain generation.

[0260] The communication device can be used to communicate with video surveillance equipment, urban spatial databases, rule databases, permit filing databases, urban management platforms, law enforcement terminals, and manual review platforms.

[0261] The storage device can be used to store video image data, camera equipment parameters, spatial semantic layers, urban management rule base, permit filing data, target object trajectory, candidate city events, violation judgment results, and violation event evidence chain.

[0262] The input / output device can be used to receive rule update instructions, manual review results, and handling feedback information input by managers, and output evidence chains, visual evidence diagrams, and structured violation records.

[0263] Example 5 This invention also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the program implements the urban violation event discrimination method based on spatiotemporal environment perception and visual semantic mapping described in any embodiment of this invention.

[0264] Computer-readable storage media may include, but are not limited to, hard disks, solid-state drives, flash memory, read-only memory, random access memory, magnetic disks, optical disks, cloud storage media, or other media capable of storing program code.

[0265] The computer-readable storage medium provided in this embodiment of the invention enables electronic devices to automatically identify urban violations, quickly screen exceptions, perform visual semantic mapping, enhance rule discrimination, synchronize rule layers, and output evidence chains by storing computer programs for executing the above methods.

Claims

1. A method for identifying violations based on spatiotemporal environment perception and visual semantic mapping, characterized in that, The method includes: Acquire multi-source sensing data of the urban monitoring area, including video image data, camera equipment parameters, shooting time information, urban spatial data, environmental status data, permit and filing data, and urban management rule data; Visual target recognition and behavioral feature extraction are performed on the video image data to obtain the visual recognition result of the target object. The visual recognition result includes at least the target object category, target object location, target object occupied area, target object trajectory, target object behavioral state, and target object identity auxiliary information. Based on the target object's identity auxiliary information and the license filing data, determine whether the target object belongs to the exception object; If the target object is not an exception object, then a mapping relationship between the video image coordinates and the urban spatial coordinate system is established based on the camera equipment parameters and the urban spatial data, and the target object is mapped to the urban spatial coordinate system based on the mapping relationship; A city spatial semantic layer is constructed, and a visual semantic mapping result is generated based on the visual semantic feature vector of the target object, the spatial semantic feature vector of the spatial unit, and the edge attribute vector between the target object and the spatial unit. By integrating the visual semantic mapping results, the target object's behavioral state, the shooting time information, the environmental state data, the permit and filing data, and the urban management rule data, event context features of candidate urban events are generated. The event context features are input into a rule-enhanced spatiotemporal semantic violation discrimination model to determine the violation category, violation level, and violation confidence of the candidate city events. Based on the violation category, the violation level, and the violation confidence level, a violation event evidence chain is generated and a judgment result is output.

2. The method according to claim 1, characterized in that, The step of performing visual target recognition and behavioral feature extraction on the video image data includes: The video image data is subjected to image enhancement, distortion correction, brightness normalization, and noise suppression processing. The object detection sub-model is used to determine the category and location of target objects in video images. The target object's occupied area in the video image is determined by an instance segmentation sub-model; The target tracking sub-model assigns a target identifier to the same target object in consecutive video frames and generates the trajectory of the target object. The behavior state of the target object is determined by the behavior recognition sub-model based on the state of the target object in consecutive video frames; The identity auxiliary information of the target object is determined through the identity recognition sub-model.

3. The method according to claim 1, characterized in that, The step of determining whether the target object belongs to the exception object based on the target object's identity auxiliary information and the license filing data includes: If the target object is identified as a vehicle with emergency support or law enforcement attributes, and the vehicle license plate, vehicle exterior markings, or registration number of the target object matches the permit registration data, then the target object is determined to be an exception object. If there are valid temporary construction permits, outdoor business permits, temporary loading and unloading permits, or major event registration information in the area where the target object is located, and the time, location, and object type of the target object meet the corresponding permit conditions, then the target object will be identified as an exception object. Otherwise, the target object is determined to be an exception object.

4. The method according to claim 1, characterized in that, The step of generating a visual semantic mapping result based on the visual semantic feature vector of the target object, the spatial semantic feature vector of the spatial unit, and the edge attribute vector between the target object and the spatial unit includes: The visual recognition results of the target object are used to encode features to generate a visual semantic feature vector of the target object; The urban monitoring area is divided into multiple spatial units, and spatial semantic features are extracted for each spatial unit to generate a spatial semantic feature vector for the spatial unit. Based on the location, trajectory, and occupied range of the target object in the urban spatial coordinates, an edge attribute vector relationship graph between the target object and the spatial unit is constructed, wherein the target object is used as an object node, the spatial unit is used as a spatial node, and the spatial relationship between the target object and the spatial unit is used as an edge; The edge attributes include at least one of the following: the overlap ratio between the target object and the spatial unit; whether the center point of the target object is located within the spatial unit; the distance between the boundary of the target object and the boundary of the spatial unit; whether the trajectory of the target object enters or crosses the spatial unit; whether the target object stays within the spatial unit; and the occupancy relationship of the target object on urban components within the spatial unit. The edge attribute vector relationship graph between the target object and the spatial unit is input into the spatial relationship graph network, and the visual semantic mapping result is output.

5. The method according to claim 4, characterized in that, The spatial relationship graph network employs a graph attention network. The step of inputting the edge attribute vector relationship graph of the target object and spatial units into a spatial relationship graph network and outputting the visual semantic mapping result includes: The visual semantic feature vectors of the object nodes, the spatial semantic feature vectors of the spatial nodes, and the edge attribute vectors of the edges are encoded using a unified dimension. Calculate the attention weights between the target object node and its adjacent spatial nodes based on the object node, the spatial node, and the edge attribute vector; Based on the attention weights, the information of adjacent spatial nodes and edge attribute information are aggregated to obtain the spatial semantic fusion representation of the target object node; Based on the spatial semantic fusion representation, the visual semantic mapping result corresponding to the target object is output. The visual semantic mapping result includes the main spatial semantics, auxiliary spatial semantics, spatial occupancy ratio, spatial relationship confidence, and potential violation prior labels corresponding to the target object. The potential violation prior label is used to indicate the possibility of a spatial semantic association between the target object and the city's violation rules.

6. The method according to claim 1, characterized in that, The method of inputting the event context features into a rule-enhanced spatiotemporal semantic violation discrimination model to determine the violation category, violation level, and violation confidence of the candidate city events includes: Establish a rule base for urban violation incidents, which includes basic rules, regional rules, time rules, exception rules, evidence rules, and layer generation rules; For the candidate city events, a spatiotemporal state sequence is constructed in chronological order; the spatiotemporal state sequence includes multiple time slices, each time slice corresponding to the state of the candidate city event at a corresponding moment; The spatiotemporal state sequence is input into the temporal behavior model, and the output is a temporal dynamic feature vector. The temporal dynamic feature vector includes event persistence score, behavior stability score, trajectory continuity score, temporal confidence, stage change state and abnormal interruption state. The event context features of the candidate city events are matched with the city violation event rule base, and the rule matching results are encoded to generate rule semantic vectors. Construct a spatiotemporal semantic violation discrimination model based on attention fusion, and encode and output visual behavior encoding vector, spatial semantic encoding vector, temporal state encoding vector, environmental state encoding vector, rule semantic encoding vector, historical event encoding vector, and temporal dynamic encoding vector; The visual behavior encoding vector, spatial semantic encoding vector, temporal state encoding vector, environmental state encoding vector, rule semantic encoding vector, historical event encoding vector, and temporal dynamic encoding vector are input into the attention fusion model to obtain a comprehensive event representation vector. Based on the comprehensive event representation vector, the violation category, violation level, and violation confidence of the candidate city event are output.

7. The method according to claim 6, characterized in that, The process involves inputting visual behavior encoding vectors, spatial semantic encoding vectors, temporal state encoding vectors, environmental state encoding vectors, rule semantic encoding vectors, historical event encoding vectors, and temporal dynamic encoding vectors into the attention fusion model to obtain a comprehensive event representation vector, including: The outputs of visual behavior encoding vector, spatial semantic encoding vector, temporal state encoding vector, environmental state encoding vector, rule semantic encoding vector, historical event encoding vector, and temporal dynamic encoding vector are unified to the same dimension through linear mapping to obtain multiple branch vectors; The multiple branch vectors are concatenated row by row to form a feature matrix; Generate a query matrix, a key matrix, and a value matrix based on the feature matrix; The attention weights between different coding branches are calculated based on the query matrix and the key matrix, and the value matrix is ​​weighted and fused based on the attention weights to obtain the attention fusion result. The attention fusion result is flattened or pooled to obtain the comprehensive event representation vector.

8. The method according to claim 1, characterized in that, After determining the violation category, violation level, and violation confidence level of the candidate city events, the method further includes: The candidate city events undergo final exception rule review and conflict resolution; The final exception rule review includes determining whether the target vehicle is a special vehicle, whether there is special permit registration information in the area where the target object is located, whether the current time is a legal time period, whether the target behavior is a passive behavior, and whether the layer version number of the dynamic state semantic layer is consistent with the rule version number. If the candidate city event meets the final exception rule review, then the violation confidence level is reduced, the violation level is adjusted, or the status of the candidate city event is adjusted to require manual review or does not constitute a violation; If there are conflicts among multiple city management rules, the conflicts will be resolved according to the priority of rules, spatial priority, time priority, security priority, and version consistency priority.

9. The method according to claim 1, characterized in that, The method further includes: Obtain rule data from the city's violation event rule base, including regional rules, time rules, exception rules, and layer generation rules; Based on the rule identifier, applicable area, applicable time, and effective status, the rule data is converted into a spatially computable dynamic state semantic layer. Configure a layer version number for each dynamic state semantic layer, and bind the layer version number to the corresponding rule version number; Before generating the evidence chain of the violation event and outputting the judgment result based on the violation category, the violation level, and the violation confidence level, the method further includes: Verify whether the layer version number of the dynamic state semantic layer to which the candidate city event is hit is consistent with the corresponding rule version number; If they match, the dynamic state semantic layer is allowed to participate in the discrimination process; If there is a discrepancy, the dynamic state semantic layer is prohibited from directly participating in the discrimination process, and the corresponding candidate city event is marked as requiring manual review.

10. A violation event detection device based on spatiotemporal environment perception and visual semantic mapping, characterized in that, The device includes: The data acquisition module is used to acquire multi-source sensing data of the urban monitoring area; the multi-source sensing data includes video image data, camera equipment parameters, shooting time information, urban spatial data, environmental status data, permit and filing data, and urban management rule data; The visual recognition module is used to perform visual target recognition and behavioral feature extraction on the video image data to obtain the visual recognition result of the target object. The visual recognition result includes at least the target object category, target object location, target object occupied area, target object trajectory, target object behavioral state and target object identity auxiliary information. The rapid exception screening module is used to determine whether the target object belongs to the exception object based on the target object's identity auxiliary information and the license filing data; The spatial mapping module is used to establish a mapping relationship between video image coordinates and urban spatial coordinate system based on the camera device parameters and urban spatial data if the target object is not an exception object, and to map the target object into the urban spatial coordinate system based on the mapping relationship. The semantic mapping module is used to generate a visual semantic mapping result based on the visual semantic feature vector of the target object, the spatial semantic feature vector of the spatial unit, and the edge attribute vector between the target object and the spatial unit. The context generation module is used to fuse the visual semantic mapping results, the target object's behavioral state, the shooting time information, the environmental state data, the permit and filing data, and the urban management rule data to generate event context features of candidate city events; The violation detection module is used to input the event context features into a rule-enhanced spatiotemporal semantic violation detection model to determine the violation category, violation level, and violation confidence of the candidate city event; The evidence chain generation module is used to generate an evidence chain for the violation event and output the judgment result based on the violation category, the violation level and the violation confidence level.

Images