Method, system, and medium for scene prior-driven object detection and event recognition
Patent Information
- Application Number
- CN202611554424.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-09
- Publication Date
- 2026-11-03
AI Technical Summary
Existing technologies in video analytics for the transportation industry suffer from problems such as fragmented labeling systems, numerous model versions, tight coupling between target detection and event recognition modules, and poor cross-scenario generalization capabilities, resulting in high system maintenance costs and low data utilization efficiency.
We adopt a scene-prior driven object detection and event recognition method. By constructing a unified task description vector to drive the perception network, and combining multi-scale feature extraction, cross-frame correlation tracking and decision functions, we can achieve cross-scene adaptation and unified object detection.
It reduces model deployment and operation costs, improves the consistency of target determination and alarm accuracy, reduces false detection rate, and enhances cross-scenario adaptability.
Smart Images
Figure CN123033209A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of surveillance, and specifically relates to a method, system and medium for scene-prior driven target detection and event recognition. Background Technology
[0002] Video analytics in the transportation industry typically covers various scenarios such as roads, railways, waterways, ports, and stations. Business objectives include target detection tasks (such as vehicle detection, personnel detection, construction machinery detection, ship detection, animal detection, foreign object detection, road surface defect detection, and water accumulation detection), text recognition tasks (such as license plate recognition, ship name identification, and hazardous materials identification), attribute recognition tasks (such as life jacket wearing status recognition, hazardous chemical identification, and day / night status recognition), and event recognition tasks based on regions, tripwires, direction lines, and time windows (such as intrusion, loitering, congestion, traffic flow statistics, road construction, and parking quantity statistics).
[0003] In existing engineering practice, the above tasks are typically achieved using the following three technical solutions: (I) Single-task target detection scheme Independent detection models are trained for vehicles, personnel, construction machinery, ships, animals, and foreign objects, with each model responsible for detecting one or a few target categories. These models are deployed according to task breakdowns. When business expands to new object categories, labels need to be redefined, data re-labeled, and new models trained separately.
[0004] (II) Separation of Detection and Rules A front-end detector outputs target bounding boxes and categories, while a back-end independent rule engine determines alerts based on region intrusion, tripwires, direction lines, and time thresholds. Data is exchanged between the detector and the rule engine via predefined interfaces (such as category names and coordinate formats). Rule configurations are tightly coupled with the labels output by the upstream detectors; when detection labels change, the rule scripts need to be modified synchronously.
[0005] (III) Identification of Task-Based Plug-in Solutions Tasks such as license plate recognition, ship name recognition, hazardous materials label recognition, and life vest attribute recognition are implemented using add-on modules. This involves secondary recognition of the image region cropped from the detection bounding box after the front-end detector detects the target. This approach is highly dependent on the quality of the upstream detection bounding box, and the recognition module, detection module, and event determination module operate independently, lacking a collaborative mechanism.
[0006] The aforementioned existing technical solutions have the following shortcomings in practical engineering applications: 1. Existing solutions are mostly built in the form of "one scenario, one model, one rule script". Different business tasks use their own independent label definitions (the same target may have different category IDs and naming conventions in different models), resulting in a fragmented label system, a large number of model versions, high computing power consumption at the edge, and high system maintenance costs.
[0007] 2. Similar targets cannot share feature representations and difficult samples in different business scenarios, making it difficult to fully utilize the common features of multi-scenario data in the transportation industry (including visible light, thermal imaging, day and night, port and shipping, road and railway data), resulting in a double loss of data utilization efficiency and model generalization ability.
[0008] 3. Target detection, attribute recognition, text recognition, and event determination are implemented by multiple independent modules connected in series. The interfaces between modules are tightly coupled. Adding new business requires modifying the detection model, recognition module, and rule script simultaneously, resulting in a long development cycle and complex configuration.
[0009] In summary, existing technologies for target recognition and analysis in various scenarios within the transportation industry suffer from problems such as fragmented labeling systems, numerous model versions, separation of target detection, recognition, and judgment links, and poor generalization ability in complex scenarios. There is an urgent need for a solution that can perform unified target detection and event recognition. Summary of the Invention
[0010] To address the aforementioned issues, this application provides a method, system, and medium for scene-prior driven target detection and event recognition. The model has low cost, strong cross-scene adaptability, and reduces the false detection rate while improving the consistency of target determination results.
[0011] The technical solution is as follows: A method for scene-prior driven object detection and event recognition includes: Acquire monitoring image data and monitoring scene configuration information, and construct a task description vector based on the target task. The task description vector shall at least include a target category set, an attribute set, spatial constraint parameters, time constraint parameters, and preset exclusion conditions. Multi-scale feature extraction is performed on the monitoring image data. Based on the extracted multi-scale feature map, a candidate region set is generated. The candidate region is a region in the monitoring image that may contain a target object. The targetability score of each candidate region represents the probability that the candidate region contains a target object. The task description vector is encoded to obtain a scene prior vector, and the visual feature vectors of each candidate region are extracted. The perceptual network calculates the association score based on the semantic similarity between the visual feature vector and the scene prior vector, the target score of the candidate region, and the spatial matching degree between the candidate region and the monitoring scene configuration information. When the association score exceeds a preset score threshold, the target detection result of the corresponding candidate region is output. Different task description vectors drive the perceptual network to output different target detection results. Cross-frame correlation tracking is performed on the target detection results of continuous image frames to obtain the target motion trajectory. An event reasoning is performed on the target motion trajectory using a decision function that fuses the spatial constraint parameters, the temporal constraint parameters and the exclusion conditions. An alarm is triggered based on the reasoning result.
[0012] A system for scene-prior driven target detection and event recognition includes: The data acquisition module is used to acquire monitoring image data and monitoring scene configuration information; The task description module is used to construct a task description vector based on the monitoring scenario configuration information and the target task. The task description vector includes at least a target category set, an attribute set, spatial constraint parameters, time constraint parameters, and preset exclusion conditions. The perception network module is used to perform multi-scale feature extraction on the monitoring image data, generate a candidate region set based on the extracted multi-scale feature maps, encode the task description vector to obtain a scene prior vector, extract the visual feature vector of each candidate region, and calculate the association score based on the semantic similarity between the visual feature vector and the scene prior vector, the target score of the candidate region, and the spatial matching degree between the candidate region and the monitoring scene configuration information. The target score of each candidate region represents the probability that the candidate region contains a target object. When the association score exceeds a preset score threshold, the target detection result of the corresponding candidate region is output. The trajectory tracking and alarm determination module is used to perform cross-frame correlation tracking on the target detection results of continuous image frames to obtain the target motion trajectory. It uses a determination function that integrates the spatial constraint parameters, the temporal constraint parameters and the exclusion conditions to perform event reasoning on the target motion trajectory, and determines whether to trigger an alarm based on the reasoning result.
[0013] Based on the same inventive concept, this application also provides a computer storage medium storing computer-executable instructions, which, when executed, implement the aforementioned method for scene prior-driven target detection and event recognition.
[0014] Compared with the prior art, this application has the following advantages: This solution constructs a unified task description vector-driven perception network, which can cover multiple scenarios across the entire transportation industry and has strong adaptability. It replaces the traditional deployment of multiple independent models with a single model, reducing model deployment and operation costs. The associated scoring mechanism expands the target judgment of the perception network from a single visual feature to a joint judgment based on visual features, task semantic priors, and scene spatial constraints, reducing the false detection rate. At the same time, the judgment function binds the event judgment logic to the task description vector, sharing the same constraint expression with the perception network, thereby improving the alarm accuracy.
[0015] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 The flowchart of the scene prior-driven target detection and event recognition method in this embodiment is shown; Figure 2 The diagram illustrates the principle of the scene prior-driven target detection and event recognition method in this embodiment. Figure 3 This embodiment illustrates the task description and unified tag space construction. Figure 4 The flowchart of online reasoning and event recognition in this embodiment is shown. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0019] See Figures 1-4 This application provides a scene prior-driven target detection and event recognition method, including: S1. Obtain monitoring image data and monitoring scene configuration information, and construct a task description vector according to the target task. The task description vector shall at least include a target category set, an attribute set, spatial constraint parameters, time constraint parameters and preset exclusion conditions. Taking the entire transportation industry as an example, the monitoring image data includes image frames, video streams, and thermal images from monitoring points such as roads, railways, waterways, and ports. The monitoring scene configuration information includes scene configurations such as monitoring point configuration, camera orientation, area frames, tripwires, direction lines, and time windows.
[0020] The task description vector is used to transform business requirements described in natural language into structured mathematical expressions.
[0021] By constructing a unified tag space for all industry scenarios, this unified tag space comprises a three-layer tag system: target layer, attribute layer, and event layer. It incorporates all target types, attribute types, and event types involved in all industry businesses into a unified tag definition framework. The target category set and attribute set in the task description vector reference the tags defined in this unified tag space.
[0022] S2. Perform multi-scale feature extraction on the monitoring image data, and generate a candidate region set based on the extracted multi-scale feature map. The candidate region is a region in the monitoring image that may contain a target object. The targetability score of each candidate region represents the probability that the candidate region contains a target object. The candidate regions are independent of the category of the target object.
[0023] S3. The task description vector is encoded to obtain the scene prior vector, and the visual feature vectors of each candidate region are extracted. The perceptual network calculates the association score based on the semantic similarity between the visual feature vector and the scene prior vector, the target score of the candidate region, and the spatial matching degree between the candidate region and the monitoring scene configuration information. When the association score exceeds the preset score threshold, the target detection result of the corresponding candidate region is output. Different task description vectors drive the perception network to output different target detection results.
[0024] S4. Perform cross-frame correlation tracking on the target detection results of continuous image frames to obtain the target motion trajectory. Use a decision function that integrates the spatial constraint parameters, the temporal constraint parameters and the exclusion conditions to perform event reasoning on the target motion trajectory, and determine whether to trigger an alarm based on the reasoning result.
[0025] By establishing a correspondence between the same target object in the time dimension through cross-frame correlation tracking, discrete detection results are organized into continuous motion trajectories.
[0026] In this embodiment, by constructing a unified task description vector-driven perception network, it can cover multiple scenarios across the entire transportation industry, has strong adaptability, and replaces the traditional deployment of multiple independent models with a single model, reducing model deployment and operating costs. The associated scoring mechanism expands the target judgment of the perception network from a single visual feature to a joint judgment based on visual features, task semantic priors, and scene spatial constraints, reducing the false detection rate. At the same time, the judgment function binds the event judgment logic to the task description vector, sharing the same constraint expression with the perception network, thereby improving the alarm accuracy.
[0027] In this embodiment, the expression for the task description vector is:
[0028] The definitions of each dimension vector are as follows: (1) : The target category set, which includes all object types involved in the target task.
[0029] This is used to define the types of target objects that the current task needs to detect. For example, target categories in the entire transportation industry include, but are not limited to: people, motor vehicles, non-motor vehicles, construction machinery, ships, animals, foreign objects, road surface defects, and water accumulation.
[0030] For different business tasks The target categories included can be one or more. For traffic congestion detection tasks, The possible value is "a set of vehicles," which can include motor vehicle categories (cars, buses, trucks, etc.), and can also exclude non-motor vehicles based on business needs; for personnel intrusion tasks, The value is "personnel"; for ship traffic statistics tasks, The value is "ship".
[0031] (2) : A set of attributes containing additional features of the target object to be detected.
[0032] This is used to define the status attributes and recognition requirements of the target object that the current task needs to determine. Common attributes in the transportation industry include, but are not limited to: hazardous chemical label status (present / absent), life jacket wearing status (worn / not worn), day / night status (daytime / nighttime), thermal imaging status (thermal imaging / visible light), and text recognition requirement signs (OCR required / OCR not required), etc.
[0033] The attribute set determines which recognition heads are activated in the unified perception network. Specifically: when When the data includes attribute constraints such as life jacket wearing status and hazardous chemical labeling status, the attribute recognition head of the perception network is triggered, and the attribute recognition result is output synchronously in the target detection result; when When the text recognition requirement flag is included, the text recognition trigger flag of the perceptual network is set to "True", which wakes up the downstream text recognition network to perform text sequence recognition on the corresponding candidate region and output text content such as license plate number and ship name.
[0034] For example, regarding inspection tasks where personnel are not wearing life jackets, The system defines "life vest wearing status = not wearing" and the unified perception network outputs the attribute label "not wearing a life vest" along with the personnel detection box. For the task of capturing license plates of hazardous materials vehicles, The system defines "Hazardous Chemical Label Status = Yes" and "Text Recognition Requirement = True". When the network detects a vehicle carrying a hazardous chemical label, it simultaneously triggers the text recognition head to perform license plate OCR recognition on the vehicle area.
[0035] (3) Spatial constraint parameters are used to limit the spatial determination rules involved in the current task, including but not limited to alarm areas, tripwires, and direction lines determined according to the monitoring scene configuration information.
[0036] The alarm area is a polygonal region defined by a set of latitude and longitude coordinates or pixel coordinates, used to limit the area within which a target object may enter or remain. For example, in a vehicle intrusion detection task, The coordinates of a polygon containing a restricted area are included, and an alarm is triggered when the target's trajectory enters this area.
[0037] A tripwire is a line segment defined by the coordinates of its two endpoints, used to define the linear boundary that a target object can cross. For example, in ship traffic statistics tasks... It includes a virtual tripwire, which is counted when a ship's trajectory crosses the tripwire.
[0038] A direction line is a vector with directional attributes defined by the coordinates of the starting and ending points, used to define the preset direction of the target object's movement. For example, in traffic flow statistics tasks... It contains a direction line vector, and the traffic statistics task is only triggered when the target object crosses the tripwire in the preset direction indicated by the direction line; it is not triggered when crossing in the opposite direction.
[0039] (4) The time constraint parameter is used to limit the timing rules involved in the current event determination, including the following three sub-parameters determined based on the monitoring scenario configuration information: Time window T: Defines the time interval for event determination, such as determining the vehicle's dwell time in the area within the past 10 seconds; Duration threshold Defines the maximum allowed dwell time of a target object within a target area. For example, in traffic congestion detection, a vehicle is required to remain within the target area for more than 5 seconds before being counted as a congested vehicle.
[0040] Quantity threshold This defines an upper limit on the cumulative number of vehicles that meet the duration condition. For example, in traffic congestion detection, a congestion alarm is triggered when the number of vehicles meeting the duration condition reaches more than 3.
[0041] In engineering implementation, the above parameters can be flexibly configured according to business scenarios. For example, urban road congestion detection can use a shorter time window and a lower vehicle number threshold (e.g., ≥3 vehicles that have been continuously stopped for more than 5 seconds within 10 seconds), while highway congestion detection can use a longer duration threshold and a higher number threshold (e.g., ≥10 vehicles that have been continuously stopped for more than 15 seconds within 30 seconds) to adapt to the traffic density and traffic speed characteristics of different scenarios.
[0042] (5) Exclusion criteria are used to filter target categories or attributes that do not need to participate in the current event determination, thus avoiding false alarms.
[0043] For example, in the task of detecting motor vehicle congestion. It can be set to "exclude non-motorized vehicles," so that non-motorized vehicles (bicycles, electric bicycles, etc.) will not be counted in the congestion vehicle count even if they remain in the target area; in road construction inspection tasks, It can be set to "exclude normally moving vehicles," only inspecting stationary or low-speed construction machinery and warning signs; in cases where personnel intrude during inspection, It can be set to "exclude security personnel and authorized personnel", triggering alarms only for unauthorized personnel.
[0044] The specific implementation of the exclusion condition is as follows: The unified decision function verifies the target category and attributes corresponding to the target's motion trajectory; if it belongs to... If the range is defined in the code, the judgment step will be skipped directly, and no event alarm will be triggered.
[0045] Suppose the business requirement of a city road monitoring point is: to detect whether traffic congestion has occurred in area A. The judgment rule is that if the number of vehicles that stay in area A for more than 5 seconds within any 10-second time window reaches 3 or more, non-motorized vehicles are excluded.
[0046] Based on the above requirements, the following is an example of a task description vector: ={Cars, buses, trucks} ={} (Congestion detection does not require attribute recognition; the attribute set is empty) ={Pixel coordinates or latitude / longitude coordinate sequence of the target area} ={Time window T=10 seconds, duration threshold} =5 seconds, quantity threshold =3 vehicles} ={Non-motorized vehicles} The task description vector is encoded as a scene prior vector and input into the perceptual network. When performing target detection on the surveillance image, the perceptual network only detects objects belonging to the scene. The association score is calculated for candidate regions of the defined target categories (cars, buses, trucks), and the target's motion trajectory is tracked.
[0047] In the unified perception network, multi-scale feature extraction is performed on the monitoring image data. Multi-scale feature extraction refers to using the backbone network to perform multi-level and multi-resolution feature abstraction on the input image to generate a set of feature maps containing semantic information at different scales.
[0048] The backbone network can adopt a convolutional neural network (CNN) architecture (such as ResNet, DarkNet, RegNet, etc.) or a Transformer architecture (such as ViT, Swin Transformer, etc.). Specifically, for the input image I, the backbone network progressively downsamples through stacked convolutional layers or attention layers, outputting multiple sets of feature maps at different resolution levels. Taking a convolutional neural network as an example, its typical structure includes multiple convolutional stages. The resolution of the feature map output by each stage decreases progressively, while the number of channels increases progressively. Each position (pixel) on the feature map corresponds to a receptive field region on the original image. The lower the resolution of the feature map, the larger the receptive field, which is suitable for detecting large targets; the higher the resolution of the feature map, the smaller the receptive field, which is suitable for detecting small targets. Since traffic monitoring scenarios contain both large targets (such as vehicles and ships at close range) and small targets at a distance (such as pedestrians and animals at a distance), a single-scale feature map is difficult to balance the detection accuracy of both. By fusing feature maps of multiple scales, both large and small targets can be effectively represented on feature layers of appropriate scales, thereby improving the target recall rate across the entire scene.
[0049] According to this embodiment, a set of candidate regions, independent of the target category, is generated based on the extracted multi-scale feature maps. The candidate regions refer to the set of bounding boxes in the surveillance image that may contain the target object. Each candidate region is represented by a four-dimensional vector: (x, y, w, h), where (x, y) are the center coordinates or top-left corner coordinates of the bounding box, and w and h are the width and height of the bounding box, respectively. "Independent of the target category" means that the candidate region generation stage only determines whether the region contains the target object, without determining which specific category the target object belongs to.
[0050] Candidate region generation methods include, but are not limited to, the following two: (1) Anchor boxes are generated on the multi-scale feature map according to a preset ratio. The anchor boxes are then subjected to foreground / background binary classification screening by a binary classification target network, and the anchor boxes classified as foreground are used as candidate regions.
[0051] Specifically, for each location (pixel) on the feature map, multiple anchor boxes with different aspect ratios and scales are preset. These anchor boxes are mapped back to the original image to form a dense set of candidate boxes covering all locations and sizes of the entire image. Subsequently, a binary classification target network is used to classify each anchor box as foreground / background. The target network typically consists of 1-2 convolutional layers, outputting a foreground confidence score for each anchor box. This score is the target score, representing the probability that the anchor box region contains any type of target. Based on a preset target score threshold (e.g., 0.5), anchor boxes with scores higher than the threshold are selected as candidate regions. Preferably, after selection, a non-maximum suppression algorithm is used to deduplicate the candidate regions, eliminating highly overlapping redundant boxes and retaining a moderate number of comprehensive candidate regions.
[0052] (2) Based on the feature points of the multi-scale feature map, perform center point position regression and bounding box size regression, and directly output candidate regions.
[0053] The basic idea of anchorless bounding box generation is as follows: for each position on the feature map, determine whether the position is located in the center region of a certain target. If so, directly predict the four side distances of the target bounding box (left, top, right, and bottom distances from the center point), thereby directly obtaining the rectangular bounding box of the candidate region.
[0054] Regardless of the method used, the output of the candidate region generator is a set of candidate region bounding boxes and the target score for each candidate region. Each candidate region is processed independently in subsequent steps, and its association score with the task description vector is calculated.
[0055] In this embodiment, the output candidate region set is completely independent of the task description vector. That is, during the inference process, regardless of the current task description vector, the candidate region generator outputs the same candidate region set for the same frame image. The task difference occurs in the association scoring stage. Different task description vectors generate different scene prior vectors with different similarities to the visual feature vectors of the candidate regions, thereby activating different candidate region outputs on the same candidate region set.
[0056] In step S3, extracting the visual feature vectors of each candidate region refers to extracting a fixed-dimensional visual semantic representation vector from the multi-scale feature map of each candidate region. Specifically, firstly, the bounding box coordinates of each candidate region (in the original image coordinate system) are mapped to the feature map coordinate system (scaled according to the downsampling factor of the feature map relative to the original image), determining the corresponding region of the candidate region on each layer of the feature map; then, using a region of interest alignment operation, the feature map region corresponding to the candidate region is pooled into a fixed-size feature map patch (such as 7×7 or 14×14) through bilinear interpolation. This feature map patch is then compressed into a fixed-length vector through a fully connected layer or a global average pooling layer, which is the visual feature vector. The dimension of the visual feature vector is consistent with the number of channels in the backbone network output feature map, and its physical meaning is: the semantic representation of the candidate region in visual space, encoding visual information such as texture, shape, color, and edges within the region.
[0057] This embodiment also encodes the task description vector to obtain a scene prior vector. The encoding process is implemented through an encoding network. The encoding network can adopt any of the following structures: Embedding layer: Maps discrete values such as the target category set and attribute set in the task description vector into a dense vector; Text encoder: Encodes the text description in the task description vector into a semantic vector; Multilayer perceptron: Input the numerical values of each dimension of the task description vector into a fully connected network, and output a scene prior vector with fixed dimensions.
[0058] The physical meaning of the scene prior vector is: the semantic template of the target to be detected by the current task, that is, what kind of target the current task needs to find in the image. The scene prior vector has the same vector dimension as the visual feature vector, so that the two can be used to calculate semantic similarity in the same embedding space.
[0059] In this embodiment, the perception network calculates an association score based on the semantic similarity between the visual feature vector and the scene prior vector, the targetability score of the candidate region, and the spatial matching degree between the candidate region and the monitoring scene configuration information. Association Score The calculation formula is as follows:
[0060] For the first i The target score of a candidate region represents the probability that the candidate region contains the target object.
[0061] For the first i Visual feature vectors of candidate regions.
[0062] For the first k Each task description vector corresponds to a scenario prior vector.
[0063] for and The semantic similarity between the candidate regions represents the degree of semantic consistency between the visual content of the candidate region and the current task target.
[0064] Preferably, the semantic similarity is and The cosine similarity or dot product similarity between them.
[0065] For the first i The spatial matching degree between each candidate region and the monitoring scene configuration information is used to measure the degree of spatial location matching between the candidate region and the spatial constraint parameters in the monitoring scene configuration information.
[0066] The candidate region is determined based on its spatial relationship with the bounding box coordinates of the candidate region and the alarm region, tripwire, or direction line. This spatial relationship includes at least one of the following: whether the candidate region is located inside the alarm region, whether the candidate region intersects with the tripwire, and whether the movement trend of the target object is consistent with the preset direction indicated by the direction line.
[0067] Preferably, if the center point or bounding box of the candidate region is located inside the polygon of the alarm region, and the intersection-over-union (IoU) ratio between the candidate region and the alarm region is high, then the spatial matching degree is high; if the candidate region intersects with the tripwire, then the spatial matching degree is high; if the movement trend of the target object is determined to be consistent with the direction indicated by the direction line based on the target movement trajectory, then the spatial matching degree is high.
[0068] , , , where are weighting coefficients, used to adjust the contribution ratios of target score, semantic similarity, and spatial matching degree to the association score, respectively.
[0069] Traditional object detectors rely solely on visual features for their output, learning a mapping between visual features and category labels. However, when environmental changes cause visual features to deviate from the training data distribution (e.g., at night, in rain or fog, or in thermal imaging), detection accuracy drops sharply. In this embodiment, the association score is jointly determined by visual features, task semantics, and scene space. When information in one dimension is insufficient, such as visual feature degradation in nighttime images, spatial constraints and task semantic priors can still provide a basis for judgment, thereby improving the model's stability in complex environments.
[0070] The preset score threshold described in this embodiment is used to control the sensitivity of the perception network activation. The higher the score threshold, the more conservative the output result (lower false positive rate, but possible missed detection); the lower the score threshold, the more aggressive the output result (higher recall rate, but the false positive rate may increase). When the association score exceeds the preset score threshold, the corresponding candidate region is activated, and the unified perception network outputs the target detection result for that candidate region. The target detection result includes at least the target bounding box and the target category.
[0071] It should be noted that different task description vectors drive the same perceptual network to output different object detection results, specifically including: When the attribute set in the task description vector is not empty, the target detection result also includes target attribute identification results (such as "life jacket worn / not worn", "carrying / not carrying hazardous chemical identification", etc.). When the attribute set in the task description vector contains a text recognition requirement flag, the target detection result also contains a text recognition trigger flag, which is used to wake up the downstream text recognition network to perform text sequence recognition on the corresponding candidate region.
[0072] Assume task description vector D k ={Target category set={vehicle}, Attribute set={hazardous chemical label status, text recognition requirements}, Spatial constraint parameter={road area}}, Association score of a candidate region The candidate region is activated when the score exceeds a preset threshold, due to D k The attribute set includes "hazardous chemical label status". The attribute recognition head determines that the vehicle carries a hazardous chemical label. It also includes "text recognition requirement". The model outputs a text recognition trigger flag = True. The downstream text recognition network performs license plate OCR recognition on the candidate area. The final output of the perception network is: bounding box coordinates + category = "vehicle" + attribute = "has hazardous chemical label" + license plate number = "XXXXX".
[0073] Furthermore, for the same candidate region and the same set of model parameters, selective output for different target objects can be achieved by changing the task description vector.
[0074] Since the target detection results output by the perceptual network are independent for each frame, each target object in each frame is assigned a bounding box and a category label, but the correspondence between the same target in different frames has not yet been established. Therefore, it is necessary to establish the correspondence of the same target in the time dimension through cross-frame correlation tracking, and organize the discrete detection results into a continuous motion trajectory.
[0075] Perform cross-frame correlation tracking on the target detection results of consecutive image frames to obtain the target motion trajectory, including: (1) Match the target detection results of the current image frame with the target detection results of the previous image frame, and determine the correspondence between cross-frame targets based on at least one of bounding box overlap, visual feature similarity and motion trend consistency; The bounding box overlap can be obtained by the intersection-over-union (IoU) ratio of the bounding boxes of the target in the current frame and the corresponding target in the previous frame in the image space. The higher the IoU, the closer the two are in the image, and the greater the probability that they belong to the same target.
[0076] Visual feature similarity is the similarity between the visual feature vectors (such as color histograms) of a target in the current frame and a target in the previous frame. Visual feature similarity is applicable when a target moves quickly in an image, resulting in low bounding box overlap, or when a target reappears after being briefly occluded.
[0077] Motion consistency is determined by predicting the position of the target in the current frame based on the target's motion velocity in the previous frame, and then calculating the distance between the target detection box in the current frame and the predicted position. The smaller the distance, the smoother and more continuous the motion trajectory, and the greater the probability that they belong to the same target.
[0078] The above matching criteria can be used in combination. The total matching score is calculated by weighted summation, and the pair with the highest matching score is taken as the cross-frame correspondence for the same target.
[0079] (2) Perform temporal correlation on the position coordinates of the same target object that was successfully matched in each image frame to generate a trajectory coordinate sequence; (3) Based on the trajectory coordinate sequence, determine the direction and speed of the target object's movement and generate the target's movement trajectory.
[0080] For the same target that is successfully matched, its position coordinates in the current frame are appended to the target's trajectory coordinate sequence, and the target's motion speed and direction are updated. Preferably, for existing trajectories that have not been matched with any detection results in the current frame and have not been updated for several consecutive frames, they are marked as "disappeared" and tracking is stopped; for a detected target that appears for the first time in the current frame and cannot be matched with any existing trajectory, a new trajectory is created for it, with its initial position as the trajectory starting point.
[0081] According to this embodiment, the determination function The expression is as follows:
[0082] For indicator functions, For the target trajectory, The direction of movement of the target object. A preset direction is specified for the spatial constraint parameters. The duration of the target object's continuous stay within the target area defined by the spatial constraint parameters. , These are the duration threshold and the quantity threshold, respectively, defined by the time constraint parameters. The cumulative number of objectives that simultaneously satisfy both time and space constraints.
[0083] The above decision function The judgment conditions included are: (1) Spatial overlap condition:
[0084] Determine whether there is spatial overlap between the target's trajectory and the alarm area defined by the spatial constraint parameters. If at least one point in the target trajectory coordinate sequence is located inside the alarm area, or if the target trajectory crosses a tripwire, then spatial overlap exists.
[0085] (2) Condition for consistent direction:
[0086] Determine whether the target object's movement direction is consistent with a preset direction. For example, if the angle between the target object's movement direction vector and the preset direction vector is less than a preset angle threshold (e.g., 30 degrees), then the directions are considered consistent. For tasks that do not require direction determination (e.g., intrusion tasks), this condition can be set to "satisfied" by default.
[0087] (3) Duration threshold condition:
[0088] Determine whether the duration of the target object's stay within the target area reaches the duration threshold. Calculate the time difference between the first moment the target object enters the target area and the current moment (or the last moment it leaves the target area) in the target trajectory coordinate sequence, and use this as the duration.
[0089] (4) Quantity threshold condition:
[0090] Determine whether the cumulative number of targets that simultaneously meet the above spatial, directional, and duration constraints has reached the quantity threshold.
[0091] (5) Filtering based on exclusion criteria:
[0092] Determine whether the target object belongs to the object type defined by the preset exclusion conditions. If the target object meets the exclusion conditions (e.g., the target category is non-motorized vehicles or the target is an authorized person), the determination process is skipped and no alarm is triggered.
[0093] According to this embodiment, a decision function is used to perform event reasoning on the target motion trajectory, and an alarm is triggered based on the reasoning result, including: An alarm is triggered when the target's motion trajectory simultaneously meets the following conditions: The target's trajectory overlaps with the alarm area, or the target object crosses a tripwire in a preset direction; the target object's continuous stay time in the target area reaches the duration threshold; the cumulative number of targets meeting the duration condition reaches the quantity threshold, that is, the cumulative number of target objects whose continuous stay time in the target area reaches the duration threshold reaches the quantity threshold; the target object does not belong to the preset exclusion conditions.
[0094] Based on the above approach, different types of business tasks can be handled by adjusting the activation status and parameter values of various conditions in the decision function.
[0095] Example 1: The target task is an intrusion event. Intrusion incidents include intrusion by people, vehicles, and vessels, and the core judgment logic is "the target has entered the alarm area".
[0096] In intrusion-type tasks: Spatial overlap condition: Enabled (requires the target object to enter the alarm area) Consistent direction condition: Not enabled (satisfied by default; intrusions are not restricted by direction). Duration threshold condition: Not enabled (meets by default, triggers upon entry). Quantity threshold condition: Not enabled (meets by default; triggers when a single target enters). Exclusion criteria: Activation as needed (e.g., excluding security personnel, authorized vehicles, etc.) The alarm determination condition for intrusion events is: determine whether there is spatial overlap between the target's movement trajectory and the alarm area, and whether the target object does not meet the preset exclusion conditions. If so, an alarm is triggered.
[0097] Example 2: The target task is a traffic statistics event. Traffic flow statistics events include ship traffic statistics, real-time vehicle traffic statistics, etc., and their core judgment logic is "the target object crosses the tripwire in a preset direction".
[0098] In traffic statistics tasks: Spatial overlap condition: Enabled (requires the target object to cross a tripwire) Consistent Direction Condition: Enabled (requires the target object to cross the tripwire in the specified direction) Duration threshold condition: Not enabled (default condition met) Quantity threshold condition: Enabled (cumulative count, counted once for each wear). Exclusion criteria: Activation will be based on demand (e.g., excluding small boats, non-motorized vehicles, etc.). The alarm judgment condition for traffic statistics events is: determine whether the target's movement trajectory crosses the tripwire in a preset direction, and whether the target object does not meet the preset exclusion conditions. If so, a single traffic count is triggered.
[0099] Example 3: The target task is a congestion or lingering event. Congestion-related events include traffic congestion and regional congestion, while loitering-related events include loitering of people and vehicles. The core judgment logic is that "the target object stays in a certain area for too long and in excessive numbers."
[0100] In congestion / wandering type tasks: Spatial overlap condition: Enabled (requires the target object to enter a certain area). Consistent direction condition: Not enabled (satisfied by default) Duration threshold condition: Enabled (requires meeting the stay time requirement) Quantity threshold condition: Enabled (requires sufficient quantity to meet the target). Exclusion criteria: Activated as needed (e.g., excluding non-motorized vehicles in motor vehicle congestion detection). The alarm determination conditions for congestion and loitering events are as follows: whether there is spatial overlap between the target's movement trajectory and the alarm area, whether the target object's continuous stay time in the alarm area reaches the duration threshold, whether the cumulative number of targets that meet the duration threshold reaches the quantity threshold, and whether the target object does not meet the preset exclusion conditions. If so, an alarm is triggered.
[0101] In traditional event determination schemes, different event types are implemented using their own independent rule scripts. The rule logic is scattered and lacks a unified framework. Each new event type requires developers to write new rule determination code and adapt it to the label names and coordinate formats output by upstream detection. This embodiment abstracts the common logic of various event determinations into a unified determination framework. All event determinations can be reduced to a joint examination of the target's motion trajectory across five dimensions: space, time, direction, quantity, and exclusion conditions. By uniformly encoding these five dimensions of determination conditions into configurable parameters (i.e., spatial constraint parameters, temporal constraint parameters, and exclusion conditions in the task description vector), a single determination function can be reused for multiple event types.
[0102] In practical engineering deployments, a challenge faced by video intelligent analysis systems in the transportation industry is that after the model is deployed to actual monitoring points, its performance gradually declines due to factors such as changes in lighting (day-night cycle, seasonal changes), weather conditions (rain, fog, snow, backlight), scene differences (different camera orientations, angles, and resolutions at different locations), and long-tail distribution (rare targets, rare poses). This leads to false alarms and missed alarms. Traditional solutions involve periodically organizing manual annotation of new data, retraining the model, and redeploying it, which is time-consuming, costly, and inefficient.
[0103] In this embodiment, according to a preferred implementation, the method further includes: (1) Collect samples whose association scores are in the preset low confidence interval, samples that have been manually confirmed as false alarms, and samples that have been re-labeled and corrected from the target detection results, and store them in the difficult sample pool; The low-confidence samples refer to edge detection box data whose association scores are in the critical uncertainty range during the online real-time inference stage. In practical applications, candidate regions with association scores exceeding a preset score threshold (e.g., 0.6) are activated, while candidate regions with association scores below a set minimum threshold (e.g., 0.3) are directly ignored (treated as background). However, when the association score is in the critical range between 0.3 and 0.6, the perceptual network is uncertain about whether the candidate region contains the target object and whether it is related to the current task, thus generating low-confidence samples.
[0104] False alarm samples refer to image data that triggered an alarm but was ultimately identified as a false positive after manual verification or business logic validation. For example, in a person intrusion alarm event, manual review might reveal that the target was actually a billboard being blown by the wind or a shadow under specific lighting conditions; in a traffic congestion alarm event, manual confirmation might reveal that the congested area was actually experiencing normal slow traffic. The difference between false alarm samples and low-confidence samples is that false alarm samples are judged as "alarm triggered" (confidence level higher than the high threshold) but are identified as erroneous alarms; low-confidence samples are judged as "uncertain" (confidence level between the low and high thresholds) and have not yet triggered an alarm.
[0105] Manually corrected samples refer to image data that has been manually and accurately re-annotated and corrected by maintenance personnel during routine spot checks for missed or false reports.
[0106] The three types of samples together cover the abnormal output states that the model may encounter during the inference process. Low confidence samples cover the ambiguous areas of the decision boundary, false alarm samples cover over-detection errors, and manually corrected samples cover under-detection and category / attribute errors, forming a complete system of difficult sample data sources.
[0107] (2) Classify and organize the samples in the difficult sample pool according to at least one of the following dimensions: monitoring location, time period, weather, day and night status, and scene category; (3) The network parameters of the perception network are periodically updated using the samples in the difficult sample pool, and the preset score threshold is calibrated simultaneously.
[0108] The training set is formed by periodically drawing samples from a difficult sample pool and then performing incremental or full-scale fine-tuning training on the unified perception network. Incremental fine-tuning refers to using the currently deployed online model parameters as initial values and performing several rounds of gradient descent optimization with difficult samples. This allows the model to retain its existing capabilities while focusing on learning the difficult example patterns represented by the difficult samples. The advantages of incremental fine-tuning over training from scratch are faster training speed, fewer required samples, and no forgetting of learned general features. In full-scale fine-tuning, samples from the difficult sample pool can be mixed with representative samples from the original training set for joint training to prevent the model from overfitting to difficult samples and sacrificing general performance.
[0109] The preset score threshold determines the sensitivity of the model output. Specific methods for threshold calibration include, but are not limited to: performing statistical analysis based on the association scores of samples in the hard sample pool to select the threshold that maximizes the F1 score; and selecting the optimal threshold that meets business constraints based on different business preferences for precision and recall (e.g., recall is more important in security scenarios, while precision is more important in efficiency scenarios).
[0110] This embodiment constructs a complete data closed-loop link: online inference → hard sample collection → periodic model update → model redeployment. As the model's runtime increases, it continuously iterates itself using high-value hard samples generated in actual operation, constantly accumulating specialized knowledge for its own scenario, continuously breaking through the performance bottleneck of long-tail scenarios, and achieving a long-term evolution effect of becoming more accurate with use.
[0111] This application also provides a system for scene-prior driven target detection and event recognition, including: a data acquisition module, a task description module, a perception network module, and a trajectory tracking and alarm determination module.
[0112] The data acquisition module is used to acquire monitoring image data and monitoring scene configuration information.
[0113] The task description module is used to construct a task description vector based on the monitoring scenario configuration information and the target task. The task description vector includes at least a target category set, an attribute set, spatial constraint parameters, time constraint parameters, and preset exclusion conditions.
[0114] The perception network module is used to extract multi-scale features from the monitoring image data, generate a candidate region set based on the extracted multi-scale feature maps, encode the task description vector to obtain a scene prior vector, extract the visual feature vector of each candidate region, and calculate the association score based on the semantic similarity between the visual feature vector and the scene prior vector, the targetability score of the candidate region, and the spatial matching degree between the candidate region and the monitoring scene configuration information. The targetability score of each candidate region represents the probability that the candidate region contains a target object. When the association score exceeds a preset score threshold, the target detection result of the corresponding candidate region is output.
[0115] Different task description vectors drive the perception network to output different target detection results.
[0116] The trajectory tracking and alarm determination module is used to perform cross-frame correlation tracking on the target detection results of continuous image frames to obtain the target motion trajectory. It uses a determination function that integrates the spatial constraint parameters, the temporal constraint parameters and the exclusion conditions to perform event reasoning on the target motion trajectory, and determines whether to trigger an alarm based on the reasoning result.
[0117] Regarding the system in the above embodiments, the specific manner in which each unit module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.
[0118] Based on the same inventive concept, this application also provides an electronic device, including: a memory and a processor, wherein the processor is used to read and execute a computer program stored in the memory to realize the aforementioned scene prior-driven unified target detection and event recognition method.
[0119] Based on the same inventive concept, embodiments of this application also provide a computer storage medium storing computer-executable instructions, which, when executed, implement the aforementioned scene prior-driven unified target detection and event recognition method.
[0120] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.
[0121] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.
[0122] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail in a particular embodiment can be found in the relevant descriptions of other embodiments. Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for scene prior-driven target detection and event recognition, characterized in that, The method includes: Acquire monitoring image data and monitoring scene configuration information, and construct a task description vector based on the target task. The task description vector shall at least include a target category set, an attribute set, spatial constraint parameters, time constraint parameters, and preset exclusion conditions. Multi-scale feature extraction is performed on the monitoring image data, and a candidate region set is generated based on the extracted multi-scale feature map. The candidate region is a region in the monitoring image that may contain a target object, and the targetability score of each candidate region represents the probability that the candidate region contains a target object. The task description vector is encoded to obtain a scene prior vector, and the visual feature vectors of each candidate region are extracted. The perceptual network calculates the association score based on the semantic similarity between the visual feature vector and the scene prior vector, the target score of the candidate region, and the spatial matching degree between the candidate region and the monitoring scene configuration information. When the association score exceeds a preset score threshold, the target detection result of the corresponding candidate region is output. Different task description vectors drive the perceptual network to output different target detection results. Cross-frame correlation tracking is performed on the target detection results of continuous image frames to obtain the target motion trajectory. An event reasoning is performed on the target motion trajectory using a decision function that fuses the spatial constraint parameters, the temporal constraint parameters and the exclusion conditions. An alarm is triggered based on the reasoning result.
2. The method according to claim 1, characterized in that, The task description vector is represented as follows: in, This represents the target category set, containing all object types involved in the target task. It represents a set of attributes, containing additional features of the target object to be detected; The spatial constraint parameters include, but are not limited to, alarm areas, tripwires, and direction lines determined according to the monitoring scene configuration information. The alarm areas are used to limit the range in which the target object enters or stays. The tripwires are used to limit the linear boundaries that the target object crosses. The direction lines are used to limit the preset direction of the target object's movement. The time constraint parameters include the time window T determined based on the monitoring scenario configuration information, and a preset duration threshold. and quantity threshold The time window T defines the time interval for event determination, and the duration threshold... Define the maximum allowed dwell time of a target object within the target area, the quantity threshold. An upper limit is defined for the cumulative number of targets that meet the duration condition; This indicates an exclusion condition.
3. The method according to claim 1, characterized in that, The generation of the candidate region set based on the extracted multi-scale feature maps includes, but is not limited to, the following two methods: Anchor boxes are generated on the multi-scale feature map according to a preset ratio. The anchor boxes are then subjected to foreground / background binary classification filtering by a binary classification target network, and the anchor boxes classified as foreground are used as candidate regions. Based on the feature points of the multi-scale feature map, center point position regression and bounding box size regression are performed to directly output candidate regions.
4. The method according to claim 2, characterized in that, Correlation score The calculation formula is as follows: in, For the first i The target score for each candidate region; For the first i Visual feature vectors of candidate regions; For the first k Each task description vector corresponds to a scenario prior vector; for and Semantic similarity between them; For the first i Spatial matching degree between each candidate region and the monitoring scene configuration information According to the i The spatial relationship between the bounding box coordinates of each candidate region and the alarm region, tripwire, or direction line is determined. , , These are the weighting coefficients.
5. The method according to claim 1, characterized in that, The target detection result includes at least the target bounding box and the target category; When the set of attributes in the task description vector is not empty, the target detection result also includes the target attribute recognition result; When the attribute set in the task description vector contains a text recognition requirement flag, the target detection result also contains a text recognition trigger flag, which is used to wake up the downstream text recognition network to perform text sequence recognition on the corresponding candidate region.
6. The method according to claim 1, characterized in that, The step of performing cross-frame correlation tracking on the target detection results output from consecutive image frames to obtain the target motion trajectory includes: The target detection results of the current image frame are matched with the target detection results of the previous image frame. The correspondence between cross-frame targets is determined based on at least one of the following: bounding box overlap, visual feature similarity, and motion trend consistency. The position coordinates of the same target object that was successfully matched in each image frame were temporally correlated to generate a trajectory coordinate sequence. Based on the trajectory coordinate sequence, the direction and speed of the target object's movement are determined, and the target's motion trajectory is generated.
7. The method according to claim 2, characterized in that, Decision function The expression is as follows: in, For indicator functions, For the target trajectory, The direction of movement of the target object. A preset direction is specified for the spatial constraint parameters. The duration of the target object's continuous stay within the target area defined by the spatial constraint parameters. , These are the duration threshold and the quantity threshold, respectively, defined by the time constraint parameters. The cumulative number of targets that simultaneously satisfy both time and space constraints; The determination function is used to perform event reasoning on the target motion trajectory, and the result of the reasoning is used to determine whether to trigger an alarm, including: An alarm is triggered when the target's motion trajectory simultaneously meets the following conditions: The target's trajectory overlaps spatially with the alarm area, or the target object crosses the tripwire in a preset direction; The target object stays in the target area for a duration that reaches the duration threshold. The cumulative number of targets that meet the duration condition reaches the specified threshold. The target object does not fall under the preset exclusion criteria.
8. The method according to claim 1, characterized in that, The method further includes: Samples with association scores in the preset low confidence interval, samples confirmed as false alarms by manual verification, and samples that have been re-labeled and corrected are collected from the target detection results and stored in the difficult sample pool. The samples in the difficult sample pool are classified and organized according to at least one of the following dimensions: monitoring location, time period, weather, day and night status, and scene category; The network parameters of the perception network are periodically updated using samples from the difficult sample pool, and the preset score threshold is calibrated simultaneously.
9. A system for scene prior-driven target detection and event recognition, characterized in that, The system includes: The data acquisition module is used to acquire monitoring image data and monitoring scene configuration information; The task description module is used to construct a task description vector based on the monitoring scenario configuration information and the target task. The task description vector includes at least a target category set, an attribute set, spatial constraint parameters, time constraint parameters, and preset exclusion conditions. The perception network module is used to perform multi-scale feature extraction on the monitoring image data, generate a candidate region set based on the extracted multi-scale feature maps, encode the task description vector to obtain a scene prior vector, extract the visual feature vector of each candidate region, and calculate the association score based on the semantic similarity between the visual feature vector and the scene prior vector, the target score of the candidate region, and the spatial matching degree between the candidate region and the monitoring scene configuration information. The target score of each candidate region represents the probability that the candidate region contains a target object. When the association score exceeds a preset score threshold, the target detection result of the corresponding candidate region is output. The trajectory tracking and alarm determination module is used to perform cross-frame correlation tracking on the target detection results of continuous image frames to obtain the target motion trajectory. It uses a determination function that integrates the spatial constraint parameters, the temporal constraint parameters and the exclusion conditions to perform event reasoning on the target motion trajectory, and determines whether to trigger an alarm based on the reasoning result.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed, implement the method described in any one of claims 1-8.