A target retrieval method and system

Abstract

The application discloses a target retrieval method and system. The method comprises the following steps: S1, data access and preprocessing, including acquiring multi-source data, cleaning, formatting, spatio-temporal standardization, target detection and cutting of images / videos, and generating structured target objects; S2, feature extraction, including extracting deep semantic feature vectors with discrimination and adaptability to complex scenes and auxiliary understanding information from target images; S3, constructing a data index, including constructing a spatio-temporal semantic hypergraph index of multi-type nodes and hyperedges based on deep features and spatio-temporal information of targets, so as to express complex relationships between targets; and S4, data query, including receiving multi-modal query information of a user, screening a candidate area, matching features and reasoning relationships by using the hypergraph index, and finally outputting a target retrieval result with high confidence. The application improves the robustness of complex scenes and target changes and deep multidimensional query.

Description

Technical Field

[0001] This application relates to the field of retrieval technology, and in particular to a target retrieval method and system. Background Technology

[0002] With the rapid development of information technology, various types of data are experiencing explosive growth, especially in fields such as video surveillance, geographic information systems, social networks, and the Internet of Things, generating massive amounts of structured and unstructured data containing target information. How to quickly and accurately retrieve targets of interest from this massive amount of data and further analyze the potential relationships between targets has become a research hotspot and a technological bottleneck in the field of information processing.

[0003] Existing target retrieval technologies, such as content-based image retrieval (CBIR), keyword-based text retrieval, or feature-specific matching algorithms, often face challenges when processing large-scale, high-dimensional, and complex scene data, mainly due to the following technical shortcomings:

[0004] Low retrieval efficiency: When dealing with massive amounts of data, traditional linear scanning or global feature matching methods struggle to meet real-time requirements. Although index structures have been introduced, the performance of existing data structure indexes (such as R-trees and some variants of KD-trees) drops sharply when faced with high-dimensional feature data, resulting in the "curse of dimensionality."

[0005] Insufficient retrieval accuracy: Targets may exhibit significant changes in appearance, pose, and lighting under different scenarios, leading to poor robustness of matching based on underlying features. Furthermore, the semantic analysis and understanding between user query intent and machine understanding also limit retrieval accuracy.

[0006] Insufficient utilization of spatial information: Many target data naturally have spatial attributes (such as GPS coordinates, relative positions in images, etc.), but existing retrieval methods do not make in-depth use of spatial information, making it difficult to support complex spatial relationship queries and efficient retrieval based on spatial proximity.

[0007] In terms of spatial data indexing, although various spatial index structures have been developed, such as R-trees, R* trees, quadtrees, and grid indexes, which have improved the query efficiency of spatial objects to some extent, these methods also have the following technical drawbacks:

[0008] Static vs. Dynamic Adaptability: Traditional spatial indexes are mostly designed for static data. For dynamically changing and frequently updated data scenarios, the maintenance cost of the index is high and the performance may degrade.

[0009] Multi-level and multi-scale issues: Targets may have different importance and relevance at different scales and levels. Single-level spatial indexes are difficult to effectively express and utilize this multi-level characteristic, resulting in insufficient flexibility in the granularity of retrieval and analysis.

[0010] Difficulty in combining with semantic information: Traditional spatial indexes mainly focus on the geometric location of the target, and the combination with the semantic features of the target (such as category and attributes) is not close enough, making it difficult to achieve efficient retrieval based on both semantic and spatial constraints.

[0011] Therefore, there is an urgent need to propose a new technical solution to achieve rapid positioning and filtering of massive target data, and ultimately achieve efficient and accurate target retrieval to meet the growing demand for intelligent information processing. Summary of the Invention

[0012] This application provides a target retrieval method to address the technical problems existing in the prior art, comprising the following steps:

[0013] Step S1: Data access and preprocessing, including acquiring multi-source data, cleaning, formatting, and spatiotemporal standardization, and performing target detection and cropping on images / videos to generate structured target objects;

[0014] Step S2: Feature extraction, including extracting deep semantic feature vectors with discriminative power and adaptability to complex scenes, as well as information to aid understanding from the target image;

[0015] Step S3: Construct a data index, including constructing a spatiotemporal semantic hypergraph index with multiple types of nodes and hyperedges based on the deep features and spatiotemporal information of the target, in order to express the complex relationships between targets;

[0016] Step S4: Data query, which includes receiving multimodal query information from users, using Hypergraph index to perform candidate region filtering, feature matching and relationship inference, and finally outputting high-confidence target retrieval results.

[0017] Furthermore, step S2: the feature extraction step specifically includes:

[0018] S21: Based on D 2 CFAN model for feature extraction;

[0019] S22: Optimize training based on the feature extraction model and set the loss function;

[0020] S23: Preprocess and normalize the trained feature extraction model during the inference stage;

[0021] S24: Feature extraction, forward propagation, and internal dynamic regulation;

[0022] S25: Obtain the output feature vector and perform quality assessment and directional transmission.

[0023] Further, step S21: based on D 2 The CFAN model constructs a feature extraction model including:

[0024] S211: Basic feature extraction and multi-level semantic representation construction, used to extract different depths of the pathway and obtain feature maps that can represent semantic information at different abstract levels;

[0025] S212: Construct a dynamic hierarchical decoupled attention submodule and apply attention to each semantic subgroup to dynamically decouple the feature channels of the feature map at each level into multiple semantic subgroups;

[0026] S213: Construct an instance-level and category-level collaborative attention submodule to allow the model to simultaneously focus on the instance uniqueness and category commonality of the target before extracting the final features;

[0027] S214: Construct a context-aware fusion attention submodule to fuse features from different processing stages;

[0028] S215: Output feature vector, which is used to pass the features obtained from the context-aware fusion attention submodule through one or more fully connected layers to obtain the final feature vector.

[0029] Further, step S22: optimizing the training based on the feature extraction model and setting the loss function specifically includes:

[0030] S221: Knowledge transfer initialization and domain adaptation fine-tuning;

[0031] S222: Optimize the main identity recognition loss and DHDA hierarchical weighted semantic decoupling orthogonality regularization, the calculation formula is as follows:

[0032]

[0033] Where: s ij ·s il Represents vector s ij and s il The dot product, ||s ij ||2 represents vector s ij The L2 norm, ∈ 0 for positive constants, ω i It is the decoupling importance weight coefficient set for the i-th semantic level;

[0034] S223:D 2 CFAN model parameter iterative optimization process.

[0035] Further, step S3, constructing the data index includes:

[0036] S31: Construct the vertices and hyperedges of the spatiotemporal semantic hypergraph index;

[0037] S32: Initial construction of the spatiotemporal semantic hypergraph index, continuous entry of data objects and dynamic evolution of the index structure, generation and association of relational hyperedges.

[0038] Furthermore, S31: The vertices and hyperedges for constructing the spatiotemporal semantic hypergraph index include:

[0039] S311: The vertices used to construct the spatiotemporal semantic hypergraph index include: macro-spatiotemporal block vertices, basic spatiotemporal unit vertices, and target instance vertices;

[0040] S312: The hyperedges for constructing the spatiotemporal semantic hypergraph index include: spatiotemporal co-occurrence hyperedges, semantic similarity hyperedges, trajectory / event association hyperedges, and region attribute association hyperedges;

[0041] S313: Determine the symbols and attributes of vertices and hyperedges.

[0042] Further, step S32 includes:

[0043] S321: Initial construction of the index;

[0044] S322: Insertion of target instance vertices and dynamic evolution of the basic structure;

[0045] S323: Dynamic generation and association of hyperedges.

[0046] Further, step S4, the data query includes:

[0047] S41: Query intent parsing and multimodal input processing;

[0048] S42: STSGHI hierarchical initial screening and candidate region localization;

[0049] S43: Fine-grained target matching and initial candidate V Object Set generation;

[0050] S44: Association Expansion and Inference Retrieval Based on STSGHI Hyperedges;

[0051] S45: Perform deduplication, fusion, and confidence assessment on direct matching and expanded results.

[0052] Further, step S1: data access and preprocessing steps include:

[0053] S11: Data source interface configuration and connection establishment;

[0054] S12: Raw data acquisition and preliminary analysis;

[0055] S13: Data cleaning and quality assessment;

[0056] S14: Spatiotemporal information extraction and standardization;

[0057] S15: Preliminary target detection;

[0058] S16: Data encapsulation and output.

[0059] This invention also provides a target retrieval module, comprising: a data access and preprocessing module, a feature extraction module, an index construction module, and a target retrieval module, wherein:

[0060] The data access and preprocessing module is used to acquire multi-source data, perform cleaning, formatting, and spatiotemporal standardization, and perform target detection and cropping on images / videos to generate structured target objects;

[0061] The feature extraction module is used to extract deep semantic feature vectors with discriminative and adaptable characteristics to complex scenes, as well as information to aid understanding from the target image;

[0062] The index building module is used to construct a spatiotemporal semantic hypergraph index with multiple types of nodes and hyperedges based on the deep features and spatiotemporal information of the target, in order to express the complex relationships between targets;

[0063] The target retrieval module receives multimodal query information from users, uses the Hypergraph index to perform candidate region filtering, feature matching and relation inference, and finally outputs high-confidence target retrieval results.

[0064] In practical applications, the modules described in the methods and systems disclosed in this application can be deployed on a single target server, or each module can be deployed independently on different target servers. In particular, as needed, to provide more powerful computing capabilities, the modules can also be deployed on a cluster of target servers.

[0065] Therefore, the technical effects achieved by the technical approach adopted in this application are as follows:

[0066] 1. Significantly improves search efficiency and accuracy: through D 2 The CFAN model extracts highly discriminative features, and combined with STSGHI's hierarchical indexing and hypergraph association, it can quickly locate targets from massive amounts of data and effectively distinguish similar targets, thereby improving the precision and recall of retrieval.

[0067] 2. Enhance robustness to complex scenarios and changing objectives: D 2 The CFAN model, through its dynamic hierarchical decoupling, context awareness, and instance-class collaboration mechanism, can effectively address the challenges of complex security scenarios such as lighting changes, occlusion, and pose diversity.

[0068] 3. Achieve deep integration and utilization of spatial, temporal, and semantic information: The STSGHI structure organically integrates the spatiotemporal attributes and deep semantic features of the target through its multi-level vertices and multi-type hyperedges, supporting more flexible and in-depth multi-dimensional queries.

[0069] 4. Supports efficient discovery and reasoning of complex relationships: STSGHI's hyperedge mechanism can explicitly express complex relationships between targets, such as spatiotemporal co-occurrence, semantic similarity, and trajectory events. This enables the system not only to find targets but also to reveal potential connections between them, supporting more advanced analysis tasks.

[0070] 5. Improve the system's intelligence level and practical application value: This invention provides a complete solution from feature extraction to index construction to intelligent retrieval, which can effectively process and analyze large-scale, multi-source urban security target data, providing strong technical support for urban security, video surveillance, geographic information systems and other fields.

[0071] To provide a clearer and more comprehensive understanding of this application, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a flowchart illustrating a target retrieval method according to an embodiment of this application.

[0074] Figure 2 This is a schematic diagram of the vertex structure in an embodiment of this application.

[0075] Figure 3 A schematic diagram of the structure of a target retrieval system according to an embodiment of this application. Detailed Implementation

[0076] Please see Figure 1 The technical solution of this application provides a target retrieval method, including the following steps:

[0077] Step S1: Data access and preprocessing steps include: acquiring multi-source data, cleaning, formatting, and spatiotemporal standardizing it, and performing target detection and cropping on images / videos to generate structured target objects.

[0078] Step S1 involves accessing raw data from diverse data sources (such as video surveillance, regulatory records, checkpoint data, mobile device positioning data, etc.), performing data cleaning, format conversion, spatiotemporal information extraction and standardization (including timestamp unification, geocoding and coordinate system unification), and conducting preliminary target detection on image / video data, cropping out target image blocks, and finally encapsulating them into a unified structured object for output.

[0079] Step S2, the feature extraction step includes: extracting deep semantic feature vectors with discriminative and adaptable characteristics to complex scenes, as well as auxiliary understanding information from the target image.

[0080] This step S2 uses dynamic hierarchical decoupling and context-aware fusion attention network (D) 2 The CFAN (Dynamic Hierarchical Decoupling and Context-Aware Fusion Attention Network) model processes pre-processed target image patches. This model extracts deep, discriminative semantic feature vectors from the target object and is adaptable to complex scenes through its core components, including Dynamic Hierarchical Decoupling Attention (DHDA), Instance-Category Collaborative Attention (ICCA), and Context-Aware Fusion Attention (CAFA).

[0081] Model training employs a composite loss function, including the primary identity recognition loss and customized auxiliary losses and regularization terms for DHDA and ICCA characteristics. During inference, the input image patches undergo normalization preprocessing, through D... 2 CFAN performs forward propagation and utilizes its internal dynamic control mechanism to output the final feature vector and optional auxiliary understanding information.

[0082] Step S3, constructing the data index includes: constructing a spatiotemporal semantic hypergraph index with multiple types of nodes and hyperedges based on the deep features and spatiotemporal information of the target, in order to express the complex relationships between targets.

[0083] This step S3 is for items with D. 2Using CFAN feature vectors and target data with precise spatiotemporal information, a Spatio-Temporal-Semantic Hypergraph Index (STSGHI) is constructed. This index structure includes multiple types of vertices (macro-spatiotemporal blocks VMBlock, adaptive semantic regions VARegion, basic spatiotemporal units VCell, and target instances V). Object (etc.) and multiple types of hyperedges (HESTC for spatiotemporal co-occurrence, HESemSim for semantic similarity, HETrack / Event for trajectory / event association, HERegionAttr for region attribute association, etc.) are used to express complex spatiotemporal co-occurrence, semantic similarity and behavioral association between targets.

[0084] Step S4, data query, includes: receiving multimodal query information from users, using the Hypergraph index to perform candidate region filtering, feature matching and relationship inference, and finally outputting high-confidence target retrieval results.

[0085] Step S4 receives multimodal query requests from users (images, text, spatiotemporal ranges, target IDs, or composite queries) and performs query intent parsing and feature extraction. It utilizes STSGHI's hierarchical structure (VMBlock, VARegion) and region semantic profiling (SemSumV) for initial screening and candidate region localization. Within the VCells of the candidate regions, based on D... 2 CFAN feature vectors are used for core semantic similarity matching, and combined with other target attributes to generate initial candidate V. Object The initial results are further expanded and inferred using various hyperedges in STSGHI (HESTC, HESemSim, HETrack / Event, HERegionAttr) to discover accompanying targets, cross-temporal and spatiotemporal similar targets, target trajectories, event participants, and targets that conform to specific region attributes. Finally, the direct matching and expanded results are deduplicated and merged, and a comprehensive confidence score is calculated based on the original detection confidence, the degree of matching with the query, and the association path score, outputting the retrieval results.

[0086] The technical solutions of this application are described below with reference to various embodiments.

[0087] Step S1: Data access and preprocessing steps include: acquiring multi-source data, cleaning, formatting, and spatiotemporal standardization, and performing target detection and cropping on images / videos to generate structured target objects.

[0088] For example, in urban security scenarios, accessing diverse data sources, including videos / images captured by fixed surveillance cameras throughout the city, videos / images captured by mobile law enforcement terminals, databases storing historical regulatory reports, vehicle checkpoint capture systems, public Wi-Fi hotspot records, and even publicly released information from social media related to public safety, requires a series of preprocessing operations. These include data cleaning, format conversion, spatiotemporal information extraction and standardization, and preliminary target (such as pedestrians, vehicles, and specific items) detection, providing standardized input for subsequent feature extraction and index construction.

[0089] In a preferred embodiment, step S1 includes:

[0090] S11: Data source interface configuration and connection establishment, specifically including:

[0091] S111: Configure the corresponding access adapter according to the data source type. For example, configure RTSP / RTMP / GB28181 clients for real-time urban surveillance video streams (such as intersection cameras, shopping mall internal monitoring); configure JDBC / ODBC connectors for regulatory databases or vehicle management databases; configure paths and access permissions for file systems storing historical surveillance video files or image libraries; and configure authentication and request parameters for interfaces that obtain public safety event APIs (such as weather warnings, traffic control information).

[0092] S112: Initialize and establish stable connections with each data source to enable continuous (e.g., real-time video stream access) or batch (e.g., daily regulatory data import) data acquisition.

[0093] S12: Raw data acquisition and preliminary analysis, specifically including:

[0094] S121: For video / image data (such as surveillance footage, images captured by checkpoints), it can be specifically divided into two types:

[0095] Real-time video stream: Decodes and captures image frames from the video stream at a preset frame rate (e.g., 5-10 frames per second for key areas, 1-3 frames per second for ordinary areas, or dynamically adjusted according to motion detection results in the image).

[0096] Historical video / image files: Batch read image files or extract frames from historical surveillance recordings.

[0097] S122: For structured / semi-structured data (such as regulatory text, sensor readings, and checkpoint records), the processing steps include:

[0098] a) Obtain data records through API calls or database queries, such as obtaining surveillance records containing the time of the incident, location description, and characteristics of the target person / vehicle.

[0099] b) Parse JSON (such as vehicle information uploaded by the checkpoint system), XML, CSV and other formats to extract key fields (such as alarm time, text description of the crime scene, event type, license plate number, vehicle color, sensor ID, and pedestrian flow statistics).

[0100] S13: Data cleaning and quality assessment, as a preferred implementation method, includes the following steps:

[0101] S131: Image Quality Assessment: Perform quality inspection on the acquired surveillance image frames or checkpoint images, and remove images that are too blurry (such as those caused by camera shake or low light at night), overexposed / underexposed, severely occluded (such as pedestrians being obscured by large objects by more than 70%), or have too low resolution (such as those that cannot recognize faces or license plates).

[0102] S132: Data Record Verification: Check the completeness (e.g., whether the crime scene location is missing), type correctness (e.g., whether the time format is standard YYYY-MM-DDHH:MM:SS, and whether the coordinates are within the city limits), and consistency (e.g., whether the vehicle entry and exit times at checkpoints are logically correct) of structured data fields such as monitoring records and checkpoint data.

[0103] S133: Missing Value Handling: For missing values ​​in key fields, mark, fill (e.g., mark as 'unknown' if the vehicle color is unknown, or infer from the historical records of the same vehicle) or delete the record without affecting the core analysis, according to security business rules.

[0104] S134: Redundant Data Removal: Identify and remove duplicate surveillance records (such as the same case being reported multiple times with identical content) or nearly identical consecutive surveillance image frames.

[0105] S14: Spatiotemporal information extraction and standardization, specifically including:

[0106] S141: Time Information Processing: Extract timestamps from filenames (e.g., surveillance video filenames often contain the time), image EXIF ​​metadata, or the time of incident field in surveillance records. Convert all timestamps to a standard time format (e.g., UTC or Beijing time, ISO 8601).

[0107] S142: Spatial information processing, specifically including:

[0108] S1421: If the data includes GPS coordinates (such as the location of an event reported by a mobile police terminal), extract it directly.

[0109] S1422: If it is an address description (such as "shop entrance at No. XX, XX Road" in the regulatory record), then call the geocoding service (such as Gaode API, Baidu API, or the address database matching service inside the regulatory department) to convert it into latitude and longitude coordinates.

[0110] S1423: If it is a fixed camera or other equipment, obtain its geographical location from the camera's installation location recorded in the equipment ledger or GIS system.

[0111] S1424: Convert all geographic coordinates to a standard coordinate system (such as WGS84 or the National 2000 coordinate system).

[0112] S15: Preliminary target detection. Preliminary target detection mainly targets image or video data, such as surveillance footage, and specifically includes:

[0113] S151: Load a pre-trained general object detection model (e.g., YOLOv7 optimized for urban scenes, Faster R-CNN in Detectron2, which have been trained on COCO, Cityscapes, and specific security datasets).

[0114] S152: Perform target detection on image frames that have passed the quality assessment, and obtain bounding boxes, preliminary category labels (such as "person", "car", "bicycle") and confidence scores for potential security-related targets (such as pedestrians, faces, motor vehicles, non-motor vehicles, backpacks, dangerous goods signs, etc.) in the image.

[0115] S153: Filter out low-confidence detection results based on a preset confidence threshold (e.g., 0.5 for regular target detection, and appropriately lower for key targets such as faces to improve recall, followed by further screening).

[0116] S154: Based on the detected bounding boxes, crop out image patches of the target region from the original image, such as cropping out individual pedestrian images or vehicle images.

[0117] S16: Data encapsulation and output, specifically including:

[0118] S161: Encapsulate the processed data into a unified structured object / message. In a security scenario, this object includes: a globally unique target ID (which can be generated based on timestamp, camera ID, and detection box position), a target image patch (if any), a preliminary target category (e.g., "pedestrian"), a precise spatiotemporal stamp (capture time, latitude and longitude), original data source information (e.g., camera number, monitoring ID), and preliminary detection confidence level.

[0119] S162: Send the encapsulated data object, especially the target image patch and related metadata (such as precise spatiotemporal stamp, preliminary category label, etc.), to D via message queue or direct API call. 2 CFAN is used for deeper semantic feature analysis.

[0120] Step S2, the feature extraction step includes: extracting deep semantic feature vectors with discriminative and adaptable characteristics to complex scenes, as well as auxiliary understanding information from the target image.

[0121] This application is approved by D. 2 The CFAN model has been improved to extract deeper, more discriminative semantic feature vectors of targets, which are more adaptable to complex security scenarios (such as changes in lighting, occlusion, small targets, and similar appearances).

[0122] In a preferred embodiment, step S2 includes:

[0123] S21: Based on D 2 The CFAN model constructs a feature extraction model, specifically including:

[0124] S211: Basic feature extraction and multi-level semantic representation construction, serving as the model's "perceptual front end" and "original data supplier," is used to extract different depths of the pathway and obtain feature maps that can represent semantic information at different levels of abstraction.

[0125] To give D 2 The CFAN model provides rich and hierarchical initial feature inputs. This application draws on the deep convolutional network architecture, but does not simply reuse a specific grid structure. Instead, it uses it as the front end of feature extraction, i.e., the basic feature extraction path. The network structure and parameter selection of this scheme depend on maximizing the efficiency of subsequent innovative sub-modules (DHDA, ICCA, CAFA).

[0126] Feature maps representing semantic information at different levels of abstraction are obtained from different depths (i.e., different computational stages) of this basic feature extraction pathway. These feature maps form the basis of the feature pyramid, defined as F_low, F_mid, and F_high. They capture visual information from low-level texture details to mid-level component structures and then to high-level abstract concepts. This multi-level representation is a prerequisite for the subsequent fine-tuning of the dynamically layered decoupled attention submodule.

[0127] S212: Construct a dynamic hierarchical decoupled attention submodule DHDA to dynamically decouple the feature channels of the feature map at each level into multiple semantic subgroups, and apply attention to each semantic subgroup to achieve initial attention focusing on dynamic semantic decoupling of features at different levels.

[0128] The implementation of multiple semantic subgroups includes dynamic subspace projection, subspace attention, and selective reorganization, among which:

[0129] Dynamic subspace projection: For each feature map F_i (i∈{low,mid,high}), it is projected onto K different feature subspaces {S_i1,S_i2,...,S_iK} through a small learnable convolutional network (called "subspace projector").

[0130] This application considers that feature maps at different levels contain different levels of semantic information abstraction, requiring different projection methods for decoupling. Therefore, F_i and SP_i have a one-to-one correspondence. SP_i can learn how to dynamically extract and group features based on the input features using its internal K processing channels, rather than using fixed channel groups. Each processing channel of SP_i is responsible for extracting and outputting a specific type of semantic information from the input F_i, and this output is S_i2 in one of the K feature subspaces.

[0131] For example, in a security scenario, when analyzing a surveillance photo of a target person, assuming the feature map at the F_mid level is being processed, the corresponding subspace projector SP_mid, upon receiving F_mid, will have its K different "analysis engines" simultaneously working: Analysis engine 1 might be specifically trained to extract and output information about the "main silhouette and folds of clothing" from F_mid, forming feature subspace S_mid1; Analysis engine 2 might focus on identifying and outputting the "shape and approximate type of carried items (such as backpacks or handbags)," forming feature subspace S_mid2; and so on, resulting in the Kth feature subspace S_midK. SP_i learns automatically to determine how to most effectively decouple the input F_i into these K meaningful feature subspaces with different focuses, rather than relying on a pre-defined fixed grouping method. These K subspaces {S_i1, S_i2, ..., S_iK} together constitute a more detailed and multi-dimensional decomposition of the original F_i information.

[0132] Subspace attention: In the above dynamic subspace projection step, the feature map F_i has been decomposed into K feature subspaces S_ij that focus on different semantic aspects through the subspace projector SP_i (e.g., S_mid1, which is decomposed from F_mid, focuses on "the main outline and folds of clothing", and S_mid2 focuses on "the shape of the carried items").

[0133] For each such feature subspace S_ij, the system independently applies a lightweight attention module. This attention module is a combination of spatial attention and channel attention. Its role is to perform a "deep focus" on the current specific semantic subspace S_ij to learn and enhance its most discriminative feature representations.

[0134] Continuing with the security example above, for the feature subspace S_mid1 focusing on the "main silhouette and folds of clothing," the attention module applied to it will carefully examine this silhouette and fold information, and may pay particular attention to a "unusually prominent shoulder profile" or "an unusual fold in a specific location," as these details may be crucial for individual identification. Similarly, for the feature subspace S_mid2 focusing on the "shape and general type of carried items," its corresponding attention module will focus on analyzing this information, and may highlight "a unique square patch on a backpack" or "a special handle design on a handbag." By applying this attention individually to each S_ij, the system can extract the most core and noteworthy details from each semantic aspect. After this step, each original subspace S_ij is refined and enhanced, forming a corresponding weighted subspace feature AS_ij. Therefore, we now have a set of "refined" semantic features {AS_i1,AS_i2,...,AS_iK}, where each AS_ij highlights the key information of its corresponding semantic aspect more effectively than the original S_ij.

[0135] Selective Reorganization: After being "refined" by subspace attention, a series of high-quality weighted subspace features {AS_i1,AS_i2,...,AS_iK} are obtained, each focusing on a specific semantic aspect (e.g., the refined "clothing silhouette" feature AS_mid1, the refined "carrying items" feature AS_mid2, etc.).

[0136] This step intelligently integrates information from different semantic perspectives to form a comprehensive and focused final representation F'_i of the current layer feature map F_i. This task is accomplished by a Gated Aggregation Unit (GAU_i). GAU_i learns a weight vector W_i = [w_i1,...,w_iK], where each weight w_ij corresponds to a weighted subspace feature AS_ij. This weight vector W_i acts like a "commander-in-chief," determining, based on the overall situation of the current input image and the importance of each "refined" semantic feature AS_ij, which determines which semantic perspective should have greater "speaking power" in the final combined feature F'_i = Σ(w_ij*AS_ij), (j = 1, 2,..., K).

[0137] Suppose that in the current target person's photo, facial details are very blurry due to lighting conditions, but their clothing and belongings are relatively clear and distinctive. In this case, GAU_mid will learn to assign higher weights w_mid1 and w_mid2 to the attention-enhanced features of "clothing silhouette" AS_mid1 and "belongings" AS_mid2. Conversely, if a semantic aspect of the photo (such as pose feature AS_mid3) becomes unreliable or lacks information due to occlusion, its corresponding weight AS_mid3 may be lower. In this way, GAU_i enables the network to dynamically and selectively aggregate decoupled and attention-enhanced semantic information based on the current specific input information, ensuring that the final output feature F'_i utilizes the most valuable cues to the maximum extent, thus providing the strongest support for subsequent recognition tasks. If all decoupled semantic information is valuable, they may all receive relatively balanced weights.

[0138] Therefore, unlike traditional multi-head self-attention, DHDA performs dynamic and learnable semantic decoupling along the feature channel dimension, and independently applies attention to the decoupled sub-features before intelligently aggregating them. This allows the model to capture and distinguish information at different levels more precisely (e.g., the texture of a pedestrian's clothing, the shape of their belongings, and their overall posture).

[0139] S213: Construct the instance-level and category-level collaborative attention submodule ICCA. This step enhances the discriminativeness of features from two dimensions: instance and category. It is a key step in improving the robustness of the model. Before extracting the final features, the model can simultaneously focus on the instance uniqueness and category commonality of the target and make the two mutually reinforcing. The specific implementation methods include the following instance branch, category branch and collaborative enhancement.

[0140] Instance Branch: First, the features F'_i, i = high (e.g., the target person features that have undergone detailed DHDA analysis) obtained in the previous stage are fed into an "instance branch". The core task of this branch is to learn and identify the subtle and unique features that distinguish this specific target ("this one" target person) from all other different individuals (other pedestrians, other vehicles, etc.). Just like an experienced detective searching for a specific target person "Zhang San" in a large number of images of people, he will try his best to search for those personal markers that are unique to "Zhang San" in his images F'_i. For example, this branch may locate "an inconspicuous mole above Zhang San's left eyebrow" or "his habitual inward turning of his right ankle when he walks" through contrastive learning (learning to distinguish individuals with different IDs during the model training phase) or a discriminant head specifically trained to identify specific instances. After processing by this branch, an "instance attention map" M_inst will be generated. This map is like a highlighted map, clearly marking which regions in the original feature F'_i are crucial for confirming "this is Zhang San, not Li Si or Wang Wu".

[0141] Category Branch: Simultaneously, the feature F'_i is also fed into another parallel "category branch." Unlike the instance branch, which focuses on "who is he / she," the category branch aims to determine "what he / she is," that is, to identify the general category to which the target belongs and its robust features. In security scenarios, this means determining whether the target in front of you is "a pedestrian," "a car," or "a backpack," etc. For example, when analyzing the image F'_i of the target person "Zhang San," this branch will focus on the common structural features that all "pedestrians" should possess, such as "a roughly human-shaped outline with a head, torso, and limbs." This is usually achieved through a category classification head. This branch generates a "category attention map" M_cat, which highlights the areas in F'_i that are most critical for determining the basic category attribute of "this is a pedestrian." This step is crucial because it helps the system first distinguish relevant target categories (such as pedestrians) from complex scenes, thereby excluding trees, buildings, or other non-target objects in the background and avoiding confusion with specific instances.

[0142] Collaborative Enhancement: Based on the instance branch and category branch mentioned above, we have two important pieces of "intelligence": one is M_inst from the instance branch, which points out the unique characteristics of "Zhang San"; the other is M_cat from the category branch, which confirms that "Zhang San" is first and foremost a "pedestrian". The next step of "collaborative enhancement" is to intelligently combine these two pieces of intelligence to form a more comprehensive and accurate judgment.

[0143] The system performs element-wise operations on the instance attention map M_inst and the category attention map M_cat (e.g., multiplying the attention values ​​of corresponding positions in the two maps, or adding them and then normalizing them through an activation function) to generate a "collaborative attention map" M_collab = f(M_inst, M_cat). This M_collab integrates two judgments; its highlighted area simultaneously satisfies "is a pedestrian" and "possesses Zhang San's unique features." This is equivalent to target locking: this is a 'pedestrian' (category confirmation), and based on its 'mole on its left eyebrow' and 'inversion of its right ankle' (instance details), it can be highly certain that it is 'Zhang San'! Finally, the system applies this collaborative attention map M_collab back to the original feature F'_i (through element-wise multiplication: F”_i = F'_i * M_collab). The resulting final feature F”_i is thus effectively "guided," focusing more on key information regions that both conform to category commonalities and reflect individual uniqueness.

[0144] ICCA (Integrated Attention Detection and Coordination) clearly separates and coordinates attention to instance-level and category-level information. This allows the model to capture individual-specific details (such as specific markers and subtle movements) while leveraging category commonalities (such as human shape and vehicle type) to resist interference from changes in lighting, partial occlusion, or viewpoint, thus significantly improving recognition accuracy and robustness when performing Re-ID tasks (such as finding people or vehicles). For example, even if "Zhang San" changes his coat (changing some visual instance features), as long as his core unique biometric features (such as moles and gait) and basic "pedestrian" form remain identifiable, the system can still accurately identify him with a high probability through this coordinated mechanism.

[0145] It can be seen that when extracting features, ICCA learns both the uniqueness of instances (distinguishing between Zhang San and Li Si) and the commonality of categories (both are pedestrians), and enhances both through an attention mechanism, so that the extracted features are both discriminative and robust.

[0146] S214: Construct the Context-Aware Fusion Attention (CAFA) submodule, which is responsible for intelligently fusing features from multiple sources and multiple levels, and dynamically adjusting them according to the real-time scene context, to fuse features from different processing stages.

[0147] Specifically, the features of F'_low, F'_mid after DHDA processing and F”_high after ICCA processing are fused, while considering the contextual information of the input image patch, including:

[0148] Contextual Feature Extraction Step S2141: In security monitoring, the image quality captured by the same camera at different times and under different weather conditions can vary drastically. CAFA's first step is like an experienced monitoring room operator evaluating the overall "shooting environment" or "context" of the current image before analyzing a specific target. It extracts a compact "context descriptor" V_ctx from the original input image patch (or features obtained from earlier stages of the basic feature extraction pathway) through a small convolutional network. This V_ctx is like a brief environmental report, encoding key contextual information such as the overall lighting conditions (e.g., "sufficient light during the day," "insufficient light at night with strong light source interference"), image sharpness (e.g., "clear image," "blurred image due to rain or snow"), and background complexity (e.g., "simple and empty background," "dense and cluttered background").

[0149] Multi-source feature preparation step S2142: After processing by the preceding DHDA and ICCA sub-modules, the system has extracted multi-level and multi-type feature information for the current target of interest (such as a suspicious pedestrian). This information may include: features F'_low from DHDA that focus on low-level texture and details (e.g., fine checkered patterns on clothing); features F'_mid that focus on mid-level components and structures (e.g., "the pedestrian is wearing a dark baseball cap" and "is carrying a rectangular briefcase in his left hand"); and high-level semantic features F"_high that have been enhanced with ICCA to improve instance uniqueness and category commonality. To facilitate subsequent unified processing and fusion, these feature maps (F'_low, F'_mid, and F"_high) from different stages and potentially with different spatial dimensions are converted into fixed-length feature vectors through adaptive pooling, referred to as the feature vector set {V_low, V_mid, V_high_collab}.

[0150] Context-guided gating fusion step S2143: This is the core of CAFA. Based on the current "shooting environment" (V_ctx) perceived in step a, it intelligently determines how to combine the various feature vectors {V_low, V_mid, V_high_collab} prepared in step b to form the most reliable and comprehensive final judgment on the target. Specifically, it includes the following steps S21431-S21433:

[0151] S21431: Concatenate the context descriptor V_ctx with each feature vector V_j to be fused. That is, the system informs each feature vector V_j (i.e., any one of {V_low, V_mid, V_high_collab}) of this "environment report" V_ctx by concatenating V_ctx with each V_j. In this way, each feature, when its importance is evaluated, can "know" under what observation conditions it was extracted.

[0152] S21432: Using a small multilayer perceptron (MLP), the system calculates a "trust level" or "importance" g_j = sigmoid(MLP(concat(V_j,V_ctx))) for each "environmentally aware" feature V_j. The magnitude of this weight g_j is significantly affected by the current environment V_ctx. For example, in a security scenario: if V_ctx indicates that it is "nighttime, extremely poor lighting, and the camera is dirty, causing the image to be very blurry," then for low-level texture features V_low that rely on high resolution (such as clothing patterns), the corresponding gating weight g_low may become very small, indicating that "under such poor conditions, this detailed information is almost unreliable, and reliance on it should be greatly reduced." Conversely, for some relatively more robust high-level semantic features V_high_collab (such as "overall human silhouette and general movement trend"), even in harsh environments, their reliability may still be relatively high, so their gating weight g_high_collab may be assigned a relatively large value. If V_ctx indicates "daytime, bright sunshine, extremely clear image", then g_low may become very large because the detail features are very reliable and highly discriminative at this time.

[0153] S21433: The system weights and sums all these feature vectors V_j according to their respective "trust level" g_j assigned in the current environment, to obtain the final fused feature V_final = Σ(g_j*V_j). This V_final is the optimal and most adaptive comprehensive feature description of the target formation after fully considering the actual observation conditions at the security monitoring site.

[0154] Therefore, the fusion weights in this application are no longer based solely on the features themselves, but are explicitly influenced by the global contextual information of the input image. This mechanism enables the model to dynamically adjust its dependence on information from different sources based on actual conditions (such as lighting, weather, and image quality), just like experienced security personnel, thereby maintaining a high level of analytical capability and judgment accuracy in complex and ever-changing urban security environments.

[0155] S215: Output feature vector, used to pass the features obtained from the context-aware fusion attention submodule through one or more fully connected layers to obtain the final feature vector, which serves as the model's output backend. This transforms complex fused features into a standardized feature vector that can be used for downstream tasks, specifically including:

[0156] S2151: Outputs a standardized feature vector, which, after intelligent fusion using CAFA, yields a feature V_final that integrates multiple information sources and considers the scene context. This feature V_final contains very rich information, but may also have high dimensionality or contain some components that are not the most essential for the final recognition task. Therefore, the system will pass it through one or more fully connected layers (FC layers). This step is similar to the final "editing and condensation" of a detailed but potentially somewhat complex "case investigation report" (V_final), removing redundant information, extracting the most essential and representative core content, and adjusting it to a dimension suitable for efficient subsequent comparison and storage.

[0157] S2152: Output a standardized feature vector and auxiliary understanding information. After the above refinement and dimensional adjustment, the system will output a final, fixed-length feature vector (e.g., depending on specific application requirements, it can be 512-dimensional, 1024-dimensional, or 2048-dimensional, etc.). This feature vector can be regarded as a highly condensed, digital "identity signature" or "feature profile" of the target being analyzed in the current monitoring screen (such as a pedestrian or a vehicle). In the security system, this "identity signature" is crucial because the subsequent target retrieval module 4 can use it to quickly and accurately find other target records with similar "identity signatures" in the vast historical database, thereby realizing key functions such as searching for people and vehicles by image.

[0158] S2153: To enhance the system's transparency and interpretability, in addition to outputting the core feature vectors, the system can selectively output some intermediate results generated during the processing of the preceding innovative sub-modules (such as DHDA, ICCA, and CAFA). These intermediate results may include: attention maps assigned to each subspace in DHDA (revealing which semantic sub-features are given priority), instance attention and category attention maps generated in ICCA (showing which regions reflect individual uniqueness and category commonality), and gating weights calculated for features from different sources in CAFA (reflecting the system's level of trust in different information in a specific scenario).

[0159] S22: Optimize training based on the feature extraction model and set the loss function, specifically including:

[0160] S221: Knowledge transfer initialization and domain adaptation fine-tuning, specifically including:

[0161] Layered pre-trained weights loading: D 2 The basic feature extraction pathway of the CFAN model (see step S211) is initialized with the weights of a mature deep convolutional network pre-trained on a large general image dataset to introduce general visual prior knowledge and accelerate subsequent training convergence.

[0162] End-to-end fine-tuning for security scenarios: the entire D 2 The CFAN model, including core innovative modules such as DHDA, ICCA, and CAFA, will undergo end-to-end fine-tuning on large-scale labeled datasets for specific security tasks (such as pedestrian re-identification and vehicle re-identification).

[0163] S222: Optimize the main identity recognition loss and DHDA hierarchical weighted semantic decoupling orthogonality regularization.

[0164] To comprehensively optimize D 2 To improve the performance of the CFAN model and fully leverage the unique advantages of its core components (DHDA, ICCA), this step includes the main recognition loss as well as innovative auxiliary losses and regularization terms for the semantic decoupling of DHDA and the collaborative mechanism of ICCA. These customized loss terms together constitute the composite loss function L_total.

[0165] The main identity recognition loss (L_ReID) employs an advanced metric learning loss function, such as Triplet Loss, Circle Loss, or ArcFace Loss, to optimize D. 2 The CFAN model ultimately outputs a feature vector V_final (S2.15.b). The core objective of this loss function is to maximize the distance between features of targets with different identities (IDs) in the learned feature space (inter-class separability), while minimizing the distance between features of the same target with different identities in different observation samples (intra-class compactness). The chosen loss function is D. 2 CFAN's final feature output directly endows it with powerful identity discrimination capabilities.

[0166] DHDA Hierarchical Weighted Semantic Decoupling Orthogonality Regularization (L_DHDA_ortho_weighted): To facilitate the learning of semantically independent (orthogonal) and informationally complementary feature subspaces {S_i1,S_i2,...,S_iK} by the dynamic subspace projector SP_i in DHDA, and considering that the decoupling requirements may differ at different semantic levels (low, mid, high), this scheme designs a hierarchically weighted orthogonality regularization term. For each layer feature map F_i (where i∈{low,mid,high}), the K subspace feature maps S_ij obtained by projecting SP_i are first transformed into corresponding feature vectors s through a global average pooling (GAP) operation. ij =GAP(S_ij). This regularization term aims to minimize the cosine similarity between feature vectors in different subspaces within the same layer (i.e., maximize their orthogonality) and allows different importance weights ω to be assigned to decoupling at different levels. i Its regularization formula is:

[0167]

[0168] Where: s ij ·s il Represents vector s ij and s il The dot product. ||s ij ||2 represents vector s ij The L2 norm (i.e., modulus) of . ∈0 is a very small positive constant (e.g., 1 × 10⁻⁶). -8 ω is used to prevent the denominator from being zero and to ensure the stability of numerical calculations. i This is a weighting coefficient assigned to the i-th semantic level (low, mid, high) to emphasize the importance of decoupling. For example, in some tasks, clear decoupling of mid-level semantics may be more critical than that of lower-level semantics; in this case, ω can be assigned... mid Relatively high values. These weights ω i This can be set empirically as a hyperparameter or optimized experimentally. This regularization term penalizes linear correlations between subspaces using squared cosine similarity, encouraging each subspace to capture unique semantic information in F_i, thereby improving the decoupling quality and feature richness of DHDA. Hierarchical weights ω are introduced. i This solution is for D 2 The specific design of CFAN's multi-level features allows model training to focus more on optimizing the decoupling effect of key semantic levels.

[0169] ICCA Instance-Class Joint Confidence Maximization Loss (L_ICCA_jcm): The core objective of ICCA is to coordinate instance uniqueness and class commonality. To more directly strengthen this "coordination" effect and ensure that the model makes correct judgments with high confidence at both levels, this scheme designs the "Instance-Class Joint Confidence Maximization Loss". This loss not only relies on the discriminant losses of the instance branch and the class branch (e.g., the standard cross-entropy losses L_inst and L_cat), but also introduces a term that directly measures and optimizes the joint prediction confidence. Assuming that for input feature F'_i, the instance branch outputs the corresponding correct instance ID... The predicted probability is The category branch output corresponds to the correct macro category. The predicted probability is Define the "joint prediction confidence score" S joint for:

[0170]

[0171] This score is only high when the model makes correct predictions with high confidence for both the instance ID and its macro-class. Based on this score, the joint confidence maximization loss L_ICCA_jcm is designed as follows:

[0172]

[0173] Where E[·] represents the expectation of the training data distribution, and ∈1 is a small positive constant to ensure numerical stability. Therefore, the total loss L_ICCA of ICCA is designed as follows:

[0174] L_ICCA=λ inst L_inst+λ cat L_cat+λ jcm L_ICCA_jcm

[0175] Where λ inst ,λ cat ,λ jcm These are the weighting coefficients for each loss term. The introduction of L_ICCA_jcm is an enhancement to standard multi-task learning. It goes beyond independent optimization of each task, explicitly incentivizing the model to learn the intrinsic consistency and synergistic relationship between instance features and class features. By maximizing joint confidence, the model is guided to a state where its accurate discrimination of instances is based on a correct understanding of their class, and vice versa, thus making ICCA's "synergistic attention" truly meaningful.

[0176] Total loss function (L_total):

[0177] Ultimately used to optimize the entire D2 The total loss function of the CFAN model is a weighted sum of the losses mentioned above:

[0178] L_total=w reid L_ReID+w ortho L_DHDA_ortho_weighted+w icca L_ICCA

[0179] Where w reid ,w ortho ,w icca These are the weight hyperparameters of each loss, which reflect the relative importance of different optimization objectives in the overall training process. They can be determined based on task requirements, model convergence, or through automated hyperparameter search methods (such as Bayesian optimization).

[0180] S223:D 2 CFAN model parameter iterative optimization process.

[0181] In each training iteration, a mini-batch of samples is drawn from the training dataset and fed into D. 2 The CFAN model performs forward propagation, calculating the outputs including the final feature vector. Then, it calculates the total loss for the current batch of data using the composite loss function L_total defined in S222 above. The gradient of this total loss is calculated relative to all learnable parameters of the model (i.e., backpropagation). Finally, optimization algorithms such as AdamW are used to update the model parameters based on the calculated gradients. This iterative process (forward propagation, loss calculation, backpropagation, parameter update) is repeated for multiple training epochs until the model's performance on the validation set reaches the preset convergence criterion or the predetermined number of training epochs is completed. Data augmentation techniques and learning rate scheduling strategies are typically used during training to improve training effectiveness and model generalization ability.

[0182] S23: Preprocess and normalize the trained feature extraction model during the inference stage.

[0183] After training and deployment, the input target image patches are preprocessed to ensure consistency with the training phase and to activate the model's optimal performance.

[0184] a. Target image patch reception and size alignment: Target image patches are received from data access and preprocessing module 1 and adjusted to D size by scaling, cropping, or padding (maintaining aspect ratio). 2 The standard input size (e.g., H×W pixels) required for the basic feature extraction path of the CFAN model.

[0185] b. Pixel value standardization: The pixel values ​​of the resized image blocks are standardized, that is, the pixel values ​​are normalized.

[0186] S24: Feature extraction, forward propagation, and internal dynamic regulation, including:

[0187] S241: Data Input and Layer-by-Layer Feature Forward Calculation: The preprocessed image patch tensor is input into D 2 The CFAN model allows data to flow through the core components defined in S2.1, including the basic feature extraction pathway, DHDA, ICCA, CAFA, and the final feature vector generation layer.

[0188] S242: DHDA Subspace Attention Dynamic Gain Adjustment: In the subspace attention processing step of DHDA, the intensity or concentration of the attention weight maps (e.g., spatial attention map M_s,ij and channel attention map M_c,ij) calculated for each semantic subspace AS_ij is quantized into a "subspace activation quality score" q_ij. For example, q_ij can be the ratio of the maximum activation value to the average activation value in M_s,ij. In the subsequent selective recombination step, the weights w_ij learned by the gated aggregation unit GAU_i can be dynamically modulated by this quality score.

[0189] w'_ij=w_ij·sigmoid(β·(q_ij-q_threshold))

[0190] Here, β is a scaling factor, and q_threshold is an activation threshold. This mechanism allows the more focused and stronger subspaces to receive higher weights during the final feature fusion.

[0191] S243: CAFA Context-Aware Gated Adaptive Sharpening: In the context-guided gated fusion step of CAFA (S2143), the gating weights g_j = sigmoid(MLP(concat(V_j,V_ctx))) generated for each feature vector V_j to be fused, whose internally computed logical value (i.e., the output of MLP, denoted as z_j = MLP(concat(V_j,V_ctx))) can be a learnable or context-determined temperature parameter τ before passing through the sigmoid activation function. j The gate weight g'_j is adjusted by (or a global temperature parameter τ). The adjusted gate weight g'_j can be expressed as: g'_j = sigmoid(z_j / τ) j When the temperature parameter τ jWhen the value is small (e.g., close to 0 but greater than 0), the sigmoid function curve becomes steeper, causing the output g'_j to be closer to 0 or 1. This means the model's decision on whether to adopt the feature source V_j is more "decisive" or "certain." Conversely, when g'_j is large, the sigmoid curve is flatter, and the output weights tend to be more intermediate, indicating a more "soft" or "uncertain" decision. This temperature parameter τ... j It can also be designed as the output of a small network h(·), with the current context descriptor V_ctx as input, i.e., τ j = h(V_ctx). This adaptive sharpening mechanism enables the model to dynamically adjust its "decision-making decisiveness" when fusing different feature sources based on the overall determinism of the scene (e.g., the quality of the image perceived by V_ctx, whether the information is sufficient, etc.), making more confident choices when information is sufficient and more conservative in fusing when information is insufficient.

[0192] S25: Obtain the output feature vector and perform quality assessment and directed propagation.

[0193] D 2 The CFAN model outputs a fixed-length feature vector V_final (S2152) for each input target image patch. This vector is normalized (e.g., L2 normalization) to adapt to subsequent tasks. The output core feature vector V_final, along with the acquired metadata (target ID, spatiotemporal stamp, preliminary category, etc.), are encapsulated together into a structured target feature data record. All records are sent to the spatiotemporal semantic hypergraph indexing module.

[0194] Step S3, constructing the data index includes: based on the deep features and spatiotemporal information of the targets, constructing a spatiotemporal semantic hypergraph index of multiple types of nodes and hyperedges to express the complex relationships between targets, specifically including:

[0195] S31: Construct the vertices and hyperedges of the spatiotemporal semantic hypergraph index;

[0196] S311: Vertices for constructing the spatiotemporal semantic hypergraph index (see also...) Figure 2 These vertices are categorized as follows, collectively forming a basic, hierarchical spatiotemporal overlay (i.e., these vertices are distinguished by "size" and "function," and they are layered, managing target information in the city from coarse to fine):

[0197] A: Macro-space-time block vertex (V) MBlock (): Represents a top-level spatiotemporal unit for coarse-grained division of the entire monitored city or region. For example, dividing the entire city map into large blocks based on time and region, such as "the large area of ​​Chaoyang District on the morning of April 25th". A V MBlockThis represents a "macro-spatiotemporal large area". This is the top layer of the index network, like the entrance to a large container, which can serve as the entry point and initial filtering layer for large-scale spatial and temporal range queries.

[0198] B: Adaptive semantic region vertex VARegion: When a V MBlock When the data density or semantic complexity within V reaches a certain threshold, this V MBlock Instead of simple uniform subdivision, it is divided into one or more VARegions. The subdivision method and boundaries of each VARegion are adaptive, and can be determined based on the spatiotemporal distribution characteristics of its internal data (such as clustering along streets) and D. 2 The semantic distribution of CFAN feature vectors (e.g., forming several semantic clusters) is dynamically determined. For example, in the large area of ​​"Chaoyang District on the morning of April 25th," some places may be particularly busy or have particularly complex situations, such as "Sanlitun Commercial Street during the evening rush hour." VARegion represents such a "key focus area" "intelligently" divided based on actual conditions (how many people, what kind of people / vehicles). Its boundary may not be a perfect square, but rather "outlined" according to the actual situation. VARegion is V MBlock This large area is further divided into smaller, more specific regions. A V MBlock There can be several different VARegions. Defining VARegion elements enables refined and intelligent management of complex regions, making the index structure more closely match the actual data distribution.

[0199] C: Vertex of the basic spacetime unit (V) Cell A VARegion can be further subdivided into the smallest granular, standardized spatiotemporal unit vertices. For example, within the VARegion of "Sanlitun Commercial Street during the evening rush hour," it can be further subdivided into very, very small standard grids, such as "the 10×10 meter area in front of a certain store during the minute from 7 PM to 7:01 PM." Cell This represents a "miniature spacetime grid." This element is the basic container that holds references to specific target instances, namely V. Cell It is the smallest cell within a VARegion, used to precisely "locate" the target.

[0200] D: Target instance vertex V ObjectVObject represents a specific, fully-informed target object instance O passed from feature extraction module 2. VObject is the most atomic vertex in STSGHI, the final point of reference for all queries and analyses. Specifically, this is the actual "person" or "vehicle" being searched for. For example, in the VCell cell above, "10×10 meters in front of the store, 7:00 to 7:01," if the camera captures a "pedestrian A wearing a red hat and a white T-shirt," this snapshot of pedestrian A is a VObject. Object V Object It is the most basic and specific information point in this information network, and it "lives" in a small VCell cell.

[0201] A target instance V Object It appears in a tiny spatiotemporal lattice VCell → this VCell belongs to a region of interest VARegion → this VARegion is located within a macroscopic spatiotemporal block VMBlock. This hierarchical relationship from large to small facilitates rapid spatial and temporal localization.

[0202] S312: Hyperedges (HE) for constructing the spatiotemporal semantic hypergraph index include: spatiotemporal co-occurrence hyperedges, semantic similarity hyperedges, trajectory / event association hyperedges, and region attribute association hyperedges.

[0203] Hyperedges (HEs) are the core of STSGHI for expressing complex multivariate relations. A hyperedge can connect two or any number of vertices (which can be VMBlock, VARegion, VCell, V...). Object (Any combination of these). In layman's terms, the role of hyperedges (HEs) is to connect the "information nodes" (vertices) in a city in various meaningful ways to inform users of a specific "story" or "connection" between them. Hyperedges primarily connect vertices, and they generally do not have direct "relationships" with each other. Instead, they are indirectly connected through the vertices they share. Alternatively, during analysis, multiple types of hyperedges may be considered simultaneously to understand the complete picture of a target. For example, pedestrian A (V... Object A) might be connected by both a HESTC (it appeared with B) and a HETrack (its own trajectory). These two hyperedges are indirectly linked because they both connect to A. In urban security scenarios, the requirement might be to "find entities that both appeared with A (check HESTC) and exhibit behavior similar to a certain pattern (check HETrack)." SemSim This combines different types of hyperedge information. The various types of hyperedges defined in the STSGHI structure are detailed below:

[0204] a. Spacetime Co-occurrence Hyperedge (HE) STC): Connects a set of target instance vertices V that coexist within a specific spatiotemporal neighborhood. Object Or fine-grained unit vertices (V) containing these instances Cell For example, if a camera captures pedestrians A, B, and C at the same intersection within one minute, then there is a traffic violation (HE). STC Take the three Vs: A, B, and C. Object Connecting them indicates that they "appeared at the same time and in the same place." This definition of the hyperedge element helps in the rapid discovery and querying of spatiotemporally associated targets.

[0205] b. Semantic similarity hyperedge (HE) SemSim ): Connect a set of its D 2 CFAN eigenvectors FVec O Target instance vertices (V) exhibiting high semantic similarity Object Even if they are not adjacent in space and time. That is, those that may not appear in the same place or at the same time, but have the same "appearance" (D) 2 Target instances V with very similar CFAN features or behavioral patterns Object Connect them. For example, if the system detects that Zhang San in the east of the city is wearing a distinctive fluorescent green jacket, and Li Si in the west is wearing the exact same jacket, then there is a HE (Hexagram / Hypertext Mark). SemSim Put Zhang San and Li Si as the two V's Object When put together, this indicates that their clothing features are highly similar. This effectively supports pure semantic similarity retrieval across time and space, uncovering potential connections.

[0206] c. Track / Event-Associated Hyperedges (HEEvent / Track): Connect a series of time-dependent V-shaped edges that form a complete motion trajectory. Object (Representing snapshots of the same target at different times), or a group of V objects identified by target retrieval module 4 as participating in the same complex event (such as group gathering or relay relay). Object That is, if a series of V Object HETrack is used to connect the continuous activity trajectories of the same person or vehicle at different points in time. If the target retrieval module 4 finds that several people are involved in a case (e.g., A, B, and C committing a crime together), HEEvent is used to link the V objects of A, B, and C. Object And connect all the vertices of potentially related locations. For example, if camera 1 captures Zhang San's image, then camera 2 captures Zhang San's image 5 minutes later, and then camera 3 captures Zhang San's image 10 minutes later, these three Zhang San's VObjects can be connected by a single VObject. Object Connecting these elements forms his movement trajectory. This hyperedge element supports organizing and querying dynamic target trajectories, expressing higher-order event relationships.

[0207] d: Region attribute associated hyperedge HERegionAttr: Connects a group of VARegions or V regions that share some common region-level attribute or semantic label. MBlock Vertices are regions that share a common "property" (such as multiple VARegions or V vertices). MBlock Connect them. For example, if the system analyzes and finds that three VARegions in a city are all "high-incidence areas of theft", then there will be a HERegionAttr connecting the vertices of these three regions. It supports queries and analysis based on macro-level regional attributes.

[0208] S313: Determine the symbols and attributes of vertices and hyperedges.

[0209] a. General Vertex Attributes (Applicable to VMBlock, VARegion, VCell): ①IDV: Unique identifier of the vertex, i.e., unique number; ②TypeV: Vertex type (MBlock, ARegion, Cell, Object); ③ExtV = (GeoBoundV, TimeBoundV): Geographic boundaries and time range covered by the vertex; ④ChildVertices V :(For non-V Object (Vertex) points to the set of its next-level child vertices; ⑤ ParentVertices V :(For non-V MBlock A set of vertices pointing to their parent vertices at the next higher level (Note: In a hypergraph structure, a lower-level vertex may participate in the formation of multiple higher-level regions, and therefore may have multiple parents); ⑥Stat V :Statistical summary of vertices, including the total number of internal targets (Count) O (V), Main category distribution Dist C at(V), such as Dist C at(V 三里屯傍晚 = {"Pedestrian": 4500, "Car": 800, "Bicycle": 500, "Bus": 50, "Van": 10, ...}

[0210] Count O (V 三里屯傍晚 ) equals the sum of the objectives in the above categories; ⑦SemSum V : Represents the semantic profile of this vertex. Where AllObjects(V) represents the region vertex V (regardless of V) Cell V ARegion Or V MBlock The final recursion includes all original target instances V. ObjectThe set, |AllObjects(V)| represents the total number of target instances in the set (note that this formula is valid when |AllObjects(V)|>0; if there are no targets in the region, SemSum V (The zero vector).

[0211] b. Target instance vertex V Object Attributes (i.e., attributes of object O): ① IDvo, the unique identifier of the specific target instance in our indexing system; ② Locvo, the precise geographical location (longitude, latitude, and possibly altitude) of the target instance when it was captured; ③ Tvo, the precise time point at which the target instance was captured; ④ FVecvo, derived from D... 2 The key feature vector extracted by the CFAN module that describes the "appearance" and "inner qualities" of this target instance.

[0212] c. Hyperedge HE attributes: ①IDHE: Unique identifier of the hyperedge; TypeHE: Hyperedge type (STC, SemSim, Track / Event, RegionAttr); ②Vertices HE ={V a V b ,...,V n}: The set of vertices connected by the hyperedge; ③WeightHE: The strength of the association represented by the hyperedge, i.e., how strong the association is. For example, the "strength of association" is obviously different between two people who just pass by each other and a group that works closely together; ④TimestampHE: The time when the hyperedge association was formed or became effective. When was this "relationship line" established, or within what time period was it effective? ⑤SemSum HE The semantic summary of the hyperedge itself, that is, the semantic profile of this "relationship line" itself. For example, if it is a HE (Head-to-Head) linking multiple "similar clothing" targets... SemSim Its SemSum HE It might just be a general description of this kind of similar clothing. If it's a HE... Event Its SemSum HE It may be the characteristics that describe the core semantics of this event (such as "group fight" or "suspicious transaction").

[0213] S32: Initial construction of the spatiotemporal semantic hypergraph index, continuous entry of data objects and dynamic evolution of the index structure, generation and association of relational hyperedges.

[0214] S321: Initial construction of the index;

[0215] a. Preliminary division of macro-spatial blocks: The system first constructs a set of top-level "macro-spatial block vertices VMBlocks" that logically covers the entire target spatiotemporal range, based on the defined city geographic area or key monitoring area and a predefined time management granularity (e.g., in days or hours). These vertices constitute the basic framework of STSGHI.

[0216] b. Preparation of storage structure: Simultaneously, the system initializes the data structure used to store various vertices and hyperedges in STSGHI.

[0217] S322: Target instance vertex (V) Object The insertion of ) and the dynamic evolution of the basic structure;

[0218] When feature extraction module 2 finishes processing a new target object and extracts its attribute information (including ID)... O ,Loc O ,T O ,FVec O Q O Cat O When the data is passed to this index module 3, the following operations will be performed:

[0219] a. Target instance vertex creation: Generates a corresponding "target instance vertex (V)" for the new target object passed in. Object ), and record all its attributes in full.

[0220] b. Target location within the index hierarchy: based on this V Object The spatiotemporal properties (Loc) O ,T O The system navigates top-down through the STSGH hierarchy (from VMBlock to VARegion), ultimately assigning the target location precisely to a leaf-level "basic spatiotemporal unit vertex VCell". If the VCell corresponding to the target location does not yet exist, it will be dynamically created to contain only this VCell. Object The VCell vertex structure.

[0221] c. Adaptive Region Structure Evolution: When the data density within a certain "macro-spatiotemporal block vertex (VMBlock)" or "adaptive semantic region vertex (VARegion)" (e.g., V...) changes... Object When the number of VCells exceeds a preset threshold (a positive integer greater than 0), or when the semantic complexity SemSumV of the internal data reaches a specific value, an adaptive adjustment mechanism for the vertices in that region will be triggered. This mechanism will comprehensively analyze the spatiotemporal distribution pattern of existing data in this region (such as street-side distribution, regional clustering) and D... 2The semantic aggregation characteristics revealed by CFAN feature vectors are used to re-subdivide the region, generating a new set of finer-grained "adaptive semantic region vertices (VARegions)" or "basic spatiotemporal unit vertices (VCells)". This process ensures that the index structure can dynamically adapt to the actual distribution of data, maintaining query efficiency.

[0222] d. Update of relevant vertex summary information: Whenever a new V is generated... Object When a region is successfully inserted, or when the region hierarchy changes due to adaptive adjustment, the system will update the statistical summary information StatV and semantic profile SemSumV of all affected parent region vertices (VCell, VARegion, VMBlock) from bottom to top to ensure the timeliness and accuracy of these summary data.

[0223] S323: Dynamic generation and association of hyperedge HE.

[0224] Hyperedge generation is the core mechanism of STSGHI for expressing complex relations. Its generation process can be based on predefined rules or driven by other analysis modules.

[0225] a. Spatiotemporal co-occurrence hyperedge generation (HESTC): When multiple "target instance vertices V" are detected... Object "When these target vertices appear simultaneously within a preset short time window ΔT and a small spatial neighborhood ΔD, a HESTC type hyperedge will be automatically created or updated to connect these target vertices with spatiotemporal co-occurrence relationships."

[0226] b. Generation of semantically similar hyperedges using HESemSim: In the new V Object During insertion, techniques such as Locality Sensitive Hash (LSH) are used to modify its D. 2 CFAN feature vector FVecVO and other V in the index Object Efficient batch similarity prediction is performed on the feature vectors. For a set of V features with similarity higher than a preset threshold... Object Even if they are not directly related in space and time, the system will create or add them to the corresponding HESemSim type hyperedge.

[0227] c. Generation of HETrack / Event associated with tracks / events:

[0228] Trajectory integration: Filtering continuous spatiotemporal observation point sequences belonging to the same moving entity (each observation point corresponds to a V). Object Create a superedge of type HETrack and connect them in an ordered manner, and explicitly express the complete motion trajectory of the target in the index.

[0229] Event Correlation: When the target retrieval module 4 identifies a specific complex security event (e.g., coordinated action by multiple individuals, the transfer of a specific item, etc.), it can associate all "target instance vertices V" related to that event. Object The event and potentially key “region vertices” (VCell or VRegion) are associated through one or more HEEvent type hyperedges. The SemSumHE attribute of the hyperedge stores descriptive information such as event type, time, and participant roles.

[0230] d. Generation of HERegionAttr hyperedges associated with regional attributes: By analyzing or externally annotating, if multiple different “region vertices” (e.g., multiple VARegions) are identified to share a certain specific macro-attribute or label (e.g., all are classified as “high-risk regions”, or their “regional semantic profiles SemSumV” show similar specific patterns), a HERegionAttr type hyperedge can be created to connect these region vertices to support macro-analysis based on common regional attributes.

[0231] Step S4, data query, includes: receiving user multimodal query information, using Hypergraph index for candidate region filtering, feature matching, and relationship inference, and finally outputting high-confidence target retrieval results, specifically including:

[0232] S41: Query intent parsing and multimodal input processing;

[0233] This step is responsible for receiving and understanding the user's diverse query requirements and transforming them into an internal representation that the system can process.

[0234] S411: Receive user query requests: The system is designed with a flexible query interface to adapt to various retrieval scenarios in urban security operations.

[0235] ① Image query: The user provides an image containing the target (such as the target person or the target vehicle).

[0236] ② Text description query: Users enter a natural language description of the target, such as "Looking for a black SUV without license plates that passed through the intersection of Zhongshan East Road and Jiefang Road between 10 pm and 11 pm last night, with obvious scratches on the roof".

[0237] ③ Spatiotemporal range query: Users specify a specific time period and geographical area, such as "retrieve all pedestrians carrying large backpacks who appeared near Gate 1 of X Park between 9:00 AM and 9:30 AM on May 24, 2025".

[0238] ④ Target ID Query: Users provide a known target instance vertex ID (IDvo) to query detailed information about the target, associated targets, or historical trajectories.

[0239] ⑤ Composite query: Any combination of the above query methods, such as "Provide a picture of a target person, and limit the last time of their appearance to within the last 24 hours, and the activity area to area Y".

[0240] S412: Query Preprocessing and Feature Extraction

[0241] Image query processing: If the query input is an image, then call "based on D". 2 CFAN's feature extraction module 2 performs the same preprocessing and forward propagation on the query image as S23 and S24, extracting its high-discrimination D. 2 CFAN feature vector FVec_query.

[0242] Text description processing: If the query input is text, key entities (e.g., "black SUV", "backpack"), attributes (e.g., "no license plate", "large"), time (e.g., "last night at 10 pm"), location (e.g., "Zhongshan East Road"), and behaviors (e.g., "passed by") are extracted through named entity recognition. Further semantic analysis is used to structure this information, and some semantic information (e.g., appearance description) can be mapped to related terms. 2 A semantic vector space compatible with the CFAN feature space, or transformed into constraints on the target attributes.

[0243] Spatiotemporal information standardization: All time and space descriptions in the query are uniformly converted into the standard format and coordinate system defined in S14.

[0244] S413: Generate a unified query object: Encapsulate all parsed and extracted information into a unified "query object". This object contains: the D of the query image. 2 The CFAN feature vector FVec_query, structured semantic attribute constraints (such as target category, color, carrying objects, etc.), standardized spatiotemporal range constraints, and other possible query parameters (such as the expected number of results, ranking preferences, etc.).

[0245] S42: STSGHI hierarchical initial screening and candidate region localization;

[0246] This step utilizes STSGHI's hierarchical structure and regional summary information to quickly narrow down the search scope and improve subsequent matching efficiency.

[0247] S421: Based on query spatiotemporal constraints, quickly filter VMBlock and VARegion vertices in STSGHI:

[0248] The system first utilizes the explicit "time range" and "geographical boundary" information in the query object.

[0249] By querying the ExtV attributes (i.e., GeoBoundV, TimeBoundV) of macro-spatiotemporal block vertices (VMBlock) and adaptive semantic region vertices (VARegion) in STSGHI, VMBlocks and VARegions that intersect with the queried spatiotemporal range are efficiently filtered out. This step leverages the hierarchical nature of STSGHI, filtering from the top level and avoiding a full scan of massive amounts of underlying data.

[0250] S422: Intelligent Pruning of Candidate Regions Based on Region Semantic Profiling (SemSumV):

[0251] For candidate VMBlocks and VARegions after spatiotemporal filtering, if the query object contains explicit semantic information (e.g., "finding areas where suspicious people gather" extracted from text descriptions, or "vehicles" being the main category of the target in an image query), then the semantic profiles of these region vertices, SemSumV, can be used for intelligent pruning.

[0252] Calculate the similarity or matching degree between the query semantics and the region SemSumV. For example, if the query is "suspicious clustering", prioritize regions in SemSumV that show "multiple targets appearing at high density in a short period of time" or contain specific risk semantic labels. If the query is "vehicles", prioritize regions in SemSumV that have a high proportion of vehicle targets or contain semantics such as "parking lot" or "major intersection".

[0253] By introducing semantic prediction at the region level, irrelevant regions can be eliminated earlier, further focusing on high-potential regions and significantly improving the accuracy and efficiency of retrieval.

[0254] S43: Fine-grained target matching and initial candidate V Object Set generation;

[0255] Within the initial candidate region, more refined target matching is performed.

[0256] S431: Locate V within the candidate VARegion / VCell Object Within the candidate VARegion determined by S42, the system further navigates to the basic spatiotemporal unit vertices (VCells) they contain. Then, it traverses all target instance vertices (V...) stored in these VCells. Object () reference.

[0257] S432: Based on D 2 Core semantic similarity matching is performed using CFAN feature vectors (FVecVO).

[0258] This is the core matching step. If the query object contains a query feature vector FVec_query (from an image query or text semantic transformation), then calculate the relationship between FVec_query and each V currently traversed. Object The semantic similarity between the feature vectors FVecVO is calculated using the cosine similarity formula.

[0259] Set a dynamic or preset similarity threshold. Only V values ​​with a similarity higher than this threshold will be considered. Object Only then is it considered a potential match. (D) 2 The high discriminative power of the CFAN model ensures that this deep semantic-based matching can effectively address the challenges of complex security scenarios such as lighting, pose, and partial occlusion.

[0260] S433: Perform precise filtering by combining other target attributes (such as preliminary category and precise spatiotemporal stamp):

[0261] For V that passed the feature similarity matching Object Further validation can be performed using other precise constraints from the query object. For example, check V Object Whether the initial category label is consistent with the query requirements, and whether its precise time stamp (Tvo) strictly falls within the query time range, etc.

[0262] After this step, an "initial match V" is obtained. Object "Set", these V Object Both the features and attributes are directly related to the query request.

[0263] S44: Association Expansion and Inference Retrieval Based on STSGHI Hyperedges;

[0264] This step is the most innovative part of this module. It goes beyond simple direct matching and uses various hyperedges defined in STSGHI to discover deeper and more multi-dimensional relationships between targets.

[0265] S441: Activation and Initial Matching V Object Associated hyperedges: for the "initial matching V" generated by S43 Object Each V in the set Object The system searches STSGHI for all direct matches of that V. Object Connected hyperedge instances. These hyperedges include: spatiotemporal co-occurrence hyperedges (HESTC), semantic similarity hyperedges (HESemSim), trajectory / event association hyperedges (HETrack / Event), and region attribute association hyperedges (HERegionAttr) that may be indirectly associated through their respective VCell / VARegion.

[0266] S442: Spatiotemporal Co-occurrence Extension (Using HESTC to Discover Accompanying Targets):

[0267] If a user's query intent involves finding other targets that appeared at the same time and place as a certain target (e.g., "find everyone who entered the building with this target"), the system will focus on analyzing the initial match V. Object HESTC.

[0268] By traversing the other V connected to these HESTCs Object Vertices can be added to the result set to represent targets with spatiotemporal co-occurrence relationships. The WeightHE property of hyperedges can be used to evaluate the strength or confidence of co-occurrence relationships.

[0269] S443: Semantic Similarity Extension (Discovering Spatiotemporally Similar Targets Using HESemSim):

[0270] If a user wants to find other target instances that are highly similar to the query target in appearance, behavior patterns, etc., but may appear at different times and locations (e.g., "find all vehicles in the city that have similar characteristics to this target vehicle"), the system will use HESemSim.

[0271] HESemSim directly connects to D 2 CFAN features are highly similar to V Object These hyperedges can be used to quickly recall targets that are visually or semantically homologous to or highly related to the query target, even if they are spatially or temporally separated.

[0272] S444: Trajectory / Event Association Extension and Reasoning (Using HETrack to track target trajectories; using HEEvent to identify event participants):

[0273] Trajectory tracking: If the initial match V Object If a target belongs to a specific HETrack (representing the trajectory of the same target), the system can trace forward or backward along that HETrack to find all V values ​​of that target at other time points. Object A snapshot is taken to fully reveal its activity trajectory.

[0274] Event participant mining: If the initial match is V Object If a user participates in a HEEvent (representing a complex security event), the system can use that HEEvent to find other users who also participated in the event. Object (Other people, vehicles, items) and related key region vertices. The SemSumHE property of a hyperedge can provide a semantic description of the event.

[0275] S445: Regional Attribute Association Extension (using HERegionAttr combined with common regional attributes for retrieval):

[0276] If the query involves macroscopic attributes of a region (e.g., "find all pedestrians loitering at night within VARegions marked as 'high-risk theft areas'"), the system can first find all VARegion vertices matching the "high-risk theft area" attribute using HERegionAttr, then limit the search scope to these regions, and finally perform target matching based on other conditions. This achieves linked queries from macroscopic regional characteristics to specific microscopic targets.

[0277] S45: Perform deduplication, fusion, and confidence assessment on direct matching and expanded results.

[0278] Directly match V from S43 Object V is obtained by combining the set with various hyperedge extensions from S44 Object Merge the sets and remove duplicate V. Object .

[0279] For each final candidate V Object A comprehensive confidence score is calculated, which is composed of the original detection confidence, the degree of matching with the query, and the association strength (WeightHE) of the associated hyperedges it participates in. The final confidence score formula is designed as follows:

[0280] Score_final(V)=Conf_detect(V)*MaxPathScore(V,Q)

[0281] Conf_detect(V) represents V Object The confidence score of V when it is initially detected in data access and preprocessing module 1; MaxPathScore(V,Q)(V Object V (maximum path association score relative to query Q) represents the strength of the path with the highest score among all association paths from user query Q to target V (Note: the result V obtained before deduplication is...). Object The collection may contain duplicate target objects, which are derived from the target objects of the query endpoint along different paths from the objects initially related to the query starting point:

[0282] MaxPathScore(V,Q)=maxPathj∈AllPaths(Q→V)[ScorePathj(Q→V)]

[0283] AllPaths(Q→V): The set of all associated paths from query Q to V; ScorePathj(Q→V) (Score of a single path Pathj):

[0284] ScorePathj(Q→V)=CalcSimilarity(FVecS,...,FVecV)*WeightHE

[0285] Where CalcSimilarity(FVecS,...,FVecV) represents the first V found that directly matches Q in the current path (S is the first V found that directly matches Q). Object D represents all objects traversed along the path from S to the query endpoint V. 2 Similarity score of CFAN feature vectors; WeightHE represents the association strength value of the hyperedge of the current path dependency.

[0286] Please see Figure 3 Based on the above embodiments, this application also provides a target retrieval module, including: a data access and preprocessing module 1, a feature extraction module 2, an index construction module 3, and a target retrieval module 4, wherein:

[0287] The data access and preprocessing module 1 is used to acquire multi-source data, perform cleaning, formatting, spatiotemporal standardization, and target detection and cropping of images / videos to generate structured target objects;

[0288] Feature extraction module 2 is used to extract deep semantic feature vectors with discriminative and adaptable characteristics to complex scenes, as well as information to aid understanding from the target image;

[0289] The index building module 3 is used to construct a spatiotemporal semantic hypergraph index with multiple types of nodes and hyperedges based on the deep features and spatiotemporal information of the target, in order to express the complex relationships between targets;

[0290] The target retrieval module 4 is used to receive multimodal query information from users, use the hypergraph index to perform candidate region filtering, feature matching and relation inference, and finally output high-confidence target retrieval results.

[0291] This application embodiment also provides a storage medium storing a computer program, which is executed by a processor to perform the target retrieval method as described above.

[0292] This application also provides an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform any of the target retrieval methods described above.

Claims

1. A target retrieval method, characterized in that, Includes the following steps: Step S1: Data access and preprocessing, including acquiring multi-source data, cleaning, formatting, and spatiotemporal standardization, and performing target detection and cropping on images / videos to generate structured target objects; Step S2: Feature extraction, which includes extracting discriminative and adaptable deep semantic feature vectors and auxiliary understanding information from the target image, specifically including: S21: Based on D 2 The CFAN model constructs a feature extraction model; S22: Optimize training based on the feature extraction model and set the loss function; S23: Preprocess and normalize the trained feature extraction model during the inference stage; S24: Feature extraction, forward propagation, and internal dynamic regulation; S25: Obtain the output feature vector and perform quality assessment and directional propagation; Step S3: Construct a data index, including building a spatiotemporal semantic hypergraph index with multiple types of nodes and hyperedges based on the deep features and spatiotemporal information of the targets, to express the complex relationships between targets. Specifically, this includes: S31: Construct the vertices and hyperedges of the spatiotemporal semantic hypergraph index, specifically: S311: The vertices used to construct the spatiotemporal semantic hypergraph index include: macro-spatiotemporal block vertices, basic spatiotemporal unit vertices, and target instance vertices; S312: The hyperedges for constructing the spatiotemporal semantic hypergraph index include: spatiotemporal co-occurrence hyperedges, semantic similarity hyperedges, trajectory / event association hyperedges, and region attribute association hyperedges; S313: Determine the symbols and attributes of vertices and hyperedges; S32: Initial construction of the spatiotemporal semantic hypergraph index, continuous entry of data objects and dynamic evolution of the index structure, generation and association of relational hyperedges; Step S4: Data query, which includes receiving multimodal query information from users, using Hypergraph index to perform candidate region filtering, feature matching and relationship inference, and finally outputting high-confidence target retrieval results.

2. The target retrieval method as described in claim 1, characterized in that, Step S21: based on D 2 The CFAN model constructs a feature extraction model, which includes: S211: Basic feature extraction and multi-level semantic representation construction, used to extract different depths of the pathway and obtain feature maps that can represent semantic information at different levels of abstraction; S212: Construct a dynamic hierarchical decoupled attention submodule and apply attention to each semantic subgroup to dynamically decouple the feature channels of the feature map at each level into multiple semantic subgroups; S213: Construct an instance-level and category-level collaborative attention submodule to allow the model to simultaneously focus on the instance uniqueness and category commonality of the target before extracting the final features; S214: Construct a context-aware fusion attention submodule to fuse features from different processing stages; S215: Output feature vector, which is used to pass the features obtained from the context-aware fusion attention submodule through one or more fully connected layers to obtain the final feature vector.

3. The target retrieval method as described in claim 1, characterized in that, Step S22: Optimize and train the feature extraction model and set the loss function, specifically including: S221: Knowledge Transfer Initialization and Domain Adaptability Fine-tuning; S222: Optimize the main identity recognition loss and DHDA hierarchical weighted semantic decoupling orthogonality regularization, the calculation formula is as follows: ； Where: low, mid, and high represent the bottom, middle, and high semantic levels, respectively, and K is the number of subspaces split from a single-layer feature projection. Representing vectors and dot product, Representing vectors L2 norm, Representing vectors L2 norm, For positive integers, It is the decoupling importance weight coefficient set for the i-th semantic level; S223: D 2 CFAN model parameter iterative optimization process.

4. The target retrieval method as described in claim 1, characterized in that, Step S32 includes: S321: Initial construction of the index; S322: Insertion of target instance vertices and dynamic evolution of the basic structure; S323: Dynamic generation and association of hyperedges.

5. The target retrieval method as described in claim 1, characterized in that, Step S4, data query includes: S41: Query intent parsing and multimodal input processing; S42: STSGHI hierarchical initial screening and candidate region localization; S43: Fine-grained target matching and initial candidate V Object Set generation; S44: Association Expansion and Inference Retrieval Based on STSGHI Hyperedges; S45: Perform deduplication, fusion, and confidence assessment on direct matching and expanded results.

6. The target retrieval method as described in claim 1, characterized in that the steps are as follows: S1: Data access and preprocessing steps include: S11: Data source interface configuration and connection establishment; S12: Raw data acquisition and preliminary analysis; S13: Data cleaning and quality assessment; S14: Spatiotemporal information extraction and standardization; S15: Preliminary target detection; S16: Data encapsulation and output.

7. A target retrieval system, applied to any one of the target retrieval methods of claims 1-6, characterized in that, include: The module includes a data access and preprocessing module, a feature extraction module, an index building module, and a target retrieval module, among which: The data access and preprocessing module is used to acquire multi-source data, perform cleaning, formatting, and spatiotemporal standardization, and perform target detection and cropping on images / videos to generate structured target objects; The feature extraction module is used to extract discriminative and adaptable deep semantic feature vectors and auxiliary understanding information from the target image, specifically including: Based on D 2 CFAN model for feature extraction; Optimize training based on the feature extraction model and define the loss function; The trained feature extraction model is preprocessed and normalized during the inference stage; Feature extraction, forward propagation, and internal dynamic regulation; The output feature vector is obtained, and quality assessment and directional propagation are performed. The index building module is used to construct a spatiotemporal semantic hypergraph index with multiple types of nodes and hyperedges based on the deep features and spatiotemporal information of the targets, in order to express the complex relationships between targets. Specifically, it includes: The vertices and hyperedges of the spatiotemporal semantic hypergraph index are constructed as follows: The vertices used to construct the spatiotemporal semantic hypergraph index include: macro-spatiotemporal block vertices, basic spatiotemporal unit vertices, and target instance vertices; The hyperedges used to construct the spatiotemporal semantic hypergraph index include: spatiotemporal co-occurrence hyperedges, semantic similarity hyperedges, trajectory / event association hyperedges, and region attribute association hyperedges; Determine the symbols and attributes of vertices and hyperedges; The initial construction of the spatiotemporal semantic hypergraph index, the continuous entry of data objects and the dynamic evolution of the index structure, and the generation and association of relational hyperedges; The target retrieval module receives multimodal query information from users, uses the Hypergraph index to perform candidate region filtering, feature matching and relation inference, and finally outputs high-confidence target retrieval results.

Images