A distance constraint-based video spatio-temporal quantization range retrieval method and system

Abstract

The present application relates to the technical field of video retrieval, and especially relates to a video space-time quantization range retrieval method and system based on distance constraint. The method comprises the following steps: acquiring video data; performing data preprocessing on the acquired video data; constructing a space-time joint index fusing distance constraint based on the preprocessed data; performing space-time range retrieval based on the constructed space-time joint index fusing distance constraint; and performing object consistency judgment and result output based on the retrieval result. The present application quantitatively represents the fine-grained space-time relationship of video objects in the same video or multiple videos, converts the space-time sensitive video data query semantics into accurate quantitative range queries in the time and space dimensions of the video objects, i.e. the spatial range queries among the same intra-frame video objects and the continuous time length range queries on the continuous video frames.

Description

Technical Field

[0001] This invention relates to the field of video retrieval technology, and in particular to a video spatiotemporal quantization range retrieval method and system based on distance constraints. Background Technology

[0002] With the widespread adoption of surveillance cameras, mobile devices, and internet applications, the scale and content of video data have grown dramatically. How to quickly and accurately retrieve data from massive amounts of video data has become an urgent need in the field of video big data applications. Unlike text and image data, video data contains rich spatiotemporal information. In many application scenarios, user needs have shifted from simply identifying "what objects appear" to focusing on quantifiable, fine-grained, and precise three-dimensional spatiotemporal relationships between video objects. For example, in the scenario reproduction and data mining of autonomous driving, researchers may need to sift through millions of kilometers of road test videos to identify all dangerous overtaking scenarios where "the distance between the vehicle and the vehicle in front is less than 0.5 meters, and this proximity lasts for more than 5 seconds," in order to train and validate avoidance algorithms. In the field of public safety, investigators may need to find surveillance footage showing "a person wearing specific clothing (such as a man in a black T-shirt) loitering near a specific vehicle (such as a white sedan), with a distance of less than 2 meters for more than 1 minute," in order to lock in key clues.

[0003] This invention abstracts the above query into a video spatiotemporal range query. The essence of the problem lies in the fact that it is not a simple pattern matching problem, but rather a problem requiring efficient range verification of arbitrarily quantified geometric constraints across continuous spatiotemporal dimensions. Specifically, it requires efficiently retrieving video segments from massive amounts of video that simultaneously satisfy the following two joint constraints: 1) Spatial dimension: Within each frame of the segment, the real physical distance between video objects with specific semantic labels must strictly satisfy a user-defined proximity constraint; 2) Temporal dimension: The same group of video objects satisfying the above spatial proximity constraint must appear continuously in a continuous image sequence, and the duration of this sequence must at least satisfy a lower time limit.

[0004] The technical prerequisite for achieving this type of video spatiotemporal range query based on real physical distance is the ability to obtain the three-dimensional spatial location of video objects. With the integration and development of deep learning and multi-view geometry technologies, it has become possible to automatically recover the real physical scale of a scene from uncalibrated monocular videos. This allows us to reliably calculate the real physical distance between any objects in a video, providing a data foundation for accurate geometric relationship retrieval.

[0005] However, existing video retrieval methods have the following problems when solving the problem of spatiotemporal range queries: (1) Cross-modal video retrieval methods mainly align high-level semantic concepts and embed the entire video segment or frame into a shared feature space, but they usually discard the precise geometric information required to verify the quantifiable distance in the spatiotemporal range query. These methods usually match the entire video, resulting in the processing of a large amount of video content that is irrelevant to the specific query; (2) Retrieval methods based on structured data processing mainly focus on filtering video object categories, quantities, etc. through lightweight models, but it is difficult to support the quantitative query of video spatiotemporal relationships. Some structured retrieval methods consider the accompanying appearance of video objects or the spatiotemporal evolution relationship between video objects, but mainly focus on specific query patterns, and the complexity is high when processing spatiotemporal range queries; (3) Existing methods based on graph structure mining (such as frequent subgraphs and relation graphs) need to pre-mine all possible object combinations when facing massive video data, resulting in high retrieval costs and difficulty in supporting range queries with arbitrary spatiotemporal constraints. Summary of the Invention

[0006] To address the challenges of existing technologies in efficiently and accurately determining the temporal and spatial proximity of video data across diverse scenarios, and the lack of indexes applicable to specific video spatiotemporal ranges, this invention provides a video spatiotemporal quantization range retrieval method and system based on distance constraints.

[0007] In a first aspect, the present invention provides a video spatiotemporal quantization range retrieval method based on distance constraints, which adopts the following technical solution:

[0008] A video spatiotemporal quantization range retrieval method based on distance constraints includes:

[0009] Acquire video data;

[0010] The acquired video data undergoes data preprocessing.

[0011] A spatiotemporal joint index with fusion distance constraints is constructed based on the preprocessed data;

[0012] Spatiotemporal range retrieval is performed based on a spatiotemporal joint index with fusion distance constraints.

[0013] Based on the search results, determine the consistency of objects and output the results.

[0014] Secondly, a video spatiotemporal quantization range retrieval system based on distance constraints includes:

[0015] The data acquisition module is configured to acquire video data;

[0016] The preprocessing module is configured to preprocess the acquired video data.

[0017] The indexing module is configured to construct a spatiotemporal joint index with fused distance constraints based on the preprocessed data;

[0018] The retrieval module is configured to perform spatiotemporal range retrieval based on a spatiotemporal joint index that incorporates distance constraints.

[0019] The output module is configured to perform object consistency judgment and output the results based on the search results.

[0020] Thirdly, the present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the aforementioned distance-constrained video spatiotemporal quantization range retrieval method.

[0021] Fourthly, the present invention provides a terminal device, including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to provide the distance-constrained video spatiotemporal quantization range retrieval method.

[0022] In summary, the present invention has the following beneficial technical effects:

[0023] 1. This invention transforms the semantics of spatiotemporally sensitive video data queries into precise quantitative range queries on both time and space dimensions of video objects by quantifying the fine-grained spatiotemporal relationships of video objects in the same or multiple videos. Specifically, it queries the spatial range between video objects within the same frame and the duration range of consecutive video frames.

[0024] 2. To address spatial proximity relationships in massive video data, this invention proposes a distance-constrained R-tree (DRR-tree) spatial index. Unlike traditional R-trees that focus on node capacity, DRR-trees do not limit the number of data elements within a node, but rather restrict the maximum spatial distance between data elements within a node. By utilizing the distance boundaries of nodes, DRR-trees provide effective range retrieval for spatial distance constraints of different granularities.

[0025] 3. This invention designs a joint range retrieval strategy based on both time and space dimensions, and proposes a spatiotemporal joint index that includes a DRR-tree. It uses temporal range coarse screening to filter invalid time intervals that do not meet the time continuity constraint. Based on this, it combines the DRR-tree to perform spatial range retrieval on time segments that have passed the temporal range screening, thus avoiding the calculation of spatial distances for a large number of invalid windows and video objects. Attached Figure Description

[0026] Figure 1 A flowchart of the method provided by the present invention;

[0027] Figure 2 This is a certain time window of the present invention. A comparative diagram showing the compression processing of all video objects within five consecutive frames based on a preset position offset threshold, where (a) is the original video object data point distribution diagram of the window; and (b) is the video object data point distribution diagram of the window after compression processing based on a position offset threshold of 0.7.

[0028] Figure 3 For the time window of this invention Distribution diagram of the two-dimensional grid of internal video objects;

[0029] Figure 4 This invention is based on a time window A schematic diagram of the distance-constrained R-tree spatial nodes of the video object after internal compression;

[0030] Figure 5 This invention is based on a time window A schematic diagram of the distance-constrained R-tree structure constructed from the video objects after internal compression;

[0031] Figure 6 This is a comparison chart of the overhead of constructing distance-constrained R-trees under different datasets according to the present invention, where (a) is a comparison chart of the time overhead of constructing distance-constrained R-trees; and (b) is a comparison chart of the memory overhead of constructing distance-constrained R-trees.

[0032] Figure 7 This is a comparison chart of the overhead of constructing different ablation index structures under different datasets in this invention, where (a) is a comparison chart of index construction time overhead; and (b) is a comparison chart of index construction memory overhead.

[0033] Figure 8 Comparison charts of the construction costs of distance-constrained R-trees for different parameters on different datasets are presented. (a) is a comparison chart of the impact of the object change threshold of window partitioning on index construction time; (b) is a comparison chart of the impact of the magnification factor of the minimum boundary rectangle distance threshold of nodes on index construction time; (c) is a comparison chart of the impact of the video object position compression threshold on index construction time; and (d) is a comparison chart of the impact of the leaf node size threshold on index construction time.

[0034] Figure 9Comparison charts of the construction costs of distance-constrained R-trees for different parameters under different datasets are presented. (a) is a comparison chart of the impact of the object change threshold of window partitioning on query time; (b) is a comparison chart of the impact of the magnification factor of the minimum boundary rectangle size threshold of nodes on query time; (c) is a comparison chart of the impact of spatial distance radius on query time; and (d) is a comparison chart of the impact of leaf node size threshold on query time. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to the accompanying drawings.

[0036] Example 1

[0037] Reference Figure 1 This embodiment of a video spatiotemporal quantization range retrieval method based on distance constraints includes:

[0038] The process includes the following four main steps: 1) video data extraction; 2) construction of a spatiotemporal joint index with distance constraints; 3) spatiotemporal range retrieval; and 4) object consistency judgment and result output. The specific technical solutions are as follows:

[0039] S1 Video Data Extraction: Monocular Vision Geometric Mapping and Video Data Quantization Representation, which involves parsing the input monocular video and generating quantization information of video objects using object detection; introducing a monocular camera calibration model to establish a mapping relationship between the image pixel coordinate system and the real-world coordinate system, converting the pixel positions of video objects into three-dimensional world coordinates with physical measurement significance, and generating structured metadata containing video object identifiers, semantic attributes, and physical coordinates.

[0040] S1.1 processes the input video frame by frame, using the YOLOv8 object detection model to obtain the unique identifier, semantic attributes (such as vehicle category, color, etc.) and two-dimensional pixel centroid coordinates on the image plane for all video objects.

[0041] S1.2 Let 𝒪 represent all video objects with unique identifiers. For each frame... We can obtain a collection of objects. .set up This is the set of all semantic attribute labels. Video objects are represented as triples. ,in A unique identifier representing a video object. These are the three-dimensional pixel coordinates of the video object. Pixels are transformed through mapping, therefore the real-world ground plane is... , It is the set of related attributes possessed by the video object;

[0042] S1.3 Employs DeepCalib, a monocular automatic calibration model based on deep neural networks, to infer the camera's geometric parameters from the initial video frames and obtain the camera's pitch angle. Camera height Camera focal length and the location of the main point ;

[0043] S1.4 Using the recovered camera parameters, the pixel coordinates of the video objects are... Projected onto the real-world coordinate system (assuming the ground plane is...) Obtain the 3D pixel coordinates of the video object;

[0044] Furthermore, the specific steps in S1.4 include:

[0045] S1.41 returns the original pixel coordinates. Relative to the camera principal point coordinates predicted by the model Perform a translation to obtain the normalized coordinates. :

[0046] ,

[0047] S1.42 Based on the normalized vertical coordinates Camera focal length predicted by the model Calculate the relative pitch angle between the ray of light from the camera's optical center to that pixel and the camera's principal optical axis. :

[0048] ,

[0049] S1.43 The above relative pitch angle The original pitch angle of the camera relative to the ground predicted by the model. Adding them together gives the angle between the ray of light from the camera's optical center to that pixel and the real-world ground. :

[0050] ,

[0051] S1.44 Utilizes triangular relationships, combined with camera height This allows you to obtain the depth coordinates of an object in the world coordinate system. (i.e., forward distance). Then, based on the principle of similar triangles, using depth... ,focal length and normalized horizontal coordinates Calculate the horizontal position of the video object This allows us to obtain the object's position in the world coordinate system. :

[0052] ,

[0053] S1.5 Since the true positions of video objects obtained based on monocular geometric reconstruction are all located on the ground plane ( Therefore, the real spatial location of video objects can be uniformly mapped to two-dimensional coordinate vectors on the ground plane. Within this plane, the spatial coordinates of points on the ground plane are uniformly represented as two-dimensional column vectors. Any two video objects... and The physical distance between them can be expressed by the L2 norm of their difference vector in two-dimensional Euclidean space, that is:

[0054] ,

[0055] S2 Spatial-Temporal Joint Index Construction with Distance Constraints: Based on quantized video object metadata, a spatio-temporal joint index with distance constraints is constructed. Specifically, this includes: adaptively dividing continuous videos into several physically stable temporal intervals based on the missing rate of the video object sequence; constructing a global attribute mapping from semantic attribute labels to video object identifiers and their duration in each temporal interval to support temporal interval range filtering based on time thresholds; and simultaneously constructing a distance-constrained R-tree (DRR-tree) in each local temporal interval. This spatial index achieves hierarchical management of the continuous physical positional relationships of video objects by limiting the maximum physical distance between objects within a tree node to no more than a preset threshold.

[0056] S2.1 divides a continuous video stream into several independent local time intervals, or time windows, with the division criteria designed to ensure that the video object sequences contained within each time window have a high degree of similarity.

[0057] Furthermore, the specific steps in S2.1 include:

[0058] S2.11 Initialize the current time window index Set time window The starting frame is the current frame to be processed in the video stream. ;

[0059] S2.12 for the start frame Extract the set of all video objects that have been detected and assigned a unique identifier in step S1. ;

[0060] S2.13 Record The total number of video objects included is .like If there is an empty frame, then skip that frame and proceed accordingly. As a new starting frame;

[0061] S2.14 Read subsequent video frames one by one according to the video time sequence Get the current frame The set of all detected video object identifiers and the total number of video objects it contains ;

[0062] S2.15 Calculate the current frame First, for the starting frame The main video object set that still exists And count their quantity :

[0063] ,

[0064] ,

[0065] S2.16 Calculate the video object missing rate of the current frame relative to the starting frame. :

[0066] .

[0067] S2.17 Set object change threshold ( ),like The system determines that the core video object within the current time window maintains a high presence rate, and the main subject of the scene remains unchanged. The current frame... Return to window And continue processing the next frame;

[0068] S2.18 If If it is determined that the core video object has been significantly missing (e.g., most of the original car has moved out of the video frame), and the current window can no longer effectively represent the continuous behavior between video objects within the set, then the current time window ends. Its included frame range is , will the current frame Set as the next time window new start frame Repeat steps S2.14-S2.18 until the video ends;

[0069] S2.2 Based on the above time window partitioning, construct a global temporal mapping index for semantic attributes;

[0070] Furthermore, the specific steps in S2.2 include:

[0071] S2.21 Decompose the semantic attributes of video objects. Establish logical association paths from each type of semantic tag to another type of semantic tag. For the same video object class with multiple semantic attribute classes, associate each of them with the end of the corresponding semantic attribute combination path;

[0072] S2.22 For each indexed video object, establish a mapping to the video frame in which it appears;

[0073] S2.23 Based on the time partition boundaries determined in S2.11, project all video frame numbers to the corresponding local time windows;

[0074] S2.3 Construct a distance-restricted R-tree (DRR-tree) for spatial range retrieval within local time intervals;

[0075] Furthermore, the specific steps in S2.3 include:

[0076] S2.31 Compresses the continuous physical spatial location information of all video objects within each time window into several location-stable spatiotemporal state data points;

[0077] Furthermore, the specific steps in S2.31 include:

[0078] S2.311 Retrieve all video objects within the time window. Each video frame within the window The corresponding real-world coordinate vector (the video object is projected onto the ground plane, so it is omitted) ):

[0079] ,

[0080] S2.312 abstracts the steady state of a video object over a continuous time interval as a triple structure. The triple represents the video object. In the time series interval It appears continuously within the space, and its spatial position remains stable. Within its neighborhood, it can be used to represent;

[0081] S2.313 in the time window Within, each video object Starting frame Corresponding spatial position vector As a baseline, subsequent video frames are traversed sequentially. Get the spatial location vector of the current frame. The physical distance offset of the current frame relative to the reference frame is calculated using the L2 norm of the difference vector in two-dimensional Euclidean space. :

[0082] ,

[0083] S2.314 Set position offset threshold According to the offset and Based on the size relationship, the following logic is performed for each video object to determine its state update:

[0084] ,

[0085] If offset Less than or equal to If the spatial position of a video object is determined to be relatively stable, meaning the deviation of the video object's position from the starting point in the current frame does not exceed the allowable error limit of the range query, then the current spatial position state of the video object is maintained, and subsequent frame numbers are merged into the time interval of the current spatiotemporal state. ,Right now ;

[0086] If offset Greater than The video object was determined to have undergone significant physical displacement, and the current representative spatiotemporal state tuple could no longer accurately describe its position. As the new starting frame, with its spatial location as the new reference, initialize the next spatiotemporal state tuple. The original spacetime state ends at frame 1. ;

[0087] S2.32 Construct a DRR-tree from bottom to top based on the compressed video object data elements within the time window;

[0088] Furthermore, the specific steps in S2.32 include:

[0089] S2.321 Define the set of levels of the DRR-tree For each level Set physical distance constraint threshold The threshold starts from the lowest leaf node layer. Initially, it increases with increasing level. The level threshold follows an increasing pattern, and the calculation formula is:

[0090] ( )

[0091] in The initial physical distance threshold for the leaf node layer. It is a preset hierarchical scaling factor used to control the spatial coverage of upper-level nodes.

[0092] S2.322 Traversing the Time Window Calculate the boundary extent of the global two-dimensional projection space for all video object data after internal compression. That is, to determine the maximum and minimum values ​​of all video object data elements in real-world coordinates (ground plane as the coordinate system). ):

[0093] ,

[0094] S2.323 Within the two-dimensional ground projection space Within this grid, a uniform two-dimensional mesh is established, where the side length of each cell is set to be equal to the actual physical distance threshold of the leaf node layer. If the grid is in The number of units in the direction is ,exist The number of units in the direction is ,but:

[0095] ,

[0096] Each grid cell is at coordinates ( The corresponding spatial interval on ) for:

[0097] ,

[0098] S2.324 Iterate through all data elements, based on each video object. Real-world coordinates Calculate its grid index position :

[0099] ,

[0100] Add the data element to the corresponding grid cell. In the list of elements, establish a mapping from grid cells to the set of data elements they contain;

[0101] S2.325 For each video object currently being processed Determine the grid cell in which it is located. Since the side length of the grid cell is less than or equal to the distance threshold All neighboring points must lie within the 3x3 region formed by the cell and its 8 neighboring cells. All data elements within this 3x3 region constitute... candidate neighbor set :

[0102] ,

[0103] S2.326 From candidate neighbor set Find one or more of the following: A subset of effective neighboring points. Each subset The Euclidean distance between any two video objects must be less than or equal to the distance threshold. ,Right now;

[0104] ,

[0105] but A valid set of neighboring points is a data point that can belong to multiple different sets of valid neighboring points.

[0106] S2.327 For each set of valid neighboring points Calculate the minimum bounding rectangle (MBR) that exactly encloses all data elements within it:

[0107] ,

[0108] The MBR, together with a list of references pointing to the video objects it contains, forms a leaf node;

[0109] S2.328 Obtain the leaf node layer After all MBRs are completed, calculate the geometric center for each MBR:

[0110] ,

[0111] According to the center point The coordinates are sorted by MBR for the leaf nodes, and a linked list structure is built in ascending order.

[0112] S2.329 will be hierarchical The set of geometric center points of all generated MBRs at the next higher level Input data;

[0113] S2.3210 Based on the hierarchical magnification factor and the leaf node layer distance threshold Determine the distance threshold for the current level. The grid classification logic of S2.323-S2.326 is reused, but the original video object coordinates are replaced with the geometric center point coordinates of the lower-level node MBR, and the grid edge length and clustering constraint threshold are updated. , will the hierarchy The actual physical distance between the center points of the internal MBR nodes does not exceed Group the MBR sets;

[0114] S2.3211 For each grouped set of MBRs, compute a new, larger MBR that completely encloses all MBRs in that set as the parent node. The parent node MBR will serve as... An internal node of a layer, and stores pointers to its contained nodes. The references to the layer MBR (i.e., its child nodes) form a tree-like hierarchical relationship;

[0115] S2.3212 Repeat the recursive process of steps S2.329-S2.3211 to generate higher-level internal MBR nodes. When the recursion reaches a certain level, if all MBRs generated at that level can be completely covered by a single parent MBR under the constraint of a higher-level distance threshold, then this single parent MBR becomes the root node of the entire DRR-tree, covering the real-world spatial extent of all objects within that time window, and the tree construction ends. The root node's MBR will cover the real-world spatial extent of all objects within that video window;

[0116] S3 Spatiotemporal Range Retrieval: This process parses the semantic attribute set, physical distance threshold, and time duration threshold from the user's query request, and performs hierarchical range retrieval based on the spatiotemporal joint index framework built in S2. This process performs a joint retrieval from both temporal and spatial dimensions, using a global attribute index to perform temporal range retrieval, filtering video objects with insufficient duration and invalid time windows. Within the valid time window DRR-tree, it uses node distance boundaries to retrieve combinations of video objects that satisfy spatial proximity constraints within the physical distance threshold range from bottom to top.

[0117] S3.1 Parse the semantic attribute set and physical distance threshold in the user query request. and time duration threshold Based on the spatiotemporal joint index framework, without accessing specific spatiotemporal coordinate data, it retrieves video objects that satisfy a specific set of semantic attributes and the time window in which the duration of the video frame sequence in which they appear meets the time duration threshold.

[0118] Furthermore, the specific implementation of S3.1 is as follows:

[0119] S3.11 Obtain the set of semantic attributes given by the user, and retrieve all video objects that meet the corresponding attribute conditions at once by calling the global semantic attribute index. and The query is represented as ;

[0120] S3.12 For and Each video object in The corresponding ordered set of video frames is formed by accessing the corresponding video frames based on the index. In each video object ordered set of frames It is then divided into several subsequences with consecutive internal frame numbers, based on a user-defined time duration threshold. (Unit: seconds, need to be divided by frame rate when calculating) (Converted to frame rate), filtering for the longest consecutive occurrence duration not less than valid video object set and :

[0121] ,

[0122] ,

[0123] Based on the selected set of valid video objects and Extract all valid video frames corresponding to it and map them to their respective time windows, each window The set of frames covered is denoted as Add them to the corresponding window collections respectively and :

[0124] ,

[0125] ,

[0126] S3.13 For these two types of video objects and Intersection operation is performed on the corresponding window sets ( ), thus obtaining the final set of candidate windows. Each window must contain at least one video object from each video object category;

[0127] S3.2 Within each selected valid time window, a spatial range query is performed using the corresponding DRR-tree index. That is, a video object class with a certain semantic attribute is used as the spatial anchor point, and the physical distance threshold set by the user is used as the query radius. The distance threshold of the tree node is used to quickly retrieve the associated video objects with specific semantic attributes that fall within the spatial range.

[0128] Furthermore, the specific implementation of S3.2 is as follows:

[0129] S3.21 For each candidate time window In the window On the DRR-tree, using one type of video object Centered on, to query distance Search for video objects within the range from bottom to top using the radius. ;

[0130] Furthermore, the specific steps in S3.21 include:

[0131] S3.211 From and Extract from window The subset of video objects appearing within are denoted as follows: and ;

[0132] S3.212 For Each video object within By utilizing the ordered linked list structure of the leaf nodes of the DRR-tree, the video object containing it can be quickly located. The leaf node MBR;

[0133] S3.213 For each leaf node MBR, if it intersects with the query range, i.e., its own threshold... If so, the MBR is considered a partial overlay. It is necessary to examine each video object within the MBR. Calculate it one by one with The precise distance, and will satisfy Video object composition and its co-occurrence frame set Add to the corresponding partial result set of the window :

[0134] ,

[0135] S3.214 If an accessed MBR has its own threshold If so, the MBR is considered fully covered. This means... With all within this MBR Video object The distances are all necessarily less than There's no need to perform point-by-point distance calculations anymore. Now, simply connect these video objects to... The combinations and their co-occurring frame sequences are added to the corresponding partial result set of the window. And expand upwards to access its parent node:

[0136] ,

[0137] S3.215 When a node's MBR is determined to be fully covered, the search will proceed upwards to its parent node to check if its sibling nodes may also be covered. If the parent node's threshold is greater than... Then, it is necessary to search downwards from the parent node for all partially covered child nodes. If the threshold of the parent node is less than or equal to... If a condition is met, then all its child nodes are fully covered. This process is repeated recursively until all fully covered and partially covered MBRs that satisfy the conditions are found and added to the partial result set. ;

[0138] S4 Object Consistency Judgment and Output: Performs temporal consistency verification on video objects scattered in different time intervals and non-continuous frames. Finally, it verifies whether the total duration of the spliced ​​segment strictly meets the user-defined time lower limit and outputs a continuous video result segment that perfectly matches semantics and spatiotemporal features.

[0139] S4.1 Assign all local results of all windows to unique video object pairs as identifiers. Perform grouping and aggregation. For each pair... The frame sequences that satisfy spatial constraints in different windows are merged into a complete, unordered set of frames. :

[0140] ,

[0141] S4.2 For each object pair From the merged set of frames, extract all consecutive frame intervals;

[0142] S4.3 Check each consecutive frame interval The duration, if it is not less than the time threshold required for the query. If the interval is true, then the interval is considered a valid final result and is added to the final result set. :

[0143] ,

[0144] S4.4 The continuous video graph sequence that has passed all spatiotemporal constraints is converted into dynamic video and returned to the user as the final retrieval result of the query.

[0145] Experimental verification

[0146] The experiments for this invention were conducted on a computer equipped with a 4.5GHz CPU and 16GB of memory. All algorithms were implemented in Python. We used YOLOv8 for object detection.

[0147] Datasets. This evaluation includes three real-world datasets: De-trac, Deep Drive, and ONCE. To mitigate the impact of errors in calculating physical distances using monocular camera self-calibration techniques on the accuracy of query results, this invention uses pixel distance measurements on De-trac and Deep Drive to validate the proposed method, and further validates it using real-world metrics on 3D annotations in ONCE. We created four video streams for experimental evaluation: for the De-trac dataset, videos were taken from the training and test sets and concatenated into two long videos, drtrain and drtest, respectively. For the Deep Drive dataset, videos were taken from the test set to form two independent long videos, bdd100kA and bdd100kB. De-trac uses object detection to obtain quantized data, while Deep Drive uses annotated data (ground values). Detailed statistics and parameter configurations are shown in Tables 1 and 2. Unless otherwise specified, the following evaluations involving these parameters will use default values.

[0148] Table 1. Dataset Statistics

[0149]

[0150] Table 2 Parameter Settings

[0151]

[0152] Benchmark algorithms: STAR, SketchQI, and EVADB. Our main implementation employs a window-based DRR-tree index, referred to as W-DRR-Tree. To validate this strategy, we include two ablation variants: a separate DRR-tree constructed for each frame, denoted as SF-DRR-Tree, and an indexless method that pre-computes pairwise spatial distances for each frame and stores them in an array, denoted as NO-DRR-Tree.

[0153] The basic idea of ​​the STAR algorithm is to achieve video data ranking and retrieval by combining spatial and temporal constraints. STAR treats videos as multi-dimensional data objects containing temporal and spatial labels. By comprehensively modeling the spatial location and temporal intervals of video segments, it achieves video query ranking subject to spatial and temporal constraints. The algorithm process is as follows: It establishes a spatiotemporal index structure for the video data and organizes the objects extracted from the video frames and their spatiotemporal attributes using graph modeling; then, in the query phase, it filters candidate video segments based on the spatial region and temporal interval specified by the user; finally, it uses a scoring function based on relevance and spatiotemporal constraints to rank the candidate segments and outputs the optimal video retrieval results that satisfy the spatial and temporal constraints.

[0154] The basic idea of ​​the SketchQL algorithm is to achieve semantic video clip retrieval based on sketches through a visual query interface. The SketchQL system uses deep neural networks to extract semantic visual features from video frames and constructs a temporal index of the videos. During the query phase, the user-input sketch or visual example is transformed into a vector representation in the same feature space. Then, the system identifies candidate video clips that are closest to the query features through similarity matching. Finally, the results are optimized and sorted through temporal constraints and a scoring function to return the video moment that best matches the user's visual intent.

[0155] The basic idea behind the EVADB algorithm is to accelerate video analysis by embedding AI model inference into the database system, combining symbolic and materialized approaches. EVADB parses AI inference operations (such as object detection and scene recognition) involved in SQL queries into logical operators. Then, the optimizer selects the optimal execution plan based on a cost model, including model inference, caching, and materialized view reuse strategies. Finally, during the execution phase, EVADB leverages the database engine and deep learning inference operators to collaboratively execute queries and reduces redundant computation by materializing intermediate results, thereby accelerating the exploratory video analysis process.

[0156] 1. Query performance evaluation

[0157] In this embodiment, we first compare our method with a benchmark algorithm to demonstrate its superiority in query performance. Then, we evaluate the impact of the relevant parameters used in our method on query performance and the impact of different index constructions on queries.

[0158] Comparison with baseline algorithms. Table 3 shows the query processing time of our method versus the baseline methods on all datasets. Our method significantly outperforms STAR and SketchQL by three to four orders of magnitude. While STAR indexes spatial relations, its sliding window method repeatedly processes overlapping frames. SketchQL's trajectory-centric index is also redundant for spatiotemporal range queries, as it only requires immediate spatial relations per frame, not complete trajectories. Compared to EVADB, our method is one to two orders of magnitude faster because EVADB relies on comprehensive pairwise distance calculations in relational joins.

[0159] Table 3. Benchmark Algorithm Query Time (seconds)

[0160]

[0161] Ablational assessment of indicator construction costs. We first evaluated the cost of constructing a DRR-tree on different datasets. For example... Figure 6 The overhead of constructing each window of the DRR-tree shown depends primarily on the object density. The dense ONCE dataset exhibits the highest time and memory costs, while the sparser bdd100k dataset has significantly lower costs. The average object duration is a secondary factor. The longer average duration in drtrain enhances our spatiotemporal compression capabilities, offsetting its higher density compared to bdd100k, resulting in similar costs.

[0162] Next, we compare the performance of index building under different strategies, such as... Figure 7 As shown, compared to SF-DRR-Tree per frame, W-DRR-Tree reduces construction time and memory usage by an order of magnitude. This demonstrates experimentally that increasing index granularity to the window level is an effective and reasonable strategy to minimize construction overhead. The NO-DRR-Tree method, by forgoing index construction and employing simple data storage, naturally incurs the lowest overhead.

[0163] Parameter sensitivity analysis. Figure 4 Key parameters were displayed. , , and Impact on DRR-tree construction time. To better compare parameter changes across different datasets, we conducted experiments and analyses on four datasets using pixel distance: drteat, drtrain, bdd100kA, and bdd100kB. Lower... This generates fewer but denser windows, resulting in frequent tree-building overhead. Conversely, higher... This results in fewer windows being generated but with excessively long spans, leading to a large number of video objects within a single window. Excessive displacement means that even after spatiotemporal compression, a significant amount of data remains, thus significantly increasing the construction and maintenance costs of a single DRR-tree and increasing index management overhead. Figure 8 (a)). Smaller This will generate deeper trees, thus incurring inter-layer construction costs. This increases... This cost can be reduced, but excessively large values ​​will amplify the threshold of upper-level nodes. In this case, the cost of partitioning nodes within the layer dominates, thereby increasing the construction time. Figure 8 (b)). Small The high build time is due to the generation of too many leaf nodes, which increases overhead. With... Increasing the number of nodes reduces the tree's depth, thus lowering the time cost. Figure 8 (c)). Larger Allows merging more object instances, reducing input cardinality and decreasing build time. Figure 8 (d)).

[0164] We analyzed the key parameters. , , and Impact on query performance. For example... Figure 9 As shown in (a), Lower values ​​generate fewer but overly complex windows, thus increasing the search time per window. Conversely, An excessively high value will result in too many windows and lead to higher costs for merging across windows. For example... Figure 9 As shown in (b), Values ​​that are too small or too large can cause the tree to be too deep or too flat, thus impairing traversal and pruning efficiency. For example... Figure 9 As shown in (c), the query time varies. The radius increases, which is because increasing the radius increases the number of candidate MBRs that must be visited in the tree. For example... Figure 9 As shown in (d), A value that is too small will increase the number of tree nodes, thus increasing traversal overhead. Conversely, If the value is too large, it will lead to an excessively large inter-layer threshold, reducing the granularity of supported queries and weakening the pruning ability.

[0165] Example 2

[0166] This embodiment provides a video spatiotemporal quantization range retrieval system based on distance constraints, including:

[0167] The data acquisition module is configured to acquire video data;

[0168] The preprocessing module is configured to preprocess the acquired video data.

[0169] The indexing module is configured to construct a spatiotemporal joint index with fused distance constraints based on the preprocessed data;

[0170] The retrieval module is configured to perform spatiotemporal range retrieval based on a spatiotemporal joint index that incorporates distance constraints.

[0171] The output module is configured to perform object consistency judgment and output the results based on the search results.

[0172] A computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the aforementioned distance-constrained video spatiotemporal quantization range retrieval method.

[0173] A terminal device includes a processor and a computer-readable storage medium, the processor being configured to implement various instructions; the computer-readable storage medium being configured to store multiple instructions adapted for loading and execution by the processor of the aforementioned distance-constrained video spatiotemporal quantization range retrieval method.

[0174] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A video spatiotemporal quantization range retrieval method based on distance constraints, characterized in that, include: Acquire video data; The acquired video data undergoes data preprocessing. A spatiotemporal joint index with fusion distance constraints is constructed based on the preprocessed data; Spatiotemporal range retrieval is performed based on a spatiotemporal joint index with fusion distance constraints. Based on the search results, determine object consistency and output the results; The construction of a spatiotemporal joint index with fusion distance constraints based on preprocessed data includes dividing the continuous video stream into several independent local temporal intervals as time windows, and first initializing the current time window index. Set time window The starting frame is the current frame to be processed in the video stream. For the starting frame Extract the collection of all video objects assigned unique identifiers. ;Record The total number of video objects included is ,like If there is an empty frame, skip that frame and proceed accordingly. As the new starting frame; read subsequent video frames one by one according to the video sequence. Get the current frame The set of all detected video object identifiers and the total number of video objects it contains ; Calculate the current frame First, for the starting frame The main set of video objects And count their quantity : ; ; Calculate the video object missing rate of the current frame relative to the starting frame. : Set object change threshold ( ),like If the core video object within the current time window is determined to maintain a high presence rate and the main scene remains unchanged, then the current frame will be... Return to window And continue processing the next frame; like The core video object has been determined to be significantly missing, and the current time window ends. Its included frame range is , will the current frame Set as the next time window new start frame Repeat the entire process from setting a new starting frame to determining the termination of the window until the video ends; The construction of a spatiotemporal joint index based on preprocessed data with fusion distance constraints also includes constructing a global temporal mapping index of semantic attributes based on time window partitioning. First, the semantic attributes of video objects are decomposed, and a logical association path is established between each type of semantic tag and another type of semantic tag. For the same video object class with multiple semantic attribute classes, it is associated with the end of its respective semantic attribute combination path. For each indexed video object, a mapping is established to its corresponding video frame, associating each video object with the video frame in which it appears. Based on the determined time partition boundaries, all video frame numbers are projected to the corresponding local time window. Then, a distance-constrained R-tree for spatial range retrieval within a local time interval is constructed. Specifically, this includes compressing the continuous physical spatial location information of all video objects within each time window into several location-stable spatiotemporal state data points. Within the time window, all video objects are retrieved. Each video frame within the window The corresponding real-world coordinate vector: The steady state of a video object over a continuous period of time is abstractly represented as a triple structure. ; within the time window Within, each video object Starting frame Corresponding spatial position vector As a baseline, subsequent video frames are traversed sequentially. Get the spatial location vector of the current frame. The physical distance offset of the current frame relative to the reference frame is calculated using the L2 norm of the difference vector in two-dimensional Euclidean space. : Set the position offset threshold. According to the offset and Based on the size relationship, the following logic is performed for each video object to determine its state update: ， If offset Less than or equal to If the spatial position of a video object is determined to be relatively stable, the current spatial position of the video object is maintained, and subsequent frame numbers are merged into the time interval of the current spatiotemporal state. ,Right now If the offset Greater than To determine if a video object has undergone significant physical displacement, in order to As the new starting frame, with its spatial location as the new reference, initialize the next spatiotemporal state tuple. The original spacetime state ends at frame 1. ; The spatiotemporal joint index based on the constructed fusion distance constraint performs spatiotemporal range retrieval, including parsing the semantic attribute set and physical distance threshold in the user query request. and time duration threshold Based on a spatiotemporal joint index framework, without accessing specific spatiotemporal coordinate data, this method retrieves video objects that satisfy a specific set of semantic attributes and the time windows in which the duration of their video frame sequences meets a duration threshold. Specifically, it obtains the user-given set of semantic attributes and retrieves all video objects that satisfy the corresponding attribute conditions at once by calling the global semantic attribute index. and The query is represented as ;for and Each video object in The corresponding ordered set of video frames is formed by accessing the corresponding video frames based on the index. In each video object ordered set of frames It is divided into several subsequences with consecutive internal frame numbers; based on the user-defined time duration threshold. Filter out those with a maximum consecutive occurrence duration of not less than valid video object set and : ; Based on the selected set of valid video objects and Extract all valid video frames corresponding to it and map them to their respective time windows, each window The set of frames covered is denoted as Add them to the corresponding window collections respectively and : ， For these two types of video objects and Intersection operation is performed on the corresponding window sets ( ), thus obtaining the final set of candidate windows. Each window must contain at least one video object from each video object class.

2. The video spatiotemporal quantization range retrieval method based on distance constraints according to claim 1, characterized in that, The acquired video data undergoes preprocessing, including frame-by-frame processing of the input video. The YOLOv8 object detection model is used to obtain the unique identifiers, semantic attributes, and two-dimensional pixel centroid coordinates of all video objects on the image plane. The DeepCalib monocular automatic calibration model, based on a deep neural network, is employed to infer the camera's geometric parameters from the initial video frames, yielding the camera's pitch angle. Camera height Camera focal length and the location of the main point Using the recovered camera parameters, the pixel coordinates of the video objects are... Projecting onto a real-world coordinate system yields the 3D pixel coordinates of the video object; specifically, the original pixel coordinates are... Relative to the camera principal point coordinates predicted by the model Perform a translation to obtain the normalized coordinates. : Based on the normalized vertical coordinates Camera focal length predicted by the model Calculate the relative pitch angle between the ray of light from the camera's optical center to that pixel and the camera's principal optical axis. : The relative pitch angle The original pitch angle of the camera relative to the ground predicted by the model. Adding them together gives the angle of the intersection point between the ray of light from the camera's optical center to the pixel and the real-world ground. : By utilizing triangular relationships and combining camera height Obtain the depth coordinates of the object in the world coordinate system. Then, based on the principle of similar triangles, using depth... ,focal length and normalized horizontal coordinates Calculate the horizontal position of the video object This allows us to obtain the object's position in the world coordinate system. : Finally, the spatial coordinates of the ground plane are uniformly represented as two-dimensional column vectors, allowing any two video objects to be represented as such. and The physical distance between them is expressed by the L2 norm of their difference vector in two-dimensional Euclidean space, i.e.: .

3. The video spatiotemporal quantization range retrieval method based on distance constraints according to claim 2, characterized in that, The construction of the distance-constrained R-tree for spatial range retrieval within a local time interval also includes constructing a DRR-tree from bottom to top based on compressed video object data elements within a time window. The specific steps include: defining the hierarchy set of the DRR-tree. For each level Set physical distance constraint threshold The threshold starts from the lowest leaf node layer. Initially, the threshold increases with the level, and the level threshold follows an increasing pattern. The calculation formula is as follows: ( )in The initial physical distance threshold for the leaf node layer. It is a preset level magnification factor; iterates through the time window. Calculate the boundary extent of the global two-dimensional projection space for all video object data after internal compression. That is, to determine the maximum and minimum values ​​of all video object data elements in real-world coordinates: ; within the two-dimensional ground projection space Within the grid, a uniform two-dimensional mesh is established, and the side length of each cell is set to be equal to the threshold of the actual physical distance of the leaf node layer. If the grid is in The number of units in the direction is ,exist The number of units in the direction is ,but: Each grid cell is in coordinates ( The corresponding spatial interval on ) for: ; Traverse all data elements, based on each video object Real-world coordinates Calculate its grid index position : And add the data elements to the corresponding grid cells. In the list of elements, establish a mapping from grid cells to the set of data elements they contain; for each video object currently being processed. Determine the grid cell in which it is located. Based on the side length of the grid cell less than or equal to the distance threshold All neighboring points are located within a 3x3 area formed by the cell and its 8 neighboring cells. All data elements within this 3x3 area constitute... candidate neighbor set : ;from candidate neighbor set Find the one containing The effective nearest neighbor subset, each subset The Euclidean distance between any two video objects within the range is less than or equal to a distance threshold. ,Right now; ,but For each valid set of neighboring points; Calculate the minimum bounding rectangle (MBR) that exactly encloses all data elements within it: The MBR, together with a list of references pointing to the video objects it contains, forms a leaf node.

4. The video spatiotemporal quantization range retrieval method based on distance constraints according to claim 3, characterized in that, The construction of the distance-constrained R-tree for spatial range retrieval within local time intervals also includes obtaining the leaf node layer. After all MBRs are completed, calculate the geometric center for each MBR: According to the center point The coordinates are sorted using the MBR for the leaf nodes, and a linked list structure is built in ascending order; the hierarchy is then... The set of geometric center points of all generated MBRs at the next higher level Input data; based on the hierarchical magnification factor and leaf node layer distance threshold. Determine the distance threshold for the current level. , will the hierarchy The actual physical distance between the center points of the internal MBR nodes does not exceed The MBR set is grouped; for each grouped MBR set, a new MBR that completely surrounds all MBRs in the set is calculated as the upper-level node, and the upper-level node MBR is used as... An internal node of a layer, and stores pointers to its contained nodes. The references to the layer MBRs form a tree-like hierarchical relationship. Finally, the recursive process is repeated to generate higher-level internal MBR nodes. When the recursion reaches a point where all the MBRs generated at a certain level are completely covered by a single parent MBR under the constraint of a higher-level distance threshold, then the single parent MBR is the root node of the entire DRR-tree, covering the real-world spatial range of all objects within that time window, and the construction of the tree ends.

5. The video spatiotemporal quantization range retrieval method based on distance constraints according to claim 4, characterized in that, The spatiotemporal range retrieval based on the constructed spatiotemporal joint index with fusion distance constraints also includes performing a spatial range query within each selected valid time window using the corresponding DRR-tree index. This query uses a specific semantic attribute video object class as the spatial anchor point, a user-defined physical distance threshold as the query radius, and utilizes the distance threshold boundaries of tree nodes to quickly retrieve associated video objects with specific semantic attributes falling within that spatial range. Specifically, for each candidate time window... In the window On the DRR-tree, using one type of video object Centered on, to query distance Search for video objects within the range from bottom to top using the radius. ;from and Extract from window The subset of video objects appearing within are denoted as follows: and ;for Each video object within By utilizing the ordered linked list structure of the leaf nodes of the DRR-tree, the video object containing it can be quickly located. The leaf node MBR; for each leaf node MBR, if it intersects with the query range, i.e., its own threshold. Then the MBR is considered a partial overlay, covering each video object within the MBR. Calculate it one by one with The precise distance, and will satisfy Video object composition and its co-occurrence frame set Add to the corresponding partial result set of the window : If an accessed MBR has its own threshold In this case, the MBR is considered a complete overlay, and these video objects are directly compared with... The combinations and their co-occurring frame sequences are added to the corresponding partial result set of the window. And expand upwards to access its parent node: ; By recursively querying, until all results within the fully covered and partially covered MBRs that meet the conditions are found, they are added to the partial result set. .

6. The video spatiotemporal quantization range retrieval method based on distance constraints according to claim 5, characterized in that, The process of determining object consistency and outputting results based on the search results includes verifying the temporal consistency of video objects scattered across different time intervals and non-continuous frames, ultimately checking whether the total duration of the spliced ​​segment meets the user-defined lower time limit, and outputting a continuous video result segment that perfectly matches semantics and spatiotemporal features. Specifically, all local results of all windows are identified by a unique video object pair identifier. Perform grouping and aggregation for each pair The frame sequences that satisfy spatial constraints in different windows are merged into a complete, unordered set of frames. : For each object pair From the merged frame set, extract all consecutive frame intervals; examine each consecutive frame interval. The duration, if it is not less than the time threshold required for the query. If the interval is true, then the interval is considered a valid final result and is added to the final result set. : The continuous video graph sequence that has passed all spatiotemporal constraints will be converted into dynamic video and returned to the user as the final retrieval result of the query.

7. A video spatiotemporal quantization range retrieval system based on distance constraints, executing the video spatiotemporal quantization range retrieval method based on distance constraints as described in claim 1, characterized in that, include: The data acquisition module is configured to acquire video data; The preprocessing module is configured to preprocess the acquired video data. The indexing module is configured to construct a spatiotemporal joint index with fused distance constraints based on the preprocessed data; The retrieval module is configured to perform spatiotemporal range retrieval based on a spatiotemporal joint index that incorporates distance constraints. The output module is configured to perform object consistency judgment and output the results based on the search results.

Images