Interactive video target space-time retrieval system based on visual similarity

Abstract

The invention provides an interactive video target space-time retrieval system based on visual similarity, which comprises a video management module for carrying out structured management on video files and related information, a target detection module for carrying out target detection, a feature extraction module for extracting visual feature vectors from detected targets, and a retrieval module for retrieving the visual feature vectors of the detected targets. The vector storage module is used for storing visual feature vectors and related information of a target to obtain a target library, the interactive display module is used for retrieving the target and carrying out manual confirmation according to a user instruction, and the path calculation module is used for calculating a target action range in a given time interval. And the vector retrieval module is used for retrieving in the target library and returning a retrieved target list. According to the system, space-time retrieval can be carried out on targets such as pedestrians and vehicles in mass video monitoring data, and investigators are assisted to quickly construct motion tracks of the targets such as the pedestrians and the vehicles.

Description

Technical Field

[0001] The present invention relates to a video object spatiotemporal retrieval system, in particular to an interactive video object spatiotemporal retrieval system based on visual similarity. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] With the widespread application of high-definition video surveillance equipment and technology in the field of public safety, massive amounts of high-definition videos have played a significant role. However, although traditional video surveillance and intelligent analysis technologies have solved the problems of video storage and playback, they are still insufficient in the rapid retrieval and rapid discovery of suspicious targets. They still need to rely on a large number of staff to rely on manpower to analyze massive amounts of video, which is labor-intensive and labor-intensive. At the same time, it is inevitable that some fleeting important information will be missed due to fatigue and negligence.

[0004] With the rapid development of technologies such as artificial intelligence and deep learning, artificial intelligence technology can effectively improve target detection, feature extraction, similarity comparison, etc. in videos. However, the relevant results are only applicable to the comparison of small scenes and small batches of data, and cannot meet the needs of pedestrian and vehicle target trajectory analysis in massive video surveillance data in urban scenes. In actual engineering applications, system functions can only be achieved through the stacking of a large amount of hardware, which requires large amounts of calculations, is time-consuming, and costly.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide an interactive video object spatiotemporal retrieval system based on visual similarity in response to the shortcomings of the existing technology.

[0007] In order to solve the above technical problems, the present invention discloses an interactive video object spatiotemporal retrieval system based on visual similarity, the system comprising:

[0008] Video management module, target detection module, feature extraction module, vector storage module, path calculation module, vector retrieval module and interactive display module; wherein,

[0009] The video management module performs structured management on video files and related information;

[0010] The target detection module performs target detection on the video content to obtain the image area of the target;

[0011] The feature extraction module extracts visual feature vectors from the detected target;

[0012] The vector storage module stores the visual feature vector of the target and the relevant information of the corresponding video frame to obtain the target library;

[0013] The interactive display module retrieves the target and performs manual confirmation by iteratively calling the path calculation module and the vector retrieval module according to the user's instructions;

[0014] The path calculation module calculates the target's movement range in a given time interval by combining geographic information data and the target's speed parameters;

[0015] The vector search module searches the target library based on the target action range and user instructions, and returns a list of retrieved targets.

[0016] Furthermore, the relevant information in the video management module includes: camera name, camera location, shooting time and whether to process.

[0017] Furthermore, the vector storage module stores the target's visual feature vector and related information of the corresponding video frame, including:

[0018] The visual feature vector of the target is stored in segments according to the target attributes, including:

[0019] Step 1-1, creating a new vector file list table according to the detected target, creating a new vector storage file, and adding a new row in the vector file list table to record the newly created vector storage file;

[0020] Step 1-2: Obtain the feature vector and corresponding information of a target from the feature extraction module, including: target type, feature vector, timestamp and camera ID;

[0021] Step 1-3, obtain the latest vector storage file from the vector file list table, and determine whether the number of vectors in the vector storage file exceeds the set threshold. If so, execute steps 1-4; if not, execute steps 1-5;

[0022] Steps 1-4: Create a new vector storage file and add a new row to the vector file list table;

[0023] Step 1-5, write the timestamp and feature vector into the vector storage file, and update the end time and vector quantity in the vector file list table;

[0024] Step 1-6, repeat step 1-2 to step 1-5 until all targets are stored.

[0025] Furthermore, the vector file list table in step 1-1 includes the following fields: file path, file name, target type, start time, end time, camera ID, and vector quantity.

[0026] Furthermore, the user instructions include:

[0027] Set the target retrieval mode, time interval, spatial range, number of targets, target starting point coordinates and target movement mode.

[0028] Furthermore, the target search mode is set to include four similarity searches: global average mode, window average mode, latest priority mode and user-defined mode; wherein,

[0029] The global average mode averages the visual features of all target images in the target trajectory and uses them as input features to retrieve similar targets.

[0030] The window averaging mode uses a set time window to average the image visual features of the target trajectory within the window interval and uses them as input features to retrieve similar targets;

[0031] The latest priority mode uses the latest image visual features in the target trajectory as input features to retrieve similar targets;

[0032] The user-defined mode is that the user selects one or more target images through the interface provided by the interactive display module, calculates the average value of visual features, and uses the average value as input features to retrieve similar targets.

[0033] Furthermore, the path calculation module combines geographic information data and target speed parameters to calculate the target movement range within a given time interval, including:

[0034] Step 2-1: Import geographic information data, including urban road and rail transit data;

[0035] Step 2-2: Take the intersections of urban roads and rail transit stations as nodes, the roads between nodes as edges, and combine the one-way or two-way properties of the roads to construct a road directed graph C. The edge weight in the road directed graph is the road length;

[0036] Step 2-3, determining the target's average motion speed v according to the target motion mode in the user's instruction;

[0037] Step 2-4, according to the target starting point coordinates in the user instruction, select the nearest node in the road directed graph as the starting point, and construct the distance matrix E and time array and the access tag array F, which are represented as follows:

[0038]

[0039] F=[F1,…,F n ]

[0040] Among them, E ij represents the shortest distance from node i to node j, represents the shortest time from node s0 to node n, F n Indicates whether node j has been visited, 0 means not visited, 1 means visited;

[0041] Initialize distance matrix E, time array and access tag array F;

[0042] Steps 2-5, based on the current node s t , combined with the access tag array F, and the adjacency matrix C, from the starting point s t Start exploring adjacent nodes step by step, calculate the shortest time to reach each adjacent node, and select the unvisited node with the minimum distance s t+1 , update the distance matrix E and time array And marked as visited nodes, as follows:

[0043]

[0044] Step 2-6, repeat steps 2-5 until all nodes are visited or the time exceeds the threshold.

[0045] Furthermore, the road directed graph C described in step 2-2 is represented as follows:

[0046]

[0047] Among them, C is a road directed graph, n is the number of nodes, C ij Represents the distance from node i to node j. If the road is bidirectional, then C ij =C ji ; If it is one-way, then C ij =∞.

[0048] Furthermore, the distance matrix E and time array initialized in steps 2-4 are And the access flag array F is as follows:

[0049] Set the distance of the starting point in the distance matrix E to 0 and the distance of other points to infinity; Set all data in the access flag array F to infinity; set all data in the access flag array F to 0 to indicate that it has not been accessed;

[0050] Furthermore, the vector search module searches the target database in combination with the target action range and the user instruction, including:

[0051] Step 3-1, obtaining user instructions from the interactive display module;

[0052] Step 3-2: According to the user's instruction, the vector file list table is searched in the target library to obtain the vector storage file to be searched;

[0053] Step 3-3: Create a new query result list, including: camera ID, timestamp, target type, feature vector and similarity, and execute steps 3-4 to 3-6 for each vector storage file to be queried;

[0054] Step 3-4: Read the vector storage file and delete irrelevant target rows according to the time interval set in the user instruction;

[0055] Step 3-5, calculate the similarity between the feature vector of the target to be retrieved and the target feature vector in the vector file, and save it in the query result list;

[0056] Step 3-6: Sort the query result list according to similarity, and retain the corresponding number of targets according to the number of targets in the user instruction;

[0057] Step 3-7, returning the query result list, completing the search.

[0058] Beneficial effects:

[0059] The present invention detects and extracts features of pedestrians and vehicles from massive video data in urban scenes, stores them in file fragments, and combines path calculation to screen valid file fragments, effectively reducing the complexity of feature calculation and improving the efficiency of target trajectory analysis through interactive iteration. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0061] Figure 1 It is a schematic diagram of the system composition of the present invention.

[0062] Figure 2 It is an interactive display process diagram.

[0063] Figure 3 It is a schematic diagram of the road node and path network.

[0064] Figure 4 It is a schematic diagram of the road nodes and path network that can be reached within a limited time.

[0065] Figure 5 This is a schematic diagram of the camera's range.

[0066] Figure 6 It is a schematic diagram of trajectory analysis.

[0067] Figure 7 is an example graph of eigenvectors. DETAILED DESCRIPTION

[0068] The overall idea of the present invention is as follows: by combining the three dimensions of time, space and visual features, an interactive retrieval system is constructed to reduce computational complexity and improve the efficiency of target trajectory analysis.

[0069] The technical solution of the present invention is as follows: Figure 1 As shown, an interactive video target spatiotemporal retrieval system based on visual similarity is composed of a video management module, a target detection module, a feature extraction module, a vector storage module, a path calculation module, a vector retrieval module, and an interactive display module.

[0070] The video management module manages files and structured data such as video files, camera names, camera locations, and shooting times;

[0071] The target detection module detects pedestrians and vehicles in the video based on a deep neural network, and obtains pedestrian and vehicle target image areas in the video frame;

[0072] The feature extraction module extracts visual feature vectors from detected pedestrians and vehicle targets based on a deep neural network;

[0073] The vector storage module supports storing visual feature vectors of pedestrians and vehicle targets, as well as data such as camera position and target appearance time;

[0074] The interactive display module supports functions such as input of target images to be retrieved, time intervals, transportation modes, similarity modes, display of retrieval result lists, and display of target trajectories. It supports retrieval of targets by iteratively calling path calculation and vector retrieval modules, and adds them to the target trajectory after manual confirmation.

[0075] The path calculation module combines geographic information data such as urban roads and rail transit, as well as speed parameters such as pedestrians, vehicles, and public transportation, to calculate the target movement range in a given time interval;

[0076] The vector retrieval module combines the target action range given by the path calculation module and the target to be retrieved given by the interactive display module, retrieves visually similar targets within a given time and space range in the target library, and returns a target list.

[0077] The vector storage module stores various target feature data in segments according to attributes such as camera ID and target appearance time. The main steps include:

[0078] Step 1-1, create a new vector file list table in the database, including fields such as file path, file name, target type, start time, end time, camera ID, and vector quantity;

[0079] Step 1-2, obtain target type, feature vector, timestamp, and camera ID data from the feature extraction module;

[0080] Step 1-3, obtain the corresponding storage file from the vector file list table, and determine whether the number of vectors exceeds the set threshold. If it exceeds the threshold, execute steps 1-4; if it does not exceed the threshold, execute steps 1-5;

[0081] Steps 1-4: Create a new vector storage file and add a new row to the vector file list table;

[0082] Steps 1-5: Write the timestamp and feature vector into the vector storage file, and update the end time, vector quantity and other fields in the vector file list table;

[0083] Step 1-6, repeat step 1-2 to step 1-5.

[0084] The target search supports four similarity searches: global average, window average, latest first, and user-defined.

[0085] The global average mode averages the visual features of all target images in the target trajectory and uses them as input features to retrieve similar targets;

[0086] The window average mode averages the image visual features of the target trajectory within the window interval through a set time window, and uses them as input features to retrieve similar targets;

[0087] The latest-first mode retrieves similar targets by using the latest image visual features in the target trajectory as input features;

[0088] The user-defined mode allows users to select one or more target images through an interactive interface, calculate the average value of visual features, and use them as input features to retrieve similar targets;

[0089] The interactive display module supports the user to switch the similarity calculation mode as needed during the retrieval process.

[0090] The path calculation module supports the calculation of the target action range based on time parameters and target motion parameters. The main steps include:

[0091] Step 2-1, import urban road, rail transit and other data;

[0092] Step 2-2: Based on the road's starting point, end point, rail transit station, and one-way / two-way attributes, a directed road graph is constructed with intersections as nodes and roads as edges. The edge weight in the graph is the road length.

[0093] Step 2-3, determining the average movement speed based on the transportation mode input by the user, including walking, cycling, electric bike riding, car, bus, subway, etc.;

[0094] Step 2-4: Based on the starting point coordinates input by the user, select the nearest node in the directed graph as the starting point, initialize the distance matrix, time array, and visit mark array, set the starting point distance to 0, and set the other points to infinity;

[0095] Steps 2-5: Select the unvisited node with the minimum distance, explore the adjacent nodes step by step from the starting point, calculate the shortest time to reach each adjacent node, update the distance matrix and time array, and mark it as a visited node;

[0096] Step 2-6, repeat steps 2-5 until all nodes are visited or the time exceeds the threshold.

[0097] The vector retrieval module supports target retrieval based on time interval, spatial range, and visual similarity. The main steps include:

[0098] Step 3-1, the vector retrieval module obtains the input feature vector, time interval, camera ID list, target type, and number of retrieval targets from the interactive retrieval module;

[0099] Step 3-2: query the vector file list table in the database, and filter by time interval and camera ID list to obtain the vector file list to be queried;

[0100] Step 3-3: Create a new query result list, which includes fields such as camera ID, timestamp, target type, feature vector, and similarity. Execute steps 3-4 to 3-6 for each file in the query vector file list.

[0101] Step 3-4: Read the vector file and delete irrelevant target rows according to the input time interval;

[0102] Step 3-5, calculate the similarity between the input feature vector and the target feature vector in the vector file, and save it in the query result list;

[0103] Step 3-6: Sort the query result list according to similarity, and based on the input retrieval target quantity parameter, retain the targets with high similarity and delete the targets with low similarity;

[0104] Step 3-7, return to the query result list.

[0105] Example:

[0106] A specific embodiment of the present invention is as follows: an interactive video object spatiotemporal retrieval system based on visual similarity, such as Figure 1 As shown, the system consists of a video management module, a target detection module, a feature extraction module, a vector storage module, a path calculation module, a vector retrieval module, and an interactive display module.

[0107] The video management module manages files and structured data such as video files, camera names, camera locations, and shooting times. The specific data format is shown in Table 1.

[0108] Table 1 Video file list data format description table

[0109] name Field Code Data Type Video files videoFile String Video ID VideoID Long Camera Name cameraName String Camera ID cameraID Long Camera longitude cameraLon Double Camera dimensions cameraLat Double Video Time videoStartTime String Whether to process isProcessed Boolean

[0110] The target detection module detects pedestrians and vehicles in the video based on the YOLO deep neural network, and obtains the pedestrian and vehicle target image areas in the video frame. The detection results are shown in Table 2.

[0111] Table 2 Test result data format description table

[0112]

[0113]

[0114] For the detected target area, the relevant area of the video frame is intercepted and saved to the file system. In order to avoid too many target images in a single directory, the target image saving path is segmented during the saving process to ensure that the number of images in a single directory does not exceed 1000. The specific path generation method is as follows:

[0115]

[0116] Where baseURL represents the base file path where the target image is saved, and floor(cameraID / 1000) represents rounding down the quotient of cameraID divided by 1000.

[0117] The feature extraction module extracts visual feature vectors V from detected pedestrians and vehicle targets based on the twin deep neural network, with a feature dimension of 1024. An example of a feature vector is Figure 7 shown.

[0118] The vector storage module supports storing visual feature vectors of pedestrians and vehicles, as well as data such as camera position and target appearance time. The vector storage module stores each type of target feature data in segments according to attributes such as camera ID and target appearance time. The storage format is hdf5. The specific steps are as follows:

[0119] Step 1-1, create a new vector file list table in the database, including fields such as file path, target type, start time, end time, camera ID, and vector quantity, as shown in Table 3;

[0120] Table 3 Vector file list table data format description table

[0121]

[0122]

[0123] Step 1-2, obtain target type, feature vector, timestamp, and camera ID data from the feature extraction module;

[0124] Step 1-3, obtain the corresponding storage file from the vector file list table, and determine whether the number of vectors exceeds the set threshold. If it exceeds the threshold, execute steps 1-4; if it does not exceed the threshold, execute steps 1-5;

[0125] Steps 1-4: Create a new vector storage file and add a new row to the vector file list table. The file path is as follows:

[0126]

[0127] Steps 1-5 write the timestamp and feature vector into the vector storage file, and update the end time, vector quantity and other fields in the vector file list table. The contents of the vector storage file are shown in Table 4:

[0128] Table 4 Vector storage file data format description table

[0129] name Field Code Data Type Target appearance time targetTime String Eigenvector featureVector Double Vector

[0130] Step 1-6, repeat step 1-2 to step 1-5.

[0131] like Figure 2 As shown, the interactive display module supports functions such as input of target images to be retrieved, time intervals, transportation modes, similarity modes, display of retrieval result lists, and display of target trajectories. It supports retrieval of targets by iteratively calling path calculation and vector retrieval modules, and adds the targets to the target trajectories after manual confirmation.

[0132] The target search supports four similarity searches: global average, window average, latest first, and user-defined.

[0133] The global average mode retrieves similar targets by averaging the visual features of all target images in the target trajectory as input features. Suppose there are T targets in the target trajectory, and their features are V t , then the input feature is V s =

[0134]

[0135] The window average mode retrieves similar targets by averaging the visual features of the images within the window interval in the target trajectory as input features. Suppose the window size is W, and W < T, then the input feature is

[0136] The latest priority mode retrieves similar targets by using the visual features of the latest image in the target trajectory as input features. V s = V t ;

[0137] The user-defined mode supports the user to select one or more target images through the interactive interface, calculate the average value of the visual features, and retrieve similar targets as input features. Suppose the selected N target features are V i , i = 1, …, N, then the input feature is

[0138] The interactive display module supports the user to switch the similarity calculation mode as needed during the retrieval process.

[0139] The path calculation module combines geographical information data such as urban roads and rail transit, as well as speed parameters of pedestrians, vehicles, public transportation, etc., to calculate the target action range within a given time interval. The main steps include:

[0140] Step 2-1: Import data such as urban roads and rail transit;

[0141] Step 2-2: As Figure 3 shown, according to the starting point, ending point of the road, rail transit stations, and attributes such as one-way / two-way, taking intersections as nodes and roads as edges, construct a road directed graph C. The edge weight in the graph is the road length. As shown below, n is the number of nodes, C ij represents the distance from node i to node j. If the road is two-way, then C ij = C ji ; if it is one-way, then C ji = ∞; The road directed graph C is represented as follows:

[0142]

[0143] Step 2-3, determining the average movement speed v based on the transportation mode input by the user, including walking, cycling, electric vehicle riding, car, bus, subway, etc.;

[0144] Step 2-4, according to the starting point coordinates input by the user, select the nearest node in the directed graph as the starting point s0 and initialize the time array To be infinite, initialize the access tag array F = [F1,…,F n ] is 0, initialize the distance matrix E, set the starting point distance to 0, and set other points to infinity; the distance matrix E is expressed as follows:

[0145]

[0146] Steps 2-5, based on the current node s t , combined with the access tag array F, and the adjacency matrix C, from the starting point s t Start exploring adjacent nodes step by step, calculate the shortest time to reach each adjacent node, and select the unvisited node with the minimum distance s t+1 , update the distance matrix E and time array P, and mark it as a visited node;

[0147]

[0148] Steps 2-6, repeat steps 2-5 until all nodes are visited or the time exceeds the threshold, e.g. Figure 4 As shown, the node range and corresponding path that can be reached from the starting point within the limited time are obtained.

[0149] The vector retrieval module combines the target action range rectangular area given by the path calculation module, such as Figure 5 As shown, the ID list within the rectangular area is obtained, and combined with the target image and features to be retrieved given by the interactive display module, visually similar targets within the given time and space range are retrieved in the target library and the target list is returned. The main steps include:

[0150] Step 3-1, the vector retrieval module obtains the input feature vector, time interval, camera ID list, target type, and number of retrieval targets from the interactive retrieval module;

[0151] Step 3-2: query the vector file list table in the database, and filter by time interval and camera ID list to obtain the vector file list to be queried;

[0152] Step 3-3: Create a new query result list, which includes fields such as camera ID, timestamp, target type, feature vector, and similarity. Execute steps 3-4 to 3-6 for each file in the query vector file list.

[0153] Step 3-4: Read the vector file p in the list of vector files to be queried, delete irrelevant target rows according to the input time interval, and obtain the feature vector matrix M p ;

[0154]

[0155] Step 3-5, calculate the similarity between the input eigenvector and the eigenvector matrix The similarity calculation uses the dot product similarity of the feature vector. in Indicates V t The transposition of is saved in the query result list. The query result list data format is shown in Table 5.

[0156] Table 5 Heartbeat link sending data description

[0157] name Field Code Data Type Feature similarity vectorSIM Float Target appearance time targetTime String Video ID videoID Long Target Type targetType Integer{1: vehicle, 2: pedestrian}

[0158] Step 3-6: Sort the query result list according to similarity, and based on the input retrieval target quantity parameter, retain the targets with high similarity and delete the targets with low similarity;

[0159] Step 3-7, return to the query result list and display it on the interface, such as Figure 6 As shown, users can click on the target image to view the associated video for analysis and add the confirmed target image to the target trajectory.

[0160] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium is capable of storing a computer program that, when executed by the data processing unit, can execute the invention content of the interactive video object spatiotemporal retrieval system based on visual similarity provided by the present invention and some or all of the steps in each embodiment. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0161] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a computer program, i.e., a software product. The computer program software product can be stored in a storage medium and includes a number of instructions for enabling a device including a data processing unit (which can be a personal computer, server, single-chip microcomputer, MCU, or network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0162] The present invention provides a concept and method for an interactive spatiotemporal retrieval system for video objects based on visual similarity. There are numerous methods and approaches for implementing this technical solution. The above is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. An interactive video object spatiotemporal retrieval system based on visual similarity, characterized by: The system comprises: Video management module, target detection module, feature extraction module, vector storage module, path calculation module, vector retrieval module and interactive display module; wherein, The video management module performs structured management on video files and related information; The target detection module performs target detection on the video content to obtain the image area of the target; The feature extraction module extracts visual feature vectors from the detected target; The vector storage module stores the visual feature vector of the target and the relevant information of the corresponding video frame to obtain the target library; The interactive display module retrieves the target and performs manual confirmation by iteratively calling the path calculation module and the vector retrieval module according to the user's instructions; The path calculation module calculates the target's movement range in a given time interval by combining geographic information data and the target's speed parameters; The vector search module searches the target library based on the target action range and user instructions, and returns a list of retrieved targets.

2. The interactive video object spatiotemporal retrieval system based on visual similarity according to claim 1, characterized in that: The relevant information in the video management module includes: camera name, camera location, shooting time and whether to process.

3. The interactive video object spatiotemporal retrieval system based on visual similarity according to claim 2, characterized in that: The vector storage module stores the target's visual feature vector and related information of the corresponding video frame, including: The visual feature vector of the target is stored in segments according to the target attributes, including: Step 1-1, creating a new vector file list table according to the detected target, creating a new vector storage file, and adding a new row in the vector file list table to record the newly created vector storage file; Step 1-2: Obtain the feature vector and corresponding information of a target from the feature extraction module, including: target type, feature vector, timestamp and camera ID; Step 1-3, obtain the latest vector storage file from the vector file list table, and determine whether the number of vectors in the vector storage file exceeds the set threshold. If it exceeds the threshold, execute steps 1-4; if it does not exceed the threshold, execute steps 1-5; Steps 1-4: Create a new vector storage file and add a new row to the vector file list table; Step 1-5, write the timestamp and feature vector into the vector storage file, and update the end time and vector quantity in the vector file list table; Step 1-6, repeat step 1-2 to step 1-5 until all targets are stored.

4. The interactive video object spatiotemporal retrieval system based on visual similarity according to claim 3, characterized in that: The vector file list table described in step 1-1 includes the following fields: file path, file name, target type, start time, end time, camera ID, and vector quantity.

5. The interactive video object spatiotemporal retrieval system based on visual similarity according to claim 4, characterized in that: The user instructions include: Set the target retrieval mode, time interval, spatial range, number of targets, target starting point coordinates and target movement mode.

6. The interactive video object spatiotemporal retrieval system based on visual similarity according to claim 5, characterized in that: The target search mode is set to include four similarity searches: global average mode, window average mode, latest priority mode and user-defined mode; The global average mode averages the visual features of all target images in the target trajectory and uses them as input features to retrieve similar targets. The window averaging mode uses a set time window to average the image visual features of the target trajectory within the window interval and uses them as input features to retrieve similar targets; The latest priority mode uses the latest image visual features in the target trajectory as input features to retrieve similar targets; The user-defined mode is that the user selects one or more target images through the interface provided by the interactive display module, calculates the average value of visual features, and uses the average value as input features to retrieve similar targets.

7. The interactive video object spatiotemporal retrieval system based on visual similarity according to claim 6, characterized in that: The path calculation module combines geographic information data and target speed parameters to calculate the target's range of action within a given time interval, including: Step 2-1: Import geographic information data, including urban road and rail transit data; Step 2-2: Take the intersections of urban roads and rail transit stations as nodes, the roads between nodes as edges, and combine the one-way or two-way properties of the roads to construct a road directed graph C. The edge weight in the road directed graph is the road length; Step 2-3, determining the target's average motion speed v according to the target motion mode in the user's instruction; Step 2-4, according to the target starting point coordinates in the user instruction, select the nearest node in the road directed graph as the starting point, and construct the distance matrix E and time array and the access tag array F, which are represented as follows: F=[F1,…,F n ] Among them, E ij represents the shortest distance from node i to node j, represents the shortest time from node s0 to node n, F n Indicates whether node j has been visited, 0 means not visited, 1 means visited; Initialize distance matrix E, time array and access tag array F; Steps 2-5, based on the current node s t , combined with the access tag array F, and the adjacency matrix C, from the starting point s t Start exploring adjacent nodes step by step, calculate the shortest time to reach each adjacent node, and select the unvisited node with the minimum distance s t+1 , update the distance matrix E and time array And marked as visited nodes, as follows: Step 2-6, repeat steps 2-5 until all nodes are visited or the time exceeds the threshold.

8. The interactive video object spatiotemporal retrieval system based on visual similarity according to claim 7, characterized in that: The road directed graph C described in step 2-2 is represented as follows: Among them, C is a road directed graph, n is the number of nodes, C ij Represents the distance from node i to node j. If the road is bidirectional, then C ij =C ji ; If it is one-way, then C ij =∞.

9. The interactive video object spatiotemporal retrieval system based on visual similarity according to claim 8, characterized in that: Initialize the distance matrix E and time array described in steps 2-4 And the access flag array F is as follows: Set the distance of the starting point in the distance matrix E to 0 and the distance of other points to infinity; All data in the access flag array F are set to infinity; all data in the access flag array F are set to 0 to indicate that they have not been accessed.

10. The interactive video object spatiotemporal retrieval system based on visual similarity according to claim 9, characterized in that: The vector search module searches the target database based on the target action range and user instructions, including: Step 3-1, obtaining user instructions from the interactive display module; Step 3-2: According to the user's instruction, the vector file list table is searched in the target library to obtain the vector storage file to be searched; Step 3-3: Create a new query result list, including: camera ID, timestamp, target type, feature vector and similarity, and execute steps 3-4 to 3-6 for each vector storage file to be queried; Step 3-4: Read the vector storage file and delete irrelevant target rows according to the time interval set in the user instruction; Step 3-5, calculate the similarity between the feature vector of the target to be retrieved and the target feature vector in the vector file, and save it in the query result list; Step 3-6: Sort the query result list according to similarity, and retain the corresponding number of targets according to the number of targets in the user instruction; Step 3-7, returning the query result list, completing the search.

Images