A camera video data retrieval method and related device

Abstract

The application discloses a kind of camera video data retrieval method and related equipment, it is related to video data retrieval field, the method includes: obtaining camera video data;The classification information extraction is carried out to the camera video data obtained, and the classification information extracted is obtained;Extracted classification information is respectively inserted into file storage, structured database and feature vector library in database according to information category;Input is obtained to be searched target information and search condition, and target feature vector to be searched is parsed according to target information to be searched, and search condition and target feature vector are searched in file storage, structured database and feature vector library based on space progressive search mode and time progressive search mode, and target multi-day travel trajectory is restored by the result searched and output.The above-mentioned method can satisfy city-level people and vehicle in whole city range, whole time section fast search.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video data retrieval, and in particular to a camera video data retrieval method and related equipment. BACKGROUND

[0002] At present, for the city-level vehicle and person retrieval scene such as public security prevention and control, the traditional way of manually checking the monitoring and tracking and identifying the target cannot realize the rapid retrieval of the target, and consumes a large amount of human resources. The existing graph search system built with the aid of a computer adopts an end-to-cloud computing architecture, and has the problems of difficulty in gathering, storing and retrieving, and cannot well realize the rapid retrieval of vehicles and persons, and cannot realize the rapid retrieval in the whole city range, continuous multi-day and full-time section.

[0003] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a camera video data retrieval method and related equipment to overcome the deficiencies of the prior art, and the present application can meet the rapid retrieval of city-level vehicles and persons in the whole city range and full-time section.

[0005] In order to solve the above-mentioned deficiencies of the prior art, the first aspect of the present application provides a camera video data retrieval method, which comprises:

[0006] obtaining camera video data;

[0007] extracting classification information from the obtained camera video data to obtain extracted classification information;

[0008] inserting the extracted classification information into a file storage library, a structured database and a feature vector library in a database according to information categories;

[0009] obtaining inputted target information to be retrieved and retrieval conditions, analyzing the target feature vector to be retrieved according to the target information to be retrieved, and using a spatial progressive retrieval method and a time progressive retrieval method to retrieve in the file storage library, the structured database and the feature vector library based on the retrieval conditions and the target feature vector, and restoring the target multi-day travel trajectory through the retrieved results and outputting.

[0010] The classification information extraction from the obtained camera video data to obtain the extracted classification information specifically comprises:

[0011] The extracted classification information is frame extraction pictures, structured data and feature vector data;

[0012] frame extraction processing is performed on the camera video data to obtain frame extraction pictures;

[0013] The base attribute information in the frame picture is extracted to obtain structured data.

[0014] The frame picture is subjected to feature information extraction to obtain feature vector data.

[0015] The extracted classification information is inserted into a file storage, a structured database and a feature vector database in the database according to information categories, and specifically includes:

[0016] The frame picture is inserted into the file storage in the database through a message queue.

[0017] The structured data is inserted into the structured database in the database through a message queue.

[0018] The feature vector data is inserted into the feature vector database in the database through a message queue.

[0019] The input target information to be searched and search conditions are obtained, and a target feature vector to be searched is parsed according to the target information to be searched, and specifically includes:

[0020] The input target information to be searched and search conditions are obtained, and target classification and a target feature vector are obtained through feature classification and feature extraction on the target to be searched.

[0021] The search conditions are obtained, and a similarity threshold, a search time range and a search space range are determined.

[0022] The similarity threshold is a preset threshold.

[0023] The search conditions and the target feature vector are used to search in the file storage, the structured database and the feature vector database in a spatial progressive search mode and a temporal progressive search mode, and specifically includes:

[0024] According to the search conditions and the target feature vector, a feature vector preliminary screening is performed on the vectors in the feature vector database to obtain K preliminary screening feature vectors.

[0025] Feature vector fusion is performed, that is, the K preliminary screening feature vectors are weighted and averaged to obtain a first fusion feature vector.

[0026] According to the obtained first fusion feature vector and the similarity threshold, a secondary search process is performed, that is, a secondary search is performed on the vectors in the feature vector database to obtain M secondary search feature vectors.

[0027] According to the obtained M secondary search feature vectors, a track restoration verification process is performed, that is, according to the obtained secondary search feature vectors, M corresponding frame extraction pictures, structured data corresponding to the frame extraction pictures and feature vector data are obtained from the database, and the target first track is restored according to the M frame extraction pictures obtained in the time and space order;

[0028] Each frame extraction picture in the target first track corresponds to a track point;

[0029] The target first track is subjected to track tracking verification, and abnormal track points are deleted;

[0030] After the track restoration verification is completed, target feature vector fusion is performed, that is, the feature vector data corresponding to the track points in the first track which are not deleted is subjected to weighted average fusion to obtain a second fusion feature vector;

[0031] Based on the obtained first track and second fusion feature vector, spatial progressive search is performed to obtain the travel track of the target in different ranges on the current date, and after the spatial progressive search is completed, the travel tracks of the target in different ranges on the current date are merged to obtain the movement track of the target on the current date;

[0032] When cross-day search is performed, corresponding frame extraction pictures are obtained based on the movement track of the target on the current date, time progressive search is performed according to the frame extraction pictures corresponding to the movement track of the current date, and the movement track of the target in the search time is obtained;

[0033] The movement tracks of different dates obtained by the spatial progressive search and the time progressive search are integrated and converged to obtain the complete track result of the target in the specified date range.

[0034] The spatial progressive search specifically includes:

[0035] The spatial progressive search takes the track end point obtained in the track restoration verification process in the last search range as the center of a circle, takes a preset distance range as the radius, and sequentially performs a secondary search process, a track restoration verification process and a target feature vector fusion process according to the last fusion feature vector, until no feature vector meeting the similarity threshold requirement is searched in the secondary search, the track search of the target on the current date is completed, a plurality of movement tracks of the target on the current date are obtained, and the movement track of the target on the current date is obtained by integrating the plurality of movement tracks of the target on the current date;

[0036] In the repeated secondary search process, track restoration verification process and target feature vector fusion process, the data corresponding to the camera which has been searched is eliminated each time the secondary search is performed.

[0037] The time progressive search specifically includes:

[0038] The local feature of the target is extracted from the frame picture corresponding to the moving track of the current date;

[0039] Based on the extracted local feature, the spatial range of the search is set as the position of the target appearing in the current date within L kilometers, the date range of the search is set as the date to be searched across days, the target is searched in the database, and a feature vector with the highest similarity is obtained;

[0040] The frame picture corresponding to the feature vector with the highest similarity is extracted, and the overall feature vector of the target in the frame picture is extracted;

[0041] According to the set spatial range of the search, the date range of the search, and the overall feature vector of the target in the extracted frame picture, the feature vector preliminary screening, feature vector fusion, secondary retrieval process, track restoration verification process, target feature vector fusion process, and spatial progressive retrieval process are sequentially performed, and the moving track of the target in the date to be searched across days is obtained.

[0042] The feature vector preliminary screening specifically includes coarse screening and fine screening;

[0043] The coarse screening is to filter the feature vectors in the feature vector library through the search conditions and the target feature vector to obtain the feature vectors corresponding to the conditions met;

[0044] The fine screening is to calculate the similarity between the feature vectors obtained by the coarse screening and the target feature vector, eliminate the feature vectors with a similarity that does not meet the set similarity threshold, and obtain the preliminary screening feature vectors.

[0045] When calculating the similarity between the feature vectors obtained by the coarse screening and the target feature vector, the Euclidean distance is used for calculation.

[0046] The second aspect of the embodiment of the application provides a camera video data retrieval device, and the camera video data retrieval device comprises:

[0047] A video acquisition module is configured to acquire camera video data;

[0048] An algorithm analysis module is configured to extract classification information from the acquired camera video data, and obtain the extracted classification information;

[0049] A data storage module is configured to insert the extracted classification information into a file storage library, a structured database, and a feature vector library in a database according to information categories.

[0050] The target retrieval module is configured to acquire input target information to be retrieved and a retrieval condition, parse a target feature vector to be retrieved according to the target information to be retrieved, perform retrieval in the file storage library, the structured database and the feature vector library based on the retrieval condition and the target feature vector by using a spatial progressive retrieval manner and a temporal progressive retrieval manner, restore a target multi-day travel trajectory through a retrieved result, and output the target multi-day travel trajectory.

[0051] The third aspect of the embodiments of the present application provides a computer readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps in the camera video data retrieval method according to any of the above.

[0052] The fourth aspect of the embodiments of the present application provides a terminal device, which includes a processor, a memory and a communication bus; the memory stores a computer readable program executable by the processor;

[0053] The communication bus realizes connection and communication between the processor and the memory.

[0054] The processor implements the steps in the camera video data retrieval method according to any of the above when executing the computer readable program.

[0055] Advantages: Compared with the prior art, the present application provides a camera video data retrieval method and related equipment, the method includes acquiring camera video data; extracting classification information from the acquired camera video data to obtain extracted classification information; inserting the extracted classification information into a file storage library, a structured database and a feature vector library in a database according to information categories; acquiring input target information to be retrieved and a retrieval condition, parsing a target feature vector to be retrieved according to the target information to be retrieved, performing retrieval in the file storage library, the structured database and the feature vector library based on the retrieval condition and the target feature vector by using a spatial progressive retrieval manner and a temporal progressive retrieval manner, restoring a target multi-day travel trajectory through a retrieved result, and outputting the target multi-day travel trajectory. Through the above method, the present application can extract features of targets such as pedestrians and vehicles in a video, store only the extracted classification information in a library, avoid the problems of storage and convergence of full-amount videos, and realize efficient retrieval of targets based on features. Specifically, the temporal progressive retrieval and the spatial progressive retrieval are used in the retrieval strategy, which guarantees retrieval efficiency and avoids blind retrieval in a full-city range and a full-time period.

[0056] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to user analysis data, user stored data, user displayed data, etc.) and signals involved in the present application are information, data and signals authorized by the user or authorized by all parties; and the collection, use and processing of related information, data and signals comply with relevant national and regional laws, regulations and standards. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0058] Figure 1 The flow chart of the camera video data retrieval method provided by the present application is shown in the figure.

[0059] Figure 2 The business logic diagram of the camera video data retrieval provided by the embodiment of the present application is shown in the figure.

[0060] Figure 3 The flow chart of the time progressive retrieval method and the space progressive retrieval method provided by the embodiment of the present application is shown in the figure.

[0061] Figure 4 The specific structure diagram of the camera video data retrieval method device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0062] The present application provides a camera video data retrieval method and related equipment, in order to make the purpose, technical scheme and effect of the present application more clear and definite, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0063] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprise" (or comprise), "comprises" (or comprises) and "comprising" (or comprising), when used in this specification, specify the presence of stated features, integers, steps, operations, elements, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof. It is further understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can be present. In addition, the word "connected" or "coupled" as used herein can include wirelessly connected or wirelessly coupled. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0064] It is to be understood that the terms so used are intended to encompass the general meaning of such terms as well as the specific meaning that can be attributed to such terms based on the present technology unless otherwise defined. It is further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0065] In addition, if the present application has a description involving "first", "second", etc., the description of "first", "second", etc. is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second" can be explicitly or implicitly included at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the scope of protection required by the present application.

[0066] At present, a large number of cameras are distributed in the city to deal with the safety of the city. However, the visual perception system used in the traditional city-level vehicle search adopts an end-cloud computing architecture, which has the problems of difficult convergence, difficult storage, and difficult search. In the corresponding city-level vehicle search, it is also difficult to achieve fast search of vehicles, and it is even more difficult to achieve fast search in the whole city range and the whole time section.

[0067] The present application provides a camera video data retrieval method and related equipment, which can meet the requirements of city-level vehicle retrieval in the whole city range and the whole time section.

[0068] Example method

[0069] As shown in Figure 1 The present application provides a camera video data retrieval method and related equipment, which can meet the requirements of city-level vehicle retrieval in the whole city range and the whole time section. Figure 1 and Figure 2 The method is described as follows:

[0070] Step S10, acquiring camera video data;

[0071] The camera video data is collected by the camera.

[0072] As shown in Figure 2 The camera video data is collected by the camera.

[0073] Step S20, extracting classification information from the acquired camera video data to obtain the extracted classification information;

[0074] The camera video data is collected by the camera. Figure 2 As shown in

[0075] The camera video data is collected by the camera.

[0076] The extracted classification information includes frame pictures, structured data and feature vector data.

[0077] The camera video data is collected by the camera.

[0078] The camera video data is collected by the camera.

[0079] The camera video data is collected by the camera.

[0080] Specifically, the camera video data is frame-extracted to obtain frame-extracted pictures;

[0081] The frame-extracted pictures are subjected to feature extraction to obtain structured data and feature vector data, and the feature extraction on the frame-extracted pictures specifically includes two parts of frame-extracted picture basic attribute information extraction and frame-extracted picture feature information extraction;

[0082] The frame-extracted picture basic information extraction mainly extracts the timestamp of the frame-extracted picture, and further, the frame-extracted picture basic information extraction obtains the collection time, collection location, collection camera ID, ID associated with the subsequently extracted feature vector data, storage address of the frame-extracted picture in the file server, and various attribute information extracted by other algorithm analysis modules from the picture, which constitute the structured data;

[0083] The feature information extraction on the frame-extracted picture obtains feature vector data, and specifically includes the target detection, target segmentation, target feature classification, target feature vector extraction, and target attribute information extraction processes in the frame-extracted picture;

[0084] Further, the target detection obtains the position of the person and vehicle targets in the picture; according to the position coordinates of different target types in the frame-extracted picture, the target image is cropped and corrected, and the target image contains the target. Preferably, in an implementation manner, the target image contains only a single person or a single vehicle. The target feature classification is performed on the obtained target image, the target feature classification supports the classification of pedestrians, faces, motor vehicles, and non-motor vehicles, and the target image is classified accordingly. The REID algorithm (Person Re-identification) is used to convert the target image into a 512-dimensional feature vector. In addition, the target image attribute is extracted, wherein the pedestrian attribute includes gender, shirt color, pants color, shoe color, whether a backpack, whether a handbag, etc., the face attribute includes hairstyle, etc., the motor vehicle attribute includes license plate color, license plate number, brand, model, vehicle length, vehicle height, vehicle width, vehicle body color, etc., and the non-motor vehicle attribute includes license plate number, brand, model, vehicle length, vehicle height, vehicle width, vehicle body color, wheels, whether to be shielded, etc.

[0085] Step S30, the extracted classification information is inserted into the file storage library, the structured database, and the feature vector library in the database according to the information categories, respectively;

[0086] Specifically, the extracted information is stored according to the frame pictures, structured data, and feature vector data, so that by extracting the features of pedestrians and vehicles in the video stream, only the structured data, feature vector data, and frame pictures are stored, the storage and aggregation of full-quantity videos can be avoided, and the efficient retrieval of subsequent targets can be realized based on the feature vector retrieval mode. Figure 2 As shown in S23, the data storage module inserts the extracted information into the file storage library, the structured database, and the feature vector library in the database according to the information categories, and the user can perform retrieval in the file storage library, the structured database, and the feature vector library.

[0087] Further, the insertion of the extracted classification information into the file storage library, the structured database, and the feature vector library in the database according to the information categories specifically includes:

[0088] The frame pictures are inserted into the file storage library in the database through the message queue.

[0089] The structured data is inserted into the structured database in the database through the message queue.

[0090] The feature vector data is inserted into the feature vector library in the database through the message queue.

[0091] Specifically, during the insertion, the data insertion strategy is that batch insertion of the feature vector data is supported according to the number of vectors and the time interval, and by default, batch insertion is performed every 100,000 feature vector data or every 60 seconds. In one embodiment, batch insertion of the frame pictures, structured data, and feature vector data is performed once every 100,000 feature vector data is generated. In another embodiment, the frame pictures and the structured data are inserted in time after being obtained, and the feature vector data is inserted every 100,000 times. In another embodiment, all data is batch inserted every 60 seconds.

[0092] During the data storage, the feature vector data is stored in a hierarchical manner, that is, the feature vector data is stored in multiple levels, based on the milvus database, the same feature classification is classified into the same vector set, and the data of the same day is placed in the same partition (partition) in the vector set, that is, the feature vector library. The basic attribute information of the frame pictures, such as the collection time, collection location, collection camera ID, ID associated with the feature vector, storage address of the picture in the file server, and various attribute information extracted by the algorithm module are stored in the structured database; the picture file processed by the algorithm module is stored in the file storage library.

[0093] Furthermore, this invention adopts a distributed data storage strategy. For city-level massive data scenarios, it uses a regional distributed cluster deployment method to store the data in the database according to the regional distribution, avoiding the simple expansion of disk and memory space, which would lead to a decrease in retrieval efficiency.

[0094] In addition, after the data storage is completed, an index is created on the feature vector data that has been stored based on the IVF_PQ quantization indexing algorithm.

[0095] Step S40: Obtain the input target information and search conditions, parse the target feature vector based on the target information, and perform a search in the file repository, structured database and feature vector library using spatial progressive search and time progressive search methods based on the search conditions and target feature vector. Reconstruct the target trajectory using the search results and output it.

[0096] After the data is stored in the database, when a user wants to perform a corresponding search, they can input the target information to be searched and the search conditions. Then, the target feature vector to be searched is parsed based on the target information. The target feature vector is then used to perform a corresponding search in the database. The target trajectory is then reconstructed from the search results and output to the user.

[0097] like Figure 2 As shown, the target retrieval module in S24 performs the corresponding data retrieval process. Specifically, the user inputs the target information to be retrieved, i.e., the image in the corresponding figure. The image is processed for features, and then the feature vector is initially screened, fused, and a second retrieval process is performed through the retrieval conditions. After trajectory verification and feature fusion, spatial progressive retrieval and temporal progressive retrieval processes are performed respectively. Finally, the trajectory is fused and the target trajectory is output.

[0098] Furthermore, the step of obtaining the input target information and search conditions, and parsing the target feature vector based on the target information, specifically includes:

[0099] The system obtains the input target information and search conditions, performs feature classification and feature extraction on the target, and obtains the target classification and target feature vector.

[0100] Obtain search criteria and determine similarity thresholds, search time range, and search spatial range.

[0101] Specifically, the input target information to be searched is information with a target input by a user. In an embodiment, the input target information to be searched is a picture containing a target, which can be obtained by a user through a camera video, a smart device, temporary shooting, or the like. The search condition includes a search time range, a search space range, a similarity threshold, and other search conditions. The search time range can be specified by a user. If the user does not specify the time range, the system defaults to searching data in the last three days. The search space range can be a specified area selected by a user on a map, or a specified location, such as an airport or a transportation hub. If no search space range is specified, the system defaults to searching within a 5-kilometer radius of an initial location, which can be marked by the user on the map. The other search conditions include various attribute information of the target, such as the type of a vehicle (motorcycle, car, pickup truck, van, truck, bus, etc.), and the attribute features of a pedestrian (backpack, handbag, hat, color of upper and lower body, etc.).

[0102] Further, the similarity threshold is a preset threshold.

[0103] That is, the user can determine the similarity threshold according to the desired search accuracy before searching, and filter vectors with low similarity through the set similarity threshold.

[0104] The search is performed in the database by using a spatial progressive search method and a temporal progressive search method based on the features, and specifically includes:

[0105] According to the search condition and the target feature vector, the vectors in the feature vector library are subjected to initial screening of feature vectors to obtain K initial screening feature vectors;

[0106] The K initial screening feature vectors are subjected to weighted average fusion to obtain a first fusion feature vector;

[0107] According to the obtained first fusion feature vector and the similarity threshold, a secondary search process is performed, that is, the vectors in the feature vector library are subjected to secondary search to obtain M secondary search feature vectors;

[0108] According to the M secondary search feature vectors, a trajectory restoration verification process is performed, that is, the corresponding M frame extraction pictures, structured data corresponding to the frame extraction pictures, and feature vector data are obtained from the database according to the secondary search feature vectors, and the target first trajectory is restored according to the M frame extraction pictures in the time and space order.

[0109] Each frame extraction picture in the target first trajectory corresponds to a trajectory point.

[0110] Trajectory tracking verification is performed on the target first trajectory, and abnormal trajectory points are deleted;

[0111] After trajectory restoration verification, target feature vector fusion is performed, that is, the feature vector data corresponding to the trajectory points not deleted in the first trajectory is weighted and averaged to obtain a second fused feature vector;

[0112] Based on the obtained first trajectory and second fused feature vector, spatial progressive retrieval is performed to obtain the travel trajectory of the target in different ranges on the same day. After the spatial progressive retrieval is completed, the travel trajectories of the target in different ranges on the same day are merged to obtain the movement trajectory of the target on the current date;

[0113] When cross-day retrieval is performed, the corresponding frame extraction picture is obtained based on the movement trajectory of the target on the current date, and time progressive retrieval is performed according to the frame extraction picture corresponding to the movement trajectory on the current date to obtain the movement trajectory of the target in the search time;

[0114] The movement trajectories of different dates obtained by the spatial progressive retrieval and the time progressive retrieval are integrated and converged to obtain the complete trajectory result of the target in the specified date range.

[0115] In terms of retrieval strategy, the present application adopts time progressive retrieval and spatial progressive retrieval to ensure retrieval efficiency and avoid blind retrieval in the whole city range and the whole time section; to ensure retrieval accuracy, the present application performs secondary iterative retrieval by fusing feature vectors of the same target in different time periods, thereby reducing target missed detection.

[0116] Further, the feature vector preliminary screening specifically includes coarse screening and fine screening;

[0117] The coarse screening is to filter the feature vectors in the feature vector library by the retrieval condition and the target feature vector to obtain the feature vectors corresponding to the satisfied conditions.

[0118] The fine screening is to calculate the similarity of the feature vectors obtained by the coarse screening and the target feature vector, and to remove the feature vectors that do not satisfy the similarity according to the set similarity threshold to obtain the preliminary screening feature vectors.

[0119] The similarity of the feature vector obtained by the coarse screening and the target feature vector is calculated by using the Euclidean distance. That is, in the feature vector preliminary screening, the feature vector of the information and the feature of the target to be searched is obtained by coarse screening in the database, and then the fine screening calculation is performed on the feature vector obtained by the coarse screening. The similarity of the feature vector obtained by the coarse screening and the target feature vector is calculated by using the Euclidean distance. According to the set similarity threshold, some feature vectors that do not meet the similarity threshold are removed, and the closest feature vector is obtained. The closest K preliminary screening feature vectors are selected for feature vector fusion. The value of K is selected by the user according to the actual situation.

[0120] Specifically, the secondary retrieval is performed according to the first fused feature vector, and the similarity calculation method used in the fine screening calculation in the feature vector preliminary screening is used again. According to the set similarity threshold, some feature vectors that do not meet the similarity threshold are removed, and the secondary retrieval feature vectors that meet the conditions are obtained. The closest M secondary retrieval feature vectors are selected for trajectory restoration verification process. The value of M is selected by the user according to the actual situation.

[0121] Specifically, in the trajectory restoration verification process, the appearance trajectory of the target is restored according to the M frame extraction pictures corresponding to the M secondary retrieval feature vectors, combined with the timestamp, spatial position and other information of the extraction pictures. In the trajectory restoration verification process, the moving speed of the target between adjacent trajectory points is calculated according to the timestamp and spatial position information of the adjacent trajectory points. If the speed is obviously abnormal, that is, the speed is significantly different from the speed before and after it, the trajectory point is discarded. In one embodiment, the displacement and time of the M trajectory points are obtained, and the average speed is calculated. If the moving speed of the target between adjacent trajectory points is different from the average speed by more than a certain threshold, the two trajectory points are discarded. The threshold is set by the user according to the actual situation.

[0122] The spatial progressive retrieval specifically includes:

[0123] The spatial progressive retrieval takes the trajectory endpoint obtained by the trajectory restoration verification process in the last retrieval range as the center of the circle, and takes the preset distance range as the radius. The secondary retrieval process, the trajectory restoration verification process and the target feature vector fusion process are performed in turn according to the last fused feature vector, until the feature vector that meets the similarity threshold requirement cannot be searched in the secondary retrieval. The trajectory retrieval of the target on the current day is ended, and a plurality of moving trajectories of the target on the current day are obtained. The moving trajectory of the target on the current day is obtained by summarizing the plurality of moving trajectories of the target on the current day.

[0124] In the repeated secondary retrieval process, trajectory restoration verification process and target feature vector fusion process, the data corresponding to the camera that has been searched is removed each time the secondary retrieval is performed.

[0125] Further description is that, in the process of spatial progressive retrieval, the end point of the first trajectory obtained by trajectory restoration verification is taken as the center of a circle, within a range of 5 kilometers, the searched camera is removed, the second retrieval process, trajectory restoration verification process and target feature vector fusion process are carried out according to the second fusion feature vector, the output trajectory and the output fusion feature vector are obtained, the next second retrieval process, trajectory restoration verification process and target feature vector fusion process are carried out again using the output trajectory and the output fusion feature vector; when the feature vector satisfying the similarity threshold requirement is not searched in the second retrieval, the trajectory retrieval of the target on the current day is ended.

[0126] In an embodiment, when the spatial progressive retrieval is carried out once, the target trajectory obtained this time is combined with the trajectory of the last time, and finally the moving trajectory of the target on the current date is obtained through multiple combinations.

[0127] The time progressive retrieval specifically includes:

[0128] The local feature of the target is extracted from the frame-extracted picture corresponding to the moving trajectory on the current date;

[0129] Based on the extracted local feature, the search space range is set to be within L kilometers around the position of the target on the current date, the search date range is set to be the date to be searched across days, the target is searched in the database, and the feature vector with the highest similarity is obtained;

[0130] The frame-extracted picture corresponding to the feature vector with the highest similarity is extracted, and the overall feature vector of the target in the frame-extracted picture is extracted;

[0131] According to the set search space range, search date range and extracted overall feature vector of the target in the frame-extracted picture, the feature vector preliminary screening, feature vector fusion, second retrieval process, trajectory restoration verification process, target feature vector fusion process and spatial progressive retrieval process are sequentially carried out, and the moving trajectory of the target on the date to be searched across days is obtained.

[0132] For the cross-sky situation, the background, light, weather, angle, pedestrian clothing and other changes will cause low retrieval accuracy, the cross-sky retrieval process needs to be optimized, therefore, the local features of the target in the frame picture corresponding to the current date moving track are extracted, the local features are stable features, for example, the stable features of pedestrians are face features, and the vehicle is a license plate feature, the size of the L is set by the user according to the actual situation, the target retrieval mode in the database is the same as the similarity calculation mode used in the secondary retrieval, so that the feature vector with the highest similarity is obtained, and the overall feature vector is all feature vector data saved in the feature vector database corresponding to the frame picture. Therefore, in order to realize one-time retrieval of a target (especially a pedestrian target) for consecutive days, the present application fuses multiple features of the target, uses the extracted face features (stable features) to locate the retrieval strategy of the pedestrian target feature vector of different dates, so as to solve the problem that the continuous cross-sky retrieval cannot be performed under the condition that the features such as clothing of pedestrians change greatly every day.

[0133] The present application combines Figure 3 The spatial progressive retrieval mode and the time progressive retrieval mode are further described. Specifically, as shown in Figure 3

[0134] Steps S31.1-S31.4: the user inputs target information to be retrieved and retrieval conditions, and enters step S32.

[0135] Step S32: target feature extraction is performed, and after the extraction is completed, step S33 is entered.

[0136] Step S33: feature vector primary screening is performed according to the retrieval conditions and the extracted target features, after the corresponding primary screening feature vector is obtained through the primary screening, step S34 is entered.

[0137] Step S34: target feature vector fusion.

[0138] Step S35: secondary retrieval is performed after the fusion is completed, and after the secondary retrieval is completed, step S36.1 is entered.

[0139] Step S36.1: trajectory verification is performed, that is, the process of obtaining the first trajectory in the trajectory restoration verification process, and after the process is completed, step S36.2 is entered.

[0140] Step S36.2: trajectory tracking is performed, that is, the process of deleting abnormal trajectory points in the trajectory restoration verification process, and step S37 is entered.

[0141] Step S37: feature fusion, after the target moving track restoration verification is completed, the target feature vectors corresponding to the trajectory points are weighted and averaged to form a new feature vector, and step S38 is entered.

[0142] ​Step S38: space progressive retrieval, i.e. taking the end point of the first trajectory in step S36.1 as the center, within a preset distance range, eliminating the retrieved camera, repeating steps S35-S37 until step S35 cannot retrieve a feature vector meeting the similarity threshold requirement, the trajectory retrieval of the target on the day is ended, the calculated trajectories are merged to obtain the movement trajectory tracking of the target on the current date, and the movement trajectory is input into steps S10 and S39.1;

[0143] The steps S39.1-S39.6 are time progressive retrieval processes.

[0144] Step 39.1: target local stable feature extraction, after completion, entering step S39.2;

[0145] Step S39.2: target local stable feature matching retrieval, i.e. based on the extracted local features, setting the search space range to within 1 km of the position of the target appearing in the previous day, setting the search date range to the whole day, performing target retrieval in the database to obtain a feature vector with the highest similarity; entering step S39.3;

[0146] Step S39.3: target overall global feature extraction;

[0147] Steps S39.4-S39.7: target overall global feature matching retrieval, trajectory verification, trajectory tracking, and feature fusion, i.e. according to the time and space range set in step S39.2, combining the extracted target overall feature vector in step S39.3, repeating steps S33-S39 to obtain the movement trajectory of the target on the day, and when the completion enters step S310;

[0148] Step S310: trajectory fusion, integrating and converging the target trajectories of different dates obtained by space progressive retrieval and time progressive retrieval to obtain a complete trajectory result of the target within a specified date range and output.

[0149] As shown in Figure 4 The second aspect of the embodiment of the application provides a camera video data retrieval device, and the camera video data retrieval device comprises:

[0150] A video acquisition module S41 is configured to acquire camera video data.

[0151] An algorithm analysis module S42 is configured to extract classification information from the acquired camera video data to obtain extracted classification information.

[0152] A data storage module S43 is configured to insert the extracted classification information into a file storage library, a structured database and a feature vector library in a database according to information categories.

[0153] The target retrieval module S44 is configured to acquire input target information to be retrieved and a retrieval condition, parse a target feature vector to be retrieved according to the target information to be retrieved, perform retrieval in the file storage, the structured database and the feature vector library based on the retrieval condition and the target feature vector by using a spatial progressive retrieval manner and a temporal progressive retrieval manner, restore a target multi-day travel trajectory through a retrieved result, and output the target multi-day travel trajectory.

[0154] The third aspect of the embodiments of the present application provides a computer readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the camera video data retrieval method.

[0155] The fourth aspect of the embodiments of the present application provides a terminal device, which comprises a processor, a memory and a communication bus; the memory stores a computer readable program which can be executed by the processor;

[0156] The communication bus realizes connection and communication between the processor and the memory;

[0157] The processor realizes the steps in the camera video data retrieval method when executing the computer readable program.

[0158] In summary, the present application provides a camera video data retrieval method and related devices, the method comprising: acquiring camera video data; extracting classification information from the acquired camera video data to obtain extracted classification information; inserting the extracted classification information into a file storage, a structured database and a feature vector library in a database according to information categories; acquiring input target information to be retrieved and a retrieval condition, parsing a target feature vector to be retrieved according to the target information to be retrieved, performing retrieval in the file storage, the structured database and the feature vector library based on the retrieval condition and the target feature vector by using a spatial progressive retrieval manner and a temporal progressive retrieval manner, restoring a target multi-day travel trajectory through a retrieved result, and outputting the target multi-day travel trajectory. Through the above method, the present application can extract features of targets such as pedestrians and vehicles in a video, store only the extracted information in a database, avoid the problem of storage and convergence of full-amount video, and realize efficient retrieval of targets based on features. Specifically, in the retrieval strategy, a spatiotemporal progressive retrieval manner is adopted to ensure retrieval efficiency and avoid blind retrieval in a full-city range and a full-time period.

[0159] It should be noted that, as used in this document, the terms "includes" and / or "containing" or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0160] Those of ordinary skill in the art will understand that all or part of the processes of the above-mentioned embodiments can be implemented by a computer program instructing relevant hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database, or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0161] It should be understood that the application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all such improvements and changes are within the scope of the appended claims of the present application.

Claims

1. A camera video data retrieval method, characterized by, The method comprises: acquiring camera video data; extracting classification information from the acquired camera video data to obtain extracted classification information; inserting the extracted classification information into a file storage, a structured database and a feature vector library in a database according to information categories; acquiring input target information to be searched and a search condition, analyzing a target feature vector to be searched according to the target information to be searched, and searching in the file storage, the structured database and the feature vector library based on the search condition and the target feature vector by using a spatial progressive search method and a temporal progressive search method, and restoring a target multi-day travel trajectory by using a searched result and outputting the target multi-day travel trajectory; the searching in the file storage, the structured database and the feature vector library based on the search condition and the target feature vector by using the spatial progressive search method and the temporal progressive search method specifically comprises: performing feature vector primary screening on vectors in the feature vector library according to the search condition and the target feature vector to obtain K primary screened feature vectors; performing feature vector fusion, that is, weighted average fusion of the K primary screened feature vectors to obtain a first fusion feature vector; performing a secondary search process according to the obtained first fusion feature vector and a similarity threshold, that is, performing secondary search on the vectors in the feature vector library to obtain M secondary search feature vectors; performing a trajectory restoration verification process according to the obtained M secondary search feature vectors, that is, acquiring corresponding M frame extraction pictures, structured data corresponding to the frame extraction pictures and feature vector data corresponding to the frame extraction pictures from the database according to the obtained secondary search feature vectors, restoring a target first trajectory according to the acquired M frame extraction pictures in a time and space order, and each frame extraction picture corresponding to a trajectory point in the target first trajectory; performing trajectory tracking verification on the target first trajectory and deleting abnormal trajectory points; performing target feature vector fusion after the trajectory restoration verification, that is, weighted average fusion of feature vector data corresponding to the trajectory points not deleted in the first trajectory to obtain a second fusion feature vector; performing spatial progressive search based on the obtained first trajectory and the second fusion feature vector to obtain a travel trajectory of the target in different ranges on the current day, merging the travel trajectories of the target in different ranges on the current day after the spatial progressive search is completed to obtain a moving trajectory of the target on the current day; when cross-day search is performed, acquiring corresponding frame extraction pictures based on the moving trajectory of the target on the current day, performing temporal progressive search on the frame extraction pictures corresponding to the moving trajectory of the current day to obtain a moving trajectory of the target in a search time; integrating and converging the moving trajectories of different days obtained by the spatial progressive search and the temporal progressive search to obtain a complete trajectory result of the target in a specified date range. the extracting classification information from the acquired camera video data to obtain extracted classification information specifically comprises:

2. The method of claim 1, wherein, the extracted classification information is frame extraction pictures, structured data and feature vector data; performing frame extraction processing on the camera video data to obtain frame extraction pictures; extracting basic attribute information in the frame extraction pictures to obtain structured data; ​ Feature information extraction is performed on the frame pictures to obtain feature vector data.

3. The method of claim 2, wherein, The extracted classification information is inserted into the file storage, the structured database and the feature vector library in the database according to information categories, and specifically includes: The frame pictures are inserted into the file storage in the database through a message queue; The structured data is inserted into the structured database in the database through a message queue; The feature vector data is inserted into the feature vector library in the database through a message queue.

4. The method of claim 1, wherein, The input target information to be searched and search conditions are obtained, and a target feature vector to be searched is parsed according to the target information to be searched, and specifically includes: The input target information to be searched and search conditions are obtained, and a target feature vector to be searched is parsed according to the target information to be searched, and specifically includes: The search conditions are obtained, and a similarity threshold, a search time range and a search space range are determined.

5. The method of claim 4, wherein, The similarity threshold is a preset threshold.

6. The method of claim 1, wherein, The space progressive search specifically includes: The space progressive search takes the track end point obtained in the last time track restoration verification process as a circle center, and a preset distance range as a radius, and sequentially performs a secondary search process, a track restoration verification process and a target feature vector fusion process according to the last time fused feature vector, until the feature vector that meets the similarity threshold requirement cannot be searched in the secondary search, the track search of the target in the current day is ended, a plurality of track of the target in the current day is obtained, and the track of the target in the current date is obtained by collecting the plurality of track of the target in the current day; In the repeated secondary search process, track restoration verification process and target feature vector fusion process, the data corresponding to the camera that has been searched is removed each time the secondary search is performed.

7. The method of claim 1, wherein, The time progressive search specifically includes: Local features of the target are extracted from the frame pictures corresponding to the track in the current date; Based on the extracted local features, the search space range is set to be within L kilometers of the position of the target in the current date, the search date range is set to be the date to be searched across days, and target search is performed in the database to obtain a feature vector with the highest similarity; A frame picture corresponding to the feature vector with the highest similarity is extracted, and a target overall feature vector in the frame picture is extracted; According to the set search space range, search date range and extracted target overall feature vector in the frame picture, a feature vector preliminary screening, a feature vector fusion, a secondary search process, a track restoration verification process, a target feature vector fusion process and a space progressive search process are sequentially performed to obtain the track of the target in the date to be searched across days.

8. The method of claim 1, wherein, The feature vector preliminary screening specifically includes coarse screening and fine screening. The coarse screening is to filter the feature vectors in the feature vector library according to the search conditions and the target feature vector to obtain the feature vectors that meet the conditions; The fine screening is to calculate the similarity between the feature vectors obtained by the coarse screening and the target feature vector, and to remove the feature vectors that do not meet the similarity threshold according to the set similarity threshold to obtain the preliminary screening feature vectors.

9. The method of claim 8, wherein, The similarity of the obtained feature vector and the target feature vector is calculated by using the Euclidean distance.

10. A camera video data retrieval apparatus, characterized by comprising: The camera video data retrieval device is used to implement the camera video data retrieval method in any one of claims 1-9, and the camera video data retrieval device comprises: a video acquisition module, configured to acquire camera video data; an algorithm analysis module, configured to extract classification information from the acquired camera video data to obtain extracted classification information; a data storage module, configured to insert the extracted classification information into a file storage, a structured database and a feature vector library in a database according to information categories; a target retrieval module, configured to acquire input target information to be retrieved and a retrieval condition, parse a target feature vector to be retrieved according to the target information to be retrieved, and perform retrieval in the file storage, the structured database and the feature vector library based on the retrieval condition and the target feature vector by using a spatial progressive retrieval mode and a time progressive retrieval mode, and restore a target multi-day travel trajectory by using the retrieved result and output the target multi-day travel trajectory.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores one or more programs which can be executed by one or more processors to implement the steps of the camera video data retrieval method in any one of claims 1-9.

12. A terminal device, comprising: comprise: a processor, a memory and a communication bus; the memory stores computer readable programs which can be executed by the processor; the communication bus realizes connection and communication between the processor and the memory; the processor executes the computer readable programs to implement the steps of the camera video data retrieval method in any one of claims 1-9.

Images