Efficient video data distributed retrieval method and system

Abstract

The invention discloses an efficient video data distributed retrieval method and system, and the method comprises the steps: converting video data into a feature vector sequence through an image feature extraction technology; then, prefix features of the sample sub-feature sequences are constructed and extracted through data indexes, and spatial indexes are constructed on all sub-feature dimensions respectively; finally, through query processing, candidate sets are queried and filtered on all the sub-indexes based on the lower bound distance, and accurate video retrieval is completed through the intersection of the candidate sets and the follow-up processing process. By applying the multivariate time series data distributed similarity query method provided by the invention, distributed storage and efficient query of large-scale video data can be realized, and query contents can be quickly identified and positioned in a massive video database for given video paragraphs or semantic descriptions. The video data distributed index structure provided by the invention can effectively support video retrieval with unequal durations, has relatively high spatial ductility, and can play an important role in the fields of big data, data mining, databases and the like.

Description

Technical Field

[0001] The present invention belongs to the fields of big data, database, data mining, etc., and specifically relates to a video data distributed retrieval method and system. Background Art

[0002] Video retrieval refers to retrieving video content related to the query conditions from large-scale video data. It is mainly divided into content-based video retrieval and text-based video retrieval. Existing retrieval methods are mainly based on inverted index and approximate nearest neighbor query methods. Both of these methods are derived from traditional text retrieval methods and have high computational complexity when processing high-dimensional video data. The latest retrieval method is based on multimodal fusion, which requires the combination of multimodal information such as visual information, audio information, and text information to improve retrieval accuracy.

[0003] The main difficulties and challenges of current video retrieval technology are: 1) The rapid increase in video data poses a great challenge to the scalability of existing retrieval methods; 2) In-depth understanding of video content and effective feature extraction are still very difficult; 3) Real application scenarios such as video surveillance and live broadcasting have high requirements for real-time retrieval technology, which is difficult to meet the query performance of existing methods. Summary of the invention

[0004] In order to solve the shortcomings of the prior art and achieve a substantial compression of the video data index space to significantly improve the query efficiency, the present invention adopts the following technical solutions:

[0005] An efficient distributed retrieval method for video data comprises the following steps:

[0006] (1) Image frame feature extraction, converting each frame of video data into a set of multi-dimensional feature vector sequences;

[0007] (2) Data index construction: separating the group of multidimensional feature vector sequences in units of each dimension to obtain sub-feature sequences of each dimension to construct an index;

[0008] (3) Query processing: convert each frame of the retrieved video into a set of multi-dimensional retrieval video feature vector sequences, separate the set of multi-dimensional retrieval video feature vector sequences by dimension, obtain retrieval video sub-feature sequences of each dimension, calculate similarity based on the retrieval video sub-feature sequence and the video data sub-feature sequence on the index, and use the video objects contained in the index nodes with high similarity as the intermediate candidate set;

[0009] (4) Subsequent processing: taking the intersection of the intermediate candidate sets queried on the sub-features of each dimension, and comparing the original video data corresponding to the final candidate set with the retrieved video for similarity to obtain the query result.

[0010] Furthermore, in the step (2), the prefix sequence features of the sub-feature sequences are extracted through the sub-feature sequences of each dimension to construct an index; in the step (3), the prefix features of the retrieval video sub-feature sequences are extracted through the retrieval video sub-feature sequences of each dimension, and the Euclidean distance between the prefix feature of the retrieval video and the prefix feature corresponding to each index node is calculated in turn on the index; the Euclidean distance based on the prefix feature is the lower bound distance of the multivariate dynamic time warping distance, which can ensure the completeness of the query result, that is, the result of the query based on the multivariate dynamic time warping distance on the original data must be included in the intermediate candidate set based on the prefix feature Euclidean distance query.

[0011] Furthermore, in step (2), the starting value, final value, maximum value, and minimum value of a prefix sequence of equal length for each sub-feature sequence are extracted as its prefix feature; in step (3), the starting value, final value, maximum value, and minimum value of a prefix sequence of equal length for each retrieved video sub-feature sequence are extracted as its prefix feature.

[0012] Furthermore, in step (2), a metric index or a spatial index is constructed on the prefix feature set of each sub-feature sequence in the video database.

[0013] Furthermore, in step (3), on the index, the Euclidean distance of the prefix feature of each sub-feature sequence of the retrieved video to the prefix feature of the video data corresponding to the index node is calculated in turn, the nodes and their subtrees with distance values ​​greater than the query threshold are pruned, and the video objects contained in all leaf nodes with distance values ​​less than or equal to the query threshold are returned as the intermediate candidate set.

[0014] Furthermore, in the step (1), the SIFT scale-invariant feature transformation method is used to reduce the dimension and extract multiple-dimensional scale-invariant feature vectors of each frame image of each video data to obtain a set of multi-dimensional feature vector sequences.

[0015] Furthermore, in step (4), since the final candidate set obtained is an address value, it is necessary to read the original video data corresponding to the final candidate set in sequence through disk I / O, calculate the multivariate dynamic time warping distance between the feature vector sequence of each original video data and the feature vector sequence of the retrieved video, and take the video objects that are less than or equal to the query threshold as the query results.

[0016] Furthermore, in step (4), the calculation of the multivariate dynamic time warping distance specifically includes the following sub-steps:

[0017] (4.1) Construct a point-to-point distance matrix of two multivariate time series. Each cell value in the matrix is ​​the Euclidean distance between the sampling vectors at the corresponding positions of the two series.

[0018] (4.2) Use the dynamic programming algorithm to find the optimal curved path on the point-to-point distance matrix so that the cumulative sum of the element values ​​on the path is minimized. The cumulative sum of the path is used as the final distance metric to determine the similarity.

[0019] A retrieval system of an efficient video data distributed retrieval method is applied to a distributed database. The system includes a video feature extraction module, a data index construction module, a query processing module and a subsequent processing module, which are respectively used for the video feature extraction, the data index construction, the query processing and the subsequent processing. The sub-feature sequences of each dimension are deployed in the distributed database.

[0020] The advantages and beneficial effects of the present invention are:

[0021] 1. The traditional video retrieval method uniformly constructs indexes on all image frames. The high-dimensional characteristics of image frames make the lower bound distance not compact enough, resulting in a large size of the intermediate candidate set. The distributed index of the present invention can significantly reduce the size of the candidate set through intersection operations, greatly improving the query efficiency.

[0022] 2. Video data takes up a large storage space, and the traditional unified index structure requires data to be stored in a centralized manner, which places high demands on the storage capacity and expansion capability of the database. The distributed indexing method proposed in the present invention can effectively realize the distributed storage and retrieval of large-scale video data. At the same time, the sub-feature indexes are independent of each other, making the index structure highly scalable.

[0023] 3. Traditional video retrieval methods only support Euclidean distance measurement, while the Euclidean distance on the prefix sequence feature used in the present invention is the lower bound of the dynamic time warping distance, which can effectively support complete queries based on the dynamic time warping distance, has a higher retrieval accuracy, and can support the retrieval of videos of different lengths; BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of a method in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0026] like Figure 1 As shown in FIG. 1 , an efficient distributed retrieval method for video data is provided. The video database stores 100,000 videos of varying lengths, all with a video frame rate of 60 fps and a resolution of 1280*720. Distributed retrieval of the database includes the following steps:

[0027] (1) Image frame feature extraction: Using the SIFT scale-invariant feature transformation method, the 32-dimensional scale-invariant feature vector (32-SIFT feature vector) of each frame of each video sample is reduced in dimension and the video data is converted into a 32-SIFT vector to obtain a vector sequence; the 32-dimensional scale-invariant feature vector sequence of each frame of the image is as follows:

[0028]

[0029] Among them, a represents the feature vector of each frame of video data, the superscript represents the dimension of each frame, and the subscript n represents the maximum number of steps that the sliding window slides on each frame.

[0030] (2) Data index construction: Separate the sub-feature sequences of the SIFT vector sequence and separate the 32-SIFT sequence into 32 sub-feature sequences, as follows:

[0031]

[0032] The starting value, final value, maximum value, and minimum value of the prefix sequence of each sub-feature sequence length of 100 are extracted as the four-dimensional prefix feature, as follows:

[0033]

[0034] Among them, the subscript max indicates that the current feature a is the maximum value of the prefix sequence, the subscript min indicates that the current feature a is the minimum value of the prefix sequence, and l indicates that the current feature a is the final value of the prefix sequence;

[0035] A metric index or a spatial index is constructed on the prefix feature set of each sub-feature sequence in the video database. In this example, an M-tree index is constructed for the prefix feature sets of 32 sub-feature sequences respectively.

[0036] (3) Query processing: SIFT dimension reduction technology is used to extract the SIFT vector of each frame of the search video, and the search video is converted into a 32-SIFT vector sequence; the four-dimensional prefix features of the 32 sub-feature sequences of the search video are extracted respectively, as follows:

[0037]

[0038] Wherein, b represents the feature vector of each frame of the retrieved video, the superscript represents the dimension of each frame of the retrieved video, the subscript m represents the maximum number of steps that the sliding window slides in each frame of the retrieved video, the subscript max represents the maximum value of the current feature b as the prefix sequence, the subscript min represents the minimum value of the current feature b as the prefix sequence, and l represents the final value of the current feature b as the prefix sequence;

[0039] The Euclidean distance from the query feature to each index node is calculated on the 32 sub-feature sequence indexes of the database in turn, and the nodes and their subtrees with distance values ​​greater than the query threshold are pruned, and the video objects contained in all leaf nodes with distance values ​​less than or equal to the query threshold are returned as the intermediate candidate set;

[0040] The Euclidean distance based on prefix features is the lower bound distance of the multivariate dynamic time warping distance, which can ensure the completeness of the query results, that is, the results of the query based on the multivariate dynamic time warping distance on the original data must be included in the intermediate candidate set of the Euclidean distance query based on the prefix feature.

[0041] (4) Subsequent processing: Take the intersection of the intermediate candidate sets retrieved on the 32 sub-feature dimensions to obtain the final candidate set, read the original video data of the final candidate objects in sequence through disk I / O, calculate the multivariate dynamic time warping distance between the 32-SIFT vector sequence of each video and the 32-SIFT vector sequence of the retrieved video, and return the video objects that are less than or equal to the query threshold as the result; wherein the calculation of the multivariate dynamic time warping distance specifically includes the following sub-steps:

[0042] (4.1) Calculate the point-to-point distance matrix of two multivariate time series, where each cell value is the Euclidean distance between the sampling vectors at the corresponding positions of the two series;

[0043] (4.2) Use the dynamic programming algorithm to find the optimal curved path on the point-to-point distance matrix so that the cumulative sum of the element values ​​on the path is minimized, and the cumulative sum of the path is used as the final distance measurement value.

[0044] A retrieval system of an efficient distributed retrieval method for video data is applied to a distributed database. The system includes a video feature extraction module, a data index construction module, a query processing module and a subsequent processing module, which are respectively used for video feature extraction, data index construction, query processing and subsequent processing. The sub-feature sequences of each dimension are deployed in a distributed database.

[0045] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An efficient distributed retrieval method for video data, characterized in that The steps include: (1) Image frame feature extraction, converting each frame of video data into a set of multi-dimensional feature vector sequences; (2) Data index construction: separating the group of multidimensional feature vector sequences in units of each dimension, obtaining sub-feature sequences of each dimension, and constructing an index; (3) Query processing: convert each frame of the retrieved video into a set of multi-dimensional retrieval video feature vector sequences, separate the set of multi-dimensional retrieval video feature vector sequences by dimension, obtain retrieval video sub-feature sequences of each dimension, calculate similarity based on the retrieval video sub-feature sequence and the video data sub-feature sequence on the index, and use the video objects contained in the index nodes with high similarity as the intermediate candidate set; (4) Subsequent processing: taking the intersection of the intermediate candidate sets queried on the sub-features of each dimension, and comparing the original video data corresponding to the final candidate set with the retrieved video for similarity to obtain the query result.

2. The efficient distributed video data retrieval method according to claim 1, characterized in that: In the step (2), the prefix features of the sub-feature sequences are extracted through the sub-feature sequences of each dimension to construct an index; in the step (3), the prefix features of the retrieval video sub-feature sequences are extracted through the retrieval video sub-feature sequences of each dimension, and the Euclidean distance between the prefix feature of the retrieval video and the prefix feature corresponding to each index node is calculated in turn on the index.

3. The efficient distributed video data retrieval method according to claim 2, characterized in that: In the step (2), the starting value, final value, maximum value, and minimum value of the prefix sequence of equal length of each sub-feature sequence are extracted as its prefix feature; In the step (3), the starting value, final value, maximum value, and minimum value of a prefix sequence of equal length to each retrieved video sub-feature sequence are extracted as its prefix feature.

4. The efficient distributed video data retrieval method according to claim 2, characterized in that: In the step (2), a metric index or a spatial index is constructed on the prefix feature set of each sub-feature sequence in the video database.

5. The efficient distributed video data retrieval method according to claim 2, characterized in that: In the step (3), on the index, the prefix feature of each sub-feature sequence of the retrieved video and the Euclidean distance of the prefix feature of the video data corresponding to the index node are calculated in turn, the nodes and their subtrees with distance values ​​greater than the query threshold are pruned, and the video objects contained in all leaf nodes with distance values ​​less than or equal to the query threshold are returned as the intermediate candidate set.

6. The efficient distributed video data retrieval method according to claim 1, characterized in that: In the step (1), the SIFT scale-invariant feature transformation method is used to reduce the dimension and extract multiple-dimensional scale-invariant feature vectors of each frame image of each video data to obtain a set of multi-dimensional feature vector sequences.

7. The efficient distributed video data retrieval method according to claim 1, characterized in that: In the step (4), the original video data corresponding to the final candidate set is read in sequence through disk I / O, the multivariate dynamic time warping distance between the feature vector sequence of each original video data and the feature vector sequence of the retrieved video is calculated, and the video objects that are less than or equal to the query threshold are taken as query results.

8. The efficient distributed video data retrieval method according to claim 7, characterized in that: In the step (4), the calculation of the multivariate dynamic time warping distance specifically includes the following sub-steps: (4.1) Construct a point-to-point distance matrix of two multivariate time series. Each cell value in the matrix is ​​the Euclidean distance between the sampling vectors at the corresponding positions of the two series. (4.2) Use the dynamic programming algorithm to find the optimal curved path on the point-to-point distance matrix so that the cumulative sum of the element values ​​on the path is minimized. The cumulative sum of the path is used as the final distance metric to determine the similarity.

9. A retrieval system of an efficient distributed retrieval method for video data according to claim 1, applied to a distributed database, characterized in that: The system includes a video feature extraction module, a data index construction module, a query processing module and a subsequent processing module, which are respectively used for the video feature extraction, the data index construction, the query processing and the subsequent processing. The sub-feature sequences of each dimension are deployed in the distributed database.

Images