A video retrieval method based on Elastic Search for cloud networks

Abstract

This invention discloses a video retrieval method based on Elastic Search for cloud networks, belonging to the field of computer vision technology. The method comprises three modules: video structured preprocessing, image feature extraction, and Elastic Search retrieval. First, target detection technology is used to perform structured preprocessing on massive amounts of video data. Second, a video structured information database is constructed based on Elastic Search. Third, Elastic Search locates key video frames by comparing the feature data of the target image with the video structured information database. Finally, a key segment localization algorithm is used to process the key video frames located by Elastic Search, ensuring the accuracy, relevance, and continuity of the key video segments. This invention extracts features from the input target image and quickly compares them with a video feature database to output key video segments. It can be applied to human-shaped surveillance scenarios in video monitoring. The video retrieval system accurately locates video segments of the target human based on the similarity between the features of the target human image and the features detected by the target in the video frame.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a video retrieval method based on ElasticSearch for cloud networks. Background Technology

[0002] With the rapid development of the Internet of Things (IoT), video plays a crucial role as a carrier of dynamic events. Tens of thousands of terminal devices capture, store, and transmit massive amounts of video information, such as vehicle cameras, surveillance cameras, and multimedia cameras. However, how to process video data, especially how to retrieve key information from complex video data, has become a major challenge in the IoT era.

[0003] Object detection technology is a key technology in the field of computer vision and plays a huge role in image and video data processing. Traditional object detection technology is based on a combination of manual feature extraction and classification, which has the disadvantages of low representation ability, poor robustness and generalization. In recent years, the progress of deep learning (DL) technology has made the image processing capabilities of neural networks more and more advanced and detailed. DL has enabled object detection technology to break through the bottleneck of traditional algorithms. Reference [1] draws on AlexNet [2] The sliding window idea was used to propose R-CNN (Regions with CNN features), a deep learning object detection algorithm for region recognition. In order to better integrate the candidate region generation process into the deep network, reference [3] proposed Faster-RCNN, which for the first time realized a true end-to-end framework. Reference [4] proposed the YOLO (You Only Look Once) algorithm, a one-stage object detection algorithm, to solve the problem of slow detection speed, but the detection accuracy was somewhat lost. Reference [5] combined the advantages of YOLO and Faster-RCNN to propose the SSD (Single Shot MultiBoxDetector) algorithm, which can ensure that the speed meets the real-time requirements and achieve high detection accuracy.

[0004] However, the target detection algorithms proposed in the above-mentioned research works [1], [3]-[5] only consider target detection in image data and ignore the detection algorithms in video data. Although video data can be processed by extracting video frame images to apply image target detection algorithms, it is still difficult to ensure the real-time performance of the detection. Considering the differences between the scene and the target in video data, the target detection algorithms for video data are often designed for specific tasks. For example, reference [6] gives a text detection algorithm in video data, reference [7] introduces a target detection algorithm for vehicle detection, recognition and tracking in video data, and reference [8] designs SeqNet (Sequential End-to-End Network) for pedestrian re-identification in video data. However, the existing video detection algorithms only realize the feature information of the detected target and do not provide a means of video retrieval. How to retrieve video segments based on the target image has become an urgent problem to be solved.

[0005] Elastic Search (ES) provides a distributed, highly scalable, and real-time data analysis solution [9]. ES can store the feature information obtained from target detection and achieve efficient feature information comparison. Therefore, ES can be well applied to the scenario of retrieving video segments based on target images. How to design a high-speed, high-real-time, and high-accuracy video retrieval algorithm based on ES, especially a video retrieval algorithm for retrieving video segments based on target images, has become a major problem to be solved by this invention.

[0006] [1]Girshick R,Donahue J,Darrell T,Malik J.Rich feature hierar-chies for accurate object detection and semantic segmentation[J].CVPR.2014:580–587.

[0007] [2]Krizhevsky A, Sutskever I, and Hinton G E.Imagenet classifi-cation with deep convolutional neural networks[J].Advances in neu-ral informationprocessing systems.2012:25.

[0008] [3]Ren S,He K,Girshick R,Sun J.Faster r-cnn:Towards real-time objectdetection with region proposal networks[J].Advances in neural informationprocessing systems,2015:91–99.

[0009] [4]Redmon J,Divvala S,Girshick R,Farhadi A.You only look once:Unified,real-time object detection[J].CVPR.2016:779–788.

[0010] [5]Liu W,Anguelov D,Erhan D,Szegedy C,Reed S,Fu C Y,Berg A C.Ssd:Single shot multibox detector[J].ECCV,Springer.2016:21–37.

[0011] [6]Yin X C,Zuo Z Y,Tian S,and Liu C L.Text detection,tracking andrecognition in video:A comprehensive survey[J].IEEE Transac-tions on ImageProcessing.2016,6(25):2752–2773.

[0012] [7]Shobha B S,Deepu R.A review on video based vehicledetec-tion,recognition and tracking[C].20183rd International Conference on ComputationalSystems and Information Technology for Sustaina-ble Solutions.2018:183–186.

[0013] [8]Li ZJ,Miao D Q.Sequential end-to-end network for efficient personsearch[C].Proceedings of the AAAI Conference on Artificial Intelligence.2021,3(35):2011-2019.

[0014] [9]Gormley C,Tong Z.Elasticsearch:The Definitive Guide:A DistributedReal-Time Search and Analytics Engine[M].O'Reilly Media,Inc.2015. Summary of the Invention

[0015] To address the aforementioned problems, this invention proposes an Elastic Search-based video retrieval method for cloud networks, capable of returning similar video segments from a video library in real time based on an input target image. First, object detection technology is used to perform structured preprocessing on massive amounts of video data. Second, a video structured information database is constructed based on Elastic Search. Third, Elastic Search locates key video frames by comparing the feature data of the target image with the video structured information database. Finally, a key segment localization algorithm is used to process the key video frames located by Elastic Search, ensuring the accuracy, relevance, and continuity of key video segments. This invention designs and implements an Elastic Search-based video retrieval system that extracts features from the input target image and quickly compares them with a video feature database to output key video segments.

[0016] This invention applies to a video retrieval method based on Elastic Search in cloud networks, and the implementation steps are as follows:

[0017] Step 1: Use object detection technology to perform structured preprocessing on massive video data.

[0018] Step 2: Store the pre-processed video data in an Elasticsearch database, including the FrameID key field of the keyframe, named as the primary key field of the video + the primary key field of the keyframe + the primary key field of the target; the target's location coordinates (BBox) and feature vector (FeatureMap).

[0019] Step 3: Use the same target detection technique as in Step 1 to extract features from the target image and store the target's location coordinates (BBox) and feature vector (FeatureMap).

[0020] Step 4: Construct the ES retrieval module of the video retrieval platform backend. The module queries data information based on the similarity between the feature vector extracted from the target image and the Feature Map in the database. This includes the construction of the ES feature comparison algorithm and the key segment localization algorithm.

[0021] The ES feature alignment algorithm is constructed as follows:

[0022] Given the feature map data after structured preprocessing of the video Where k represents the target number; p is the Feature Map data after feature extraction from the target image; and the cosine similarity between vectors is defined as the feature similarity, denoted as...

[0023]

[0024] The similarity is defined as -1 ≤ θ ≤ 1; the closer θ is to 1, the more similar the two vectors are, and vice versa. A feature similarity threshold θ is defined. t Then θ≥θ t The target image is considered to be similar to the target in the video frame.

[0025] The key segment localization algorithm is constructed as follows:

[0026] Using the ES feature comparison algorithm, ES outputs Frame ID data similar to the target image from the database containing structured preprocessed video data. This data includes the corresponding video ID data i and frame ID data j. i The frame ID is combined with the time interval Δt set by the keyframe extraction method based on time interval to calculate the time T of the frame. j , represented as:

[0027] T j =j×Δt

[0028] This yields the timestamps of all video frames similar to the target image, which can then be used to locate the segment in the original video where the target appears.

[0029] Step 5: Build the front end of the video retrieval platform based on Elasticsearch.

[0030] The video retrieval platform front-end has a target image input section. Based on the target image input by the user at the front-end, it compares the data of the video library after video structured preprocessing. It outputs the start and end times of the target image appearing in the original video through the ES feature comparison algorithm and key segment localization algorithm, and outputs the start and end times of the target image to the front-end. The front-end plays similar segments in the original video according to the start and end times.

[0031] This invention applies to a video retrieval method based on Elastic Search in cloud networks, and the specific steps are as follows:

[0032] The advantages of this invention are:

[0033] 1. This invention applies to a video retrieval method based on Elastic Search in cloud networks, which realizes the segment retrieval and output of key video segments based on target images;

[0034] 2. This invention applies to the video retrieval method based on Elastic Search in cloud networks, effectively ensuring the real-time performance and accuracy of the retrieval.

[0035] 3. The video retrieval method based on Elastic Search applied to cloud networks in this invention has low system resource consumption;

[0036] 4. The video retrieval method based on Elastic Search applied to cloud networks in this invention can retrieve different target images and can be widely used in various fields such as surveillance camera video retrieval and vehicle camera video retrieval. Attached Figure Description

[0037] Figure 1 This is a system flowchart of the video retrieval algorithm based on Elastic Search applied to cloud networks according to the present invention;

[0038] Figure 2 This is an example of a human-shaped surveillance video scene using the method of the present invention;

[0039] Figure 3 This is a schematic diagram illustrating the retrieval effect of the video retrieval algorithm based on Elastic Search applied to cloud networks in a human surveillance scenario.

[0040] Figure 4 This is a performance test table of video structured preprocessing in the video retrieval algorithm based on Elastic Search applied to cloud networks.

[0041] Figure 5 This is a system storage pressure test table for the video retrieval algorithm based on Elastic Search applied to cloud networks, based on the present invention.

[0042] Figure 6 This is a test table for the real-time performance of video retrieval in a system based on Elastic Search video retrieval algorithm applied to cloud networks.

[0043] Figure 7This is a test table of system retrieval accuracy in the video retrieval algorithm based on Elastic Search applied to cloud networks. Detailed Implementation

[0044] The present invention will now be described in further detail with reference to the accompanying drawings.

[0045] This invention applies to a video retrieval method based on Elastic Search in cloud networks. The overall model is as follows: Figure 1 As shown, the method comprises three modules: video structured preprocessing, image feature extraction, and Elasticsearch (ES) retrieval. First, target detection technology is used to perform structured preprocessing on massive amounts of video data. Second, a video structured information database is constructed based on ES. Third, ES locates key video frames by comparing the feature data of the target image with the video structured information database. Finally, a key segment localization algorithm is used to process the key video frames located by ES, ensuring the accuracy, relevance, and continuity of the key video segments. This invention's method extracts features from the input target image and quickly compares them with a video feature database to output key video segments, and can be applied to applications such as... Figure 2 In the human-shaped surveillance scene shown in the video, the video retrieval system accurately locates the video segment of the target human based on the similarity between the features of the target human image and the features of the target detection in the video frame.

[0046] This invention applies to a video retrieval method based on Elastic Search in cloud networks. The specific implementation steps are as follows:

[0047] Step 1: Construct a video structured preprocessing module to transform raw video data into data that is easy to store, extract, and compare.

[0048] Step 101: Given a massive video database, this video database stores N... video There are three video data sets in different formats, each with a duration of T. i The video structuring preprocessing module first converts video data into frame data and stores it in a frame database, then extracts keyframes from each video frame. There are two methods for keyframe extraction: one is based on inter-frame differences, and the other is based on time intervals. To ensure video retrieval performance, the time interval-based keyframe extraction method is used. Given a time interval Δt, the number of keyframes is:

[0049]

[0050] In the formula, This is for rounding down.

[0051] Step 102: Considering the similarity between video frame data and image data for target detection, a target detection algorithm is used to perform target detection on the data in the frame database. Given an accuracy metric τ, target detection can label, locate, and extract features from all targets in the video frame data that meet the accuracy (Score ≥ τ). Assume that for each keyframe j, there are n targets that meet the accuracy. j If there are 1, then the total target number is:

[0052]

[0053] Step 2: Preprocess the video using structured preprocessing (N) target Each piece of data is stored in an Elasticsearch database. To facilitate data storage after structured video preprocessing, the key data definitions are as follows:

[0054]

[0055] This data contains relevant data for all targets in the original video data. FrameID is the primary key field of the keyframe, named as: Video Primary Key Field + Keyframe Primary Key Field + Target Primary Key Field. For example, the 4th target in the keyframe at the 8th second of video ID 2 would have FrameID 2_8_4. BBox represents the target's location coordinates. For example, if a keyframe is 1920*1080 pixels, and the target's bounding box is located at the top left corner (20, 100), with a length of 50 and a height of 150, the coordinates would be stored as (20, 100, 50, 150). FeatureMap is the feature vector, a floating-point vector, resulting from compressing the target image captured from the BBox coordinates using a target detection neural network. It is a vector with values ​​such as [0.3, 0.2, ..., 0.7].

[0056] It is worth noting that video structuring preprocessing is performed in the background, and only one video structuring preprocessing is needed to generate relevant data for all targets in the video library.

[0057] Step 3: Construct an image feature extraction module. Similar to video structured preprocessing, in order to realize image retrieval of video data, image feature extraction is required to convert the image into a feature vector that is easy to compare.

[0058] Given a target image to be searched, to avoid interference from feature extraction algorithms, the same target detection algorithm used in video structured preprocessing is employed for feature extraction. Since there is only one target in the image, the output data after target detection does not need to distinguish IDs; only the bounding box and feature map data are stored.

[0059] Step 4: Construct the ES retrieval module in the backend of the ES-based video retrieval platform. This module queries data information based on the similarity between the feature vector extracted from the target image and the Feature Map in the database. This includes the construction of the ES feature comparison algorithm and the key segment localization algorithm.

[0060] The construction method of the ES feature matching algorithm is as follows:

[0061] Given the feature map data after structured preprocessing of the video Where k represents the target number; p is the Feature Map data after feature extraction from the target image. Since the Feature Map is the feature vector of the image, the cosine similarity between vectors can be defined as the feature similarity, expressed as:

[0062]

[0063] The similarity threshold is -1 ≤ θ ≤ 1. The closer θ is to 1, the more similar the two vectors are; conversely, the closer θ is to -1, the less similar the two vectors are. Define the feature similarity threshold θ. t Then θ≥θ t The target image is considered similar to the target in the video frame. The specific steps of the ES feature comparison algorithm are as follows:

[0064]

[0065]

[0066] The algorithm described above provides a method for calculating the similarity of feature matching, and it is based on a similarity threshold θ. t Similar video frames were distinguished.

[0067] The key segment localization algorithm is constructed as follows:

[0068] Using the ES feature comparison algorithm, ES can output Frame ID data similar to the target image from the database containing structured preprocessed video data. This data includes the corresponding video ID data i and frame ID data j. i The frame ID can be calculated from the time interval T set by the keyframe extraction method based on the time interval. j , represented as

[0069] T j =j×Δt

[0070] This yields the timestamps of all video frames similar to the target image, which can then be used to locate the segment in the original video where the target appears. The specific steps are as follows.

[0071]

[0072]

[0073] The algorithm described above can accurately locate the start and end times of the target's appearance in the original video based on the data output by the ES feature comparison algorithm. Based on these start and end times, the video can be cropped to output the video segments in which the target appears.

[0074] Step 402: Build the front end of the video retrieval platform based on Elasticsearch.

[0075] The video retrieval platform front-end has a target image input section. Based on the target image input by the user at the front-end, it compares the data of the video library after video structured preprocessing. It outputs the start and end times of the target image in the original video through the ES feature comparison algorithm and key segment localization algorithm, and outputs the start and end times of the target image to the front-end. The front-end plays similar segments in the original video according to the start and end times.

[0076] The schematic diagram of the results obtained by the video retrieval algorithm based on Elastic Search for cloud networks proposed in this invention for retrieving surveillance videos from a video library is shown below. Figure 3 As shown. In Figure 3 In this study, the pre-trained SeqNet pedestrian re-identification algorithm was selected as the object detection algorithm, and surveillance videos from busy downtown streets were used as the video database. The accuracy metric for SeqNet was set to τ = 0.9, and the similarity threshold for the ES retrieval algorithm was set to θ. t =0.74, the segment interval weight of the key segment localization algorithm is w g =20, fragment length weight is w l =5. In this process, the constructed humanoid monitoring platform can perform structured preprocessing of video data in the video library in the background, retrieve randomly captured humanoid target images in real time, and accurately output the video clips in which the target appears. This invention conducted similar experiments on different targets in different numbers of different monitoring videos, demonstrating that the proposed algorithm can effectively retrieve video clips in which the target appears.

[0077] The video structuring preprocessing performance test of the video retrieval algorithm based on Elastic Search proposed in this invention and applied to cloud networks is as follows: Figure 4 As shown. In Figure 4In this system, an NVIDIA GeForce RTX 3090 graphics card and an Intel Xeon 4216 CPU were selected. Keyframe extraction speed and average video structuring preprocessing speed were calculated based on the total number of frames, while object detection speed was calculated based on the number of keyframes. The video structuring preprocessing background processing speed is relatively fast, requiring only 7 hours for a 20-hour video, which creates favorable conditions for real-time retrieval.

[0078] The storage stress test of the video retrieval algorithm system based on Elastic Search proposed in this invention and applied to cloud networks is as follows: Figure 5 As shown, the 26.3GB of video data in the video library, after video structuring preprocessing, only occupies about 4.6GB, and this data is stored in the Elasticsearch database. Therefore, the proposed algorithm has a low computer resource consumption.

[0079] The video retrieval real-time performance test of the video retrieval algorithm system based on Elastic Search proposed in this invention and applied to cloud networks is as follows: Figure 6 As shown. Real-time performance is an essential requirement for video retrieval systems. In the algorithm proposed in this invention, the video structured preprocessing module runs in the background and will not significantly impact the system's real-time performance. Therefore, considering the image feature extraction and ES retrieval modules, both have relatively fast algorithmic efficiency, with a real-time video retrieval latency of approximately 3 seconds, which meets the real-time requirement.

[0080] The video retrieval accuracy test of the video retrieval algorithm system based on Elastic Search proposed in this invention and applied to cloud networks is as follows: Figure 7 As shown, the number of incorrectly retrieved segments is defined as FP (False Positive); the number of correctly retrieved segments is defined as TP (True Positive); and the number of correctly retrieved segments that were not found is defined as FN (False Negative), i.e., the number of missed detections.

[0081] Define recall as

[0082]

[0083] Precision is defined as...

[0084]

[0085] This invention performs video retrieval on a specific target within a limited-length video, achieving a recall and precision rate both exceeding 90%. This demonstrates that the system meets the accuracy requirements.

[0086] In summary, by implementing the Elastic Search-based video retrieval algorithm for cloud networks proposed in this invention, key video segments can be retrieved and output based on target images. While effectively ensuring the real-time performance and accuracy of the retrieval, it also has low system resource consumption. The video retrieval algorithm proposed in this invention can retrieve different target images and can be widely applied in various fields such as surveillance camera video retrieval and vehicle-mounted camera video retrieval.

Claims

1. A video retrieval method based on Elastic Search applied to cloud networks, characterized in that: The implementation steps are as follows: Step 1: Use object detection technology to perform structured preprocessing on massive amounts of video data; Step 2: Store the pre-processed video data in an ES database, including the FrameID key field of the keyframe, named as the primary key field of the video + the primary key field of the keyframe + the primary key field of the target; the target's location coordinates (BBox) and feature vector (FeatureMap); the structuring and processing method is as follows: first, convert the video data into frame data and store it in the frame database, and then extract the keyframes of each video frame data. Subsequently, given an accuracy metric, target detection is used to label, locate, and extract features from all targets in the video frame data that meet the accuracy target. Step 3: Use the same target detection technique as in Step 1 to extract features from the target image and store the target's location coordinates (BBox) and feature vector (FeatureMap). Step 4: Construct the ES retrieval module of the video retrieval platform backend based on ES. The module queries data information based on the similarity between the feature vector extracted from the target image and the Feature Map in the database. This includes the construction of the ES feature comparison algorithm and the construction of the key segment localization algorithm. The ES feature alignment algorithm is constructed as follows: Given the feature map data after structured preprocessing of the video ,in Indicates the target number; Feature Map data after target image feature extraction. Define the cosine similarity between vectors as feature similarity, denoted as: Among similarity ;when The closer to The more similar two vectors are, the less similar they are; conversely, the less similar they are, the more similar they are. The closer The less similar the two vectors are, the more similar they are; define a feature similarity threshold. Then it satisfies The target image is considered to be... The corresponding keyframe targets are similar; The key segment localization algorithm is constructed as follows: Using the ES feature comparison algorithm, ES outputs Frame ID data similar to the target image from the database containing structured preprocessed video data. This data includes the corresponding video ID data. and frame ID data The frame ID is combined with the time interval set by the keyframe extraction method based on the time interval to calculate the frame's time. , is represented as: Thus, the timestamps of all video frames similar to the target image are obtained, and the segments in the original video in which the target appears can be located using these timestamps. Step 5: Build the front end of the video retrieval platform based on Elasticsearch; The video retrieval platform front-end has a target image input section. Based on the target image input by the user at the front-end, it compares the data of the video library after video structured preprocessing. It outputs the start and end times of the target image appearing in the original video through the ES feature comparison algorithm and key segment localization algorithm, and outputs the start and end times of the target image to the front-end. The front-end plays similar segments in the original video according to the start and end times.

Images