An edge-cloud collaborative approximate redundancy removal system and method for multi-camera video analytics

Abstract

This invention proposes an edge-cloud collaborative approximate deduplication system and method for multi-camera video analysis, belonging to the field of edge-cloud collaborative optimization technology in the Internet application layer. The invention proposes an edge-cloud collaborative caching architecture. The cloud cache provides more storage resources at a lower cost, while an edge cache is constructed using a maximum coverage cache selection algorithm. The edge cache is located closer to the camera, resulting in lower latency. The camera connects to the edge server and continuously transmits the captured video stream to it. By combining the advantages of both types of caching, the system quickly and accurately identifies the target. The same target is only identified once by the system, and subsequently, the target's features and the identification result are stored in the cache. When the target reappears, the system can retrieve the corresponding identification result from the cache without calling the model for repeated calculations, thus greatly saving computational resources.

Description

Technical Field

[0001] This invention belongs to the field of edge-cloud collaborative optimization technology in the Internet application layer, specifically an edge-cloud collaborative approximate redundancy removal system and method for multi-camera video analysis. Background Technology

[0002] In recent years, the deployment of cameras has surged, making the need to fully utilize their potential increasingly urgent. As a result, video analytics is rapidly evolving into an indispensable tool for both public and private organizations, helping them improve efficiency, reduce costs, and enhance security through Internet of Things (IoT) applications. The emergence of deep neural networks (DNNs) has ushered in a new era of accuracy in video analytics, but it has also brought higher resource demands.

[0003] When multiple cameras are deployed in the same environment, computational redundancy arises in video analytics applications due to the repeated occurrence of the same target object in the footage. For example... Figure 1 As shown, in a group of cameras distributed at an intersection or along a road, the same target object may be captured by different cameras or by the same camera at different times. This leads to repeated capture and analysis by the DNN model, resulting in computational redundancy and wasted computing resources. Eliminating the repeated calculation of the same object between cameras can reduce computational costs while maintaining low latency and high accuracy.

[0004] Currently, a large amount of research is dedicated to improving the performance of multi-camera video analytics pipelines. These studies can be divided into two categories: (1) camera clusters with overlapping viewpoints; and (2) camera clusters with non-overlapping viewpoints. The first category primarily utilizes the overlapping fields of view (FoV) between multiple cameras to establish correlations between camera capture areas, thereby enabling the sharing of analysis results. However, these methods are not suitable for camera scenarios with non-overlapping viewpoints, limiting their ability to comprehensively cover the monitored area. The second category focuses on scheduling the activation and deactivation of geographically dispersed cameras to track predetermined target objects among the cameras. However, these methods are mainly designed for retrospective queries and lack real-time inference capabilities. Summary of the Invention

[0005] To address the computational redundancy caused by multiple cameras capturing the same target in the same or different time and space scenarios, this invention proposes an edge-cloud collaborative approximate redundancy removal system and method for multi-camera video analysis.

[0006] An edge-cloud collaborative approximate redundancy removal method for multi-camera video analytics, with the following specific steps:

[0007] Step 1: Build an edge-cloud collaborative caching architecture, including edge servers and a cloud platform, and establish an initial cloud cache in the cloud platform.

[0008] The initial cloud cache setup process is as follows:

[0009] In the initial stage, the video capture system consisting of multiple cameras transmits the video stream over a period of time to the edge-cloud collaborative caching architecture for target recognition, and stores the recognized targets in the cloud platform to establish a cloud cache.

[0010] Step 2: Select features from the cloud cache using the maximum coverage cache selection algorithm to build an edge cache and store it on the edge server;

[0011] The process of building an edge cache using the maximum coverage cache selection algorithm is as follows:

[0012] First, determine the number of feature points k and the coverage radius r of the feature points required for the edge cache;

[0013] Then, all feature points P in the cloud cache are input into the maximum coverage cache selection algorithm, and the hierarchical navigable small-world method (HNSW) is used to find each feature point p in the cloud cache. i The nearest neighbors of each feature point are selected, and all neighbor points within the radius r covered by each feature point are selected to form feature point p. i The set of covered points S i Traverse all feature points in the cloud cache to obtain a coverage set S containing the coverage range of each feature point.

[0014] Finally, using a greedy strategy, the coverage point set of all feature points is iteratively selected until the number of feature points corresponding to the selected coverage point set reaches the number k required for the edge cache. The specific iterative process is as follows:

[0015] In the initial iteration, the set of cover points containing the most feature points is selected from the cover set S, denoted as Si. imax Then remove S from all feature points P. imax The included elements, while also including S imax Remove from S. Then repeat the above iterative selection process for the remaining set of covering points and feature points, with each selected set of covering points having the highest coverage rate. Finally, select the feature points p corresponding to each set of covering points. i Add to result set p result In, until P result It contains k feature points.

[0016] Step 3: In the formal operation phase of the edge-cloud collaborative caching architecture, the video stream is transmitted to the edge server for processing, and key frames are extracted.

[0017] The edge server processes the video stream as follows:

[0018] First, the video stream is decoded into consecutive video frames, and the video frames are transmitted to the video buffer.

[0019] Then, the video buffer classifies the video frames into key frames and non-key frames. The key frames are processed by model recognition, and the non-key frames are analyzed by a lightweight target tracking algorithm in the target tracking module.

[0020] Step 4: Use a convolutional network to extract the target bounding box in the keyframe, and further use a feature extractor to extract features from the target bounding box to obtain the target features;

[0021] Step 5: Use a fast approximate matching algorithm to query whether the extracted target features have already been stored in the edge cache as the corresponding recognition result. If they exist, return the corresponding recognition result immediately; otherwise, send the extracted target features to the cloud platform for further processing.

[0022] Step 6: The cloud platform uses a fast approximate matching algorithm to search the cloud cache based on the target features. If the target exists, the retrieved recognition result is returned to the edge server. If no corresponding result is found, the target recognition model deployed on the cloud platform is executed to calculate the corresponding result and returned to the edge server. At the same time, the result is added to the cloud cache.

[0023] Step 7: For the updated cloud cache, return to Step 2 to periodically update the edge cache and perform target recognition on the new video stream.

[0024] Furthermore, the specific process of the fast approximate matching algorithm based on HNSW in the above process is as follows:

[0025] Step 1: Initialize the cache using HNSW. During initialization, find multiple nearest neighbors for each feature point and construct a hierarchical graph containing the connections between neighbors.

[0026] Step 2: Using the hierarchical graph, find the m nearest neighbors of the new query feature to obtain a candidate set P for approximate matching. cand .

[0027] Step 3, for P cand The algorithm performs filtering, retaining only feature points whose distance to the query feature is less than a threshold. The filtered neighbors are then used as candidate approximate matching features. If the candidate set is empty after filtering, a message indicating that there is no corresponding result in the cache is returned.

[0028] The initial value of the threshold is determined by the distance between features of the same type.

[0029] Step 4: Examine the labels of the candidate approximate matching features and select the label with the highest frequency as the query result.result If multiple labels have the same frequency, select the label closest to the query feature as the query result.

[0030] Step 5: Based on the query results, dynamically adjust the matching threshold according to the feedback from the cloud platform's recognition model to adapt to the changing video content.

[0031] (1) The query result is l result

[0032] After completing the above recognition process, the fast approximate matching algorithm generates a random number between 0 and 1 and determines whether the random number is less than the fixed parameter η. If so, the threshold is adjusted; otherwise, the threshold remains unchanged.

[0033] The process of adjusting the threshold is as follows:

[0034] First, obtain the true labels of the query features from the recognition model. g If the query results have the tag l result With real labels g If there is a mismatch, it indicates that the current threshold is too lenient and needs to be tightened. In this case, divide the threshold by α (α>1) to adjust it to a smaller value.

[0035] The adjusted threshold is expressed as follows:

[0036]

[0037] (2) The query result is empty.

[0038] If the corresponding result is not found in the cache, the cache will be checked for features p that match the query target. q The nearest feature point p nrst The label is used to determine the threshold. If the distance between these two feature points is greater than the threshold, but the label matches the true label result of the recognition model, then the current threshold is too strict and needs to be relaxed. In this case, the threshold is adjusted using an exponentially weighted moving average, as shown in the following formula:

[0039] t = β × threshold + (1 - β) × |p nrst -p q |

[0040] Where β is a parameter used to control the rate of threshold adjustment. Threshold is the current threshold.

[0041] Step 6: Return to Step 3 and use the threshold for the next target matching.

[0042] The edge-cloud collaborative approximate redundancy reduction system for multi-camera video analytics is an edge-cloud collaborative architecture consisting of edge servers and a cloud platform. Cameras connect to the edge servers and transmit captured video streams to them, reducing computational redundancy by building caches across multiple cameras. The cloud platform connects to the edge servers, providing additional storage resources through cloud caching. The same target is only identified by the system once; subsequently, the target's features and the identification result are stored in the cache. When the target reappears, the system can retrieve the corresponding identification result from the cache without needing to call the model for repeated calculations.

[0043] Edge caching and cloud caching use a key-value pair structure to store relevant information. The key represents the feature vector extracted from the target object, while the value corresponds to the final result of the object recognition model.

[0044] The advantages and beneficial effects of this invention are as follows:

[0045] (1) In this invention, a collaborative caching framework is established. Edge caching is closer to the camera and has a shorter latency, while cloud caching has a lower cost and larger storage capacity. By combining the advantages of the two types of caching, the collaborative framework optimizes the performance and cost-effectiveness of the caching system.

[0046] (2) This invention proposes a maximum coverage cache selection algorithm and a fast approximate matching algorithm. The former can effectively select cached content to improve the cache hit rate, while the latter achieves efficient cache lookup. Combining these two algorithms allows this invention to reduce computational cost and end-to-end latency while maintaining accuracy. Attached Figure Description

[0047] Figure 1 This is a diagram illustrating the same target object being captured by different cameras.

[0048] Figure 2 This is a flowchart illustrating the edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis according to the present invention.

[0049] Figure 3 This is a flowchart of the approximate matching algorithm in this invention. Detailed Implementation

[0050] This invention first proposes an edge-cloud collaborative caching architecture to achieve near-redundancy removal in edge-cloud collaborative caching for multi-camera video analysis, such as... Figure 2As shown, the camera connects to an edge server and continuously transmits captured video streams to it. Since multiple cameras capturing the same target leads to redundant computation, this invention proposes a collaborative caching framework to reduce computational redundancy by building a cache across multiple cameras. The same target is only recognized once by the system, and subsequently, the target's features and recognition result are stored in the cache. When the target reappears, the system can retrieve the corresponding recognition result from the cache without calling the model for repeated computation, thus significantly saving computational resources. Edge caching is located closer to the camera, resulting in lower latency, but it has limited storage space and higher cost. Cloud caching provides more storage resources at a lower cost, but suffers from higher latency due to long transmission distances. By combining the advantages of both types of caching, the collaborative framework aims to optimize the performance and cost-effectiveness of the caching system.

[0051] Edge caching and cloud caching use a key-value pair structure to store relevant information. The key represents the feature vector extracted from the target object, while the value corresponds to the final result of the object recognition model, such as the label obtained by the model.

[0052] The specific implementation of the edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis includes the following main steps:

[0053] Step (1) During the initialization phase, the system runs for a specific period of time and stores the detection results corresponding to the identified targets into the cloud cache. During this process, the cloud cache will be continuously updated in real time.

[0054] Step (2) After the cloud cache is established, run the maximum coverage cache selection algorithm to select appropriate features to establish the edge cache, so as to maximize the hit rate of the edge cache.

[0055] To improve edge cache hit rate, selecting the most valuable cloud cache content is crucial. The construction of the edge cache is transformed into a maximum coverage problem, aiming to select a subset of cloud cache content that covers as many feature points as possible. Therefore, this invention proposes a maximum coverage cache selection algorithm, which aims to find the optimal subset of feature points that maximizes cumulative coverage within a given radius. By carefully selecting cache content according to the maximum coverage method, the edge cache hit rate can be effectively improved. This algorithm not only considers the importance of feature points but also determines the optimal subset to maximize coverage, thereby increasing the cache hit probability of queries related to these feature points.

[0056] like Figure 2 As shown in Algorithm 1, the maximum coverage cache selection algorithm is as follows:

[0057] The algorithm takes all feature points in the cloud cache (P) as input. Additionally, it needs to determine two parameters: the number of feature points (k) required for the edge cache and the coverage radius (r) of the feature points. k is used to balance cache hit rate and end-to-end latency, while r is determined based on the distribution size of feature points of the same type. A radius is determined by the distance between each feature point; feature points within this radius are considered to be of the same type. The algorithm's output is a subset of P, denoted as P0. result This result will become the content of the edge cache. The algorithm identifies the k feature points with the largest coverage as P through two steps. result :

[0058] First, for each point p in the cache i Construct a set of cover points S within a specified radius r i The algorithm uses the Hierarchical Navigable Small World (HNSW) method to find the nearest neighbors of feature points, and then determines whether the neighbors are within a specified radius r. By iteratively running HNSW on the neighbors, the algorithm finally obtains a coverage set S containing the coverage range of each feature point.

[0059] Secondly, the algorithm employs a greedy strategy, iteratively selecting the feature point with the largest intersection with the remaining uncovered points until k feature points have been selected. In each iteration, the feature point S with the largest intersection with P is selected. i Then delete S from P. i The elements contained in S, and remove S from S. i The selected S i The feature point p with the highest coverage is selected. i Add to result set P result Middle. Repeat this process until P. result It contains k feature points. This greedy selection process ensures that the selected feature points can achieve the maximum coverage of cached feature points, thereby effectively optimizing the use of edge cache.

[0060] Step (3) During the formal operation phase, the system selects key frames from the video frames captured by the end-side camera for subsequent processing.

[0061] After the edge server acquires the video stream transmitted from the camera, the video decoding module decodes the video stream into consecutive video frames and inputs the frames into the video buffer. Considering the temporal redundancy of the video frames, the system selects key frames exhibiting significant changes from the video buffer for subsequent model recognition processing. Non-key frames are analyzed using a lightweight target tracking algorithm in the target tracking module. Using target tracking reduces the number of video frames requiring further processing, thus reducing computational load, while ensuring that the accuracy of video analysis does not excessively decrease.

[0062] Step (4) Extract the features of the target in the keyframe and use a fast approximate matching algorithm to search for the corresponding results in the edge buffer.

[0063] The system uses a convolutional network to extract target bounding boxes from keyframes and a feature extractor to extract features from the identified objects. Based on the extracted features, the system uses a fast approximate matching algorithm to check in the edge cache whether the recognition result corresponding to the target has already been stored.

[0064] Step (5) If the search is successful, return the result; if the search fails, send the target features to the cloud platform for processing.

[0065] Step (6) Use the fast approximate matching algorithm on the cloud platform to search for the corresponding results of the identified target in the cloud cache.

[0066] Step (7) If the search is successful, the result is returned to the edge server; if the search fails, the target recognition model deployed on the cloud platform is used to identify the corresponding result and return it to the edge server, and the result is added to the cloud cache.

[0067] Step (8) The fast approximate matching algorithm adjusts the threshold based on the search results and updates the cached content.

[0068] The system periodically runs the maximum coverage cache selection algorithm to update the edge cache, ensuring that the content of the edge cache adapts to real-time changes in video content.

[0069] like Figure 2 As shown in Algorithm 2, in order to efficiently implement steps (4) and (6) of querying the edge / cloud cache, this invention proposes a fast approximate matching algorithm based on HNSW. By utilizing HNSW, the approximate matching algorithm can quickly identify potential matches in the cache. In addition, in order to better adapt to different video content, the algorithm dynamically adjusts the matching threshold according to environmental changes.

[0070] Fast approximate matching algorithms based on HNSW, such as Figure 3 As shown, the specific process is as follows:

[0071] First, the cache is initialized using HNSW. During initialization, multiple nearest neighbors are found for each feature point, and a hierarchical graph containing the connections between neighbors is constructed. Using this graph, the algorithm can quickly find the m nearest neighbors of a new query feature, thus obtaining a candidate set P for approximate matching. cand .

[0072] Next, the algorithm applies P candThe algorithm filters out feature points, retaining only those whose distance to the query feature is less than a threshold. Since feature matching is approximate, selecting the nearest neighbor within the threshold may not be accurate. Therefore, the algorithm uses the filtered neighbors as candidate approximate matching features and ultimately selects the label that appears most frequently among these points as the result. If the candidate set is empty after filtering, a message indicating that no corresponding result is in the cache is returned.

[0073] Finally, examine the labels of the remaining features in the candidate set and select the label with the highest frequency as the query result. result If multiple labels have the same frequency, select the label closest to the query feature as the final result.

[0074] After completing an approximate matching, the fast approximate matching algorithm dynamically adjusts the matching threshold based on feedback from the recognition model to adapt to changing video content. The algorithm first generates a random number between 0 and 1. If this number is less than η, the algorithm adjusts the threshold. The true label l of the query feature is then obtained from the recognition model. g If the queried tag is l result With l g A mismatch indicates that the current threshold is too lenient and needs to be tightened. In this case, the threshold will be divided by α (α>1) to adjust it to a smaller value. If the algorithm cannot find the corresponding result in the cache, it checks the cache for a result matching the query feature p. q The nearest feature point p nrst The label. If the distance between feature points does not reach the threshold, but the label matches the result of the recognition model, it means that the current threshold is too strict and needs to be relaxed. The threshold is too strict, meaning the threshold is less than the distance between the query feature and the nearest feature point. The adjustment here is to increase the threshold to a value that does not exceed the distance between the query feature and the nearest feature point. This is achieved by using an exponentially weighted moving average: β×threshold+(1-β)×|p nrst -p q Here, β is a parameter used to control the rate at which the threshold is adjusted.

[0075] In summary, this invention proposes an edge-cloud collaborative caching architecture for multi-camera video analysis. Addressing the cache construction and feature point approximation matching problems, this invention proposes a maximum coverage cache selection algorithm and a fast approximation matching algorithm. The former effectively selects cached content to improve cache hit rate, while the latter achieves efficient cache lookup. Ultimately, the system reduces computational cost and end-to-end latency while maintaining accuracy.

Claims

1. An edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis, characterized in that, The specific steps are as follows: Step 1: Build an edge-cloud collaborative caching architecture, including edge servers and a cloud platform, and establish an initial cloud cache in the cloud platform; The initial cloud cache setup process is as follows: In the initial stage, the video capture system composed of multiple cameras transmits the video stream over a period of time to the edge-cloud collaborative caching architecture for target recognition, and stores the recognized targets in the cloud platform to establish a cloud cache; Step 2: Select features from the cloud cache using the maximum coverage cache selection algorithm to build an edge cache and store it on the edge server; Step 3: In the formal operation phase of the edge-cloud collaborative caching architecture, the video stream is transmitted to the edge server for processing, and key frames are extracted. Step 4: Use a convolutional network to extract the target bounding box in the keyframe, and further use a feature extractor to extract features from the target bounding box to obtain the target features; Step 5: Use the fast approximate matching algorithm to query whether the extracted target features have already stored the corresponding recognition result in the edge cache. If they exist, return the corresponding recognition result immediately; otherwise, send the extracted target features to the cloud platform for further processing. Step 6: The cloud platform uses a fast approximate matching algorithm to search the cloud cache based on the target features. If the target exists, the retrieved recognition result is returned to the edge server. If no corresponding result is found, the target recognition model deployed on the cloud platform is executed to calculate the corresponding result and returned to the edge server. At the same time, the result is added to the cloud cache. Step 7: For the updated cloud cache, return to Step 2 to periodically update the edge cache and perform target recognition on the new video stream.

2. The edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis according to claim 1, characterized in that, The process of building an edge cache using the maximum coverage cache selection algorithm is as follows: First, determine the number of feature points k and the coverage radius r of the feature points required for the edge cache; Then, all feature points P in the cloud cache are input into the maximum coverage cache selection algorithm, and HNSW is used to find each feature point in the cloud cache. The nearest neighbors of each feature point are selected, and all neighbor points within the radius r covered by each feature point are selected to form the feature points. Coverage point set ; Traverse all feature points in the cloud cache to obtain a coverage set S containing the coverage range of each feature point; Finally, a greedy strategy is used to iteratively select the coverage point set of all feature points until the number of feature points corresponding to the selected coverage point set reaches the number k required for the edge cache.

3. The edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis according to claim 2, characterized in that, The iterative process using a greedy strategy is as follows: In the initial iteration, the set of cover points containing the most feature points is selected from the cover set S, denoted as . Then delete from all feature points P The included elements, and at the same time Remove from S; then repeat the above iterative selection process for the remaining set of covering points and feature points, with each selected set of covering points having the highest coverage rate; finally, select the feature points corresponding to each set of covering points. Add to result set In the middle, until It contains k feature points.

4. The edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis according to claim 1, characterized in that, The edge server processes the video stream as follows: First, the video stream is decoded into consecutive video frames, and the video frames are transmitted to the video buffer. Then, the video buffer classifies the video frames into key frames and non-key frames. Key frames are processed by model recognition, while non-key frames are analyzed using a lightweight target tracking algorithm in the target tracking module.

5. The edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis according to claim 1, characterized in that, The specific process of the fast approximate matching algorithm searching edge / cloud cache is as follows: Step 1: Initialize the cache using HNSW. During the initialization process, find multiple nearest neighbors for each feature point and construct a hierarchical graph containing the connections between neighbors. Step 2: Find the new query features using the hierarchy diagram. We obtain the nearest neighbors to obtain a candidate set for approximate matching. ; Step 3, for Filter the feature points and retain only those that are less than the threshold distance from the query feature. Use the filtered neighbors as candidate approximate matching features. If the candidate set is empty after filtering, return a message that there is no corresponding result in the cache; Step 4: Examine the labels of the candidate approximate matching features and select the label with the highest frequency as the query result. If multiple labels have the same frequency, select the label closest to the query feature as the query result. Step 5: Based on the query results, dynamically adjust the matching threshold according to the feedback from the cloud platform recognition model to adapt to the changing video content; Step 6: Return to Step 3 and use the threshold for the next target matching.

6. The edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis according to claim 5, characterized in that, The process of adjusting the threshold for different query results is as follows: (1) The query result is After completing the above recognition process, the fast approximate matching algorithm generates a random number between 0 and 1, and determines whether the random number is less than a fixed parameter. If yes, then the threshold is adjusted; otherwise, the threshold remains unchanged. The process of adjusting the threshold is as follows: First, obtain the true labels of the query features from the recognition model. If the tags in the query results With real labels If there is a mismatch, it indicates that the current threshold is too lenient and needs to be tightened. At this point, divide the threshold by Adjust it to a smaller value; The adjusted threshold is expressed as follows: (2) The query result is empty If the corresponding result is not found in the cache, the cache will be checked for features matching the query target. nearest feature point The label; if the distance between these two feature points is greater than the threshold, but the label matches the true label result of the recognition model, it indicates that the current threshold is too strict and needs to be relaxed; in this case, the threshold adjustment is achieved by using an exponentially weighted moving average, as shown in the following formula: in, `threshold` is a parameter used to control the rate at which the threshold is adjusted; `threshold` is the current threshold.

7. An edge-cloud collaborative approximate redundancy removal system for multi-camera video analysis, characterized in that, It is an edge-cloud collaborative caching architecture consisting of edge servers and cloud platforms. Cameras are connected to edge servers and transmit captured video streams to them. By building a cache among multiple cameras, computational redundancy is reduced. The cloud platform is connected to the edge servers and provides more storage resources through cloud caching. The same target will only be recognized by the system once, and the target's features and recognition results will be stored in the cache. When the target reappears, the system retrieves the corresponding recognition result from the cache, eliminating the need to call the model for repeated calculations.

8. The edge-cloud collaborative approximate redundancy removal system for multi-camera video analysis according to claim 7, characterized in that, Edge caching and cloud caching use a key-value pair structure to store relevant information, where the key represents the feature vector extracted from the target object and the value corresponds to the final result of the recognition model.

Images