End-side cloud collaborative approximate redundancy removal system and method for multi-camera video analysis

Abstract

The invention provides an end-side cloud collaborative approximate redundancy removal system and method for multi-camera video analysis, and belongs to the technical field of end-side cloud collaborative optimization of an internet application layer. According to the invention, an end-side cloud collaborative cache architecture is provided, the cloud cache provides more storage resources with lower cost, the edge cache is constructed through a maximum coverage cache selection algorithm, the edge cache is located at a place closer to the camera, and the delay is shorter. The camera is connected to the edge server and continuously transmits captured video streams to the edge server, and the system quickly and accurately identifies a target result by combining the advantages of the two caches. The same target is only identified once by the system, and then the features and the identification result of the target are stored in the cache. And when the target appears again, the system can query the corresponding identification result from the cache without calling the model for repeated calculation, so that the calculation resources are greatly saved.

Description

Technical Field

[0001] The present invention belongs to the technical field of edge-cloud collaboration optimization in the Internet application layer, and specifically relates to an edge-cloud collaborative approximate redundancy removal system and method for multi-camera video analysis. Background Art

[0002] In recent years, the number of deployed cameras has increased exponentially, and the need to fully utilize their potential has become even more urgent. Therefore, video analysis is rapidly developing into an indispensable tool for public and private institutions, helping these organizations improve efficiency, reduce costs, and enhance security through Internet of Things (IoT) applications. The emergence of deep neural networks (DNNs) has opened a new era for the accuracy of video analysis, but at the same time has brought higher resource requirements.

[0003] In the presence of multi-camera deployments in the same environment, due to the repeated appearance of the same target object being captured, there is computational redundancy in video analysis applications. As Figure 1 shown, in a group of cameras distributed at an intersection or along a road, the same target object may be captured by different cameras or the same camera at different times, resulting in repeated capture and analysis by the DNN model, generating computational redundancy and wasting computing resources. Eliminating the redundant calculations of the same object between cameras can reduce the computational cost while maintaining low latency and high accuracy.

[0004] Currently, a large number of studies are dedicated to improving the performance of multi-camera video analysis pipelines, and these studies can be divided into two categories: (1) camera clusters with overlapping viewpoints; (2) camera clusters with non-overlapping viewpoints. The first category of studies mainly utilizes the overlapping fields of view (FoV) between multiple cameras to establish the correlation between camera capture areas, thereby enabling the sharing of analysis results. However, these methods are not applicable to camera scenarios with non-overlapping viewpoints, limiting their ability to comprehensively cover the monitored area. The second category of studies focuses on scheduling the enabling and disabling of geographically dispersed cameras to track a predetermined target object between cameras. However, these methods are mainly designed to handle retrospective queries and lack real-time inference capabilities. Summary of the Invention

[0005] Aiming at the computational redundancy caused by multi-cameras capturing the same target in the same or different time-spaces, resulting in repeated calculations, the present invention proposes an edge-cloud collaborative approximate redundancy removal system and method for multi-camera video analysis.

[0006] The edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis comprises the following specific steps:

[0007] Step 1, construct an edge-cloud collaborative caching architecture, including an edge server and a cloud platform, and establish an initial cloud cache in the cloud platform.

[0008] The establishment process of the initial cloud cache is as follows:

[0009] In the initial stage, the video capture system composed of multiple cameras transmits the video stream within a period of time to the edge-cloud collaborative caching architecture for target recognition, and stores the recognized targets in the cloud platform to establish the cloud cache.

[0010] In step two, the maximum coverage cache selection algorithm is used to select features from the cloud cache to construct the edge cache and store it in the edge server;

[0011] The process of constructing the edge cache by the maximum coverage cache selection algorithm is as follows:

[0012] First, determine the number of feature points k required for the edge cache and the coverage radius r of the feature points;

[0013] Then, input all the feature points P in the cloud cache into the maximum coverage cache selection algorithm, and use the hierarchical navigable small world method HNSW to find the i neighbors of each feature point p in the cloud cache, and filter out all the neighbor points within the coverage radius r of each feature point from the neighbor points to form the coverage point set S i of the feature point p i . Traverse all the feature points in the cloud cache to obtain the coverage set S containing the respective coverage ranges of all the feature points.

[0014] Finally, through the greedy strategy, perform iterative selection on the coverage point sets of all the feature points until the number of feature points corresponding to the selected coverage point set reaches the required number k for the edge cache. The specific iterative process is as follows:

[0015] In the first iteration, select the coverage point set containing the most feature points from the coverage set S and denote it as S imax , then delete the elements contained in S imax from all the feature points P, and at the same time delete S imax from S. Then repeat the above iterative screening process for the remaining coverage point sets and feature points. Each selected coverage point set has the highest coverage rate. Finally, add the feature point p i corresponding to each selected coverage point set to the result set p result until P result contains k feature points.

[0016] In step three, during the formal operation stage of the edge-cloud collaborative caching architecture, the video stream is transmitted to the edge server for processing, and the key frames are extracted;

[0017] The processing process of the edge server for the video stream is as follows:

[0018] First, decode the video stream into consecutive video frames and transmit the video frames to the video buffer;

[0019] Then, the video buffer classifies the video frames into key frames and non-key frames, performs model recognition processing on the key frames, and obtains analysis results for the non-key frames through the lightweight object tracking algorithm in the object tracking module;

[0020] Step Four, use the convolutional network to extract the object bounding box in the key frame, and further use the feature extractor to extract features from the object bounding box to obtain object features;

[0021] Step Five, use the fast approximate matching algorithm to query whether the extracted object features have stored the recognition results corresponding to the object in the edge cache. If so, immediately return the corresponding recognition results; otherwise, send the intercepted object features to the cloud platform for further processing.

[0022] Step Six, the cloud platform uses the fast approximate matching algorithm to search the cloud cache according to the object features. If there is a match, return the retrieved recognition results to the edge server; if there is no corresponding result, execute the object recognition model deployed on the cloud platform to calculate the corresponding results and return them to the edge server, and at the same time add the results to the cloud cache.

[0023] Step Seven, for the updated cloud cache, return to Step Two to periodically update the edge cache and perform object recognition on the new video stream.

[0024] Furthermore, in the above process, the specific process of the fast approximate matching algorithm based on HNSW is as follows:

[0025] Step1, perform HNSW initialization on the cache. During the initialization process, find multiple nearest neighbors for each feature point and construct a hierarchical graph containing the connections between the neighbors.

[0026] Step2, through the hierarchical graph, find the m nearest neighbors of the new query feature to obtain the candidate set P for approximate matching cand .

[0027] Step3, filter P cand and only retain the feature points whose distance from the query feature is lower than the threshold. The filtered neighbors are used as candidate approximate matching features. If the candidate set is empty after filtering, return a message indicating that there is no corresponding result in the cache.

[0028] Among them, the initial value of the threshold is determined by the distance between features of the same type.

[0029] Step4, check the labels of the candidate approximate matching features and select the label with the highest frequency of occurrence as the query result lresult ; If the frequencies of multiple tags are the same, select the tag closest to the query feature as the query result.

[0030] Step5. For the query result, dynamically adjust the matching threshold based on the feedback of the cloud platform recognition model to adapt to the changing video content.

[0031] (1) The query result is l result

[0032] After the fast approximate matching algorithm completes the above recognition process, generate a random number between 0 and 1, and determine whether the random number is less than the fixed parameter η. If so, adjust the threshold; otherwise, the threshold remains unchanged.

[0033] The process of adjusting the threshold is specifically as follows:

[0034] First, obtain the true tag l of the query feature from the recognition model g . If the tag l of the query result result does not match the true tag l g , it indicates that the current threshold is too loose and needs to be tightened. At this time, divide the threshold by α (α > 1) and adjust it to a smaller value.

[0035] The adjusted threshold is expressed as:

[0036]

[0037] (2) The query result is empty

[0038] If the corresponding result cannot be found in the cache, check the tag of the feature point p q closest to the query target feature p nrst in the cache. If the distance between these two feature points is greater than the threshold, but the tag matches the true tag result of the recognition model, it means that the current threshold is too strict and needs to be relaxed. At this time, use the exponential weighted moving average to achieve threshold adjustment, and the formula is as follows:

[0039] t = β × threshold + (1 - β) × |p nrst - p q |

[0040] where β is a parameter used to control the rate of threshold adjustment. threshold is the current threshold.

[0041] Step6. Return to Step3 and use the threshold for the next target matching.

[0042] The edge-cloud collaborative approximate redundancy removal system for multi-camera video analysis is an edge-cloud collaborative architecture composed of an edge server and a cloud platform. Cameras are connected to the edge server and transmit the captured video streams to it. By building caches among multiple cameras, computational redundancy is reduced. The cloud platform is connected to the edge server and provides more storage resources through cloud caches. The same target will only be recognized by the system once, and then the features and recognition results of the target will be stored in the cache. When the target appears again, the system can query the corresponding recognition results from the cache without invoking the model for repeated calculations.

[0043] The edge cache and the cloud cache use a key-value pair structure to store relevant information. The key represents the feature vector extracted from the target object, and the value corresponds to the final result of the object recognition model.

[0044] The advantages and beneficial effects of the present invention are as follows:

[0045] (1) In the present invention, a collaborative cache framework is established. The edge cache is closer to the cameras, with short latency, and the cloud cache has low cost and large storage. By combining the advantages of the two caches, the collaborative framework optimizes the performance and cost-effectiveness of the cache system.

[0046] (2) In the present invention, the maximum coverage cache selection algorithm and the fast approximate matching algorithm are proposed. The former can effectively select cache contents to improve the cache hit rate, and the latter realizes efficient cache query. Combining these two algorithms enables the present invention to reduce the computational cost and end-to-end latency while maintaining accuracy. Description of the Drawings

[0047] Figure 1 Schematic diagram of the same target object captured by different cameras;

[0048] Figure 2 Specific flowchart of the edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis of the present invention;

[0049] Figure 3 Flowchart of the approximate matching algorithm in the present invention. Detailed Embodiments

[0050] The present invention first proposes an edge-cloud collaborative cache architecture to achieve edge-cloud collaborative approximate redundancy removal for multi-camera video analysis, as Figure 2As shown in the figure. The camera is connected to the edge server and continuously transmits the captured video stream to it. Due to multiple cameras capturing duplicate targets resulting in duplicate calculations, the present invention proposes a collaborative caching framework to reduce computational redundancy by building caches among multiple cameras. The same target will only be recognized once by the system, and then the features and recognition results of the target will be stored in the cache. When the target appears again, the system can query the corresponding recognition result from the cache without calling the model for duplicate calculations, thus greatly saving computational resources. Edge caches are located closer to the cameras, with shorter latency, but have limited storage space and higher costs. Cloud caches provide more storage resources at a lower cost, but have a higher latency due to the long transmission distance. By combining the advantages of the two types of caches, the collaborative framework aims to optimize the performance and cost-effectiveness of the cache system.

[0051] Edge caches and cloud caches use a key-value pair structure to store relevant information. The key represents the feature vector extracted from the target object, and the value corresponds to the final result of the object recognition model, such as the label obtained by the model recognition.

[0052] The specific implementation of the edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis mainly includes the following steps:

[0053] Step (1) In the initialization stage, the system runs for a specific period of time, and stores the detection results corresponding to the recognized targets in the cloud cache. During this process, the cloud cache will be continuously updated in real time.

[0054] Step (2) After the cloud cache is established, run the maximum coverage cache selection algorithm to select appropriate features to build the edge cache, so as to improve the hit rate of the edge cache as much as possible.

[0055] To improve the hit rate of the edge cache, it is crucial to select the most valuable cloud cache content. Transforming the construction of the edge cache into a maximum coverage problem, the goal is to select a subset of cloud cache content that can cover as many feature points as possible. Therefore, the present invention proposes a maximum coverage cache selection algorithm, aiming to find the optimal subset of feature points that maximizes the cumulative coverage within a given radius. By carefully selecting cache content according to the maximum coverage method, the hit rate of the edge cache can be effectively improved. This algorithm not only considers the importance of feature points, but also determines the best subset to maximize coverage, thereby increasing the cache hit probability for queries related to these feature points.

[0056] As Figure 2 shown in Algorithm 1 below, the maximum coverage cache selection algorithm is specifically as follows:

[0057] This algorithm takes all the feature points in the cloud cache (P) as input. In addition, two parameters need to be determined: the number of feature points (k) required for the edge cache and the coverage radius (r) of the feature points. k is used to balance the cache hit rate and the end-to-end latency, while r is determined by the size of the feature point distribution of the same type of target. By calculating the distances between individual feature points, a radius is determined, and feature points with a mutual distance within this radius are considered of the same type. The output of the algorithm is a subset of P, denoted as P result , which will become the content of the edge cache. The algorithm identifies the k feature points with the maximum coverage as P result :

[0058] First, for each point p in the cache i construct a set of covered points S within the specified radius r i . The algorithm uses the Hierarchical Navigable Small World method (HNSW) to find the nearest neighbors of the feature points and then determines whether the neighbors are within the specified radius r. By iteratively running HNSW on the neighbors, the algorithm finally obtains a coverage set S that contains the coverage ranges of all the feature points respectively.

[0059] Second, the algorithm applies a greedy strategy, iteratively selecting the feature point with the largest intersection with the remaining uncovered points until k feature points are selected. In each iteration, select the S with the largest intersection with P i . Then delete S i from P i and delete S i from S i . The selected S result has the highest coverage rate, so select the corresponding feature point p result and add it to the result set P result . Repeat this process until P result contains k feature points. This greedy selection process ensures that the selected feature points can achieve the maximum coverage of the cached feature points, thus effectively optimizing the use effect of the edge cache.

[0060] In step (3), during the formal operation phase, the system selects key frames from the video frames captured by the end-side camera for subsequent processing.

[0061] After the edge server obtains the video stream transmitted by the camera, the video decoding module decodes the video stream into continuous video frames and inputs the video frames into the video buffer. Considering the temporal redundancy of the video frames, the system selects key frames that exhibit significant changes from the video buffer for subsequent model recognition processing. Non-key frames are analyzed using a lightweight object tracking algorithm in the object tracking module. Using object tracking can reduce the number of video frames that need to be further processed, thereby reducing the computational load while ensuring that the accuracy of video analysis does not decline excessively.

[0062] Step (4) Extract the features of the target in the key frame, and use the fast approximate matching algorithm to search for the corresponding results in the edge cache.

[0063] The system uses a convolutional network to extract the target bounding box in the key frame, and uses a feature extractor to extract the features of the recognized object. Based on the extracted features, the system uses the fast approximate matching algorithm to query in the edge cache whether the recognition result corresponding to the target has been stored.

[0064] Step (5) If the search is successful, return the result; if the search fails, send the target features to the cloud platform for processing.

[0065] Step (6) Use the fast approximate matching algorithm on the cloud platform to search for the corresponding results of the recognized target in the cloud cache.

[0066] Step (7) If the search is successful, return the result to the edge server; if the search fails, use the target recognition model deployed on the cloud platform to identify the corresponding result and return it to the edge server, and add the result to the cloud cache.

[0067] Step (8) The fast approximate matching algorithm adjusts the threshold according to the search results and updates the cache content.

[0068] The system will regularly run the maximum coverage cache selection algorithm to update the edge cache to ensure that the content of the edge cache adapts to the real-time changes of the video content.

[0069] As Figure 2 shown in Algorithm 2, in order to efficiently implement steps (4) and (6) to query the edge / cloud cache, the present invention proposes a fast approximate matching algorithm based on HNSW. By using HNSW, the approximate matching algorithm can quickly identify potential matching items in the cache. In addition, in order to better adapt to different video contents, the algorithm dynamically adjusts the matching threshold according to environmental changes.

[0070] The fast approximate matching algorithm based on HNSW, as Figure 3 shown, the specific process is as follows:

[0071] First, initialize the cache with HNSW. During the initialization process, find multiple nearest neighbors for each feature point and construct a hierarchical graph containing the connections between the neighbors. Through this graph, the algorithm can quickly find the m nearest neighbors of the newly queried feature, so as to obtain the candidate set P for approximate matching cand .

[0072] Next, the algorithm processes P candFiltering is performed to retain only the feature points whose distance from the query feature is below the threshold. Since feature matching is approximate, selecting the nearest neighbor within the threshold may not be accurate. Therefore, the algorithm uses the filtered neighbors as candidate approximate matching features and finally selects the label that appears most frequently among these points as the result. If the candidate set is empty after filtering, a message indicating that there is no corresponding result in the cache is returned.

[0073] Finally, check the labels of the remaining features in the candidate set and select the label with the highest frequency as the query result l result . If the frequencies of multiple labels are the same, select the label closest to the query feature as the final result.

[0074] After the fast approximate matching algorithm completes the approximate matching, it dynamically adjusts the matching threshold based on the feedback of the recognition model to adapt to the changing video content. The algorithm first generates a random number between 0 and 1. If this number is less than η, the algorithm will perform threshold adjustment. Obtain the true label l of the query feature from the recognition model g . If the queried label l result does not match l g , it indicates that the current threshold is too loose and needs to be tightened. In this case, the threshold will be divided by α (α > 1) and adjusted to a smaller value. If the algorithm cannot find the corresponding result in the cache, check the label of the feature point p q nearest to the query feature p nrst . If the distance between the feature points does not reach the threshold, but the label matches the result of the recognition model, it means that the current threshold is too strict and needs to be relaxed. The threshold is too strict, and the threshold is less than the distance between the query feature and the nearest feature point. The adjustment here is to enlarge the threshold to a value not exceeding the distance between the query feature and the nearest feature point. This is achieved by using an exponentially weighted moving average: β × threshold + (1 - β) × |p nrst - p q |. Here, β is a parameter used to control the rate of threshold adjustment.

[0075] In summary, the present invention proposes an edge-cloud collaborative caching architecture for multi-camera video analysis. For the cache construction and feature point approximate matching problems therein, the present invention proposes a maximum coverage cache selection algorithm and a fast approximate matching algorithm. The former can effectively select cache content to improve the cache hit rate, and the latter realizes efficient cache query. The system finally reduces the computational cost and end-to-end latency while maintaining accuracy.

Claims

1. An edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis, characterized in that The specific steps are as follows: Step 1, construct an edge-cloud collaborative caching architecture, including an edge server and a cloud platform, and establish an initial cloud cache in the cloud platform; The process of establishing the initial cloud cache is as follows: In the initial stage, a video capture system composed of multiple cameras transmits the video stream within a period of time to the edge-cloud collaborative caching architecture for target recognition, and stores the recognized targets in the cloud platform to establish a cloud cache; Step 2, select features from the cloud cache through the maximum coverage cache selection algorithm to construct an edge cache, and store it in the edge server; Step 3, in the official operation stage of the edge-cloud collaborative caching architecture, the video stream is transmitted to the edge server for processing, and the key frames are extracted; Step 4, use a convolutional network to extract the target bounding boxes in the key frames, and further use a feature extractor to extract features from the target bounding boxes to obtain target features; Step 5, use the fast approximate matching algorithm to query whether the edge cache has stored the recognition result corresponding to the target for the extracted target features. If it exists, immediately return the corresponding recognition result; otherwise, send the intercepted target features to the cloud platform for further processing; Step 6, the cloud platform uses the fast approximate matching algorithm to search the cloud cache according to the target features. If it exists, return the retrieved recognition result to the edge server; if there is no corresponding result, execute the target recognition model deployed on the cloud platform to calculate the corresponding result and return it to the edge server, and at the same time add the result to the cloud cache; Step 7, for the updated cloud cache, return to Step 2 to update the edge cache regularly and perform target recognition on the new video stream.

2. The edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis according to claim 1, wherein The process of constructing the edge cache through the maximum coverage cache selection algorithm is as follows: First, determine the number of feature points k required for the edge cache and the coverage radius r of the feature points; Then, input all the feature points P in the cloud cache into the maximum coverage cache selection algorithm, and use HNSW to find each feature point p in the cloud cache i 's neighbors, and filter out all neighbor points within the coverage radius r of each feature point from the neighbor points to form the coverage point set S of the feature point p i ; Traverse all feature points in the cloud cache to obtain a coverage set S that contains the respective coverage ranges of all feature points; i ; Traverse all feature points in the cloud cache to obtain a coverage set S that contains the respective coverage ranges of all feature points; Finally, through a greedy strategy, iteratively select the coverage point sets of all feature points until the number of feature points corresponding to the selected coverage point set reaches the required number k of the edge cache.

3. The edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis according to claim 2, wherein The process of iterating through the greedy strategy is as follows: At the first iteration, select the covering point set with the most feature points from the covering set S and denote it as S imax , then delete S from all feature points P imax The elements included, and at the same time delete S from S imax ; Then repeat the above iterative screening process for the remaining covering point sets and feature points. Each selected covering point set has the highest coverage rate; Finally, add the feature points p corresponding to each selected covering point set i to the result set P result , until P result contains k feature points.

4. The edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis according to claim 1, wherein The process of the edge server processing the video stream is as follows: First, decode the video stream into continuous video frames and transmit the video frames to the video buffer; Then, the video buffer classifies the video frames into key frames and non-key frames, performs model recognition processing on the key frames, and obtains analysis results for the non-key frames through the lightweight target tracking algorithm in the target tracking module.

5. The edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis according to claim 1, wherein The specific process of the fast approximate matching algorithm searching the edge / cloud cache is as follows: Step1, initialize the HNSW for the cache; during the initialization process, find multiple nearest neighbors for each feature point and construct a hierarchical graph containing the connections between the neighbors; Step 2, find the m nearest neighbors of the new query feature through the hierarchical graph, so as to obtain the candidate set P for approximate matching cand ; Step3, filter P cand by retaining only the feature points whose distance from the query feature is below the threshold, and use the filtered neighbors as candidate approximate matching features; If the candidate set is empty after filtering, return a message that there is no corresponding result in the cache; Step4, check the labels of the candidate approximate matching features and select the label with the highest occurrence frequency as the query result l result ; if the frequencies of multiple labels are the same, select the label closest to the query feature as the query result; Step5, for the query result, dynamically adjust the matching threshold based on the feedback of the cloud platform recognition model to adapt to the changing video content; Step6, return to Step3 and use the threshold for the next target matching.

6. The edge-cloud collaborative approximate redundancy removal method for multi-camera video analysis according to claim 5, characterized in that The processes of adjusting the threshold for different query results are as follows: (1) The query result is l result After the fast approximate matching algorithm completes the above recognition process, it generates a random number between 0 and 1 and determines whether the random number is less than the fixed parameter η. If so, the threshold is adjusted; otherwise, the threshold remains unchanged; The process of adjusting the threshold is specifically as follows: First, obtain the true label l of the query feature from the recognition model g , if the label l of the query result result does not match the true label l g , it indicates that the current threshold is too loose and needs to be tightened; at this time, divide the threshold by α (α > 1) and adjust it to a smaller value; The adjusted threshold is expressed as: (2) The query result is empty If the corresponding result cannot be found in the cache, the tags of the feature point p q q nearest to the query target feature p nrst nrst will be checked; if the distance between these two feature points is greater than the threshold, but the tag matches the true tag result of the recognition model, it indicates that the current threshold is too strict and needs to be relaxed; at this time, the threshold adjustment is achieved by using the exponentially weighted moving average, and the formula is as follows: t = β×threshold+(1 - β)×|p nrst -p q | where β is a parameter used to control the rate of threshold adjustment; threshold is the current threshold.

7. Edge-cloud collaborative approximate redundancy removal system for multi-camera video analysis, characterized in that It is an edge-cloud collaborative caching architecture composed of edge servers and cloud platforms. The camera is connected to the edge server and transmits the captured video stream to it. By building caches between multiple cameras, computational redundancy is reduced; the cloud platform is connected to the edge server to provide more storage resources through cloud caching; the same target will only be recognized by the system once, and then the features and recognition results of the target will be stored in the cache; When the target appears again, the system queries the corresponding recognition result from the cache without calling the model for repeated calculations.

8. The edge-cloud collaborative approximate redundancy removal system for multi-camera video analysis according to claim 7, characterized in that Edge caching and cloud caching use a key-value pair structure to store relevant information. The key represents the feature vector extracted from the target object, and the value corresponds to the final result of the recognition model.

Images