A video monitoring method based on distributed edge cooperation

Abstract

This invention provides a video surveillance method based on distributed edge collaboration. The method includes deploying pedestrian tracking and pedestrian re-identification services on different edge devices and using the Apache Kafka communication protocol for bidirectional communication. Initial feature extraction is performed on pedestrians entering the monitored area to generate pedestrian features. The pedestrian tracking service detects and tracks pedestrians within the monitored area, generating pedestrian feature vectors. The pedestrian re-identification service calculates the similarity between the received pedestrian feature vectors and pedestrian features stored in a shared server to determine the pedestrian's identity. Based on the current computing resource usage of the heterogeneous edge devices, the edge devices running the video analysis service and the pedestrian re-identification service are dynamically scheduled. This invention achieves pedestrian tracking and re-identification through collaborative reasoning of edge devices within the monitored area, thereby avoiding single points of failure and privacy leaks, and improving the real-time performance and stability of video surveillance.

Description

Technical Field

[0001] This invention relates to the field of intelligent video surveillance technology, and more specifically to a video surveillance method based on distributed edge collaboration. Background Technology

[0002] In the field of video surveillance, with the rapid development of big data, artificial intelligence, and the Internet of Things (IoT) technologies, the application of video surveillance systems is becoming increasingly widespread, covering many important areas such as public safety, intelligent transportation, and smart cities. Most existing video surveillance systems adopt a traditional centralized architecture, which centrally transmits monitoring data to cloud servers for processing and storage. This architecture relies on a high-bandwidth, low-latency network environment to ensure real-time transmission and processing of video data. However, under current network conditions, centralized video surveillance systems cannot guarantee that video data from all monitoring points can be transmitted to the central server in real time and smoothly, resulting in a significant reduction in monitoring effectiveness. Furthermore, delays in the return of results from cloud processing can lead to missed or false alarms for critical events. In addition, there is a risk of data theft and privacy breaches during the transmission of video data from cameras to cloud servers for processing.

[0003] Existing distributed edge collaborative video analytics methods based on multi-agent reinforcement learning employ a distributed architecture design, dividing video surveillance tasks into multiple sub-services or tasks and deploying them across multiple edge devices. While this addresses the latency issue caused by long-distance data transmission, it fails to explicitly divide the video analytics task into pedestrian video analytics and pedestrian re-identification services, and then deploy them on different edge devices. Furthermore, in task partitioning, agent modeling, and service decision-making, it fails to adequately address the increased computational load and the inability of edge devices to handle increased pedestrian activity within the area, leading to a decline in video surveillance performance. Summary of the Invention

[0004] To address the issues of high latency, poor privacy and security, and limited resources inherent in traditional centralized video surveillance and edge computing methods, as well as the unclear task division, poor service deployment, and poor startup decisions in reinforcement learning distributed edge collaboration methods, this invention aims to provide a video surveillance method based on distributed edge collaboration. The core of this method lies in achieving pedestrian tracking and re-identification through collaborative inference of edge devices within the monitoring area, thereby avoiding single points of failure and privacy leaks, and improving the real-time performance and stability of video surveillance.

[0005] The present invention achieves the above objectives through the following technical solutions:

[0006] A video surveillance method based on distributed edge collaboration includes the following steps:

[0007] The video surveillance task is divided into pedestrian tracking service and pedestrian re-identification service;

[0008] The pedestrian tracking service and the pedestrian re-identification service are deployed on different edge devices to achieve collaborative inference between heterogeneous edge devices, and the Apache Kafka communication protocol is used to achieve bidirectional communication between the video analysis service and the pedestrian re-identification service.

[0009] The acquired real-time video stream signal is pushed to the edge device cluster, and the pedestrians entering the monitoring area are subjected to initial feature extraction to generate pedestrian features. The pedestrian features are then sent to the shared server for storage.

[0010] The pedestrian tracking service detects and tracks pedestrians within the monitored area, generating pedestrian feature vectors.

[0011] The pedestrian re-identification service calculates the similarity between the received pedestrian feature vector and the pedestrian features stored in the shared server to determine the pedestrian's identity.

[0012] Based on the current computing resource usage of heterogeneous edge devices, dynamically schedule the edge devices running video analytics service and pedestrian re-identification service;

[0013] The shared server periodically synchronizes data with the edge devices running the pedestrian re-identification service.

[0014] A video surveillance method based on distributed edge collaboration provided by the present invention further includes:

[0015] The feature extraction service performs initial feature extraction on pedestrians entering the monitored area, including:

[0016] Assuming the input frame of the video stream is I with size H×W, the convolution kernel is K with size m×n (height m and width n), and the output feature map is O with size H′×W′, then it can be expressed as the following formula:

[0017]

[0018] Where O(i,j) represents the pixel value at position (i,j) in the output feature map, I(i+u,j+v) represents the pixel value at position (i+u,j+v) in the input frame, K(u,v) represents the weight of the convolution kernel at position (u,v), m and n represent the height and width of the convolution kernel, and H′ and W′ represent the width and height of the output feature map.

[0019] According to the video surveillance method based on distributed edge collaboration provided by the present invention, the pedestrian tracking service achieves target tracking by tracking high-confidence and low-confidence detected targets, specifically including:

[0020] Input definition:

[0021] High confidence test set D H The set of detection targets that meet the confidence threshold θ;

[0022] Low confidence test set D L : The set of detection targets below the confidence threshold θ;

[0023] The confidence score is provided by the target detection algorithm.

[0024] According to the present invention, a video surveillance method based on distributed edge collaboration is provided, which achieves pedestrian tracking through high-confidence matching and low-confidence matching, specifically including:

[0025] High-confidence matching: Hungarian matching is performed using the high-confidence detection set D1 to match the detected target in the current frame with the tracked target in the previous frame, expressed as the following formula:

[0026] C = {c ij |c ij =1-IOU(T) i D j )},

[0027] Among them, T i D represents the bounding box of the currently tracked target i. j Represents the bounding box of the detected target j, IOU(T) i D j ) represents the target T i and detection D j Intersection over Union (IOU)

[0028] The optimal match for high-confidence targets is achieved by minimizing the cost matrix C.

[0029] Low-confidence matching: Matching the remaining unmatched tracked targets with low-confidence detected targets D. L The matching is performed using the same Hungarian algorithm, but only those targets that were not matched are considered, as expressed in the following formula:

[0030] C′={c′ ij |c′ ij =1-IOU(T′) i ,D′ j )},

[0031] Among them, T′ i D′ represents the tracked target that did not match after the first round of matching. j This indicates a target with low confidence.

[0032] According to the video surveillance method based on distributed edge collaboration provided by the present invention, updating and managing trajectories are also performed, specifically including:

[0033] Trajectory Update: For detected targets and tracked targets that are successfully associated through the first round of high-confidence matching and the second round of low-confidence matching, update the bounding box position or motion state information of the detected targets to the corresponding tracking trajectory;

[0034] New trajectory initialization: For high-confidence detected targets that do not match any existing tracked targets in the current frame, they are identified as newly appearing targets, and new tracking trajectories are initialized. A unique trajectory identifier is assigned to the target, and its status information is recorded in the new trajectory.

[0035] Low-confidence target handling: For low-confidence detected targets that do not match any tracked targets, a decision is made based on a preset strategy to either ignore these targets or mark them as potential targets for continuous tracking; if continuous tracking is selected, temporary trajectories are created for these potential targets, and attempts are made to match them in subsequent frames to verify whether they are real targets.

[0036] According to the video surveillance method based on distributed edge collaboration provided by the present invention, when generating pedestrian feature vectors, the feature extraction formula is expressed as the following vector:

[0037] f i =g(I i ),

[0038] Among them, I i The detected pedestrian bounding box is represented by g(·), which represents the feature extraction network, and f is the function of the detection network. i It is the extracted 2048-dimensional pedestrian feature vector.

[0039] According to the video surveillance method based on distributed edge collaboration provided by the present invention, the following further steps are performed:

[0040] Establish a trajectory lifecycle management mechanism to perform trajectory termination and aging processing steps, specifically including:

[0041] Unmatched frame count: A counter for unmatched frames is maintained for each active trajectory. When the trajectory fails to match any detected target in a frame, the counter is incremented by one; otherwise, if a match is successful, the counter is reset to zero.

[0042] Aging threshold determination: A preset threshold for the number of unmatched frames is used. When the counter for the number of unmatched frames of a certain trajectory reaches or exceeds the threshold, the trajectory is marked as an aging trajectory.

[0043] Aging treatment: For items marked as aging tracks, perform one or more of the following aging treatment measures:

[0044] Reduce the priority or weight of this trajectory in subsequent tracking and decision-making processes; move the trajectory to a dedicated aging trajectory storage area; send an aging notification to the system administrator or relevant applications;

[0045] The system sets retention periods or conditions for aging tracks. When these conditions are met, the aging tracks are deleted to completely free up system resources. After deleting the aging tracks, the resource allocation of the remaining active tracks is dynamically adjusted to ensure that tracking efficiency and accuracy are not affected, and to prepare for the initialization and management of new tracks.

[0046] According to a video surveillance method based on distributed edge collaboration provided by the present invention, in the pedestrian re-identification service, the cosine similarity between the received pedestrian feature vector and the features synchronized with the shared server data is calculated, expressed as the following formula:

[0047]

[0048] Among them, f i ·f j Represents vector f i sum vector f j The dot product, ||f i ‖ and ||f j || represents the vector f i sum vector f j The norm of a vector is its length.

[0049] According to the present invention, a video surveillance method based on distributed edge collaboration includes an edge device for dynamically scheduling the operation of video analytics services and pedestrian re-identification services, comprising:

[0050] Periodically collect and analyze the resource utilization rate of each edge device, as well as the currently running service type and service load;

[0051] Based on the collected resource usage data, a resource usage threshold is set. When the resource usage rate of a certain device exceeds the preset threshold, while the resource usage rate of another device is far below the average level, it is determined that there is a load imbalance.

[0052] Based on the results of the load imbalance assessment, select the task types and target devices that need to be migrated to maximize resource utilization efficiency and load balance.

[0053] This involves using containerization technology to launch and run the migration task on the target device.

[0054] A video surveillance method based on distributed edge collaboration provided by the present invention further includes:

[0055] During the task migration process, ensure data synchronization between the source device and the target device;

[0056] For pedestrian re-identification or pedestrian tracking services that require collaborative processing, feature extraction and result sharing between devices are supported. That is, after device C processes part of the task, it returns the intermediate or final result to device B, or uploads it directly to the central server.

[0057] After task migration and reallocation are completed, the resource usage and system performance of each device are continuously monitored. If a new load imbalance or resource bottleneck is detected, a new round of load assessment and task migration decision is automatically triggered, or some migration tasks are rolled back to achieve optimal resource utilization and continuous stable operation of the system.

[0058] Therefore, this invention, through its distributed architecture design, innovatively divides video surveillance tasks into two independent services: pedestrian tracking and pedestrian re-identification, and deploys them separately on different heterogeneous edge devices, achieving an efficient and flexible collaborative reasoning mechanism. Consequently, the technical solution of this invention brings the following significant benefits:

[0059] 1. Improved Monitoring Efficiency and Accuracy: This invention separates and optimizes pedestrian tracking and pedestrian re-identification services, allowing each service to focus on its strengths, thereby improving the overall efficiency and accuracy of monitoring. The pedestrian tracking service can detect and track pedestrian targets in real time and accurately, while the pedestrian re-identification service utilizes powerful feature comparison capabilities to accurately identify the same pedestrian under different cameras.

[0060] 2. Enhanced System Flexibility and Scalability: The distributed architecture design of this invention enables the system to easily handle monitoring scenarios of varying scales and complexities. By dynamically adjusting task allocation and collaboration, the system can flexibly deploy and migrate services across different edge devices based on actual needs and resource availability, thereby achieving high scalability and flexibility.

[0061] 3. Optimized resource utilization and reduced energy consumption: This invention dynamically adjusts task allocation and collaboration based on network status and device resource usage, ensuring maximum resource utilization. It avoids resource waste or performance bottlenecks caused by uneven resource allocation, while also reducing the overall energy consumption of the system.

[0062] 4. Improved system stability and reliability: The distributed architecture and collaborative reasoning mechanism make the system more fault-tolerant to the failure of individual devices. Even if one device fails, other devices can continue to provide services, thus ensuring the stability and reliability of the system.

[0063] 5. Promoting Technological Innovation and Industrial Upgrading: The implementation of this invention not only promotes the innovation of video surveillance technology, but also provides strong support for the upgrading of related industries. By optimizing the video surveillance process and improving monitoring efficiency and accuracy, this invention injects new vitality into the development of smart cities, intelligent security and other fields.

[0064] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0065] Figure 1 This is a flowchart of an embodiment of a video surveillance method based on distributed edge collaboration according to the present invention.

[0066] Figure 2 This is a schematic diagram of the system architecture of an embodiment of a video surveillance method based on distributed edge collaboration according to the present invention.

[0067] Figure 3 This is a flowchart illustrating the pedestrian tracking service, feature extraction service, and pedestrian re-identification service in an embodiment of a video surveillance method based on distributed edge collaboration according to the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0069] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0070] See Figures 1 to 3 This embodiment provides a video surveillance method based on distributed edge collaboration, which includes the following steps:

[0071] Step S1: Divide the video surveillance task into pedestrian tracking service and pedestrian re-identification service;

[0072] Step S2: Deploy the pedestrian tracking service and the pedestrian re-identification service on different edge devices to achieve collaborative inference between heterogeneous edge devices, and use the Apache Kafka communication protocol to achieve bidirectional communication between the video analysis service and the pedestrian re-identification service.

[0073] Step S3: The acquired real-time video stream signal is pushed to the edge device cluster, the pedestrians entering the monitoring area are subjected to the first feature extraction, pedestrian features are generated, and the pedestrian features are sent to the shared server for storage.

[0074] Step S4: The pedestrian tracking service detects pedestrians within the monitored area and tracks them, generating pedestrian feature vectors.

[0075] Step S5: The pedestrian re-identification service calculates the similarity between the received pedestrian feature vector and the pedestrian features stored in the shared server to determine the pedestrian's identity.

[0076] Step S6: Based on the current computing resource usage of heterogeneous edge devices, dynamically schedule the edge devices running the video analytics service and the pedestrian re-identification service;

[0077] The shared server periodically synchronizes data with the edge devices running the pedestrian re-identification service.

[0078] Therefore, the video surveillance method based on distributed edge collaboration provided in this embodiment includes RTSP video stream pushing, feature extraction service, pedestrian tracking service (video analysis), pedestrian re-identification service, Apache Kafka communication protocol, load balancing algorithm, and data synchronization. Specifically, RTSP streaming is responsible for pushing the footage captured by the camera to the edge device cluster; the feature extraction service performs initial feature extraction on pedestrians entering the monitored area and sends it to the shared server for storage; the video analysis service detects and tracks pedestrians within the area; the pedestrian re-identification service determines the pedestrian's identity by calculating the similarity between the received pedestrian features and the features stored on the shared server; the Apache Kafka communication protocol handles communication between the video analysis service and the pedestrian re-identification service; the load balancing request dynamically schedules the edge devices running the video analysis service and the pedestrian re-identification service based on the computing resource usage of the heterogeneous edge devices, ensuring optimal overall system performance; and data synchronization involves the shared server synchronizing the stored features with the edge devices running pedestrian re-identification.

[0079] The RTSP video stream is connected to the feature extraction service and the video analysis service to acquire and analyze the video stream in real time; the feature extraction service is connected to the shared server to store the extracted pedestrian features; the video analysis service and the pedestrian re-identification service are connected bidirectionally via the Kafka communication protocol; the pedestrian re-identification service communicates bidirectionally with the shared server, with the former acquiring the features synchronized from the latter's data, comparing them, and returning the results.

[0080] In this embodiment, the initial feature extraction of pedestrians entering the monitored area (entrance) is performed using a feature extraction service, including:

[0081] Assuming the input frame of the video stream is I, with a size of H×W (height H and width W), the convolution kernel is K, with a size of m×n (height m and width n), and the output feature map is O, with a size of H′×W′, then it can be expressed as the following formula:

[0082]

[0083] Where O(i,j) represents the pixel value at position (i,j) in the output feature map, I(i+u,j+v) represents the pixel value at position (i+u,j+v) in the input frame, K(u,v) represents the weight of the convolution kernel at position (u,v), m and n represent the height and width of the convolution kernel, and H′ and W′ represent the width and height of the output feature map.

[0084] The feature extraction service sends the extracted pedestrian features to a shared server for storage. The RTSP video stream is connected to the video analysis service, which tracks pedestrians within the area and sends the tracking and analysis results (i.e., pedestrian feature vectors) to the pedestrian re-identification service via the Kafka communication protocol. The pedestrian tracking service achieves target tracking by tracking high-confidence and low-confidence detected targets, specifically including:

[0085] (1) Input definition:

[0086] High confidence test set D H The set of detection targets that meet the confidence threshold θ;

[0087] Low confidence test set D L : The set of detection targets below the confidence threshold θ;

[0088] The confidence score is provided by the target detection algorithm; in this embodiment, YOLOv8 is used.

[0089] (2) Matching steps

[0090] Pedestrian tracking is primarily achieved through two steps: high-confidence matching and low-confidence matching. Specifically, this includes:

[0091] High-confidence matching (first round matching): using the high-confidence detection set D H The Hungarian algorithm is used to match the detected target in the current frame with the tracked target in the previous frame, expressed as the following formula:

[0092] C = {c ij |c ij =1-IOU(T) i D j )},

[0093] Among them, T i D represents the bounding box of the currently tracked target i. j Represents the bounding box of the detected target j, IOU(T) i D j ) represents the target T i and detection D j Intersection over Union (IOU) ratio.

[0094] The optimal matching of a high-confidence objective is achieved by minimizing the cost matrix C. Specifically, minimizing the cost matrix to achieve the optimal matching of a high-confidence objective typically involves optimization techniques, especially in task assignment, pattern recognition, or classification problems in machine learning and data mining. First, the cost matrix is ​​defined. The cost matrix, also known as the loss matrix, is a two-dimensional array where each element represents the cost of assigning one class to another. Assuming there are N true classes and M predicted classes, the cost matrix C is of size N×M, where C[i][j] represents the cost of assigning true class i to predicted class j. Next, the objective function is determined. The objective is to minimize the overall cost, i.e., to find an assignment scheme that minimizes the total cost of all matches. Let ytrue be the true class vector and ypred be the predicted class vector, then the objective function can be expressed as: Minimize ∑i=1n C[ytrue[i]][ypred[i]].

[0095] Low-confidence matching (second round matching): Matching the remaining unmatched tracking targets and low-confidence detection targets D. L The matching is performed using the same Hungarian algorithm, but only those targets that were not matched are considered, as expressed in the following formula:

[0096] C′={c′ ij |c′ ij =1-IOU(T′) i ,D′ j )},

[0097] Among them, T′ i D′ represents the tracked target that did not match after the first round of matching.j This indicates a target with low confidence.

[0098] In this embodiment, updating and managing the trajectory are also performed, specifically including:

[0099] Trajectory Update: For detected targets and tracked targets that are successfully associated through the first round of high-confidence matching and the second round of low-confidence matching, update the bounding box position or motion state information of the detected targets to the corresponding tracking trajectory.

[0100] New trajectory initialization: For high-confidence detected targets that do not match any existing tracked targets in the current frame, they are identified as newly appearing targets, and new tracking trajectories are initialized. A unique trajectory identifier is assigned to the target, and its status information is recorded in the new trajectory.

[0101] Low-confidence target handling: For low-confidence detected targets that do not match any tracked targets, a decision is made based on a preset strategy to either ignore these targets or mark them as potential targets for continuous tracking; if continuous tracking is selected, temporary trajectories are created for these potential targets, and attempts are made to match them in subsequent frames to verify whether they are real targets.

[0102] As can be seen, successfully matched detection targets will be updated to the corresponding tracking trajectory, unmatched high-confidence detection targets will be initialized to a new trajectory, and unmatched low-confidence detection targets will be ignored or continued to be tracked as possible targets.

[0103] When generating pedestrian feature vectors, the feature extraction formula is expressed as the following vector:

[0104] f i =g(I i ),

[0105] Among them, I i The detected pedestrian bounding box is represented by g(·), which represents the feature extraction network. In this embodiment, OSNet is used. i It is the extracted 2048-dimensional pedestrian feature vector.

[0106] In this embodiment, the following is also performed:

[0107] A trajectory lifecycle management mechanism is established to perform trajectory termination and aging processing steps. For tracking trajectories that have not matched any detected target for multiple consecutive frames (e.g., exceeding a preset frame threshold, such as 5 frames), the system marks them as aging trajectories, specifically including:

[0108] Unmatched frame count: A counter for unmatched frames is maintained for each active trajectory. When the trajectory fails to match any detected target in a frame, the counter is incremented by one; otherwise, if a match is successful, the counter is reset to zero.

[0109] Aging threshold determination: A preset threshold for the number of unmatched frames (e.g., 5 frames) is used. When the counter for the number of unmatched frames of a certain trajectory reaches or exceeds this threshold, the trajectory is marked as an aging trajectory.

[0110] Aging treatment: For items marked as aging tracks, perform one or more of the following aging treatment measures:

[0111] Reduce the priority or weight of this trajectory in subsequent tracking and decision-making processes; move the trajectory to a dedicated aging trajectory storage area; send an aging notification to the system administrator or relevant applications.

[0112] The system sets retention periods or conditions for aging trajectories (such as the number of aging trajectories exceeding a certain threshold or system resource shortages). When these conditions are met, the aging trajectories are deleted to completely release system resources. After deleting the aging trajectories, the resource allocation of the remaining active trajectories is dynamically adjusted to ensure that tracking efficiency and accuracy are not affected, and to prepare for the initialization and management of new trajectories.

[0113] In the pedestrian re-identification service, the cosine similarity between the received pedestrian feature vector and the features synchronized with the shared server data is calculated to confirm the pedestrian's identity. The result is then returned to the video analysis service via the Kafka communication protocol. The cosine similarity calculation is expressed as the following formula:

[0114]

[0115] Among them, f i ·f j Represents vector f i sum vector f j The dot product, ||f i ‖ and ||f j || represents the vector f i sum vector f j The norm of a vector is its length.

[0116] After receiving the returned results, the video analytics service displays the footage in real time. During this process, the load balancing request dynamically schedules the edge devices running the video analytics service and the pedestrian re-identification service based on the resource usage of each edge device, including:

[0117] The system periodically collects and analyzes the resource utilization of each edge device, including the utilization of key resources such as CPU, GPU, memory, and bandwidth of devices A, B, and C, as well as the current service types and service load. Based on the collected resource utilization data, a resource utilization threshold is set. When the resource utilization of a device exceeds the preset threshold (e.g., CPU or GPU utilization exceeds 80%), while the utilization of similar resources of another device is far below the average level (e.g., below 30%), a load imbalance is determined. Based on the load imbalance determination, the system selects the task type to be migrated (e.g., video analytics service or pedestrian re-identification service) and the target device (e.g., migrate from device A to device C) to maximize resource utilization efficiency and load balance.

[0118] Containerization technology is used to start and run migration tasks on the target device, ensuring the smoothness and continuity of service migration.

[0119] During task migration, ensure data synchronization between the source device (e.g., device A) and the target device (e.g., device C), especially the continuous transmission of video stream data; use Kafka or other efficient messaging protocols to achieve low-latency, high-reliability transmission of video segments, ensuring the continuity and accuracy of the analysis process.

[0120] For pedestrian re-identification or pedestrian tracking services that require collaborative processing, feature extraction and result sharing between devices are supported. That is, after device C processes part of the task, it returns the intermediate or final results to device B, or directly uploads them to the central server, ensuring the collaborative completion of the re-identification task and seamless integration of results.

[0121] After task migration and reallocation are completed, the resource usage and system performance of each device are continuously monitored. If a new load imbalance or resource bottleneck is detected, a new round of load assessment and task migration decision is automatically triggered, or some migration tasks are rolled back to achieve optimal resource utilization and continuous stable operation of the system.

[0122] Specifically, suppose an edge computing cluster contains edge devices A, B, and C, with video streams from two regions, each containing 2-3 cameras. The initial scenario is as follows:

[0123] Edge device A: Receives the video stream from region one and runs video analytics services.

[0124] Edge device B: Receives the video stream from region two and runs video analytics and pedestrian re-identification services.

[0125] Edge device C: Currently under low load, serving as a backup only.

[0126] At this point, devices A and B collaborate. Device A performs video analysis on the received video stream and sends pedestrian feature vectors to device B. Device B's pedestrian re-identification service receives and processes the vectors, returning the results to device A. Device B sends a portion of the received video stream to device A for video analysis and then performs pedestrian re-identification on the results. The remaining portion is analyzed and re-identified locally. Device C remains idle. However, as time progresses, the number of pedestrians in the area increases, and the video stream density rises. The resource consumption of devices A and B increases dramatically, and the demand for video analysis and pedestrian re-identification services also grows.

[0127] To prevent any device from becoming a bottleneck, the load balancing request will perform the following actions:

[0128] (1) Resource monitoring: The load balancing system continuously monitors the usage of resources such as CPU, GPU, memory, and bandwidth of device A and device B, and detects that they are approaching the upper limit of resource usage.

[0129] (2) Load Assessment: Based on the current load, the system assesses whether tasks need to be reallocated. For example, device A's GPU utilization is 85%, while device C's GPU utilization is only 20%.

[0130] (3) Task migration:

[0131] The system decided to migrate some video analytics tasks from device A to device C. The video analytics service was quickly launched on device C using containerization technology.

[0132] Similarly, some tasks in the pedestrian re-identification service of device B can be migrated to device C, thereby freeing up the resources of device B.

[0133] (4) Data synchronization and collaboration:

[0134] Data synchronization after task migration: During task migration, video stream data needs to be synchronized between device A and device C to ensure the continuity of the analysis process. Kafka or other messaging protocols can be used to transfer video clips between devices.

[0135] Feature extraction and re-identification collaboration: Device B and Device C collaborate. After processing part of the pedestrian re-identification task, Device C returns the result to Device B or directly sends the result to the central server. The two devices need to ensure seamless integration between the intermediate state and the final result of the re-identification task.

[0136] (5) Feedback mechanism:

[0137] After the task migration and reallocation are completed, the load balancing system will continuously monitor the new resource usage to ensure that the migrated system runs efficiently under load balance.

[0138] If a device becomes overloaded again, the system can continue to adjust task allocation and even roll back some migration tasks to achieve optimal resource utilization.

[0139] Through the above example, each edge device can dynamically collaborate through a load balancing request system to rationally allocate computing tasks and achieve optimal utilization of computing resources. This dynamic adjustment mechanism ensures efficient operation in the edge computing environment, avoids device overload and resource waste, effectively addresses network changes, single points of failure, and privacy protection within the monitored area, and significantly reduces inference latency for video surveillance tasks. In practical applications, inference latency can reach below 41ms. Furthermore, this method optimizes the collaboration algorithm and data stream processing of edge devices, ensuring rapid real-time processing even when the computational load increases due to a sudden increase in pedestrians within the monitored area. It is suitable for security monitoring scenarios with limited resources, privacy protection, and the need for rapid response. In summary, this embodiment proposes a low-latency, highly reliable, and secure distributed edge collaborative video surveillance method, enabling it to better serve the needs of smart city security.

[0140] In practical applications, through real-world testing, the distributed edge collaboration-based video surveillance method in this embodiment significantly reduces inference latency. Test results show that in a video surveillance system consisting of ten cameras and six edge devices, the inference latency can be reduced to below 41ms. Furthermore, the data transmitted between edge devices is pedestrian feature vectors rather than pedestrian photographs, effectively preventing the leakage of pedestrian privacy.

[0141] In summary, this embodiment innovatively divides the video surveillance task into two independent services—pedestrian tracking and pedestrian re-identification—by adopting a distributed architecture design, and deploys them on different heterogeneous edge devices, thereby achieving an efficient and flexible collaborative reasoning mechanism.

[0142] Furthermore, this embodiment separates and optimizes pedestrian tracking and pedestrian re-identification services, allowing each service to focus on its strengths, thereby improving the overall efficiency and accuracy of monitoring. The pedestrian tracking service can detect and track pedestrian targets in real time and accurately, while the pedestrian re-identification service utilizes powerful feature comparison capabilities to accurately identify the same pedestrian under different cameras.

[0143] Furthermore, the distributed architecture design of this embodiment enables the system to easily handle monitoring scenarios of varying scales and complexities. By dynamically adjusting task allocation and collaboration, the system can flexibly deploy and migrate services across different edge devices based on actual needs and resource availability, thereby achieving high scalability and flexibility.

[0144] Furthermore, this embodiment dynamically adjusts task allocation and collaboration based on network status and device resource usage, ensuring maximum resource utilization. This avoids resource waste or performance bottlenecks caused by uneven resource allocation, while also reducing the overall energy consumption of the system.

[0145] Furthermore, the distributed architecture and collaborative reasoning mechanism enable the system to have higher fault tolerance to the failure of individual devices. Even if one device fails, other devices can continue to provide services, thereby ensuring the stability and reliability of the system.

[0146] Furthermore, this embodiment not only promotes innovation in video surveillance technology, but also provides strong support for the upgrading of related industries. By optimizing the video surveillance process and improving monitoring efficiency and accuracy, this invention injects new vitality into the development of smart cities, intelligent security and other fields.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A video surveillance method based on distributed edge collaboration, characterized in that, Includes the following steps: The video surveillance task is divided into pedestrian tracking service and pedestrian re-identification service; The pedestrian tracking service and the pedestrian re-identification service are deployed on different edge devices to achieve collaborative inference between heterogeneous edge devices, and the Apache Kafka communication protocol is used to achieve bidirectional communication between the video analysis service and the pedestrian re-identification service. The acquired real-time video stream signal is pushed to the edge device cluster, and the pedestrians entering the monitoring area are subjected to initial feature extraction to generate pedestrian features. The pedestrian features are then sent to the shared server for storage. The pedestrian tracking service detects and tracks pedestrians within the monitored area, generating pedestrian feature vectors. The pedestrian re-identification service calculates the similarity between the received pedestrian feature vector and the pedestrian features stored in the shared server to determine the pedestrian's identity. Based on the current computing resource usage of heterogeneous edge devices, the edge devices running the video analytics service and the pedestrian re-identification service are dynamically scheduled. Specifically, this includes: periodically collecting and analyzing the resource utilization rate of each edge device, as well as the currently running service type and service load; setting resource utilization thresholds based on the collected resource utilization data; determining that there is a load imbalance when the resource utilization rate of one device exceeds the preset threshold, while the resource utilization rate of another device is far below the average level; selecting the task types and target devices to be migrated based on the load imbalance determination results to maximize resource utilization efficiency and load balance; and using containerization technology to start and run the migrated tasks on the target devices. The shared server periodically synchronizes data with the edge devices running the pedestrian re-identification service.

2. The method according to claim 1, characterized in that, Also includes: The feature extraction service performs initial feature extraction on pedestrians entering the monitored area, including: Assuming the video stream input frame is Size is W, the convolution kernel is Size is That is, height m and width n, the output feature map is Size is Then it can be expressed as the following formula: in, Indicates the position in the output feature map pixel values, Indicates the position of the input frame pixel values, Indicates the position of the convolution kernel The weight, and Indicates the height and width of the convolution kernel. and This represents the width and height of the output feature map.

3. The method according to claim 1, characterized in that: Pedestrian tracking services achieve target tracking by tracking detected targets with high confidence and low confidence levels, specifically including: Input definition: High confidence test set : Satisfies the confidence threshold The set of targets to be detected; Low confidence detection set Below the confidence threshold The set of targets to be detected; The confidence score is provided by the target detection algorithm.

4. The method according to claim 3, characterized in that: Pedestrian tracking is achieved through high-confidence matching and low-confidence matching, specifically including: High-confidence matching: using a high-confidence detection set Hungarian matching is performed, which matches the detected target in the current frame with the tracked target in the previous frame. This is expressed by the following formula: in, Indicates the current tracking target bounding box, Indicates the target of detection j bounding box, Indicate target and detection Intersection over Union (IOU) By minimizing the cost matrix To achieve the best match for high-confidence objectives; Low-confidence matching: Matching the remaining unmatched tracked targets and low-confidence detected targets. The matching is performed using the same Hungarian algorithm, but only those targets that were not matched are considered, as expressed in the following formula: in, This indicates the tracked targets that did not match after the first round of matching. This indicates a target with low confidence.

5. The method according to claim 4, characterized in that, It also performs updates and manages tracks, specifically including: Trajectory Update: For detected targets and tracked targets that are successfully associated through the first round of high-confidence matching and the second round of low-confidence matching, update the bounding box position or motion state information of the detected targets to the corresponding tracking trajectory; New trajectory initialization: For high-confidence detected targets that do not match any existing tracked targets in the current frame, they are identified as newly appearing targets, and new tracking trajectories are initialized. A unique trajectory identifier is assigned to the target, and its status information is recorded in the new trajectory. Low-confidence target handling: For low-confidence detected targets that do not match any tracked targets, a decision is made based on a preset strategy to either ignore these targets or mark them as potential targets for continuous tracking; if continuous tracking is selected, temporary trajectories are created for these potential targets, and attempts are made to match them in subsequent frames to verify whether they are real targets.

6. The method according to claim 5, characterized in that: When generating pedestrian feature vectors, the feature extraction formula is expressed as the following vector: in, This indicates the detected pedestrian bounding boxes. This represents a feature extraction network. It is the extracted 2048-dimensional pedestrian feature vector.

7. The method according to claim 5, characterized in that, Also execute: Establish a trajectory lifecycle management mechanism to perform trajectory termination and aging processing steps, specifically including: Unmatched frame count: A counter for unmatched frames is maintained for each active trajectory. When the trajectory fails to match any detected target in a frame, the counter is incremented by one; otherwise, if a match is successful, the counter is reset to zero. Aging threshold determination: A preset threshold for the number of unmatched frames is used. When the counter for the number of unmatched frames of a certain trajectory reaches or exceeds the threshold, the trajectory is marked as an aging trajectory. Aging treatment: For items marked as aging tracks, perform one or more of the following aging treatment measures: Reduce the priority or weight of this trajectory in subsequent tracking and decision-making processes; move the trajectory to a dedicated aging trajectory storage area; send an aging notification to the system administrator or relevant applications; The system sets retention periods or conditions for aging tracks. When these conditions are met, the aging tracks are deleted to completely free up system resources. After deleting the aging tracks, the resource allocation of the remaining active tracks is dynamically adjusted to ensure that tracking efficiency and accuracy are not affected, and to prepare for the initialization and management of new tracks.

8. The method according to any one of claims 1 to 7, characterized in that: In the pedestrian re-identification service, the cosine similarity between the received pedestrian feature vector and the features synchronized with the shared server data is calculated, expressed as the following formula: in, Representing vectors sum vector dot product, and Representing vectors sum vector The norm of a vector is its length.

9. The method according to claim 1, characterized in that, Also includes: During the task migration process, ensure data synchronization between the source device and the target device; For pedestrian re-identification or pedestrian tracking services that require collaborative processing, feature extraction and result sharing between devices are supported. That is, after device C processes part of the task, it returns the intermediate or final result to device B, or uploads it directly to the central server. After task migration and reallocation are completed, the resource usage and system performance of each device are continuously monitored. If a new load imbalance or resource bottleneck is detected, a new round of load assessment and task migration decision is automatically triggered, or some migration tasks are rolled back to achieve optimal resource utilization and continuous stable operation of the system.

Images