Video monitoring method based on distributed edge collaboration

Abstract

The invention provides a video monitoring method based on distributed edge cooperation, and the method comprises the steps: enabling a pedestrian tracking service and a pedestrian re-recognition service to be disposed on different edge devices, and achieving the bidirectional communication through an Apache Kafka communication protocol; performing first feature extraction on pedestrians entering the monitoring area to generate pedestrian features; the pedestrian tracking service detects pedestrians in the monitoring area, realizes pedestrian tracking and generates pedestrian feature vectors; the pedestrian re-identification service performs similarity calculation on the received pedestrian feature vector and the pedestrian features stored in the shared server, and judges the identity of the pedestrian; and dynamically scheduling the edge devices running the video analysis service and the pedestrian re-identification service according to the current computing resource use condition of the heterogeneous edge devices. According to the invention, tracking and re-identification of pedestrians are realized through cooperative reasoning of edge devices in a monitoring area, so that single-point faults and privacy leakage are avoided, and the real-time performance and stability of video monitoring are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent video surveillance, and in particular to a video surveillance method based on distributed edge collaboration. Background Art

[0002] In the field of video surveillance, with the rapid development of big data, artificial intelligence and Internet of Things technologies, the application of video surveillance systems has become increasingly widespread, covering many important fields such as public safety, intelligent transportation, and smart cities. Most of the existing video surveillance systems adopt the traditional centralized architecture, that is, the monitoring data is centrally transmitted to the cloud server for processing and storage. This architecture relies on a high-bandwidth, low-latency network environment to ensure the real-time transmission and processing of video data. However, under the existing network conditions, it is difficult for the centralized video surveillance system to ensure that the video data of all monitoring points can be transmitted to the central server in real time and smoothly, resulting in a significant reduction in the monitoring effect. At the same time, the delay in the return result of cloud processing may also lead to the omission or false alarm of key events. In addition, there is a risk of data theft and privacy leakage in the process of video data being transmitted from the camera to the cloud server for processing.

[0003] The existing distributed edge collaborative video analysis method based on multi-agent reinforcement learning adopts a distributed architecture design, divides the video surveillance task into multiple sub-services or tasks, and deploys them on multiple edge devices. Although it solves the delay problem caused by long-distance data transmission, it does not clearly divide the video analysis task into pedestrian video analysis service and pedestrian re-identification service, and deploys them on different edge devices. In addition, in task division, agent modeling and service decision-making, it fails to respond well to the increase in computing power caused by the increase in pedestrians in the area and the difficulty of edge devices to load, resulting in a decline in video surveillance performance. Summary of the invention

[0004] In order to solve the problems of high latency, poor privacy and security, and limited resources in traditional centralized video surveillance methods and edge computing methods, as well as unclear task division, poor service deployment and startup decision-making in reinforcement learning distributed edge collaboration methods, the purpose of the present invention is to provide a video surveillance method based on distributed edge collaboration. The core of this method is to achieve tracking and re-identification of pedestrians through collaborative reasoning of edge devices in the monitoring area, thereby avoiding single point failures and privacy leakage, and improving the real-time and stability of video surveillance.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] A video surveillance method based on distributed edge collaboration includes the following steps:

[0007] Divide the video surveillance task into pedestrian tracking service and pedestrian re-identification service;

[0008] Deploy the pedestrian tracking service and pedestrian re-identification service on different edge devices to achieve collaborative reasoning between heterogeneous edge devices, and use the Apache Kafka communication protocol to achieve two-way communication between the video analysis service and the pedestrian re-identification service;

[0009] The acquired real-time video stream signal is pushed to the edge device cluster, and the pedestrians entering the monitoring area are first extracted to generate pedestrian features, which are then sent to the shared server for storage.

[0010] The pedestrian tracking service detects and tracks pedestrians in the monitoring area and generates pedestrian feature vectors;

[0011] The pedestrian re-identification service calculates the similarity between the received pedestrian feature vector and the pedestrian features stored in the shared server to determine the pedestrian's identity;

[0012] Dynamically schedule edge devices running video analysis services and person re-identification services based on the current computing resource usage of heterogeneous edge devices;

[0013] Among them, the shared server regularly synchronizes data with the edge devices running the pedestrian re-identification service.

[0014] A video monitoring method based on distributed edge collaboration provided by the present invention also includes:

[0015] Feature extraction service, which performs the first feature extraction on pedestrians entering the monitoring area, including:

[0016] Assuming that the video stream input frame is I, with a size of H×W, the convolution kernel is K, with a size of m×n, that is, height m and width n, and the output feature map is O, with a size of H′×W′, it can be expressed as the following formula:

[0017]

[0018] Among them, O(i, j) represents the pixel value at position (i, j) in the output feature map, I(i+u, j+v) represents the pixel value at position (i+u, j+v) of the input frame, K(u, v) represents the weight of the convolution kernel at position (u, v), m and n represent the height and width of the convolution kernel, and H′ and W′ represent the width and height of the output feature map.

[0019] According to a video surveillance method based on distributed edge collaboration provided by the present invention, a pedestrian tracking service realizes target tracking by tracking high-confidence and low-confidence detection targets, specifically including:

[0020] Input definition:

[0021] High confidence detection set D H : The set of detected targets that meet the confidence threshold θ;

[0022] Low confidence detection set D L : The set of detected targets below the confidence threshold θ;

[0023] Among them, the confidence value is provided by the target detection algorithm.

[0024] According to a video surveillance method based on distributed edge collaboration provided by the present invention, pedestrian tracking is implemented through high confidence matching and low confidence matching, specifically including:

[0025] High confidence matching: Use the high confidence detection set D1 for Hungarian matching to match the detected target in the current frame with the tracked target in the previous frame, expressed as the following formula:

[0026] C={c ij |c ij =1-IOU(T i ,D j )},

[0027] Among them, T i represents the bounding box of the current tracked target i, D j represents the bounding box of the detected target j, IOU(T i ,D j ) represents the target T i and detection D j Intersection over Union (IOU);

[0028] By minimizing the cost matrix C, the best match of high confidence targets is achieved;

[0029] Low confidence matching: For the remaining unmatched tracking targets and low confidence detection targets D L For matching, the same Hungarian algorithm is used, but only those targets that have not been matched are considered, expressed as the following formula:

[0030] C′={c′ ij |c′ ij =1-IOU(T′ i ,D′ j )},

[0031] Among them, T′ i represents the tracking target that is not matched after the first round of matching, D′ j Indicates a low confidence detection target.

[0032] According to a video monitoring method based on distributed edge collaboration provided by the present invention, updating and managing trajectories are also performed, specifically including:

[0033] Track update: For the detection targets and tracking targets that are successfully associated through the first round of high confidence matching and the second round of low confidence matching, the bounding box position or motion state information of the detection target is updated to the corresponding tracking track;

[0034] New track initialization: For high-confidence detection targets that are not matched to any existing tracking targets in the current frame, identify them as new targets, initialize new tracking tracks, assign a unique track identifier to the target, and record its status information in the new track;

[0035] Low-confidence target processing: For low-confidence detection targets that are not matched to any tracking targets, the preset strategy decides whether to ignore these targets or mark them as potential targets for continuous tracking; if continuous tracking is selected, temporary tracks are created for these potential targets, and attempts are made to match them in subsequent frames to verify whether they are real targets.

[0036] According to a video surveillance method based on distributed edge collaboration provided by the present invention, when generating a pedestrian feature vector, the feature extraction formula is expressed as the following vector:

[0037] f i =g(I i ),

[0038] Among them, I i represents the detected pedestrian box, g(·) represents the feature extraction network, and f i It is the extracted 2048-dimensional pedestrian feature vector.

[0039] According to a video monitoring method based on distributed edge collaboration provided by the present invention, the following steps are further performed:

[0040] Set up a trace lifecycle management mechanism, which is used to perform trace termination and aging processing steps, including:

[0041] Unmatched frame count: A unmatched frame counter is maintained for each active track. When the track fails to match any detection target in a frame, the counter is incremented by one; otherwise, if the match is successful, the counter is reset to zero.

[0042] Aging threshold judgment: a threshold of unmatched frames is preset. When the unmatched frame counter of a certain track reaches or exceeds the threshold, the track is marked as an aged track.

[0043] Aging processing: For entries marked as aging tracks, perform one or more of the following aging processing measures:

[0044] Reduce the priority or weight of the track in subsequent tracking and decision-making processes; move the track to a dedicated aging track storage area; send an aging notification to the system administrator or related applications;

[0045] Among them, the retention period or conditions of the aging tracks are set. When these conditions are met, the aging tracks are deleted to completely release system resources. After deleting the aging tracks, the resource allocation of the remaining active tracks is dynamically adjusted to ensure that the tracking efficiency and accuracy are not affected, and to prepare for the initialization and management of new tracks.

[0046] According to a video surveillance method based on distributed edge collaboration provided by the present invention, in a pedestrian re-identification service, a cosine similarity calculation is performed between a received pedestrian feature vector and a feature synchronized with shared server data, which is expressed as the following formula:

[0047]

[0048] Among them, f i ·f j Represents the vector f i and vector f j The dot product of i ‖ and ||f j || represents the vector f i and vector f j The norm of , which is the length of the vector.

[0049] According to a video surveillance method based on distributed edge collaboration provided by the present invention, the edge device for dynamically scheduling the operation of the video analysis service and the pedestrian re-identification service includes:

[0050] Periodically collect and analyze the resource usage of each edge device, as well as the type of services currently running and the service load;

[0051] According to the collected resource usage data, a resource usage threshold is set. When the resource usage rate of a certain device exceeds the preset threshold, while the resource usage rate of the same type of another device is far below the average level, it is determined that there is load imbalance;

[0052] Based on the load imbalance determination result, select the task type and target device to be migrated to maximize resource utilization efficiency and load balance;

[0053] Among them, containerization technology is used to start and run the migration task on the target device.

[0054] A video monitoring method based on distributed edge collaboration provided by the present invention also includes:

[0055] During the task migration process, ensure data synchronization between the source device and the target device;

[0056] For tasks that require collaborative processing in pedestrian re-identification services or pedestrian tracking services, feature extraction and result sharing between devices are supported. That is, after device C processes part of the task, it returns the intermediate results or final results to device B, or directly uploads them to the central server.

[0057] After task migration and reallocation are completed, the resource usage and system performance of each device are continuously monitored; if a new load imbalance or resource bottleneck is detected, a new round of load assessment and task migration decisions will be automatically triggered, or some migration tasks will be rolled back to achieve optimal resource utilization and continuous and stable operation of the system.

[0058] It can be seen that the present invention innovatively divides the video surveillance task into two independent services, pedestrian tracking and pedestrian re-identification, by adopting a distributed architecture design, and deploys them on different heterogeneous edge devices, thereby realizing an efficient and flexible collaborative reasoning mechanism. Therefore, the technical solution of the present invention brings the following significant beneficial effects:

[0059] 1. Improve monitoring efficiency and accuracy: The present invention separates pedestrian tracking and pedestrian re-identification services and optimizes them separately, so that each service can focus on its own areas of expertise, thereby improving the efficiency and accuracy of overall monitoring. The pedestrian tracking service can detect and track pedestrian targets in real time and accurately, while the pedestrian re-identification service uses powerful feature comparison capabilities to accurately identify the same pedestrian under different cameras.

[0060] 2. Enhance system flexibility and scalability: The distributed architecture design of the present invention enables the system to easily cope with monitoring scenarios of different scales and complexities. By dynamically adjusting task allocation and collaboration, the system can flexibly deploy and migrate services between different edge devices according to actual needs and resource conditions, thereby achieving high scalability and flexibility.

[0061] 3. Optimize resource utilization and reduce energy consumption: The present invention dynamically adjusts the allocation and collaboration of tasks according to the network status and device resource usage, ensuring the maximum utilization of resources. It avoids resource waste or performance bottlenecks caused by uneven resource allocation, and also reduces the overall energy consumption of the system.

[0062] 4. Improve system stability and reliability: The distributed architecture and collaborative reasoning mechanism make the system more tolerant to the failure of a single device. Even if a device fails, other devices can continue to provide services, thus ensuring the stability and reliability of the system.

[0063] 5. Promote technological innovation and industrial upgrading: The implementation of the present invention not only promotes the innovation of video surveillance technology, but also provides strong support for the upgrading of related industries. By optimizing the video surveillance process and improving monitoring efficiency and accuracy, the present invention has injected new vitality into the development of smart cities, smart security and other fields.

[0064] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flow chart of an embodiment of a video monitoring method based on distributed edge collaboration of the present invention.

[0066] Figure 2 It is a system architecture schematic diagram of an embodiment of a video surveillance method based on distributed edge collaboration of the present invention.

[0067] Figure 3 It is a process schematic diagram of pedestrian tracking service, feature extraction service and pedestrian re-identification service in an embodiment of a video surveillance method based on distributed edge collaboration of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0070] See also Figures 1 to 3 , this embodiment provides a video monitoring method based on distributed edge collaboration, the method comprising the following steps:

[0071] Step S1, dividing the video surveillance task into pedestrian tracking service and pedestrian re-identification service;

[0072] Step S2: Deploy the pedestrian tracking service and the pedestrian re-identification service on different edge devices respectively to achieve collaborative reasoning between heterogeneous edge devices, and use the Apache Kafka communication protocol to achieve two-way communication between the video analysis service and the pedestrian re-identification service;

[0073] Step S3: Push the acquired real-time video stream signal to the edge device cluster, perform the first feature extraction on the pedestrians entering the monitoring area, generate pedestrian features, and send the pedestrian features to the shared server for storage;

[0074] Step S4, the pedestrian tracking service detects and tracks pedestrians in the monitoring area and generates a pedestrian feature vector;

[0075] Step S5, the pedestrian re-identification service calculates the similarity between the received pedestrian feature vector and the pedestrian features stored in the shared server, thereby determining the identity of the pedestrian;

[0076] Step S6, dynamically scheduling edge devices running video analysis services and pedestrian re-identification services according to current computing resource usage of heterogeneous edge devices;

[0077] Among them, the shared server regularly synchronizes data with the edge devices running the pedestrian re-identification service.

[0078] Therefore, the video surveillance method based on distributed edge collaboration provided in this embodiment includes RTSP video streaming, feature extraction service, pedestrian tracking service (video analysis), pedestrian re-identification service, Apache Kafka communication protocol, load balancing algorithm, and data synchronization. Among them, RTSP streaming is responsible for streaming the images captured by the camera to the edge device cluster; the feature extraction service performs the first feature extraction of pedestrians entering the monitoring area and sends it to the shared server for storage; the video analysis service performs detection of pedestrians in the area and realizes pedestrian tracking; the pedestrian re-identification service determines the identity of the pedestrian by calculating the similarity between the received pedestrian features and the features stored in the shared server; the Apache Kafka communication protocol is responsible for the communication between the video analysis service and the pedestrian re-identification service; the load balancing request dynamically schedules the edge devices running the video analysis service and the pedestrian re-identification service according to the usage of the computing resources of the heterogeneous edge devices, so that the overall system maintains the best performance; data synchronization, the shared server synchronizes the stored features with the edge devices running the pedestrian re-identification.

[0079] Among them, the RTSP video stream is connected to the feature extraction service and the video analysis service to obtain the video stream in real time and analyze it; the feature extraction service is connected to the shared server to store the extracted pedestrian features; the video analysis service and the pedestrian re-identification service are connected through a two-way communication connection via the Kafka communication protocol; the pedestrian re-identification service communicates with the shared server in a two-way manner, the former obtains the features of the latter's data synchronization for comparison and returns the results.

[0080] In this embodiment, the feature extraction service is used to extract the first feature of pedestrians entering the monitoring area (entrance), including:

[0081] Assuming that the video stream input frame is I, the size is H×W (height H and width W), the convolution kernel is K, the size is m×n, that is, height m and width n, the output feature map is O, the size is H′×W′, it can be expressed as the following formula:

[0082]

[0083] Among them, O(i, j) represents the pixel value at position (i, j) in the output feature map, I(i+u, j+v) represents the pixel value at position (i+u, j+v) of the input frame, K(u, v) represents the weight of the convolution kernel at position (u, v), m and n represent the height and width of the convolution kernel, and H′ and W′ represent the width and height of the output feature map.

[0084] The feature extraction service sends the extracted pedestrian features to the shared server for storage. The RTSP video stream is connected to the video analysis service. The video analysis service tracks pedestrians in the area and sends the tracking analysis results (i.e., pedestrian feature vectors) to the pedestrian re-identification service through the Kafka communication protocol. The pedestrian tracking service implements target tracking by tracking high-confidence and low-confidence detection targets, including:

[0085] (1) Input definition:

[0086] High confidence detection set D H : The set of detected targets that meet the confidence threshold θ;

[0087] Low confidence detection set D L : The set of detected targets below the confidence threshold θ;

[0088] The confidence value is provided by the target detection algorithm, and this embodiment uses YOLOv8.

[0089] (2) Matching steps

[0090] When tracking pedestrians, it is mainly achieved through two steps of high confidence matching and low confidence matching, including:

[0091] High confidence matching (first round matching): Use high confidence detection set D H Perform Hungarian matching (Hungarian algorithm) to match the detected target in the current frame with the tracked target in the previous frame, which is expressed as the following formula:

[0092] C={c ij |c ij =1-IOU(T i ,D j )},

[0093] Among them, T i represents the bounding box of the current tracked target i, D j represents the bounding box of the detected target j, IOU(T i ,D j ) represents the target T i and detection D j Intersection over Union (IOU).

[0094] By minimizing the cost matrix C, the best match of the high confidence target is achieved. Specifically, minimizing the cost matrix to achieve the best match of the high confidence target usually involves optimization techniques, especially in task assignment, pattern recognition or classification problems in machine learning and data mining. First, define the cost matrix. The cost matrix (Cost Matrix), also known as the loss matrix, is a two-dimensional array in which each element represents the cost of assigning one category to another category. Assuming there are N true categories and M predicted categories, the size of the cost matrix C is N×M, where C[i][j] represents the cost of assigning the true category i to the predicted category j. Then determine the objective function. The goal is to minimize the overall cost, that is, to find an assignment scheme that minimizes the total cost of all matches. Let ytrue be the true category vector and ypred be the predicted category vector, then the objective function can be expressed as: Minimize∑i=1n C[ytrue[i]][ypred[i]].

[0095] Low confidence matching (second round matching): for the remaining unmatched tracking targets and low confidence detection targets D L For matching, the same Hungarian algorithm is used, but only those targets that have not been matched are considered, expressed as the following formula:

[0096] C′={c′ ij |c′ ij =1-IOU(T′ i ,D′ j )},

[0097] Among them, T′ i represents the tracking target that is not matched after the first round of matching, D′j Indicates a low confidence detection target.

[0098] In this embodiment, updating and managing tracks are also performed, specifically including:

[0099] Track update: For the detection targets and tracking targets that are successfully associated through the first round of high-confidence matching and the second round of low-confidence matching, the bounding box position or motion state information of the detection target is updated to the corresponding tracking track.

[0100] New track initialization: For high-confidence detection targets that are not matched to any existing tracking targets in the current frame, identify them as newly appeared targets and initialize new tracking tracks. A unique track identifier is assigned to the target and its status information is recorded in the new track.

[0101] Low-confidence target processing: For low-confidence detection targets that are not matched to any tracking targets, the preset strategy decides whether to ignore these targets or mark them as potential targets for continuous tracking; if continuous tracking is selected, temporary tracks are created for these potential targets, and attempts are made to match them in subsequent frames to verify whether they are real targets.

[0102] It can be seen that the successfully matched detection targets will be updated to the corresponding tracking trajectory, the unmatched high-confidence detection targets will be initialized as new trajectories, and the unmatched low-confidence detection targets will be ignored or continue to be tracked as possible targets.

[0103] When generating a pedestrian feature vector, the feature extraction formula is expressed as the following vector:

[0104] f i =g(I i ),

[0105] Among them, I i represents the detected pedestrian box, g(·) represents the feature extraction network, and this embodiment uses OSNet, f i It is the extracted 2048-dimensional pedestrian feature vector.

[0106] In this embodiment, the following is also performed:

[0107] Set a track lifecycle management mechanism, which is used to execute the track termination and aging processing steps. For a track that does not match any detection target for multiple consecutive frames (such as exceeding a preset frame number threshold, such as 5 frames), the system marks it as an aged track. Specifically, it includes:

[0108] Unmatched frame count: A unmatched frame counter is maintained for each active track. When the track fails to match any detection target in a frame, the counter is incremented by one; otherwise, if the match is successful, the counter is reset to zero.

[0109] Aging threshold judgment: a threshold of unmatched frames (such as 5 frames) is preset. When the unmatched frame counter of a certain track reaches or exceeds the threshold, the track is marked as an aged track.

[0110] Aging processing: For entries marked as aging tracks, perform one or more of the following aging processing measures:

[0111] Reduce the priority or weight of the track in subsequent tracking and decision-making processes; move the track to a dedicated aging track storage area; send an aging notification to a system administrator or related applications.

[0112] Among them, the retention period or conditions of aging trajectories are set (such as the number of aging trajectories exceeds a certain threshold, system resources are tight, etc.). When these conditions are met, the aging trajectories are deleted to completely release system resources. After deleting the aging trajectories, the resource allocation of the remaining active trajectories is dynamically adjusted to ensure that the tracking efficiency and accuracy are not affected, and prepare for the initialization and management of new trajectories.

[0113] In the pedestrian re-identification service, the cosine similarity calculation is performed on the received pedestrian feature vector and the features synchronized with the shared server data to confirm the pedestrian's identity, and the result is returned to the video analysis service through the Kafka communication protocol. The cosine similarity calculation is expressed as the following formula:

[0114]

[0115] Among them, f i ·f j Represents the vector f i and vector f j The dot product of i ‖ and ||f j || represents the vector f i and vector f j The norm of , which is the length of the vector.

[0116] After receiving the returned results, the video analysis service displays the images in real time. During this process, the load balancing request dynamically schedules the edge devices running the video analysis service and pedestrian re-identification service according to the resource usage of each edge device, including:

[0117] Periodically collect and analyze the resource usage of each edge device, including the usage of key resources such as CPU, GPU, memory, bandwidth, etc. of device A, device B, device C, etc., as well as the currently running service types and service load conditions; set resource usage thresholds based on the collected resource usage data. When the resource usage of a device exceeds the preset threshold (such as CPU or GPU usage exceeding 80%), and the resource usage of the same type of another device is far below the average level (such as below 30%), it is determined that there is load imbalance; based on the load imbalance determination result, select the task type that needs to be migrated (such as video analysis service or pedestrian re-identification service) and the target device (such as migrating from device A to device C) to maximize resource utilization efficiency and load balance.

[0118] Among them, containerization technology is used to start and run the migration tasks on the target device to ensure the smoothness and continuity of service migration.

[0119] During the task migration process, ensure data synchronization between the source device (such as device A) and the target device (such as device C), especially the continuous transmission of video stream data; use Kafka or other efficient messaging protocols to achieve low-latency and high-reliability transmission of video clips to ensure the continuity and accuracy of the analysis process.

[0120] For tasks that require collaborative processing of pedestrian re-identification services or pedestrian tracking services, feature extraction and result sharing between devices are supported. That is, after device C processes part of the task, it returns the intermediate results or final results to device B, or directly uploads them to the central server, ensuring the collaborative completion of the re-identification task and the seamless connection of the results.

[0121] After task migration and reallocation are completed, the resource usage and system performance of each device are continuously monitored; if a new load imbalance or resource bottleneck is detected, a new round of load assessment and task migration decisions will be automatically triggered, or some migration tasks will be rolled back to achieve optimal resource utilization and continuous and stable operation of the system.

[0122] Specifically, assume an edge computing cluster, including edge devices A, B, and C, with video streams from two areas, each area containing 2-3 cameras, and the initial scene is:

[0123] Edge device A: Receives the video stream from region 1 and runs the video analysis service.

[0124] Edge device B: Receives the video stream from area 2 and runs the video analysis service and the pedestrian re-identification service.

[0125] Edge device C: Currently under low load and only serves as a backup.

[0126] At this time, device A and device B cooperate with each other. After device A performs video analysis on the received video stream, it sends the pedestrian feature vector to device B. The pedestrian re-identification service of device B receives and processes the data and returns the result to device A. Device B will send part of the received video stream to device A for video analysis, and then receive the result for pedestrian re-identification. The other part will be directly analyzed and re-identified locally, while device C is idle. As time goes by, the number of pedestrians in the area increases, the density of video streams increases, the resource consumption of devices A and B increases sharply, and the demand for video analysis and pedestrian re-identification services also increases.

[0127] To prevent any one device from becoming a bottleneck, load balancing requests will do the following:

[0128] (1) Resource monitoring: The load balancing system continuously monitors the usage of CPU, GPU, memory, bandwidth and other resources of device A and device B, and detects that they are approaching the upper limit of resource usage.

[0129] (2) Load evaluation: Based on the current load situation, the system evaluates whether tasks need to be reallocated. For example, the GPU utilization of device A reaches 85%, while the GPU utilization of device C is only 20%.

[0130] (3) Task Migration:

[0131] The system decides to migrate some video analysis tasks from device A to device C. The video analysis service is quickly started on device C using containerization technology.

[0132] Similarly, some tasks in the pedestrian re-identification service of device B can also be migrated to device C, so that the resources of device B can be released.

[0133] (4) Data synchronization and collaboration:

[0134] Data synchronization after task migration: During the task migration process, the video stream data needs to be synchronized between device A and device C to ensure the continuity of the analysis process. Kafka or other messaging protocols can be used to transfer video clips between devices.

[0135] Feature extraction and re-identification collaboration: Device B collaborates with device C. After device C processes part of the pedestrian re-identification task, it returns the result to device B, or directly sends the result to the central server. The two devices need to ensure that the intermediate state and final result of the re-identification task can be seamlessly connected.

[0136] (5) Feedback mechanism:

[0137] After task migration and reallocation are completed, the load balancing system will continue to monitor the new resource usage to ensure that the migrated system runs efficiently in a load-balanced state.

[0138] If a device is overloaded again, the system can continue to adjust task allocation or even roll back some migration tasks to achieve optimal utilization of resources.

[0139] Through the above examples, each edge device can dynamically collaborate through the load balancing request system, reasonably allocate computing tasks, and achieve optimal utilization of computing resources. This dynamic adjustment mechanism ensures efficient operation in the edge computing environment, avoids equipment overload and resource waste, can effectively respond to network changes, single point failures and privacy protection in the monitoring area, and greatly reduces the inference delay of video surveillance tasks. In practical applications, the inference delay time can be less than 41ms. In addition, the method optimizes the collaborative algorithm and data stream processing of edge devices, ensuring that even if the amount of calculation caused by a sudden increase in pedestrians in the monitoring area increases, it can be processed quickly and in real time, which is suitable for security monitoring scenarios with limited resources, privacy protection, and rapid response. In summary, this embodiment proposes a low-latency, highly reliable and secure distributed edge collaborative video surveillance method, so that it can better serve the needs of urban smart security.

[0140] In actual applications, through actual application tests, the video surveillance method based on distributed edge collaboration in this embodiment significantly reduces the inference delay. The test results show that in a video surveillance system consisting of ten cameras and six edge devices, the inference delay can be reduced to less than 41ms. In addition, the data transmitted between edge devices is the feature vector of pedestrians rather than their photos, which effectively prevents the privacy leakage of relevant pedestrians.

[0141] In summary, this embodiment adopts a distributed architecture design to innovatively divide the video surveillance task into two independent services: pedestrian tracking and pedestrian re-identification, and deploy them on different heterogeneous edge devices respectively, thereby realizing an efficient and flexible collaborative reasoning mechanism.

[0142] Furthermore, this embodiment separates pedestrian tracking from pedestrian re-identification services and optimizes them separately, so that each service can focus on its own areas of expertise, thereby improving the efficiency and accuracy of overall monitoring. The pedestrian tracking service can detect and track pedestrian targets in real time and accurately, while the pedestrian re-identification service uses powerful feature comparison capabilities to accurately identify the same pedestrian under different cameras.

[0143] Furthermore, the distributed architecture design of this embodiment enables the system to easily cope with monitoring scenarios of different scales and complexities. By dynamically adjusting task allocation and collaboration, the system can flexibly deploy and migrate services between different edge devices according to actual needs and resource conditions, thereby achieving high scalability and flexibility.

[0144] Furthermore, this embodiment dynamically adjusts the allocation and collaboration of tasks according to the network status and device resource usage, ensuring maximum utilization of resources, avoiding resource waste or performance bottlenecks caused by uneven resource allocation, and also reducing the overall energy consumption of the system.

[0145] Furthermore, the distributed architecture and collaborative reasoning mechanism make the system more tolerant to the failure of a single device. Even if a device fails, other devices can continue to provide services, thus ensuring the stability and reliability of the system.

[0146] Furthermore, this embodiment not only promotes the innovation of video surveillance technology, but also provides strong support for the upgrading of related industries. By optimizing the video surveillance process and improving monitoring efficiency and accuracy, the present invention has injected new vitality into the development of smart cities, smart security and other fields.

[0147] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. A video surveillance method based on distributed edge collaboration, characterized in that: The following steps are involved: Divide the video surveillance task into pedestrian tracking service and pedestrian re-identification service; Deploy the pedestrian tracking service and pedestrian re-identification service on different edge devices to achieve collaborative reasoning between heterogeneous edge devices, and use the Apache Kafka communication protocol to achieve two-way communication between the video analysis service and the pedestrian re-identification service; The acquired real-time video stream signal is pushed to the edge device cluster, and the pedestrians entering the monitoring area are first extracted to generate pedestrian features, which are then sent to the shared server for storage. The pedestrian tracking service detects and tracks pedestrians in the monitoring area and generates pedestrian feature vectors; The pedestrian re-identification service calculates the similarity between the received pedestrian feature vector and the pedestrian features stored in the shared server to determine the pedestrian's identity; Dynamically schedule edge devices running video analysis services and person re-identification services based on the current computing resource usage of heterogeneous edge devices; Among them, the shared server regularly synchronizes data with the edge devices running the pedestrian re-identification service.

2. The method according to claim 1, characterized in that Also includes: Feature extraction service, which performs the first feature extraction on pedestrians entering the monitoring area, including: Assume that the video stream input frame is I, with a size of H×W, the convolution kernel is K, with a size of m×n, that is, height m and width n, and the output feature map is O, with a size of H ′ ×W ′ , it is expressed as the following formula: Among them, O(i, j) represents the pixel value at position (i, j) in the output feature map, I(i+u, j+v) represents the pixel value at position (i+u, j+v) in the input frame, K(u, v) represents the weight of the convolution kernel at position (u, v), m and n represent the height and width of the convolution kernel, and H ′ and W ′ Represents the width and height of the output feature map.

3. The method according to claim 1, characterized in that: The pedestrian tracking service implements target tracking by tracking high-confidence and low-confidence detection targets, including: Input definition: High confidence detection set D H : The set of detected targets that meet the confidence threshold θ; Low confidence detection set D L : The set of detected targets below the confidence threshold θ; Among them, the confidence value is provided by the target detection algorithm.

4. The method according to claim 3, characterized in that: When tracking pedestrians, it is achieved through high confidence matching and low confidence matching, including: High confidence matching: Use high confidence detection set D H Perform Hungarian matching to match the detected target in the current frame with the tracked target in the previous frame, which can be expressed as the following formula: C={c ij |c ij =1-IOU(T i ,D j )}, Among them, T i represents the bounding box of the current tracked target i, D j represents the bounding box of the detected target j, IOU(T i ,D j ) represents the target T i and detection D j Intersection over Union (IOU); By minimizing the cost matrix C, the best match of high confidence targets is achieved; Low confidence matching: For the remaining unmatched tracking targets and low confidence detection targets D L For matching, the same Hungarian algorithm is used, but only those targets that have not been matched are considered, expressed as the following formula: C ′ ={c i ′ j |c i ′ j =1-IOU(T i ′ ,D j ′ )}, Among them, T i ′ represents the tracking target that is not matched after the first round of matching, D j ′ Indicates a low confidence detection target.

5. The method according to claim 4, characterized in that Also performs updates and manages trajectories, including: Track update: For the detection targets and tracking targets that are successfully associated through the first round of high confidence matching and the second round of low confidence matching, the bounding box position or motion state information of the detection target is updated to the corresponding tracking track; New track initialization: For high-confidence detection targets that are not matched to any existing tracking targets in the current frame, identify them as new targets, initialize new tracking tracks, assign a unique track identifier to the target, and record its status information in the new track; Low-confidence target processing: For low-confidence detection targets that are not matched to any tracking targets, the preset strategy decides whether to ignore these targets or mark them as potential targets for continuous tracking; if continuous tracking is selected, temporary tracks are created for these potential targets, and attempts are made to match them in subsequent frames to verify whether they are real targets.

6. The method according to claim 5, characterized in that: When generating pedestrian feature vectors, the feature extraction formula is expressed as the following vector: f i =g(I i ), Among them, I i represents the detected pedestrian box, g(·) represents the feature extraction network, and f i It is the extracted 2048-dimensional pedestrian feature vector.

7. The method according to claim 5, characterized in that Also execute: Set up a trace lifecycle management mechanism, which is used to perform trace termination and aging processing steps, including: Unmatched frame count: A unmatched frame counter is maintained for each active track. When the track fails to match any detection target in a frame, the counter is incremented by one; otherwise, if the match is successful, the counter is reset to zero. Aging threshold judgment: a threshold of unmatched frames is preset. When the unmatched frame counter of a certain track reaches or exceeds the threshold, the track is marked as an aged track. Aging processing: For entries marked as aging tracks, perform one or more of the following aging processing measures: Reduce the priority or weight of the track in subsequent tracking and decision-making processes; move the track to a dedicated aging track storage area; send an aging notification to the system administrator or related applications; Among them, the retention period or conditions of the aging tracks are set. When these conditions are met, the aging tracks are deleted to completely release system resources. After deleting the aging tracks, the resource allocation of the remaining active tracks is dynamically adjusted to ensure that the tracking efficiency and accuracy are not affected, and to prepare for the initialization and management of new tracks.

8. The method according to any one of claims 1 to 7, characterized in that: In the person re-identification service, the cosine similarity calculation is performed between the received pedestrian feature vector and the features synchronized with the shared server data, which is expressed as the following formula: Among them, f i ·f j Represents the vector f i and vector f j The dot product of i ‖ and ||f j || represents vector f i and vector f j The norm of , which is the length of the vector.

9. The method according to any one of claims 1 to 7, characterized in that: The edge device for dynamically scheduling the video analysis service and the pedestrian re-identification service to run includes: Periodically collect and analyze the resource usage of each edge device, as well as the type of services currently running and the service load; According to the collected resource usage data, a resource usage threshold is set. When the resource usage rate of a certain device exceeds the preset threshold, while the resource usage rate of the same type of another device is far below the average level, it is determined that there is load imbalance; Based on the load imbalance determination result, select the task type and target device to be migrated to maximize resource utilization efficiency and load balance; Among them, containerization technology is used to start and run the migration task on the target device.

10. The method according to claim 9, characterized in that Also includes: During the task migration process, ensure data synchronization between the source device and the target device; For tasks that require collaborative processing in pedestrian re-identification services or pedestrian tracking services, feature extraction and result sharing between devices are supported. That is, after device C processes part of the task, it returns the intermediate results or final results to device B, or directly uploads them to the central server. After task migration and reallocation are completed, the resource usage and system performance of each device are continuously monitored; if a new load imbalance or resource bottleneck is detected, a new round of load assessment and task migration decisions will be automatically triggered, or some migration tasks will be rolled back to achieve optimal resource utilization and continuous and stable operation of the system.

Images