Real-time figure tracking system based on multiple cameras

Abstract

The invention relates to the technical field of monitoring, in particular to a real-time figure tracking system based on multiple cameras. Comprising an edge end, a server and a client. The edge end is used for executing a calculation task according to the camera; the server receives the video stream and the feature data based on a transmission protocol, and processes a data request; the client side is used for displaying a final result of processing in real time through a webpage interface, and the feature value matching module is used for carrying out similarity calculation on feature vectors extracted by the edge end or the server side and historical features in a database and judging whether the feature vectors are the same person or not. According to the real-time figure tracking system based on the multiple cameras provided by the invention, the technologies of target detection, feature extraction and efficient matching are fused, so that the system has remarkable effects in the aspects of precision, real-time performance and resource efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of monitoring, in particular to a real-time character tracking system based on multiple cameras. BACKGROUND

[0002] Smart city and intelligent security are developing rapidly, and with them comes the need for multi-camera coordinated character tracking technology in the digital transformation of cities. According to market research agency Statista, the global intelligent security market is expected to exceed $80 billion in 2025, with a compound annual growth rate of 12.4%, of which the application of cross-camera target tracking technology accounts for more than 30%. In many scenarios, the demand for multi-camera character tracking has emerged, such as the following:

[0003] Public safety is a typical scenario, such as in airports, stations, and large venues, where traditional single-camera monitoring can only capture a fixed angle of view. This will result in a blind spot, and once there is an abnormal behavior in a crowded scene (such as abandoned items or people gathering), the traditional camera will be powerless. However, a multi-camera system can track the movement path of the target by coordinating across different angles of view, and combined with spatiotemporal data analysis, it can quickly locate abnormal targets and generate alarm information if necessary.

[0004] Commercial retail or consumer analysis is also a classic scenario: the retail industry needs to analyze customer behavior, and this demand for refinement has given rise to multi-camera tracking technology, which has also driven the deployment of multi-camera tracking technology. When we deploy cross-regional camera coordination, the system can track customer flow, dwell time, and hotspots of interest, which can provide data support for store layout optimization and precise marketing. According to a report by IBM's Business Value Institute, retailers who use user analysis have an average conversion rate increase of 22% and customer satisfaction improvement of 17%.

[0005] Traffic management and smart travel: In the context of urban transportation hubs, deploying a multi-camera system can track pedestrians in real time and identify flow trends to help alleviate congestion areas. For example, in a subway station, after obtaining camera footage, the system can analyze peak passenger flow times and personnel distribution to dynamically adjust the number of turnstiles to reduce the risk of stampedes. In addition, it can also use facial recognition technology to locate key personnel such as missing children.

[0006] Industrial production and personnel safety management: In factory and construction site scenarios, multi-camera tracking technology can monitor personnel activity in real time, alert strangers or employees who enter dangerous areas, and detect behaviors such as not wearing safety equipment and provide early warnings.

[0007] Traditional human recognition techniques need to rely on feature extraction methods. These methods are mostly manually designed. Early studies focused on features such as color histograms, histograms of oriented gradients (HOG), and local binary patterns (LBP), and iterated on these features to achieve target detection and tracking through template matching or classifiers such as SVM and AdaBoost. For example, Dalal et al. proposed a method that combines HOG features with linear SVM, which achieved good results in pedestrian detection tasks, but the feature expression ability was not perfect, and was limited by the manual design rules. It was not good at adapting to changes in light and posture, and was also not good at adapting to occlusion scenarios. In cross-camera scenarios, traditional methods usually use color-based checks or texture similarity measures. Common methods include histogram intersection kernel and chi-square distance. However, due to differences in camera angles, resolution, and environmental interference, feature matching can fail, making it difficult to achieve stable cross-camera human tracking.

[0008] In recent years, with the rise of convolutional neural networks (CNN), the mainstream method of human recognition technology has gradually become a modified version based on deep learning. First, there was a breakthrough in target detection, with algorithms such as Faster R-CNN and YOLO series emerging. These algorithms can achieve significant improvements in detection accuracy through end-to-end training, while also achieving a balance with speed. Among them, the YOLO series has become the preferred solution for target detection due to its simple design and efficient detection performance. In the task of pedestrian re-identification (ReID), networks such as OSNet use mechanisms such as multi-scale feature fusion to significantly improve the robustness of feature expression.

[0009] Compared with traditional methods, deep learning models can automatically learn discriminative high-level semantic features. As a result, the accuracy of recognition in complex scenarios such as occlusion and low light has been greatly improved. Existing related research is mostly focused on centralized processing architectures, which have a large number of parameters and require high computational resources. These issues limit the implementation of real-time applications on edge devices, and the efficiency and accuracy of cross-camera feature matching still need to be further optimized.

[0010] With the development of technology, the multi-camera tracking system has been established. It is known that under the condition of limited resources, how to fully combine the advantages of edge computing to achieve efficient feature extraction and matching is still one of the current research difficulties.

[0011] To this end, a multi-camera-based real-time person tracking system is designed to provide a technical solution to the above technical problems. SUMMARY

[0012] Therefore, it is necessary to provide a multi-camera-based real-time person tracking system to solve the technical problems in the background art.

[0013] To solve the above technical problems, the present application adopts the following technical solutions:

[0014] The multi-camera-based real-time person tracking system comprises an edge terminal, a server and a client.

[0015] The edge terminal is used to perform computing tasks according to the camera.

[0016] The server receives video streams and feature data based on a transmission protocol and processes data requests.

[0017] The client is used to display the final result of processing in real time through a web interface.

[0018] As a preferred embodiment of the multi-camera-based real-time person tracking system provided by the present application, the camera device running mode of the edge terminal is as follows:

[0019] A. Complete target detection and feature extraction locally, and upload after generating a relatively compact feature vector.

[0020] B. Directly transmit the original video stream without processing.

[0021] As a preferred embodiment of the multi-camera-based real-time person tracking system provided by the present application, the server comprises a matching module, an inference module and an ID management database.

[0022] The matching module is used to send feature data to the FAISS library for fast matching.

[0023] The inference module is used to analyze video streams frame by frame.

[0024] The ID management database is used to record the appearance time of each identity and the associated camera, and to dynamically clean up records that have not been active for a long time.

[0025] As a preferred embodiment of the multi-camera-based real-time person tracking system provided by the present application, the working steps of the client are as follows:

[0026] Real-time video panel receives MJPEG stream from server, picture superposition detection frame and identity label, delay control within 200 milliseconds;

[0027] Statistical dashboard integrates each camera working status and system load, historical data trend chart helps administrator master people flow change rule;

[0028] ID search interface supports uploading local pictures, system returns matching identity information and activity record, query result is automatically cleared after a period of time to ensure privacy security.

[0029] As a preferred embodiment of the multi-camera based real-time character tracking system provided by the application, the edge device periodically packs and sends the feature vector and metadata, the server quickly completes the matching work after receiving, updates the database, and the client synchronously refreshes the displayed content.

[0030] As a preferred embodiment of the multi-camera based real-time character tracking system provided by the application, the inference module includes an image preprocessing module, a target detection module and a feature extraction module

[0031] The image preprocessing module is used to convert the original picture into a normalized format that can be processed by the model.

[0032] The target detection module is used to locate the area where the pedestrian is located according to the YOLOv5s model.

[0033] The feature extraction module is used to convert the pedestrian image into a discriminative feature vector according to the OSNet_x0.25 network.

[0034] As a preferred embodiment of the multi-camera based real-time character tracking system provided by the application, it further includes a feature value matching module, which is used to calculate the similarity of the feature vectors extracted by the edge or server and the historical features in the database to determine whether they are the same person.

[0035] As a preferred embodiment of the multi-camera based real-time character tracking system provided by the application, when the new feature vector reaches the server, the matching operation is performed during similarity calculation:

[0036] L2 distance is used as a similarity measure to search the K closest candidate features to the query feature in the feature library.

[0037] Convert L2 distance to cosine similarity, the expression is as follows:

[0038]

[0039] The numerical range is between -1, 1-1, 1, and the closer the value is to 1, the more similar the features are.

[0040] It can be seen without doubt that the above technical solutions of the present application can certainly solve the technical problems to be solved by the present application.

[0041] At the same time, through the above technical solutions, the present application at least has the following beneficial effects:

[0042] The real-time character tracking system based on multiple cameras provided by the present application combines target detection, feature extraction and efficient matching, and achieves remarkable results in terms of accuracy, real-time performance and resource efficiency, and can successfully realize cross-camera character trajectory continuous tracking and abnormal behavior warning function in campus monitoring actual application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0044] Figure 1 It is a model symmetry quantization diagram of the present application;

[0045] Figure 2 It is a normal operation scene diagram of the present application;

[0046] Figure 3 It is a user picture query scene diagram of the present application;

[0047] Figure 4 It is a test effect diagram of the present application;

[0048] Figure 5 It is a query function display diagram of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0050] In order to make those skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings.

[0051] It should be noted that the embodiments in the present application and the features and technical solutions in the embodiments can be combined with each other without conflict.

[0052] It should be noted that similar reference numerals and letters refer to like items throughout the accompanying drawings, and once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings.

[0053] Reference Figures 1-5 , a multi-camera-based real-time person tracking system.

[0054] I. Hardware and software selection

[0055] 1. Pedestrian Re-Identification (REID) technology

[0056] Pedestrian Re-Identification (ReID) technology belongs to a very important research direction category in the field of computer vision. Its most core goal is to match the features of pedestrians in cross-camera scenes, so as to achieve continuous tracking and accurate identification of the identity of a specific target. In the traditional monitoring system, the range covered by the camera is usually limited, and the view angle is fixed. When the target to be tracked moves back and forth between multiple cameras without overlapping view, it is very difficult to achieve continuous tracking. ReID technology extracts robust features from pedestrian images and establishes cross-camera identity association, which can compensate for the limitations of single-camera monitoring and become a key technical support in multi-camera collaborative tracking systems.

[0057] Early pedestrian Re-Identification technology is not perfect, and mostly relies on manually designed feature descriptors. These feature descriptors are designed manually, such as color histogram, local binary pattern (LBP), and histogram of oriented gradients (HOG). These methods can only compare and measure the similarity by quantifying the local texture, color distribution, and other shallow features of the appearance of pedestrians, but they are not universal in complex scenes. They are easily affected by changes in light, differences in posture, and changes in viewing angle, and have weak generalization ability. In recent years, with the development of deep learning, ReID methods based on convolutional neural networks (CNN) have gradually become mainstream. This method well compensates for the shortcomings of traditional methods. Through an end-to-end feature learning framework, the model can automatically capture high-level semantic information in pedestrian images, and combine with metric learning to optimize the feature space distribution, significantly improving the ability of cross-camera systems and the identity discrimination ability in multiple scenes.

[0058] In the design of the system, YOLOv5 is used to realize pedestrian detection, which combines with the OSNet_x0_25 lightweight network to extract the pedestrian feature vector. And OSNet effectively fuses semantic information at different abstraction levels by constructing full-scale feature representation. At this time, its compact model structure meets the real-time inference requirements of the single-chip microcomputer end, while maintaining adaptability to clothing changes and perspective differences. The server end can realize cross-camera identity matching through feature vector similarity calculation. At this time, this distributed processing architecture can reduce the data transmission bandwidth while performing centralized feature comparison, thus ensuring the scalability of the system. This technical route fully embodies the evolution trend of pedestrian re-identification algorithm from theoretical research to engineering landing, providing a reliable technical implementation scheme for multi-camera cooperative tracking.

[0059] 2. yolov5

[0060] The implementation of pedestrian re-identification (ReID) technology now has multiple steps, which relies on efficient and accurate pedestrian detection as a pre-step. The YOLO (You Only Look Once) series algorithm is a very representative method in the field of target detection, with its unique single-stage detection framework and advantages in real-time, it has been widely used in person tracking systems. The most core principle of YOLO is to convert the target detection task into a regression problem, specifically by dividing the input image into grid cells, and then letting each cell directly predict the target bounding box coordinates, confidence and class probability. This end-to-end processing form completely eliminates the time-consuming candidate region generation step in traditional two-stage detection methods, greatly improving the detection speed, which can meet the related needs of processing real-time video streams. In the specific scene of pedestrian detection, YOLO uses convolutional neural networks to extract multi-scale features, and combines with the non-maximum suppression (NMS) algorithm to filter overlapping boxes, which can quickly locate pedestrian targets in the image and output accurate bounding box coordinates, providing strong support for subsequent feature extraction and identity matching.

[0061] Compared with other object detection models, YOLO has obvious advantages in the balance between speed and accuracy. Its single-stage detection architecture, which takes advantage of global context information, can effectively avoid interference from complex backgrounds. Moreover, it also enhances the detection ability of small-scale pedestrians by setting multi-scale prediction layers. In the ReID system, the real-time and accuracy of the detection link directly affect the overall tracking performance. Specifically, on the one hand, the detection speed must match the camera frame rate, otherwise data accumulation will occur; on the other hand, the accuracy of the detection box must ensure that the pedestrian image is complete, and prevent the problem of feature extraction distortion caused by truncation or background mixing. YOLO's high frame rate processing capability and stable detection quality make it an ideal choice for the front-end detection link of the ReID system.

[0062] YOLOv5, as an important iterative version of the YOLO series, has implemented systematic optimization measures in model structure, training strategy, and deployment efficiency. Compared with earlier versions, YOLOv5 introduces the Focus structure to replace the traditional down-sampling layer, which can reduce computational complexity and retain more spatial information through slicing operations. It uses the CSPNet (Cross Stage Partial Network) module to reconstruct the backbone network, reducing parameter redundancy through feature reuse and stage-by-stage fusion. At the same time, its neck network integrates PANet (Path Aggregation Network) to achieve multi-scale feature interaction, enhancing the detection robustness for pedestrians of different scales. During the training phase, YOLOv5 relies on the Mosaic data augmentation strategy to improve the model's generalization ability, and combines adaptive anchor box calculation and GIoU loss function optimization to further improve the positioning accuracy in dense scenes. These improvements enable YOLOv5 to maintain high detection speed while significantly improving the average precision (mAP) compared to previous models, especially in complex conditions such as multi-camera scenarios with varying lighting and different angles.

[0063] Among the many sub-versions contained in YOLOv5, such as v5n, v5s, v5m, v5l, and v5x, v5s has become a popular choice in edge computing scenarios due to its lightweight design. This version compresses the model width (number of channels) and depth (number of layers), successfully controlling the parameter size to about 7.2M and the model volume to about 14MB, significantly reducing the demand for hardware computing power. Despite the reduction in model size, v5s still relies on improved feature fusion mechanisms and efficient network structure design to maintain detection accuracy similar to larger versions.

[0064] For a multi-camera-based person tracking system, the lightweight nature of v5s allows it to achieve real-time inference on embedded devices such as Raspberry Pi and Jetson Nano, and can further optimize inference efficiency with the TensorRT acceleration engine. This low-latency and low-power feature not only meets the resource constraints of single-chip microcomputers in distributed deployment, but also leaves enough computing resources for the server side to perform feature comparison and trajectory fusion operations, supporting a complete end-to-end real-time tracking link. Therefore, the efficient deployment capability of YOLOv5s on embedded devices and its seamless integration with ReID feature extraction networks are important technical foundations for multi-camera person tracking systems.

[0065] 3. OSnet

[0066] In the pedestrian re-identification (ReID) system, the performance of the feature extraction network directly affects the accuracy of cross-camera identity matching. For example, OSNet (Omni-Scale Network) is a lightweight deep neural network designed specifically for ReID tasks. It can break through the limitations of traditional models in multi-scale feature fusion through full-scale feature learning mechanisms.

[0067] The core innovation lies in the construction of a network structure that dynamically aggregates multi-scale features, which can adaptively capture full-scale information from local details (such as clothing texture, accessories) to global semantics (such as body shape, posture) in pedestrian images. Compared with general backbone networks such as ResNet and MobileNet, OSNet has many enhancements. By introducing a unified full-scale residual block (OSblock), OSNet realizes multi-granularity features in a single network. This optimization is very effective and significantly improves the ability of feature expression. In the presence of perspective changes, occlusions, and low resolution, the robustness of complex scenes is improved.

[0068] The network architecture of OSNet focuses on the construction of "full-scale" feature fusion. Inside each OSblock, a multi-branch structure is used to extract multi-scale features in parallel with convolution layers of different expansion rates. A lightweight channel attention mechanism is also introduced to dynamically adjust the weights of each branch, allowing the network to automatically focus on the most relevant feature scale for the current input. This design avoids the tedious operation of manually presetting scale groups and effectively reduces computational redundancy through feature reuse mechanisms. In addition, OSNet uses a combination of depthwise separable convolution and point convolution to replace standard convolution, greatly reducing the number of parameters while ensuring the ability of feature expression. For example, the OSNet_x0_25 version used in this system has a model parameter of only about 0.22M and a volume of less than 1MB. However, it can achieve recognition accuracy comparable to large models on mainstream ReID datasets such as Market-1501 and DukeMTMC-reID, fully embodying the design concept of "small model with great ability".

[0069] OSNet_x0_25 is selected as the feature extraction network, mainly because it has many advantages in edge computing scenarios. First, its full-scale feature fusion mechanism can better alleviate the problem of detail loss caused by excessive compression in traditional lightweight models, even in low-resolution pedestrian images, it can still maintain relatively high feature discrimination. Second, the model uses dynamic weight allocation to enhance adaptability to changes in perspective and local occlusion. When part of the pedestrian's body is occluded, the network can automatically enhance the feature weight of the visible area to ensure that the identity information can be expressed stably. Third, it has extremely low computational complexity, and the time-consuming of a single image on an embedded device is only about 10 ms, which matches the efficient detection of YOLOv5s in time sequence, avoiding the bottleneck of system throughput caused by delay in feature extraction. In actual deployment, OSNet_x0_25 can be directly loaded into the AI inference framework of the single-chip microcomputer, such as TensorFlow Lite, NCNN, etc. It and YOLOv5s together form a "detection-feature extraction" localized processing pipeline, only the 512-dimensional feature vector needs to be uploaded to the server, which can greatly reduce the occupation of network bandwidth. This design not only meets the real-time requirements of distributed systems, but also ensures the accuracy of cross-camera identity association through high-discriminative features, providing a lightweight solution for multi-node collaborative person tracking.

[0070] 4. Edge computing

[0071] During the engineering practice of the multi-camera person tracking system, edge computing technology was introduced, which effectively resolved a series of problems such as poor real-time performance, excessive bandwidth load, and privacy leakage risk encountered by traditional centralized processing architecture. Edge computing sinks the computing task to the terminal device close to the data source, such as the single-chip microcomputer of the camera, so as to achieve local processing and preliminary analysis of data, and only the key features or decision results are transmitted to the cloud server, thus building an efficient computing mode of 'end-edge-cloud' cooperation. In the specific scenario of person tracking, the core value of this technical route is to use the distributed computing power of edge nodes to transfer the high-computing-density target detection and feature extraction tasks from the server to the terminal, which significantly reduces the delay caused by network transmission and reduces the computing pressure of the center node, while improving the system's tolerance to network fluctuations and device heterogeneity.

[0072] In the system design stage of the present application, edge computing is specifically implemented by embedded hardware (such as Raspberry Pi, Jetson Nano and other such devices) equipped with lightweight models. The strategy of deploying YOLOv5s and OSNet_x0_25 together fully considers the power constraints of edge devices: YOLOv5s achieves a volume of 14MB by compressing the model width and depth, and with the inference acceleration function of the TensorRT engine, it can achieve real-time detection speed of more than 30FPS on embedded GPU; while OSNet_x0_25 relies on the full-scale dynamic feature fusion mechanism to extract high-discriminative features with only 0.22M parameters, its computational complexity is reduced by about 95% compared to traditional ResNet-50, but it still maintains more than 80% Rank-1 accuracy on datasets such as Market-1501. This collaborative optimization method of making the model lightweight and accelerating through hardware enables the single-chip microcomputer to complete the whole process of processing from raw video stream to feature vector locally, only needing to upload 512-dimensional feature data (the single transmission volume is about 2KB), which reduces the bandwidth consumption by 99.6% compared to transmitting raw video stream (1080P frame is about 500KB), while avoiding privacy risks in public network transmission.

[0073] Once edge computing is introduced, it gives the system better scalability and robustness. Specifically, by dispersing the computing load to each camera node, the server side only needs to focus on tasks such as feature matching and trajectory fusion, which are relatively lightweight. In this way, the system can rely on the increase in the number of edge nodes to achieve linear expansion of the monitoring range, and in this process, the hardware of the central server does not need to be upgraded simultaneously. In addition, the edge node itself has the ability of local caching and breakpoint resume, even in the case of network interruption, it can still maintain offline inference for a short time, and after the network is restored, the feature data is supplemented, which undoubtedly effectively improves the stability of the system in weak network environment. This design method of distributed architecture is in line with the needs of smart city, industrial inspection and other large-scale monitoring scenarios, on the one hand, and on the other hand, it successfully reduces the risk of privacy leakage according to the principle of data minimization, thereby providing strong technical support for the compliant landing of person tracking technology.

[0074] 5. Domestic substitute chip and model

[0075] The system of the application runs on a domestic chip RK3588s, which has three NPU cores, and the developer is Rockchip, which developed the RKNN model format for the RK series AI chips produced by Rockchip, and the Rockchip npu development platform, which can run on RK series chips and fully utilize the performance of the NPU computing core.

[0076] NPU (Neural Networks Process Units) neural network processing unit, which is specially designed for neural networks, can quickly perform convolution, matrix multiplication and other most basic operation units in NN, and has lower power consumption. Under the same power consumption, the performance of NPU is higher than that of GPU. NPU is different from GPU acceleration, which is embodied in that the calculation results of each layer of neurons do not need to be output to the main memory, but are transmitted to the lower layer of neurons for further calculation according to the connection of the neural network, so that the operation performance and power consumption are greatly improved.

[0077] Yolov5 and osnet are both.pth format models based on pytorch, and most of them are float32 floating point operations. RK3588s does not support the running of such models, and at the same time, gradient back propagation is not needed during model inference, so it does not need such high calculation precision. At this time, the model can be mapped to lower precision operation to reduce the amount of operation and improve the operation speed. The process of converting the model from high precision operation to low precision operation is called model quantization. When converting general models such as.pth to RKNN models, pruning, quantization and other optimization operations will be performed to improve the running efficiency of the model.

[0078] Model ( Figure 1 ) calculation expression is as follows:

[0079]

[0080] II. System construction

[0081] In the scenarios of smart campus and public security, real-time person tracking system needs to meet multi-dimensional comprehensive requirements. The system needs to build a distributed computing architecture to realize the collaborative operation of edge and center servers, complete real-time processing of front-end sensing data through domestic edge devices, reduce the load of the center server and protect data privacy. In terms of performance, it needs to meet the basic requirements of real-time video stream processing, that is, to maintain an inference speed of not less than 5FPS under the resolution of 640x640, to ensure the timeliness of target detection and feature extraction. The system needs to have continuous tracking ability across camera perspectives, support at least 8 concurrent video stream processing, and the ID matching accuracy needs to be higher than 80% to maintain tracking continuity. In terms of robustness, it needs to adapt to complex light changes, local occlusion and other realistic interference factors, and maintain stable operation in typical scenes such as school corridors and playgrounds. The scalability requirement is reflected in the support of flexible access of different types of camera devices, including smart cameras with local inference capability and traditional devices with only video transmission function. The security needs to meet the specifications of data encryption transmission and feature vector anonymization storage, and meet the technical implementation requirements of national privacy protection law. The economic constraint requires that the edge hardware cost be controlled within 1000 yuan per node, ensuring the feasibility of large-scale deployment.

[0082] 2.1 System framework

[0083] The overall architecture of the system mainly revolves around the collaborative work of multiple cameras. Camera devices perform key computing tasks on the edge, while video streams or feature data are transmitted to the server side through different protocols. The server acts as the core hub, handling data requests from multiple sources, including feature matching, video stream analysis, and responding to user queries. The client displays the final results in real time through a web interface, allowing users to view surveillance footage or search for specific person's movement trajectory. The camera devices on the edge have two operating modes: one mode performs target detection and feature extraction locally, and uploads the compact feature vectors; the other mode directly transmits the raw, unprocessed video stream to adapt to various demands under different hardware conditions.

[0084] The request routing module on the server side is responsible for distributing various types of requests. The feature data is sent to the FAISS library for fast matching, and the video stream is sent to the inference module for frame-by-frame analysis. The FAISS library uses L2 distance to achieve efficient search, and the matching results are combined with a time decay strategy to optimize the accuracy of identity association. The ID management database records the appearance time of each identity and the associated camera, and dynamically cleans up records that have not been active for a long time to ensure system efficiency. When the user uploads a picture, the server triggers a complete inference process, including detection, feature extraction, and matching, which can be completed in a few seconds. The final result is sent back to the front-end interface through an encrypted channel.

[0085] The real-time video panel on the client side continuously receives the MJPEG stream from the server, with detection boxes and identity labels overlaid on the screen, and the delay is controlled within 200 milliseconds. The dashboard integrates the working status of each camera and the system load, and the historical data trend chart helps administrators understand the rules of people flow changes. The ID search interface supports uploading local pictures, and the system returns matching identity information and activity records in multiple cameras. The query results are automatically cleared after a short period of caching to ensure privacy security.

[0086] In normal operation scenarios, the edge device periodically sends feature vectors and metadata, the server quickly completes matching and updates the database, and the client synchronously refreshes the displayed content. When the user submits a picture query request, the server starts an independent inference thread to avoid interfering with real-time processing. In pure video stream mode, the server handles all computing loads, including video decoding, target detection, and feature extraction. This setup is suitable for older or resource-constrained edge devices. The switching between different modes is dynamically adjusted based on network bandwidth and device computing power, ensuring stable operation in complex environments.

[0087] The core innovation of the system is that the edge and the cloud can flexibly cooperate. When the characteristic level data is transmitted, the occupation of the bandwidth can be greatly reduced. The FAISS library has an efficient retrieval mechanism, which can effectively guarantee the real-time response. The system supports multiple protocols, which enhances the compatibility of the device. Whether it is uploading features by HTTP or transmitting video streams by RTSP, the system can achieve seamless connection and proper handling. The security mechanism is throughout the entire architecture. The data is encrypted during transmission, and regular cleaning strategies are implemented, which effectively reduces the risk of privacy leakage. In actual deployment, the system has verified its reliability and practicality in campus and shopping mall scenarios, providing a scalable technical solution for intelligent security.

[0088] 2.2 Data stream analysis

[0089] Reference Figure 2 and Figure 3 , Figure 2 For normal operation scenarios (edge inference), Figure 3 For user picture query scenarios in data stream, refer to Table 1;

[0090] Table 1: Application scenarios

[0091]

[0092] 2.3 Inference calculation process

[0093] 2.3.1 Image preprocessing

[0094] The image preprocessing stage aims to convert the original picture into a standardized format that the model can process. The input picture is first scaled to a fixed size (640x640 pixels) through a bilinear interpolation algorithm. This size is based on the default input design of YOLOv5s, balancing detection accuracy and computational efficiency. Then, the pixel values are normalized from the integer range 0, 255 0, 255 to the floating-point range 0, 10, 1 to avoid numerical overflow and speed up model convergence.

[0095]

[0096] Finally, the image data format is converted from height x width x channel (HWC) to channel x height x width (CHW), and a batch dimension is added to form a four-dimensional tensor (1x3x640x640), which meets the input requirements of the PyTorch model.

[0097] 2.3.2 Object detection

[0098] In the target detection phase, the YOLOv5s model is used to locate the area where the pedestrian is located. The CSPDarknet backbone network in the model uses cross-stage partial connection, that is, the CSP structure, which divides the input features into two parts, processes them respectively, and then combines them together, thus greatly reducing the computational redundancy. The PANet feature pyramid further fuses the shallow high-resolution features and deep high-semantic features, which improves the model's detection ability for different scale targets. The bounding box coordinates, confidence and class probability of the detection output are processed by the non-maximum suppression (NMS) post-processing operation. In this process, the prediction results with a confidence higher than 0.25 and an intersection over union (IoU) lower than 0.45 are retained, and finally the cropped person region image is output.

[0099] 2.3.3 Feature extraction

[0100] In the feature extraction phase, the OSNet_x0.25 network is used to convert the pedestrian image into a discriminative feature vector. For the cropped region, it is first scaled to 256x128 pixels, and standardized according to the mean (0.485, 0.456, 0.406, 0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225, 0.229, 0.224, 0.225) specified by the ImageNet dataset, which aims to eliminate the effects of light differences. The OSNet network extracts features of different scales through a multi-branch convolution structure. The 1x1 convolution captures local details such as clothing texture, while the 7x7 convolution captures global contour information and dynamically fuses the outputs of each branch through adaptive weights. At the end of the network, there is a generalized mean pooling (GeM) operation that aggregates spatial features, highlights prominent areas such as backpacks and hats, and suppresses background interference, ultimately generating a 512-dimensional normalized feature vector. After L2 normalization, the length of the vector is 1, so cosine similarity can be used directly for cross-camera matching.

[0101] 2.4 Feature value matching process

[0102] 2.4.1 Feature library construction

[0103] Feature matching plays a key role in realizing cross-camera person identity association. It mainly calculates the similarity between the feature vector extracted from the edge or server and the historical features in the database to determine whether it is the same person. This process includes feature library construction, similarity calculation, and ID decision management. The specific process is as follows:

[0104] First, the feature library construction work needs to be done. When the system is initialized, the FAISS library is empty.

[0105] When the first feature vector appears, such as the one uploaded by the edge device or generated on the server after relevant processing, the feature import operation needs to be performed. Specifically, the 512-dimensional normalized feature vector needs to be stored in the FAISS index, and a unique ID needs to be assigned to it, such as ID = 1. At the same time, metadata record work needs to be done, that is, a metadata entry corresponding to the ID needs to be created in the database, which contains the first appearance timestamp, last active timestamp, and associated camera list information. As the system runs continuously, the feature library will be dynamically expanded, and each new feature will be processed according to the process described above.

[0106] 2.4.2 Similarity calculation

[0107] When a new feature vector (query feature) arrives at the server, the following matching operations are performed: FAISS index search: use L2 distance (Euclidean distance) as the similarity measure to search for the K closest candidate features (default K = 1) in the feature library. Similarity conversion: convert L2 distance to cosine similarity. Since the feature vector has been normalized (length 1), the conversion formula is:

[0108]

[0109] The value range is between -1, 1-1, 1, and the closer the value is to 1, the more similar the features are.

[0110] 2.4.3 ID decision management

[0111] Identity determination based on similarity results: if similarity >= threshold (default 0.8), determine as the same ID, update the metadata of this ID: refresh the last_seen field to the current timestamp; if the current camera ID is not recorded in the cameras list, add it. If similarity < threshold, consider it as a new person, assign a globally unique ID (such as the current maximum ID + 1), and insert the feature vector into the FAISS library.

[0112] III. System implementation and testing

[0113] 3.1 Experimental environment

[0114] The edge device selected is Orange Pi 5Pro (equipped with RK3588S and 8GB of memory), on which the YOLOv5s and OSNet_x0.25 quantization models are deployed.

[0115] The server is equipped with an Intel Core i7-10700K processor, accompanied by an RTX 3060 GPU, with 32GB of memory, and uses Ubuntu 20.04 as the operating system.

[0116] The camera configuration is as follows: it has 3 analog cameras with a resolution of 1080P and a frame rate of 30FPS. With the help of the camera1.py program, a video stream for testing can be generated.

[0117] In terms of data sets, there is the COCO data set, which can be used to train yolov5. In addition, there is the Market-1501 data set, which contains up to 32,668 pedestrian images and 1,501 different IDs, which can be used when fine-tuning the OSNet model.

[0118] Test data:

[0119] Scenario 1: Static background (indoor corridor), simulate system working conditions under low occlusion and uniform lighting conditions;

[0120] Scenario 2: Dynamic background (mall entrance), contains lighting changes and partial occlusion, test dynamic effect;

[0121] Scenario 3: High-density crowd (subway platform), test the anti-interference ability and robustness of the system.

[0122] 3.2 Evaluation indicators

[0123] ID matching accuracy (Rank-1 Accuracy): the proportion of correctly matched query features;

[0124] End-to-End Latency: The time cost of the whole process from frame input to ID update.

[0125] Resource Consumption: Memory usage on edge and GPU utilization on server.

[0126] Cross-camera Consistency: The continuity of the same ID's trajectory in different cameras.

[0127] 3.3 System Test

[0128] 3.3.1 Performance Test

[0129] Based on the specific comparison in Table 2, the analysis is as follows:

[0130] The Rank-1 accuracy of this system is improved by 20% compared to a single model, mainly due to the multi-scale features of OSNet and the efficient retrieval of FAISS.

[0131] The host memory usage is only 35% of DeepSORT, making it suitable for resource-constrained devices.

[0132] The end-to-end latency is significantly optimized (48.2ms vs. 112.5ms), meeting the real-time requirements.

[0133] Table 2: Performance Test Comparison

[0134] Method Rank-1 accuracy Processing latency (ms) Edge memory footprint (MB) The system (edge computing) 86.7％ 48.2 520 Traditional hand-crafted features 30.4％ 29.5 1480 FairMOT 90.1％ 103.6 3249

[0135] 3.3.2 Scene Robustness Test

[0136] Based on the test in Table 3, we know that:

[0137] In static scenes, the system performs stably with a false match rate below 3%.

[0138] In high-density crowd scenes, the ID switching rate increases, and space-time constraints need to be optimized (such as limiting the ID switching distance between adjacent cameras).

[0139] Table 3: Scene Robustness Test

[0140] Scene ID switch rate (times / minute) False match rate Static background 0.3 2.1％ Dynamic background 1.8 6.7％ High-density crowd 4.5 12.3％

[0141] 3.3.3 Resource Consumption Test

[0142] Edge:

[0143] YOLOv5s peak memory usage: 380MB

[0144] OSNet single inference time: 150ms

[0145] Server:

[0146] Matching latency of FAISS index feature library: 22ms

[0147] GPU utilization rate of 6-way video stream parallel processing: 72%.

[0148] 3.4 Test visualization

[0149] Cross-camera trajectory: The movement path of ID = 205 between camera A (entrance) and camera B (exit) is recorded completely, reference Figure 4 ;

[0150] User search: After uploading the picture, the user successfully calculates the person ID in the system and knows where the person appears using this ID, reference Figure 5 .

[0151] IV. Conclusion

[0152] Focusing on the demand for real-time person tracking in multi-camera scenes, a lightweight system relying on edge computing and central database collaborative architecture was carefully designed and successfully implemented. In this system, by integrating target detection, feature extraction, and efficient matching technologies, significant results were achieved in terms of accuracy, real-time performance, and resource efficiency. The specific conclusions are as follows:

[0153] The two models YOLOv5s and OSNet_x0.25 were used for deployment, and finally the real-time inference effect was achieved on the edge device Orange Pi 5Pro.

[0154] Through related experiments, it can be seen that when running these models on the edge, the performance of the device itself can be fully utilized to achieve the goal of domestic substitution. The single-frame processing delay can be stably controlled within 48ms, which is 57% lower than the traditional centralized solution like DeepSORT. When tested on the Market-1501 test set, the Rank-1 accuracy of the system reached 86.7%, which was 8.3% higher than the baseline method, thus verifying the robustness of lightweight models in more complex scenarios.

[0155] The real-time feature retrieval and ID management module involved in FAISS can support the matching of 100,000-level feature vectors in milliseconds, and the average delay can reach 1.2 ms. By using the dynamic similarity threshold adjustment method combined with the time decay strategy, the cross-camera ID mismatch rate can be controlled within 2.1% in a static scene, and even in a high-density crowd scene, the mismatch rate can still be maintained at a level of 12.3%. Compared with the traditional method based on color histogram (the mismatch rate is higher than 25%), this method has obvious advantages.

[0156] The flexibility system of the multi-modal computing architecture can support both edge feature uploading and server-side full-process processing, which can adapt to various devices with different hardware conditions. When actually deploying, for the case of 3 parallel 1080P cameras, the GPU utilization of the server is only 72%, and the system can also give a quick response to sudden user queries (such as image retrieval), and the average query delay is less than 200 ms.

[0157] In the actual application scenario of campus monitoring, the system successfully realizes continuous tracking of cross-camera human trajectories and realizes the early warning function for abnormal behavior.

[0158] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details and limit the application to the specific embodiments described. Obviously, according to the content of the specification, many modifications and changes can be made. The specification selects and describes these embodiments in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and utilize the application. The application is limited by the claims and their entire scope and equivalents.

Claims

1. A multi-camera based real-time person tracking system, characterized in that, The edge end, the service end and the client are included; The edge end is used for performing a computing task according to a camera; The service end receives a video stream and feature data based on a transmission protocol and processes a data request; The client is used for displaying a final result of processing in real time through a webpage interface.

2. The multi-camera based real-time character tracking system of claim 1, wherein, The camera device of the edge end runs in the following modes: A. Completes target detection and feature extraction locally, and then uploads a relatively compact feature vector; B. Directly transmits an original video stream without processing.

3. The multi-camera based real-time character tracking system of claim 1, wherein, The service end includes a matching module, an inference module and an ID management database; The matching module is used for sending feature data to a FAISS library for fast matching; The inference module is used for analyzing a video stream frame by frame; The ID management database is used for recording an occurrence time of each identity and a camera associated with the identity, and dynamically cleaning records that have not been active for a long time.

4. The multi-camera based real-time character tracking system of claim 1, wherein, The working steps of the client are as follows: A real-time video panel receives an MJPEG stream transmitted by the service end, a picture is superimposed with a detection box and an identity label, and a delay is controlled within 200 milliseconds; A statistical dashboard integrates working states of each camera and system load, and a historical data trend chart helps an administrator master a people flow change rule; An ID search interface supports uploading a local picture, the system returns matching identity information and activity records, and query results are automatically cleaned after a period of time to ensure privacy security.

5. The multi-camera based real-time character tracking system according to any of claims 1-4, characterized in that, The edge end device periodically packages and sends feature vectors and metadata, the service end quickly completes matching work after receiving the feature vectors and metadata, updates a database, and the client synchronously refreshes displayed content.

6. The multi-camera based real-time character tracking system of claim 3, wherein, The inference module includes an image preprocessing module, a target detection module and a feature extraction module The image preprocessing module is used for converting an original picture into a standardized format that can be processed by a model; The target detection module is used for positioning a region where a pedestrian is located according to a YOLOv5s model; The feature extraction module is used for converting a pedestrian image into a feature vector with discriminative characteristics according to an OSNet_x0.25 network.

7. The multi-camera based real-time character tracking system of claim 1, wherein, A feature value matching module is further included, which is used for calculating a similarity between a feature vector extracted by the edge end or the server end and a historical feature in the database to determine whether the feature vector is of the same person.

8. The multi-camera based real-time character tracking system of claim 7, wherein, When a new feature vector arrives at the server, a matching operation is performed during similarity calculation: K candidate features closest to a query feature are searched in a feature library by taking L2 distance as a similarity measurement standard; L2 distance is converted into cosine similarity, and an expression is as follows: A numerical range is between -1 and 1, and a value closer to 1 indicates that features are more similar.

Images