End-cloud collaborative vehicle detection and tracking method, system, device, and storage medium

Abstract

The present invention relates to the field of smart transportation technology, and discloses an end-cloud collaborative vehicle detection and tracking method, system, device and storage medium. The method includes tracking the target vehicle in the target detection result of the current video frame according to the first model to determine the current first tracking result when the number of vehicles in the current video frame is less than or equal to the first preset threshold; the first model is deployed on the edge terminal of the end-cloud collaborative architecture; when the number of vehicles is greater than or equal to the second preset threshold, the current video frame is detected and tracked according to the second model to determine the current second tracking result; the computing resources occupied by the first model are less than the second model; the second model is deployed on the cloud; when the number of vehicles is greater than the first preset threshold and less than the second preset threshold, tracking is performed based on the most recently triggered working mode. The present invention can achieve dynamic resource adaptation and improve computing resource utilization by dynamically switching models that occupy different computing resources based on vehicle density.

Description

Technical Field

[0001] The present invention relates to the field of smart transportation technology, and in particular to a vehicle detection and tracking method, system, device, and storage medium for end-cloud collaboration. Background Art

[0002] With the acceleration of urbanization, smart transportation has become a key means of alleviating traffic congestion and improving management efficiency. Vehicle tracking technology, a core component of smart transportation, relies primarily on computer vision and artificial intelligence algorithms to accurately locate and continuously track vehicles using video data collected by cameras.

[0003] Traditional vehicle tracking technology mainly uses a fixed computing model and cannot dynamically adjust computing resources according to real-time traffic flow. As a result, equipment resources are not fully utilized during non-peak hours or on remote roads, and it is difficult to meet demand during peak hours.

[0004] Therefore, there is an urgent need for a detection and tracking method that can adapt to dynamically changing traffic flow and improve the utilization of computing resources. Summary of the Invention

[0005] In order to solve the above technical problems, in a first aspect, the present invention provides a vehicle detection and tracking method with end-cloud collaboration, comprising:

[0006] Determine a current video frame based on the real-time video stream of the target intersection, and detect the current video frame to determine a target detection result, wherein the target detection result includes a target vehicle and the number of vehicles in the current video frame;

[0007] When the number of vehicles is less than or equal to a first preset threshold, triggering a first working mode;

[0008] When the number of vehicles is greater than or equal to a second preset threshold, triggering a second operating mode;

[0009] When the number of vehicles is greater than a first preset threshold and less than a second preset threshold, a third working mode is triggered; the third working mode is the working mode that was triggered most recently;

[0010] In the first working mode, the target vehicle in the target detection result is tracked according to the first model, and a current first tracking result is determined; the first model is deployed on the edge terminal of the end-cloud collaborative architecture;

[0011] In the second working mode, the target vehicle in the current video frame is detected and tracked according to the second model, and a current second tracking result is determined; the computing resources occupied by the first model are less than the computing resources occupied by the second model; the second model is deployed in the cloud of the end-cloud collaborative architecture;

[0012] In the third working mode, the target vehicle in the current video frame is detected, or detected and tracked, based on the most recently triggered working mode, and a current third tracking result is determined.

[0013] In an optional implementation, tracking the target vehicle in the target detection result according to the first model includes:

[0014] Obtain a target list; the target list is obtained by detecting and tracking a previous video frame; the previous video frame is a video frame before the current video frame; the target list includes identification information and location information of the target; the target is a vehicle in the previous video frame; the location information of the target includes: location information of a detection box corresponding to the target in the previous video frame; the detection box is marked when the target detection result is determined by detecting the current video frame; a single target corresponds to a single tracking instance;

[0015] When the target vehicle does not exist in the target list and the number of targets in the target list is less than a first preset number, creating a target tracking instance corresponding to the target vehicle and adding relevant information of the target vehicle to the target list; the relevant information of the target vehicle includes identification information and location information of the target vehicle; the location information of the target vehicle includes: location information of a detection box corresponding to the target vehicle in the current video frame;

[0016] When the target vehicle does not exist in the target list and the number of targets in the target list is greater than or equal to a first preset number, ending tracking of the target vehicle;

[0017] When the target vehicle exists in the target list, relevant information of the target vehicle is added to the target list.

[0018] In an optional embodiment, the method further includes:

[0019] When there is a first target in the target list, the identification information and location information of the first target in the target list are deleted, and the first tracking instance corresponding to the first target is released; the first target is a target that has not been detected in a second preset number of consecutive video frames.

[0020] In an optional embodiment, the method further includes:

[0021] In the second working mode, triggering a delayed exit mechanism;

[0022] The delayed exit mechanism includes: obtaining a target list; and tracking targets in the target list according to a first model until the target list is empty.

[0023] In an optional embodiment, the method further includes:

[0024] Different quantity ranges and tracking processing frequencies corresponding to different quantity ranges are preset; different quantity ranges correspond to different tracking processing frequencies; the quantity range is the quantity range of the tracking instance;

[0025] Determine a quantity range and a target processing frequency based on the number of current tracking instances, so as to execute the step of tracking the target vehicle in the target detection result according to the first model based on the target processing frequency; the target processing frequency is a tracking processing frequency corresponding to the quantity range to which the number of current tracking instances belongs.

[0026] In an optional embodiment, the method further includes:

[0027] Determining a vehicle trajectory of the target vehicle based on the tracking result corresponding to each video frame in the target video stream; the target video stream includes a third preset number of consecutive video frames;

[0028] Determining whether the target vehicle is a traffic violation vehicle according to the vehicle trajectory of the target vehicle;

[0029] When the target vehicle is determined to be a traffic violation vehicle based on its vehicle trajectory, details of the traffic violation event are determined, the details of the traffic violation event are visualized, and a real-time heat map of the target intersection is generated; the traffic violation event details include the license plate information of the traffic violation vehicle, the time of the violation, the location of the violation, and a video clip of the violation.

[0030] In an optional embodiment, the first model is a model using the MobileNetV3-Small neural network as the backbone network and the Ghost-FPN feature pyramid as the enhanced feature extraction network.

[0031] In a second aspect, the present invention provides a vehicle detection and tracking system with end-cloud collaboration, the vehicle detection and tracking system with end-cloud collaboration comprising:

[0032] Edge terminals, including:

[0033] An intersection density assessment unit is configured to determine a current video frame based on a real-time video stream of a target intersection, detect the current video frame to determine a target detection result, the target detection result including the target vehicle and the number of vehicles in the current video frame; trigger a first operating mode when the number of vehicles is less than or equal to a first preset threshold; trigger a second operating mode when the number of vehicles is greater than or equal to a second preset threshold; trigger a third operating mode when the number of vehicles is greater than the first preset threshold and less than the second preset threshold; the third operating mode being the most recently triggered operating mode;

[0034] Unit 1, including the first model;

[0035] The first unit is configured to track the target vehicle in the target detection result according to the first model in the first working mode and determine a current first tracking result;

[0036] The first unit is further configured to, in the third operating mode, when the most recently triggered operating mode is the first operating mode, detect the target vehicle in the current video frame based on the most recently triggered operating mode, and determine a current third tracking result;

[0037] Cloud, including:

[0038] Unit 2, including the second model;

[0039] The second unit is configured to detect and track the target vehicle in the current video frame according to the second model in the second working mode, and determine a current second tracking result; the computing resources occupied by the first model are less than the computing resources occupied by the second model;

[0040] The second unit is further configured to detect and track the target vehicle in the current video frame based on the most recently triggered working mode when the most recently triggered working mode is the second working mode in the third working mode, and determine the current third tracking result.

[0041] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the end-cloud collaborative vehicle detection and tracking method of the first aspect or any corresponding embodiment thereof.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the end-cloud collaborative vehicle detection and tracking method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0043] The technical solution provided by the present invention has the following technical effects:

[0044] The technical solution of the embodiment of the present invention can dynamically switch the operating mode based on the number of vehicles in the video frame. When the number of vehicles is small (less than or equal to a first preset threshold), the first model deployed on the edge terminal is activated for detection and tracking. The first model consumes less computing resources and is suitable for low-density scenarios, avoiding waste of cloud resources. When the number of vehicles is large (greater than or equal to a second preset threshold), it switches to the second model deployed on the cloud. Although the second model consumes more resources, it can handle complex high-density vehicle tracking tasks, fully utilizing the powerful computing power of the cloud to achieve reasonable resource allocation and improve overall resource utilization. A third operating mode is set. When the number of vehicles is in the middle range (greater than the first preset threshold and less than the second preset threshold), the most recently triggered operating mode is maintained for detection and tracking. This effectively avoids frequent model switching due to fluctuations in the number of vehicles near the threshold, reduces instability in the tracking process, ensures the continuity and stability of tracking, and prevents problems such as tracking interruptions and jumps. The technical solution of the present invention can adapt to different traffic flow scenarios. Whether it is off-peak hours with sparse vehicles, remote sections of road, or peak hours with dense vehicles, the appropriate model and operating mode can be selected for vehicle detection and tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in related technologies, the following briefly introduces the drawings required for use in the specific embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 1 is a flow chart of a vehicle detection and tracking method using end-cloud collaboration according to an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of a specific example of the end-cloud collaborative vehicle detection and tracking method according to an embodiment of the present invention;

[0048] Figure 3 1 is a schematic diagram of the structure of a vehicle detection and tracking system with end-cloud collaboration according to an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0051] In urban traffic management, many intersections experience long periods of low traffic density (e.g., during off-peak hours or on remote roads), where vehicle traffic far exceeds the equipment's capacity. In theory, these scenarios could be addressed through independent target tracking and analysis using edge terminals, eliminating the need for cloud resources. However, traditional solutions typically utilize a fixed cloud or standalone edge machine model, which is unable to dynamically adapt to traffic flow changes and results in wasted computing resources.

[0052] Therefore, the present invention designs a technical solution for vehicle detection and tracking with end-cloud collaboration, which can dynamically switch detection and tracking models based on vehicle density, and is suitable for scenarios such as real-time monitoring of urban roads, intelligent analysis of traffic flow, and identification of vehicle violations. The technical solution of the present invention realizes dynamic resource adaptation by integrating the differentiated advantages of the SiamFC twin convolution tracking algorithm and the DeepSORT multi-target tracking algorithm: at low-density intersections or idle periods, the improved lightweight SiamFC algorithm is preferentially used to independently complete target tracking at the edge terminal, significantly reducing the cloud computing load. In high-density scenarios, it automatically switches to the DeepSORT algorithm running in the cloud, and improves tracking robustness through multi-target association and feature fusion. This dynamic collaborative mechanism effectively solves the contradiction between the traditional solution's waste of resources in low-load scenarios and insufficient performance in high-load scenarios, taking into account both real-time performance and computational efficiency.

[0053] An embodiment of the present invention provides an embodiment of a vehicle detection and tracking method with end-cloud collaboration. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer device such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0054] Figure 1 It is a flow chart of the end-cloud collaborative vehicle detection and tracking method according to an embodiment of the present invention.

[0055] like Figure 1 As shown, an embodiment of the present invention provides an end-cloud collaborative vehicle detection and tracking method, which is implemented based on an end-cloud collaborative architecture.

[0056] The end-cloud collaborative vehicle detection and tracking method includes:

[0057] S101: Determine a current video frame according to a real-time video stream of a target intersection, and detect the current video frame to determine a target detection result.

[0058] In this embodiment, the target detection result includes the target vehicle and the number of vehicles in the current video frame. The target detection result also includes the marked current video frame.

[0059] In this embodiment, an edge terminal is deployed at the target intersection. The edge terminal is an intelligent edge computing terminal. By deploying intelligent edge computing terminals at key nodes such as urban road intersections, a distributed vehicle tracking infrastructure is constructed. Existing intelligent traffic cameras can be reused, and the edge terminal can support access to heterogeneous devices (cameras of different brands, models, and specifications). The edge terminal must meet certain computing power and storage space requirements to support high-resolution video capture, thereby improving the real-time performance of detection and tracking and enhancing the recognition accuracy of the detection and tracking model. The edge terminal can access the video acquisition equipment deployed at the target intersection, such as the existing intelligent traffic camera.

[0060] In this embodiment, the real-time video stream is continuously collected by a camera at the target intersection, and a frame at the current moment can be selected from the real-time video stream as the current video frame.

[0061] In this embodiment, the target detection result can be determined by detecting the current video frame based on the first model deployed in the edge terminal. For example, the target detection result can be obtained by detecting based on a lightweight target detection model (e.g., a YOLO-Nano model).

[0062] The first model can detect the target vehicle in the current video frame, mark it with a detection frame, and obtain a marked current video frame. One detection frame corresponds to one target vehicle, and the number of detection frames can be determined as the number of vehicles in the current video frame, thereby determining the number of vehicles in the current video frame. In this embodiment, when the edge terminal is started, the first model is started to detect the target vehicle in the current video frame to determine the number of vehicles in the current video frame, so as to choose whether to track it at the edge terminal or in the cloud according to the number of vehicles. And the adaptive smooth switching of the working mode can be achieved according to the change of the real-time number of vehicles, ensuring the continuity of tracking and improving the utilization efficiency of computing resources. The target detection result also includes the marked current video frame, and the marked current video frame includes the detection frame corresponding to the target vehicle.

[0063] In this embodiment, the number of vehicles in the current video frame represents vehicle density or vehicle flow.

[0064] As an example, the frequency of detecting and determining the target detection result for the current video frame may be the same as or different from the target processing frequency described below. However, in the present embodiment, the target detection result is determined for each real-time video frame. That is, the frequency of detecting and determining the target detection result for the current video frame is 1 frame / time (processing once per frame), and the frequency of executing the step of tracking the target vehicle in the target detection result according to the first model may be 1 frame / time or 2 frames / time (processing once every 2 frames).

[0065] S102: When the number of vehicles is less than or equal to a first preset threshold, triggering a first working mode.

[0066] In this embodiment, as an example, when the number of vehicles is less than or equal to a first preset threshold, the first working mode may be triggered at a target processing frequency determined as follows.

[0067] S103: When the number of vehicles is greater than or equal to a second preset threshold, triggering the second working mode.

[0068] S104: When the number of vehicles is greater than the first preset threshold and less than the second preset threshold, triggering the third working mode.

[0069] In this embodiment, the third operating mode is the most recently triggered operating mode.

[0070] In this embodiment, when the number of vehicles is greater than a first preset threshold and less than a second preset threshold, the most recently triggered operating mode is maintained, thereby avoiding frequent switching of operating modes. The adaptive switching strategy includes: triggering the first operating mode when the number of vehicles is less than or equal to the first preset threshold. Triggering the second operating mode when the number of vehicles is greater than or equal to the second preset threshold. Maintaining the most recently triggered operating mode when the number of vehicles is greater than the first preset threshold and less than the second preset threshold.

[0071] In this embodiment, the first and second preset thresholds are dynamically adjusted in the cloud based on historical traffic flow data. When it is determined that the first and second preset thresholds need to be adjusted, the first and second preset thresholds are dynamically adjusted based on the historical traffic flow data. The first and second preset thresholds can be dynamically adjusted over time within a preset time period (e.g., a day). The types of time periods at the current moment include normal times and peak times. During normal times, the first preset threshold is the first preset value, e.g., 5. During peak times, the first preset threshold is the second preset value, e.g., 3, with the first preset value being greater than the second preset value. During normal times, the second preset threshold is the third preset value, e.g., 15. During peak times, the second preset threshold is the fourth preset value, e.g., 10, with the third preset value being greater than the fourth preset value. The corresponding first and second preset thresholds can be determined based on the type of time period at the current moment. If the type of time period at the current moment is different from the type of time period at the previous moment, the first and second preset thresholds need to be adjusted.

[0072] As an example, during the morning rush hour (7:00-9:00) or the evening rush hour (17:00-19:00), when traffic volume and the number of vehicles are generally large and the traffic situation is more complicated, the first preset threshold and the second preset threshold can be appropriately lowered to allow more detection and tracking to be performed through the cloud to cope with complex traffic conditions.

[0073] S105: In the first working mode, the target vehicle in the target detection result is tracked according to the first model, and a current first tracking result is determined.

[0074] In this embodiment, the first model is deployed on the edge terminal of the end-cloud collaborative architecture, and is a lightweight vehicle target detection and tracking model. The first model includes a first target detection model and a first target tracking model. The first target detection model is a lightweight target detection model, which can be specifically a YOLO-Nano model. The first target tracking model is a lightweight target tracking model, which can be specifically constructed based on the lightweight SiamFC algorithm. The first target detection model is used to detect the current video frame and determine the target detection result. The first target tracking model is used to track the target vehicle in the target detection result according to the first model in the first working mode, and determine the current first tracking result.

[0075] As an example, the target vehicle in the current video frame is detected based on the YOLO-Nano model. The specific method to obtain the target detection result is as follows:

[0076] Data preprocessing: The input video frame image (the current video frame) is normalized and pixel values ​​are scaled to the range of 0 to 1 to meet the model's input requirements. At the same time, some data augmentation operations may be performed on the image, such as random cropping, flipping, and adjusting brightness and contrast, to increase data diversity and improve the model's generalization ability.

[0077] Feature Extraction: The YOLO-Nano model uses a lightweight convolutional neural network architecture to extract features from video frames. It progressively extracts image features through a series of convolutional layers, batch normalization layers, and activation functions, transforming the image from its original pixel space to a feature space. In this process, the model automatically learns various vehicle characteristics, such as shape, color, and texture.

[0078] Object Prediction: After feature extraction, the model performs object prediction on the feature map. YOLO-Nano divides the image into multiple grids, each of which is responsible for predicting objects within a certain range. For each grid, the model predicts the presence of a target vehicle, its category, the location of its bounding box, and its confidence score. The bounding box's location is typically expressed as an offset relative to the grid, while the confidence score indicates the likelihood of the target vehicle being present in that grid.

[0079] Post-processing: The prediction results output by the model need to be post-processed to obtain the final detection results. First, according to the set confidence threshold, the prediction boxes with lower confidence are filtered out, and the boxes with higher confidence are retained as possible target detection results. Then, the non-maximum suppression algorithm (NMS) is used to remove detection boxes with high overlap. The NMS algorithm calculates the intersection-over-union ratio between each detection box. If the intersection-over-union ratio of two detection boxes exceeds a certain threshold, the box with higher confidence is retained and the remaining boxes are removed to obtain the final target detection result.

[0080] Specifically, in this embodiment, the labeled current video frame obtained based on the technical solution corresponding to S101 can be input into the first target tracking model of the first model to track the target vehicle. In the first operating mode, tracking is performed at the edge terminal, and the input of the first target tracking model is the labeled current video frame. The labeled current video frame obtained when determining the number of vehicles can be directly used for tracking, reducing the amount of computation, reducing the waste of computing resources, and improving the efficiency of detection and tracking.

[0081] S106: In the second working mode, the target vehicle in the current video frame is detected and tracked according to the second model, and a current second tracking result is determined. The computing resources occupied by the first model are less than the computing resources occupied by the second model.

[0082] In this embodiment, the computing resources occupied by the first model are smaller than those occupied by the second model. The first model is deployed on the edge terminal of the end-cloud collaborative architecture, and the second model is deployed on the cloud of the end-cloud collaborative architecture. The first model is built based on a lightweight single-target tracking algorithm, and the second model is built based on a multi-target tracking algorithm. The number of vehicles is detected in real time, and the operating mode is switched based on the real-time number of vehicles (vehicle density) to dynamically adjust the detection and tracking strategy. When the vehicle density is low, the edge terminal can independently rely on the first model to complete vehicle detection and tracking tasks. Due to the lightweight nature of the first model, it can respond quickly with limited computing resources and provide timely feedback on vehicle information to meet real-time requirements. Once the vehicle density increases, the complex situation brought about by a large number of targets exceeds the processing capacity of the first model, and it will automatically switch to the operating mode dominated by the second model. At this time, the edge terminal uploads the collected video frame data to the cloud and processes it using the powerful multi-target tracking capabilities of the second model. With its abundant computing resources, the cloud can accurately analyze the trajectories and status of numerous vehicles in complex scenarios.

[0083] Ensure that under different vehicle densities, the end-cloud collaborative architecture can efficiently and stably complete vehicle detection and tracking tasks.

[0084] In this embodiment, considering that a large number of vehicles requires higher detection accuracy and more computing resources, in the second working mode, the edge terminal uploads the current video frame to the cloud. The cloud is used to receive the current video frame uploaded by the edge terminal. The input of the second model is the current video frame. The target vehicle in the current video frame is detected and tracked by the second model with higher accuracy and more computing resources, which can meet the requirements of multi-target detection and tracking. In order to reduce the amount of data transmission, improve transmission efficiency and reduce network bandwidth pressure, the edge terminal encodes the current video frame and uploads the encoded current video frame to the cloud. The cloud includes a corresponding data receiving and processing module for receiving the encoded current video frame and decoding the encoded current video frame.

[0085] In this embodiment, a conventional encoding method in the art can be used to encode the video frame, and the specific encoding method is not limited here. As an example, the current video frame can be encoded using the H.264 encoding standard or the H.265 / HEVC encoding standard. When the edge terminal encodes the current video frame, the encoded current video frame is decoded, the target vehicle in the current video frame is marked according to the second model, and the detection frame corresponding to the target vehicle is determined.

[0086] In this embodiment, when the second working mode is triggered, the second working mode is maintained for a preset time period to avoid oscillation. The preset time period can be set and modified according to actual needs. For example, it is 3 minutes, that is, when the second working mode is triggered, the second working mode is locked for 3 minutes.

[0087] S107: In the third working mode, the target vehicle in the current video frame is detected, or detected and tracked, based on the most recently triggered working mode, and a current third tracking result is determined.

[0088] In this embodiment, the most recently triggered working mode is the first working mode. In the third working mode, the edge terminal performs tracking, tracks the target vehicle in the target detection result according to the first model, and determines the current third tracking result.

[0089] In this embodiment, the most recently triggered working mode is the second working mode. In the third working mode, detection and tracking are performed by the cloud, and the target vehicle in the current video frame is detected and tracked according to the second model, and the current third tracking result is determined.

[0090] In the present invention, the first tracking result, the second tracking result, and the third tracking result may include the location information of the detection box, and may also include information such as the driving direction, time, and the geographic location of the vehicle, for example, the latitude and longitude of the center of the vehicle. The present invention does not limit the information included in the tracking results.

[0091] In the present invention, there are cases where the current video frame does not include a vehicle (the number of vehicles is 0) and includes a vehicle. When the current video frame includes multiple target vehicles, the technical solutions of S101-S107 above can be executed for each of the multiple target vehicles to determine the current tracking result of each target vehicle in the current video frame.

[0092] In the present invention, the edge terminal and the cloud interact through instructions to achieve the uploading of video frames, the uploading of tracking results, and the adjustment of the first preset threshold and the second preset threshold.

[0093] In an optional implementation, tracking the target vehicle in the target detection result according to the first model in S105 includes:

[0094] S1051: Obtain a target list.

[0095] In this embodiment, the target list is obtained by tracking the preceding video frame. The preceding video frame is the video frame preceding the current video frame. The target list includes the target's identification information and location information. The target is a vehicle in the preceding video frame. The target's location information includes the location information of the detection box corresponding to the target in the preceding video frame. The detection box is marked when the target detection result is determined in the current video frame. A single target corresponds to a single tracking instance. As an example, the tracking instance can be a SiamFC instance. Tracking instances can be isolated using Docker containers to avoid resource contention. Tracking instances can be stored in the SiamFC tracking instance pool of the edge terminal, and lifecycle management of tracking instances is performed, including the creation and release of tracking instances. Specifically, for a single target, the maximum number of preceding video frames corresponding to the target can be a maximum preset number (the number of preceding video frames corresponding to a single target is less than or equal to the maximum preset number), for example, 60. The number of target location information is less than or equal to the preset number of locations. If a target appears only once in a video frame, the corresponding target location information includes the location information of at most the preset number of detection boxes (i.e., the maximum preset number of 60). The maximum preset quantity and the number of preset positions can be designed and modified according to actual needs.

[0096] In this embodiment, the target in the target list is the tracked vehicle. The maximum number of targets in the target list is less than a first preset number, for example, the first preset number is 10. The target list is used to record the identification information and location information of the target currently being tracked by the tracking instance. As an example, the identification information is used to distinguish different targets, such as digital information such as 01, 02, 03, etc. The location information is used to determine the position of the target in the previous video frame. For example, in a coordinate system established with the upper left corner vertex of the video frame as the coordinate origin, the horizontal direction to the right of the origin as the horizontal coordinate, and the vertical direction downward from the origin as the vertical coordinate, the location information of the detection frame can be the coordinate information of the four vertices of the detection frame. For example, the location information of a detection frame includes the coordinates of the upper left vertex (x1, y1), the coordinates of the upper right vertex (x2, y1), the coordinates of the lower right vertex (x2, y2), and the coordinates of the lower left vertex (x1, y2). Through these four coordinate values, the position and size of the detection frame in the video frame can be accurately described, thereby accurately locating the position of the target in the previous video frame.

[0097] S1052: When the target vehicle does not exist in the target list and the number of targets in the target list is less than a first preset number, create a target tracking instance corresponding to the target vehicle and add relevant information of the target vehicle to the target list.

[0098] In this embodiment, the target vehicle's related information includes its identification information and location information. The target vehicle's location information includes the location information of a detection frame corresponding to the target vehicle in the current video frame. Each detection frame can be assigned corresponding identification information, and the detection frame's identification information can be used as the target vehicle's identification information.

[0099] In this embodiment, when there is no target vehicle in the target list and the number of targets in the target list is less than a first preset number, it indicates that the target vehicle is a new target that can be tracked, and a target tracking instance corresponding to the target vehicle is created to continuously track the target vehicle.

[0100] In this embodiment, a tracking instance is a logical unit constructed to achieve continuous tracking of a single target. It integrates functions such as target identification, positioning, and status update to ensure stable tracking of the target in the video stream. Each target corresponds to an independent tracking instance. Just like equipping each vehicle to be tracked with a dedicated "tracker", for example, a tracking instance 01 will be created for target vehicle 01. Tracking instance 01 is specifically responsible for recording and updating the tracking information of vehicle 01 in different video frames, and will not be confused with the tracking information of other vehicles. This ensures the independence and accuracy of tracking of each target. The tracking instance is responsible for recording relevant information about the target vehicle, including identification information (such as the target vehicle ID, such as 01, 02, etc.) and location information (i.e., the location information of the detection box, such as the detection box described by the four vertex coordinates). During the video stream processing, the tracking instance continuously updates this information as the vehicle moves. For example, if a vehicle moves from one position in the current video frame to another position in the next frame, the tracking instance will record the vehicle's new location information based on the new tracking information, thereby achieving continuous recording of the vehicle's motion trajectory. The tracking instance uses a specific tracking algorithm to track the target vehicle. Taking SiamFC as an example, the new position of the target vehicle is determined by finding the area most similar to the target vehicle in different video frames. In this process, the tracking instance uses an algorithm to analyze and process the image data of the previous and next video frames. For example, the characteristics of the target vehicle are determined in the previous video frame. In the current video frame, the tracking instance uses the SiamFC algorithm to match similar features, and then updates the location information of the target vehicle to ensure that the vehicle is always accurately tracked in the video stream. The tracking instance has its own life cycle. When a new target vehicle appears and meets the tracking conditions (such as the target list is not full, and the number of targets in the target list is less than the first preset number), a tracking instance is created for it. In actual applications, a SiamFC tracking instance is a specific operating entity that uses the SiamFC algorithm to track the target vehicle.

[0101] S1053: When the target vehicle does not exist in the target list and the number of targets in the target list is greater than or equal to a first preset number, the tracking of the target vehicle is terminated.

[0102] In this embodiment, when the target list is full, even if a new target appears, the new target will not be tracked.

[0103] S1054: When the target vehicle exists in the target list, relevant information of the target vehicle is added to the target list.

[0104] In this embodiment, the presence of a target vehicle in the target list indicates that the target vehicle has been previously tracked. If the number of target vehicle location information in the target list is less than or equal to the preset number of locations, the relevant information of the target vehicle is directly added to the target list. If the number of target vehicle location information in the target list is equal to the preset number of locations, the earliest added location information is deleted and the relevant information of the target vehicle is added to the target list to update the target list to the latest location information.

[0105] In this embodiment, whether the target vehicle exists in the target list can be determined by performing an Intersection over Union (IoU) match between the target vehicle and the targets in the target list. When the matching result of the detection frame corresponding to the target vehicle and a detection frame corresponding to the target in the target list is greater than or equal to the first matching threshold (or the average value of the matching results of the detection frame corresponding to the target vehicle and all detection frames corresponding to the targets in the target list is greater than or equal to the second matching threshold), it is determined that the target vehicle exists in the target list. When the matching results of the detection frame corresponding to the target vehicle and all detection frames corresponding to all targets in the target list are less than the first matching threshold, it is determined that the target vehicle does not exist in the target list. The first matching threshold and the second matching threshold can both be set and modified according to actual needs.

[0106] As an example, the matching result between the detection box A corresponding to the target vehicle and a detection box B corresponding to a target in the target list is calculated as follows:

[0107] .

[0108] in, Indicates the matching result between the detection frame A corresponding to the target vehicle and a detection frame B corresponding to the target in the target list. It represents the area of ​​the intersection between the detection box A corresponding to the target vehicle and a detection box B corresponding to the target in the target list. The area of ​​the detection box A corresponding to the target vehicle, Indicates the area of ​​a detection box B corresponding to the target in the target list.

[0109] In an optional embodiment, the end-cloud collaborative vehicle detection and tracking method further includes:

[0110] When the first target exists in the target list, the identification information and location information of the first target in the target list are deleted, and the first tracking instance corresponding to the first target is released.

[0111] In this embodiment, the first target is a target that is not detected in a second preset number of consecutive video frames. The second preset number can be set and modified according to actual needs.

[0112] In this embodiment, if a target vehicle leaves the monitoring range and is not detected for a certain number of frames, the corresponding tracking instance is released. For example, if a vehicle 04 is not detected for a second predetermined number of consecutive video frames (e.g., 5 frames), the identification and location information of vehicle 04 is deleted from the target list, and the tracking instance corresponding to vehicle 04 is released, freeing up related resources. This helps to efficiently manage tracking resources and avoid ineffective tracking.

[0113] In an optional embodiment, the first model is a model using the MobileNetV3-Small neural network as the backbone network and the Ghost-FPN feature pyramid as the enhanced feature extraction network.

[0114] In this embodiment, the first model is specifically a first target tracking model, which is a model built based on the SiamFC algorithm with the MobileNetV3-Small neural network as the backbone network and the Ghost-FPN feature pyramid as the enhanced feature extraction network. The target vehicle in the target detection result can be tracked based on the lightweight target tracking model built based on the SiamFC algorithm. As an example, the SiamFC algorithm is based on a twin network and includes two branches of the same structure. One branch (the first branch network) is used to extract the features of the target vehicle in the initial frame, and the other branch (the second branch network) is used to extract the features of the search area image in subsequent frames. In this embodiment, the backbone networks of both branches use the MobileNetV3-Small neural network. For example, based on the position information of the detection frame corresponding to the target vehicle in the previous video frame, a preset ratio (such as 1.5 times) is expanded outward to obtain a search area frame, and then the image corresponding to the search area frame is cropped in the current video frame to obtain a search area image. The image of the detection frame corresponding to the target vehicle in the initial frame (first frame) is respectively input into the first branch network and the Ghost-FPN feature pyramid for feature extraction and enhancement to obtain a first enhanced feature map. The search area image is respectively input into the second branch network and the Ghost-FPN feature pyramid for feature extraction and enhancement to obtain a second enhanced feature map. The first enhanced feature map and the second enhanced feature map are correlated and calculated for similarity (for example, cosine similarity). The position with the highest similarity to the first enhanced feature map is found in the second enhanced feature map, and the position with the highest similarity to the first enhanced feature map in the second enhanced feature map is determined as the detection frame corresponding to the target vehicle in the current video frame, that is, the new position of the target vehicle in the current video frame.

[0115] Replacing the backbone network: The AlexNet backbone of the original SiamFC algorithm is replaced with the lightweight MobileNetV3-Small network, accelerating feature extraction while improving its capabilities. The main advantages are: MobileNetV3-Small's reverse residual block increases the number of channels by expanding the layer, and combines different convolution kernels (3x3, 5x5) to extract multi-granular features, enhancing adaptability to vehicle scale changes. Lightweight SE modules are integrated at key levels (such as Stages 5, 6, and 7) to dynamically enhance the response of the target area. The h-swish activation function is used. Compared to ReLU, h-swish is more stable under low-precision quantization (for example, INT8), reducing accuracy loss in edge deployment.

[0116] Introducing the Ghost-FPN feature pyramid to fuse multi-scale features between levels:

[0117] The Ghost-FPN feature pyramid is an improved structure that combines the Ghost module and the Feature Pyramid Network (FPN). Its lightweight design improves model efficiency while maintaining multi-scale feature fusion capabilities. Although MobileNetV3-Small captures features of different scales through convolution operations at different layers, these features are not explicitly fused. When the target vehicle moves from far to near, relying solely on a single layer of features may cause the tracking frame to drift or be lost. Therefore, in scenarios where multi-scale targets need to be processed, adding an additional multi-scale fusion mechanism, FPN, can improve the robustness of the model. To adapt to the inference performance of terminal devices, low-cost cross-layer feature fusion is constructed through Ghost convolution, addressing the lack of multi-scale interaction in MobileNetV3-Small.

[0118] In this embodiment, the first target tracking model in the first model implements single target tracking through tracking instances.

[0119] In an optional implementation, the second model includes a second target detection model and a second target tracking model.

[0120] The second target detection model is used to detect the current video frame to determine the target detection result, and the second target tracking model is used to track the target vehicle in the target detection result.

[0121] The second object detection model may be a YOLOv8 model.

[0122] When the YOLOv8 model and the DeepSORT algorithm are used together, the YOLOv8 model is only responsible for target detection and does not require modification of the network structure. The model built based on the DeepSORT algorithm then extracts features from multiple targets detected by the YOLOv8 model to achieve feature matching of multiple targets across frames. As an example, the feature extraction part can choose a ResNet network or other networks. For example, depending on the real-time inference speed requirements and the complexity of the scene, a lightweight ResNet-18 or a deeper ResNet-101 can be selected. Other network structures can also be used.

[0123] The second target tracking model can be a model built based on the DeepSORT algorithm, which implements multi-target tracking using the DeepSORT algorithm. Target detection can be performed based on the YOLOv8 model, and multi-target tracking can be performed using the target tracking model built based on the DeepSORT algorithm. As an example, the network structure of the YOLOv8 model includes a backbone network, a neck network, and a head network. The backbone network is responsible for extracting the underlying features of the image, the neck network fuses and adjusts the features, and the head network is used to predict the category and position of the target. In the multi-target tracking task, the YOLOv8 model is only responsible for the target detection task, and can quickly and accurately identify various types of targets in the video frame, such as vehicles, pedestrians, etc., and output the target's detection box and category information (category information such as vehicles and pedestrians). In this embodiment, the target detection result with the category information of vehicle can be input into the second target tracking model to track the target vehicle. The specific implementation scheme can refer to the conventional methods in the field. For example, the ResNet-18 network can be used to extract features of the target vehicle to obtain the target's appearance feature vector. At the same time, the motion information of the target vehicle is predicted using the Kalman filter algorithm. The appearance feature vector is used to describe the appearance features of the target vehicle, and the motion information is used to predict the position of the target vehicle in the next video frame. Combining appearance features and motion information to achieve multi-target tracking is a conventional method in this field and will not be described in detail here.

[0124] When the number of vehicles is greater than or equal to the second preset threshold, it switches to the second working mode, and the cloud takes over the analysis task. The edge terminal encodes the video frame and uploads it to the cloud, and the DeepSORT algorithm combines the target detection results to achieve multi-target tracking. In order to cope with occlusion and interference from similar targets in dense scenes, the cloud introduces appearance feature learning, such as extracting vehicle appearance features through ResNet-18 to improve tracking robustness. Taking the above example as an example, the specific example schematic diagram of the vehicle detection and tracking method of end-cloud collaboration of the present invention is shown as follows Figure 2 shown.

[0125] In the present invention, the structure of the model is the structure designed by the technical solution of the present invention. The model can be trained in a conventional manner in the art, for example, a training method based on historical data, which will not be described in detail here.

[0126] In an optional embodiment, the end-cloud collaborative vehicle detection and tracking method further includes:

[0127] In the second working mode, the delayed exit mechanism is triggered.

[0128] In this embodiment, the delayed exit mechanism includes: obtaining a target list, and tracking targets in the target list according to the first model until the target list is empty.

[0129] In this embodiment, in the second operating mode, the edge terminal triggers a delayed exit mechanism. The delayed exit mechanism specifically includes: obtaining a target list, tracking targets in the target list according to the first model, and when a first target exists in the target list, deleting the identification information and location information of the first target in the target list and releasing the first tracking instance corresponding to the first target until the target list is empty.

[0130] In this embodiment, that is, in the second working mode, the cloud performs detection and tracking, and the edge terminal triggers the delayed exit mechanism and continues to track all targets in the target list. During this period, no new tracking instances are added. When all tracking instances are released, the edge terminal exits tracking, leaving only the cloud to track.

[0131] In this embodiment, when switching from the first working mode to the second working mode, if the delayed exit mechanism is not adopted, there may be a momentary interruption in tracking, resulting in the loss of some targets. However, through the delayed exit mechanism, the edge terminal continues to track all targets in the target list until all tracking instances are released, ensuring the continuity and smoothness of target tracking. After triggering the delayed exit mechanism, the edge terminal continues to use its own computing resources to track the target until all tracking instances are released. This allows the computing resources of the edge terminal to be fully utilized during the mode switching process, avoiding waste of resources. The delayed exit mechanism allows the edge terminal to continue tracking the target for a certain period of time, ensuring continuous tracking of the target even in the event of network fluctuations or interruptions. This improves the adaptability to network fluctuations.

[0132] In an optional embodiment, the end-cloud collaborative vehicle detection and tracking method further includes:

[0133] Different quantity ranges and tracking processing frequencies corresponding to different quantity ranges are preset.

[0134] Different quantity ranges correspond to different tracking processing frequencies. The quantity range refers to the number of tracking instances. The greater the number of tracking instances, the lower the tracking processing frequency.

[0135] A number range and a target processing frequency are determined based on the number of current tracking instances, and the step of tracking the target vehicle in the target detection result according to the first model is performed based on the target processing frequency. The target processing frequency is a tracking processing frequency corresponding to the number range to which the current number of tracking instances belongs.

[0136] In this embodiment, in order to adapt to the lightweight reasoning of the edge terminal, a frame rate adaptive method can be used to dynamically adjust the tracking processing frequency according to the number of current tracking instances.

[0137] As an example, the preset different quantity ranges and the corresponding tracking processing frequencies for the different quantity ranges are:

[0138] The number of trace instances ranges from , the corresponding tracking processing frequency is 1 frame / time (processing once per frame, 30FPS).

[0139] The number of trace instances ranges from , the corresponding tracking processing frequency is 2 frames / time (processing once every two frames, 15FPS), Indicates the number of tracking instances as an integer. For a tracking processing frequency of 2 frames / time, the real-time video stream is video frame 1, video frame 2, video frame 3, video frame 4, video frame 5, etc. As an example, the edge terminal detects video frame 1, video frame 2, video frame 3, etc. in sequence according to the first model to determine the target detection result. When the number of vehicles is less than or equal to the first preset threshold, the first working mode is triggered according to the target processing frequency of 2 frames / time, and the target vehicles in the target detection results corresponding to video frame 1, video frame 3, video frame 5, etc. are tracked according to the first model.

[0140] Edge terminals have limited computing resources and processing power. Different numbers of tracking instances imply different computational loads. By setting different tracking processing frequencies corresponding to different number ranges, resources can be dynamically allocated based on the actual load. When the number of tracking instances is small, the number of tracking instances is within the range corresponding to 1 frame / time, and video frames are processed at a higher frequency, fully utilizing the idle resources of the edge terminal and ensuring real-time and accurate tracking of the target vehicle. As the number of tracking instances increases and the computational load increases, the tracking processing frequency is adjusted to 2 frames / time. This prevents edge terminal lag or crashes due to exhausted computing resources, thereby ensuring rational resource utilization and efficient allocation.

[0141] Based on the number of current tracking instances, the number range and target processing frequency are determined. The step of tracking the target vehicle in the target detection results according to the first model is then executed based on the target processing frequency. This balances tracking accuracy and real-time performance. Vehicle detection and tracking tasks require both tracking accuracy and real-time performance. For a small number of tracking instances, a higher tracking processing frequency can promptly capture subtle changes in the target vehicle, improve tracking accuracy, and accurately record the vehicle's trajectory and status. When the number of tracking instances is large, lowering the tracking processing frequency, while reducing the number of processed video frames, maintains system operation within limited resources and ensures that processing delays caused by excessive pursuit of accuracy do not affect real-time performance. This dynamic adjustment mechanism can find the right balance between tracking accuracy and real-time performance under varying load conditions. It also improves system stability. If the edge terminal consistently processes video frames at a fixed frequency, insufficient resources may lead to system instability when there are too many tracking instances, resulting in frame loss, tracking errors, and other issues. Dynamic frame rate adaptation automatically optimizes the processing frequency based on the number of tracking instances. During the video frame processing process, the edge terminal detects and tracks the video frames according to the adjusted tracking processing frequency, avoiding system failures caused by resource overload, improving the stability and reliability of the system, and ensuring that vehicle detection and tracking tasks can be carried out continuously and stably.

[0142] In an optional embodiment, the end-cloud collaborative vehicle detection and tracking method further includes:

[0143] The vehicle trajectory of the target vehicle is determined according to the tracking results corresponding to each video frame in the target video stream.

[0144] In this embodiment, the target video stream includes a third preset number of consecutive video frames. The third preset number can be set and modified according to actual needs, for example, 5, and the vehicle trajectory of the target vehicle is determined based on the tracking results corresponding to the 5 consecutive video frames. In this embodiment, the tracking result corresponding to each video frame is determined according to the technical solution of S101-S107 above.

[0145] Determine whether the target vehicle is a traffic violation vehicle based on the vehicle trajectory of the target vehicle.

[0146] When the target vehicle is determined to be a traffic violation vehicle based on the vehicle trajectory of the target vehicle, details of the traffic violation event are determined, the details of the traffic violation event are visualized, and a real-time heat map of the target intersection is generated.

[0147] When it is determined that the target vehicle is not a traffic violation vehicle according to the vehicle trajectory of the target vehicle, the next video frame is used for detection and tracking.

[0148] In this embodiment, the details of the traffic violation event include the license plate information of the violating vehicle, the time of the violation, the location of the violation, and a video clip of the violation.

[0149] In this embodiment, when the target vehicle is determined to be a traffic violation vehicle based on the vehicle trajectory of the target vehicle, the corresponding alarm method is triggered. The alarm methods include: the first-level alarm (such as wrong-way driving) triggers the immediate linkage of the traffic police platform, and the second-level alarm (such as illegal parking) sends a notification and records it. The details of the violation event can also include an alarm record, and the alarm record includes the time of the violation, the triggered alarm level, the type of violation event, etc. In this embodiment, all details of the violation event are stored in a cloud-based data warehouse, which can support retrospective query and visual analysis. The real-time heat map can be used to display in the real-time monitoring screen of the target intersection in the form of a pop-up window, or displayed in the visual alarm interface of the system. Visual analysis and alarms can be performed based on the above embodiments.

[0150] In this embodiment, the vehicle trajectory of the target vehicle may be generated using conventional methods in the art.

[0151] As an example, a specific implementation method of determining the vehicle trajectory of the target vehicle according to the tracking result corresponding to each video frame in the target video stream is as follows:

[0152] When a target vehicle is first detected in the target video stream, its trajectory is initialized. The target vehicle's initial position (the center coordinates of the detection box or other coordinates representing the location) is obtained from the tracking results corresponding to the first frame and used as the starting point of the trajectory. The timestamp of this video frame is also recorded for subsequent calculations of the target vehicle's velocity and time series relationship. As the target video frame advances, the vehicle trajectory is updated based on the tracking results corresponding to subsequent frames. For a target vehicle with an initialized trajectory, the associated detection box is found in each frame and its new position is obtained. Based on the new position and the position of the previous frame, the target vehicle's motion parameters, such as displacement and velocity, are calculated. For example, by calculating the difference between the center coordinates of the detection boxes in two adjacent video frames, the target vehicle's displacement during that time period is obtained. The target vehicle's velocity is then calculated based on the time interval between the two frames. The new position, velocity, and other information are added to the trajectory data structure to form an ever-expanding vehicle trajectory. Directly obtained vehicle trajectories may exhibit fluctuations and discontinuities. To obtain a smoother and more accurate trajectory, trajectory smoothing is required. A common method is the moving average method. For each point in the trajectory, the average of the positions of several points before and after it (for example, three points before and after) is taken as the new position of the point, thereby reducing the impact of noise on the trajectory. Alternatively, spline interpolation can be used to connect trajectory points by fitting a smooth curve, making the trajectory smoother and more natural. In practical applications, the appropriate smoothing method is selected based on specific needs and scenarios to improve trajectory quality and facilitate subsequent analysis of vehicle behavior and violation determination.

[0153] In this embodiment, determining whether a target vehicle is a traffic violation vehicle based on its vehicle trajectory specifically includes: determining whether the target vehicle is a traffic violation vehicle based on its vehicle trajectory based on traffic violation determination rules. If at least one of the traffic violation determination rules is satisfied, the target vehicle is determined to be a traffic violation vehicle. If none of the traffic violation determination rules are satisfied, the target vehicle is determined not to be a traffic violation vehicle.

[0154] Traffic violation determination rules include rules for wrong-way driving and illegal parking, etc.

[0155] As an example, the rule for determining wrong-way driving: if the angle between the motion direction vector fitted by the trajectory points for more than 5 consecutive frames and the road direction vector is greater than 150°, it is determined to be wrong-way driving. As an example, the rule for determining illegal parking: if the target vehicle is stationary (the speed is continuously lower than the speed threshold, such as 0.5m / s and the stationary time exceeds the preset stationary time, such as 5 minutes) and is located in a no-parking area (such as the edge of the road, the area covered by the no-parking marking), it is determined to be illegal parking. The rules for determining other violations are no longer listed one by one.

[0156] In this embodiment, if the target vehicle is determined to be a traffic violation vehicle based on its trajectory, the license plate information of the traffic violation vehicle is identified and recorded. In this embodiment, a lightweight OCR model suitable for edge computing can be selected to identify the license plate information of the traffic violation vehicle based on conventional license plate recognition methods. As an example, at least one detection frame including the target vehicle can be determined, the license plate area within the detection frame can be extracted, and the image of the license plate area can be enhanced. The license plate information can be identified using the lightweight OCR model to determine the target vehicle's license plate information. The legitimacy of the information can be verified using the vehicle database to ensure the accurate recording of the identity of the traffic violation vehicle.

[0157] As a specific example, a video frame with high definition and a relatively complete license plate area can be selected to obtain a detection frame containing the target vehicle. If multiple consecutive frames contain the target vehicle, the information from multiple detection frames can be combined to improve the accuracy of subsequent license plate region extraction. Based on prior knowledge, license plates are typically located in a specific position on a vehicle (such as the front or rear) and have a certain aspect ratio. These rules can be used to roughly determine the area within the detection frame where the license plate is likely to be located. For example, for a typical small car, the license plate is typically located at the lower part of the front or the lower middle of the rear, with an aspect ratio of approximately 2:1 to 3:1. A pretrained license plate localization model, such as one based on a convolutional neural network, is used. The image within the detection frame is fed into the model, which then outputs the precise location of the license plate within the image, thereby extracting the license plate region.

[0158] In order to improve the accuracy of license plate information recognition, it is necessary to perform image enhancement processing on the extracted license plate area, which mainly includes the following aspects:

[0159] Grayscale conversion: Converting a color license plate image to a grayscale image reduces color information interference and computational complexity. A weighted average method can be used to sum the pixel values ​​of the three RGB channels according to certain weights to obtain the grayscale value.

[0160] Filtering and denoising: Use methods such as Gaussian filtering and median filtering to remove noise from the image, making the license plate characters clearer. For example, median filtering can effectively remove salt and pepper noise by replacing the current pixel value with the median of the neighboring pixel values.

[0161] Histogram equalization: Enhances the contrast of the image, making the distinction between the license plate characters and the background more distinct. By adjusting the grayscale distribution of the image, the grayscale values ​​of the image are more evenly distributed across the entire grayscale range.

[0162] Tilt correction: If the license plate is tilted in the image, it needs to be corrected. You can use methods such as Hough transform to detect the tilt angle of the license plate and then correct the tilt by rotating the image.

[0163] Select a lightweight OCR model suitable for edge computing. This model has a simple structure, low computational effort, and can quickly process license plate images while maintaining a certain level of recognition accuracy.

[0164] Model training: Use a large dataset of license plate images to train the lightweight OCR model, enabling it to learn the characteristics and distribution patterns of license plate characters. The training data should include license plate images in different fonts, colors, and lighting conditions to improve the model's generalization capabilities.

[0165] Recognition process: The license plate area after image enhancement is input into the trained lightweight OCR model. The lightweight OCR model will output the recognition result, that is, the license plate information, that is, the character information of the license plate.

[0166] Verify the validity of the license plate against the vehicle database: The license plate is compared with the information in the vehicle database to check whether the license plate exists in the database and whether the vehicle information corresponding to the license plate matches the characteristics of the current target vehicle (such as model and color). If the license plate information is valid and matches the vehicle characteristics, the identity is considered to be successful. Otherwise, re-identification or further verification may be required.

[0167] In this embodiment, when the tracking result corresponding to each video frame in the target video stream is obtained in the first working mode, it is determined whether the violation judgment is performed at the edge terminal or in the cloud based on the judgment complexity corresponding to the type of violation event to be evaluated and the computing power constraint corresponding to the edge terminal. When the tracking result corresponding to each video frame in the target video stream is obtained in the second working mode, the violation judgment is performed in the cloud. When the tracking result corresponding to each video frame in the target video stream is obtained in the first working mode and the second working mode, it can be determined whether the violation judgment is performed at the edge terminal or in the cloud based on the judgment complexity corresponding to the type of violation event to be evaluated and the computing power constraint corresponding to the edge terminal. It can be directly selected to perform the violation judgment at the edge terminal, or it can be directly selected to perform the violation judgment in the cloud. The violation judgment includes: determining the vehicle trajectory of the target vehicle based on the tracking result corresponding to each video frame in the target video stream, and determining whether the target vehicle is a violation vehicle based on the vehicle trajectory of the target vehicle.

[0168] As an example, the types of traffic violation events to be evaluated include: driving against traffic, running a red light, speeding, etc.

[0169] The complexity of judging violations such as driving against traffic and running a red light is relatively low. If the violation type to be assessed is driving against traffic or running a red light, and the edge terminal has sufficient computing power, while meeting the task of real-time video frame tracking, there is still residual computing power to handle the violation judgment (the computing power required for violation judgment is less than or equal to the residual computing power, indicating that the computing power constraint of the edge terminal is met). In this case, the violation judgment can be performed on the edge terminal. This avoids network delays when uploading video frames to the cloud, improving judgment efficiency.

[0170] The judgment corresponding to speeding is relatively complex. If the type of violation to be evaluated is speeding and the computing power of the edge terminal is insufficient (the computing power required for violation judgment is greater than the remaining computing power, indicating that the computing power constraint corresponding to the edge terminal is not met), the speeding violation judgment can be performed in the cloud.

[0171] Complex violations, such as speeding violations with changing conditions, require a comprehensive assessment based on multiple video frames, vehicle speed changes, and the relative position of the vehicle to surrounding vehicles. This involves complex algorithms and extensive data processing, resulting in a relatively high level of complexity. However, violations with relatively simple rules, such as driving against traffic and running red lights, can be considered relatively low in complexity.

[0172] This invention addresses the issue of wasted computing resources in low-density urban traffic management scenarios by proposing a vehicle detection and tracking method that leverages end-to-end collaboration and dynamic vehicle density switching, reducing cloud resource usage and network costs. Through dynamic resource adaptation, smooth end-to-end switching, and lightweight algorithm optimization, this technical solution addresses the rigid resource allocation issues of traditional systems while balancing real-time performance with cost-effectiveness, improving resource utilization and system stability.

[0173] The main improvements of the present invention are as follows:

[0174] Traditional solutions generally adopt fixed computing modes, such as full cloud processing or stand-alone edge operation, which cannot dynamically adapt to changes in traffic flow and waste computing resources. This invention proposes an adaptive switching strategy based on the real-time number of vehicles and achieves resource optimization through a dynamic selection algorithm: low-density scenarios are processed at the edge terminal using the lightweight SiamFC algorithm, and high-density scenarios are switched to cloud processing using the DeepSORT algorithm. Compared with fixed algorithm solutions, the technical solution of the invention can improve the utilization of computing resources.

[0175] To address the stability issues of end-cloud collaboration, an adaptive switching strategy is proposed, which achieves seamless connection through containerized resource isolation and progressive instance management technology. When switching to the cloud, the delayed exit mechanism of the edge terminal is used to maintain tracking continuity, eliminate frame loss or ID jumping problems, improve system tracking stability, and reduce additional bandwidth usage during switching.

[0176] To address the real-time limitations and feature expression flaws of the original SiamFC, a three-pronged improvement was implemented: using the MobileNetV3-Small neural network as the backbone network, the Ghost-FPN feature pyramid as the enhanced feature extraction network, and adjusting the target processing frequency based on the number of currently tracked instances. This balances lightweight design with high precision. First, AlexNet was replaced with MobileNetV3-Small to reduce the number of model parameters. Second, the Ghost-FPN feature pyramid was introduced to lightweightly fuse cross-layer features, addressing tracking drift caused by single-layer dependencies. The target processing frequency was adjusted based on the number of currently tracked instances, improving detection accuracy while maintaining high detection frequency at the edge.

[0177] The traditional solution using a single tracking algorithm has limitations in scenarios with changing traffic flow. For example, although DeepSORT can handle dense multi-target tracking scenarios, its complex structure that combines Kalman filtering with appearance feature learning leads to computational redundancy at low density. Although SiamFC is lightweight, the original version relies on the AlexNet backbone network, and the cross-layer feature fusion is missing, resulting in a high tracking drift rate and difficulty in achieving real-time performance on edge devices. Although lightweight improvements (such as replacing the backbone network with MobileNet) reduce the amount of computation, they do not solve the problem of multi-scale feature fusion. For example, when the target vehicle moves from far to near, the single-layer feature dependency causes the tracking frame to offset, and the cross-layer fusion of traditional FPN requires additional computational overhead, which may affect the edge inference speed. The technical solution of the present invention uses the MobileNetV3-Small neural network as the backbone network and the Ghost-FPN feature pyramid as the enhanced feature extraction network. Adjusting the target processing frequency according to the number of current tracking instances can solve the above problems.

[0178] It should be noted that any content not described in detail in this specification is well known to those skilled in the art. The acquisition, storage, use, and processing of data such as video streams, video frames, images, and license plate areas in the technical solution of this invention comply with relevant national laws and regulations.

[0179] In this embodiment, a vehicle detection and tracking system with end-cloud collaboration is also provided. A single system is used to implement the above-mentioned embodiments and optional implementation methods. The details that have been explained will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0180] In the present invention, an end-cloud collaborative vehicle detection and tracking system is also constructed based on the end-cloud collaborative architecture.

[0181] Figure 3 It is a structural diagram of the end-cloud collaborative vehicle detection and tracking system according to an embodiment of the present invention.

[0182] The present invention provides a vehicle detection and tracking system with end-cloud collaboration, such as Figure 3 As shown, the end-cloud collaborative vehicle detection and tracking system includes: edge terminal and cloud.

[0183] The edge terminal 11 includes:

[0184] The intersection density assessment unit 111 is configured to determine a current video frame based on a real-time video stream of a target intersection, and to detect the current video frame to determine a target detection result, wherein the target detection result includes the target vehicle and the number of vehicles in the current video frame. When the number of vehicles is less than or equal to a first preset threshold, a first operating mode is triggered. When the number of vehicles is greater than or equal to a second preset threshold, a second operating mode is triggered. When the number of vehicles is greater than the first preset threshold and less than the second preset threshold, a third operating mode is triggered. The third operating mode is the most recently triggered operating mode.

[0185] The first unit 112 includes a first model.

[0186] The first unit 112 is configured to track the target vehicle in the target detection result according to the first model in the first working mode, and determine a current first tracking result.

[0187] The first unit 112 is further configured to detect the target vehicle in the current video frame based on the most recently triggered working mode and determine a current third tracking result when the most recently triggered working mode is the first working mode in the third working mode.

[0188] Cloud 12, including:

[0189] The second unit 121 includes a second model.

[0190] The second unit 121 is configured to detect and track the target vehicle in the current video frame according to the second model in the second working mode, and determine a current second tracking result. The computing resources occupied by the first model are smaller than the computing resources occupied by the second model.

[0191] The second unit 121 is further configured to detect and track the target vehicle in the current video frame based on the most recently triggered working mode when the most recently triggered working mode is the second working mode, and determine a current third tracking result.

[0192] In an optional embodiment, the first unit 112 is specifically configured to obtain a target list. The target list is obtained by tracking a preceding video frame. The preceding video frame is the video frame preceding the current video frame. The target list includes target identification information and location information. The target is a vehicle in the preceding video frame. The target location information includes the location information of a detection box corresponding to the target in the preceding video frame. The detection box is marked when the target detection result is determined during detection of the current video frame. A single target corresponds to a single tracking instance.

[0193] If the target vehicle does not exist in the target list and the number of targets in the target list is less than a first preset number, a target tracking instance corresponding to the target vehicle is created, and relevant information about the target vehicle is added to the target list. The relevant information about the target vehicle includes identification information and location information of the target vehicle. The location information of the target vehicle includes the location information of a detection box corresponding to the target vehicle in the current video frame.

[0194] When the target vehicle does not exist in the target list and the number of targets in the target list is greater than or equal to a first preset number, tracking of the target vehicle is terminated.

[0195] When the target vehicle exists in the target list, relevant information of the target vehicle is added to the target list.

[0196] In an optional embodiment, the edge terminal 11 of the end-cloud collaborative vehicle detection and tracking system further includes:

[0197] The management unit is configured to delete identification information and location information of the first target in the target list and release a first tracking instance corresponding to the first target when the first target exists in the target list. The first target is a target that has not been detected in a second preset number of consecutive video frames.

[0198] In an optional embodiment, the edge terminal 11 of the end-cloud collaborative vehicle detection and tracking system further includes:

[0199] The delayed exit unit is used to trigger the delayed exit mechanism in the second working mode.

[0200] The delayed exit mechanism includes: obtaining a target list, and tracking targets in the target list according to a first model until the target list is empty.

[0201] In an optional embodiment, the edge terminal 11 of the end-cloud collaborative vehicle detection and tracking system further includes:

[0202] The adaptive adjustment unit is used to pre-set different quantity ranges and corresponding tracking processing frequencies. Different quantity ranges correspond to different tracking processing frequencies. The quantity range is the quantity range of the tracking instance.

[0203] A number range and a target processing frequency are determined based on the number of current tracking instances, and the step of tracking the target vehicle in the target detection result according to the first model is performed based on the target processing frequency. The target processing frequency is a tracking processing frequency corresponding to the number range to which the current number of tracking instances belongs.

[0204] In an optional embodiment, the cloud 12 of the end-cloud collaborative vehicle detection and tracking system further includes:

[0205] The analyzing unit is configured to determine a vehicle track of the target vehicle based on a tracking result corresponding to each video frame in the target video stream. The target video stream includes a third preset number of consecutive video frames. The analyzing unit is configured to determine whether the target vehicle is a traffic violation vehicle based on the vehicle track of the target vehicle.

[0206] The visualization unit is configured to determine the details of the violation event, visualize the details of the violation event, and generate a real-time heat map of the target intersection when the target vehicle is determined to be a traffic violation vehicle based on its trajectory. The violation details include the license plate information of the violating vehicle, the time of the violation, the location of the violation, and a video clip of the violation.

[0207] In an optional embodiment, the first model is a model using the MobileNetV3-Small neural network as the backbone network and the Ghost-FPN feature pyramid as the enhanced feature extraction network.

[0208] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0209] The end-cloud collaborative vehicle detection and tracking system in this embodiment is presented in the form of a functional unit, where the unit refers to an application-specific integrated circuit (ASIC) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0210] The present invention also provides a computer device. Figure 4 , Figure 4 Schematic diagram of the hardware structure of the computer device according to the embodiment of the present invention. Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In an optional embodiment, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor device). Figure 4 A processor 10 is taken as an example.

[0211] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0212] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable at least one processor 10 to execute the method shown in the above embodiment.

[0213] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating device, an application required for at least one function. The data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In an optional embodiment, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0214] The memory 20 may include volatile memory, such as random access memory. The memory may also include non-volatile memory, such as flash memory, a hard disk, or a solid-state drive. The memory 20 may also include a combination of the above types of memory.

[0215] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0216] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or downloaded through a network and originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc. Further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0217] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A vehicle detection and tracking method with end-cloud collaboration, characterized in that: include: Determine a current video frame based on the real-time video stream of the target intersection, and detect the current video frame to determine a target detection result, wherein the target detection result includes a target vehicle and the number of vehicles in the current video frame; When the number of vehicles is less than or equal to a first preset threshold, triggering a first working mode; When the number of vehicles is greater than or equal to a second preset threshold, triggering a second operating mode; When the number of vehicles is greater than a first preset threshold and less than a second preset threshold, a third working mode is triggered; the third working mode is the working mode that was triggered most recently; In the first working mode, the target vehicle in the target detection result is tracked according to the first model, and a current first tracking result is determined; the first model is deployed on the edge terminal of the end-cloud collaborative architecture; In the second working mode, the target vehicle in the current video frame is detected and tracked according to the second model, and a current second tracking result is determined; the computing resources occupied by the first model are less than the computing resources occupied by the second model; the second model is deployed in the cloud of the end-cloud collaborative architecture; In the third working mode, the target vehicle in the current video frame is detected, or detected and tracked, based on the most recently triggered working mode, and a current third tracking result is determined.

2. The method according to claim 1, characterized in that Tracking the target vehicle in the target detection result according to the first model includes: Obtain a target list; the target list is obtained by tracking a previous video frame; the previous video frame is a video frame before the current video frame; the target list includes identification information and location information of the target; the target is a vehicle in the previous video frame; the location information of the target includes: location information of a detection box corresponding to the target in the previous video frame; the detection box is marked when the target detection result is determined by detecting the current video frame; a single target corresponds to a single tracking instance; When the target vehicle does not exist in the target list and the number of targets in the target list is less than a first preset number, creating a target tracking instance corresponding to the target vehicle and adding relevant information of the target vehicle to the target list; the relevant information of the target vehicle includes identification information and location information of the target vehicle; the location information of the target vehicle includes: location information of a detection box corresponding to the target vehicle in the current video frame; When the target vehicle does not exist in the target list and the number of targets in the target list is greater than or equal to a first preset number, ending tracking of the target vehicle; When the target vehicle exists in the target list, relevant information of the target vehicle is added to the target list.

3. The method according to claim 2, characterized in that The method further comprises: When there is a first target in the target list, the identification information and location information of the first target in the target list are deleted, and the first tracking instance corresponding to the first target is released; the first target is a target that has not been detected in a second preset number of consecutive video frames.

4. The method according to claim 3, characterized in that The method further comprises: In the second working mode, triggering a delayed exit mechanism; The delayed exit mechanism includes: obtaining a target list; and tracking targets in the target list according to a first model until the target list is empty.

5. The method according to claim 2, characterized in that The method further comprises: Different quantity ranges and tracking processing frequencies corresponding to different quantity ranges are preset; different quantity ranges correspond to different tracking processing frequencies; the quantity range is the quantity range of the tracking instance; Determine a quantity range and a target processing frequency based on the number of current tracking instances, so as to execute the step of tracking the target vehicle in the target detection result according to the first model based on the target processing frequency; the target processing frequency is a tracking processing frequency corresponding to the quantity range to which the number of current tracking instances belongs.

6. The method according to claim 1, characterized in that The method further comprises: Determining a vehicle trajectory of the target vehicle based on the tracking result corresponding to each video frame in the target video stream; the target video stream includes a third preset number of consecutive video frames; Determining whether the target vehicle is a traffic violation vehicle according to the vehicle trajectory of the target vehicle; When the target vehicle is determined to be a traffic violation vehicle based on its vehicle trajectory, details of the traffic violation event are determined, the details of the traffic violation event are visualized, and a real-time heat map of the target intersection is generated; the traffic violation event details include the license plate information of the traffic violation vehicle, the time of the violation, the location of the violation, and a video clip of the violation.

7. The method according to claim 1, characterized in that The first model uses the MobileNetV3-Small neural network as the backbone network and the Ghost-FPN feature pyramid as the enhanced feature extraction network.

8. A vehicle detection and tracking system with end-cloud collaboration, characterized in that: include: Edge terminals, including: An intersection density assessment unit is configured to determine a current video frame based on a real-time video stream of a target intersection, detect the current video frame to determine a target detection result, the target detection result including the target vehicle and the number of vehicles in the current video frame; trigger a first operating mode when the number of vehicles is less than or equal to a first preset threshold; trigger a second operating mode when the number of vehicles is greater than or equal to a second preset threshold; trigger a third operating mode when the number of vehicles is greater than the first preset threshold and less than the second preset threshold; the third operating mode being the most recently triggered operating mode; Unit 1, including the first model; The first unit is configured to track the target vehicle in the target detection result according to the first model in the first working mode and determine a current first tracking result; The first unit is further configured to, in the third operating mode, when the most recently triggered operating mode is the first operating mode, detect the target vehicle in the current video frame based on the most recently triggered operating mode, and determine a current third tracking result; Cloud, including: Unit 2, including the second model; The second unit is configured to detect and track the target vehicle in the current video frame according to the second model in the second working mode, and determine a current second tracking result; the computing resources occupied by the first model are less than the computing resources occupied by the second model; The second unit is further configured to detect and track the target vehicle in the current video frame based on the most recently triggered working mode when the most recently triggered working mode is the second working mode in the third working mode, and determine the current third tracking result.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the end-cloud collaborative vehicle detection and tracking method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the end-cloud collaborative vehicle detection and tracking method according to any one of claims 1 to 7.

Images