Vehicle detection tracking method, device and equipment of quasi-real-time digital twin display

Abstract

The application provides a vehicle detection tracking method, device and equipment for quasi-real-time digital twinborn display, which comprises the following steps: collecting a video stream from a traffic camera; using a detection classification neural network to detect the detection frame of a vehicle in each frame of the video stream in real time, and obtaining the color classification of the vehicle type; matching the vehicle positions between different video frames based on the detection frame information of the vehicle, tracking the path sequence of the vehicle, and recording the vehicle identification serial number; projecting and converting the coordinates of the obtained path sequence to a virtual twinborn space based on the internal and external parameters of the camera, matching the projected trajectory, and setting a unified vehicle identification serial number; performing smoothing filtering and completion on the trajectory in the virtual twinborn space, and outputting the three-dimensional trajectory, vehicle type, color and vehicle serial number of the vehicle with constant delay for the vehicle animation display of the twinborn scene. The application provides tracking data to a digital twinborn system with sub-second delay, and the obtained vehicle driving trajectory is continuous and stable.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of intelligent transportation digital twin technology, in particular to a vehicle detection and tracking method, device and equipment for quasi-real-time digital twin display. BACKGROUND

[0002] In the field of intelligent transportation, the main purpose of digital twin is to make full use of the traffic data equipment deployed on the road, collect and analyze the behavior of traffic participants, and reproduce their full life cycle process in the virtual twin scene. In addition, traffic flow prediction, abnormal behavior capture alarm, etc. are also the pain points and difficulties of intelligent transportation, and digital twin provides the necessary scene and massive real data for realizing the above functions. In traffic digital twin, vehicle tracking based on roadside cameras is an important part of its function.

[0003] The existing vehicle tracking method usually extracts the detection box of the vehicle by the vehicle detector first, and then uses the detection box extracted by the detector in combination with the features and image data to analyze and draw the driving path of the vehicle.

[0004] Digital twin requires not only good accuracy of detection and tracking, but also good reproduction ability of real scene, and uninterrupted processing of collected data for 24 hours a day. Limited by the performance of the detector and the tracking module, most of the existing methods cannot meet the needs of large-scale and long-time development of digital twin. The IoUT-based tracker can track the vehicle trajectory in real time and quickly, but these tracking algorithms are heavily dependent on the accuracy of the detector, and when the detector produces noise, the IoUT-based tracker is prone to interruption, resulting in fragmented trajectories. The SORT-based tracking method uses Kalman filter prediction in combination with the Hungarian algorithm to match vehicle trajectories. This method is suitable for pedestrian tracking, but for fast-moving vehicles, the tracking sequence number of the vehicle is prone to jump due to the speed difference caused by the perspective of the camera. The tracking algorithm using image features often needs to use deep neural networks, which is not efficient. At the same time, because it relies too much on image features, the probability of vehicle error matching is large, and it is not suitable for application in traffic digital twin system.

[0005] In addition, most real-time tracking methods do not use data buffering, and there will be obvious loss tracking and re-identification process during execution, mainly manifested as the sudden disappearance and appearance of the target vehicle, which has an adverse effect on the display effect. For the digital twin display system, a new feasible method is urgently needed. SUMMARY

[0006] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a vehicle detection and tracking method, device and equipment for quasi-real-time digital twin display, which solves the technical problem of poor vehicle detection and tracking effect in the prior art.

[0007] To achieve the above object and other related objects, embodiments of the present application provide a vehicle detection and tracking method for quasi-real-time digital twin display, which comprises: vehicle detection and classification: collecting a video stream from a traffic camera, using a detection and classification neural network to detect a detection frame of a vehicle in each frame of the video stream in real time, and obtaining a vehicle model and color classification; real-time local matching tracking: matching vehicle positions between different video frames based on the detection frame information of the vehicle to track a path sequence of the vehicle and record a vehicle identification number; global matching of the trajectory: projecting and converting coordinates of the path sequence obtained to a virtual twin space based on internal and external parameters of the camera, matching the projected trajectory, and setting a unified vehicle identification number; delay output: smoothing and completing the trajectory in the virtual twin space to output a three-dimensional trajectory, a vehicle model, a color, and a vehicle number of the vehicle with constant delay for vehicle animation display in a twin scene.

[0008] In an embodiment of the present application, in the vehicle detection and classification, the detection and classification neural network detects a detection frame, a target confidence, and a feature vector of all vehicles appearing in each frame of the video stream in real time; semantic information of a vehicle model and a color of the vehicle is obtained based on the feature vector.

[0009] In an embodiment of the present application, in the real-time local matching tracking: the detection result of each frame is used as the input of the detection and classification neural network, the continuity information between the detection frame and the path sequence is used to match the detection frame in real time, and the detection frame is added to the path sequence; the remaining detection frames that are not matched are initialized as a path sequence, and a unique vehicle identification number is configured; the path sequence that is not matched in the local matching is updated in position by using speed prediction and vehicle detection frame, and the matching process is continued in the next frame.

[0010] In an embodiment of the present application, in the global matching of the trajectory: the path sequence is converted to a virtual twin space by using an inverse projection method, each possible path sequence within a certain time after a trajectory interruption is detected based on the continuity of the trajectory in the virtual twin space coordinate system and the consistency of the vehicle features; if a path sequence satisfies the continuity and consistency test, it is considered that the two trajectories belong to the same vehicle, and the same vehicle identification number is assigned.

[0011] In an embodiment of the present application, in the delay output: the delay output and the matching process are performed simultaneously, the path sequence before a longest global matching time is buffered, the trajectory and the feature vector in the twin world coordinate system are read, calculated, and output; the path sequence is smoothed by using a smoothing filter, and the missing trajectory is completed by using an interpolation method; the output feature is calculated incrementally during the tracking process.

[0012] The embodiment of the present application also provides a vehicle detection tracking device of quasi-real-time digital twinborn display, comprising: a vehicle detection classification module, configured to collect a video stream from a traffic camera, detect a detection frame of a vehicle in each frame of the video stream in real time by using a detection classification neural network, and obtain a vehicle type and color classification; a real-time local matching tracking module, configured to match vehicle positions between different video frames based on the detection frame information of the vehicle, track a path sequence of the vehicle, and record a vehicle identification serial number; a global matching module of a trajectory, configured to project and convert coordinates of the path sequence obtained to a virtual twinborn space based on internal and external parameters of the camera, match projected trajectories, and set a unified vehicle identification serial number; and a delay output module, configured to perform smoothing filtering and completion on the trajectories in the virtual twinborn space, and output three-dimensional trajectories, vehicle types, colors, and vehicle serial numbers of the vehicle at a constant delay for vehicle animation display in a twinborn scene.

[0013] In an embodiment of the present application, in the real-time local matching tracking module, the detection result of each frame is taken as an input of the detection classification neural network, the continuity information between the detection frame and the path sequence is used to match the detection frame in real time, and the detection frame is added to the path sequence; the remaining detection frames that are not matched are initialized as a path sequence, and a unique vehicle identification serial number is configured; in the local matching, the path sequence that is not matched is updated in speed prediction and vehicle detection frame position, and the matching process is continued in the next frame.

[0014] In an embodiment of the present application, in the global matching module of the trajectory: the path sequence is converted to the virtual twinborn space by using an inverse projection method; based on the continuity of the trajectory in the virtual twinborn space coordinate system and the consistency of the vehicle characteristics, each possible path sequence within a certain time after the trajectory is interrupted is detected; if a path sequence satisfies the continuity and consistency test, it is considered that the two trajectories belong to the same vehicle, and the same vehicle identification serial number is given.

[0015] In an embodiment of the present application, in the delay output module: the delay output and the matching process are performed simultaneously; the path sequence before a longest global matching time is buffered, and the trajectory and the feature vector in the twinborn world coordinate system are read, calculated and output; the path sequence is smoothed and the missing trajectory is completed by interpolation; and the output feature vector is calculated incrementally in the tracking process.

[0016] The embodiment of the present application also provides an electronic device, comprising a processor and a memory, wherein the memory stores program instructions; and the processor runs the program instructions to implement the vehicle detection tracking method of quasi-real-time digital twinborn display.

[0017] The vehicle detection tracking method, device and equipment of the quasi-real-time digital twin display have the following beneficial effects:

[0018] The application provides tracking data to the digital twin system with sub-second delay, and the obtained vehicle driving track is continuous and stable, and the appearance consistency of the same track vehicle is good. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A flowchart of the vehicle detection tracking method of the quasi-real-time digital twin display is shown.

[0020] Figure 2 A data flow diagram of local track matching in the vehicle detection tracking method of the quasi-real-time digital twin display is shown.

[0021] Figure 3 A data flow diagram of global track matching in the vehicle detection tracking method of the quasi-real-time digital twin display is shown.

[0022] Figure 4 A three-dimensional projection schematic diagram of a traffic camera in the vehicle detection tracking method of the quasi-real-time digital twin display is shown.

[0023] Figure 5 A schematic diagram of spatial continuity of local matching in the vehicle detection tracking method of the quasi-real-time digital twin display is shown.

[0024] Figure 6 A schematic diagram of the principle structure of the vehicle detection tracking device of the quasi-real-time digital twin display is shown.

[0025] Figure 7 A schematic diagram of the structure of an electronic device in an embodiment of the application is shown.

[0026] Element number explanation

[0027] 10 electronic device

[0028] 1101 processor

[0029] 1102 memory

[0030] 100 vehicle detection tracking device of quasi-real-time digital twin display

[0031] 110 vehicle detection classification module

[0032] 120 real-time local matching tracking module

[0033] 130 global matching module of track

[0034] 140 delay output module

[0035] S100-S400 steps DETAILED DESCRIPTION

[0036] Other advantages and benefits of the present application will become apparent to those skilled in the art upon consideration of the disclosure herein. The present application can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Thus, the present disclosure is not intended to limit the application, as the described embodiments are presented solely for purposes of illustration. It is to be understood that features of the embodiments described herein can be combined with features of other embodiments described herein without departing from the spirit or essential characteristics of the present application. It is further understood that the features of the embodiments described herein can be combined with features of other embodiments described herein without departing from the spirit or essential characteristics of the present application.

[0037] The purpose of the present embodiment is to provide a vehicle detection tracking method, device and equipment for quasi-real-time digital twin display, which solves the technical problem of poor vehicle detection tracking effect in the prior art.

[0038] The principles and implementation modes of the vehicle detection tracking method, device and equipment for quasi-real-time digital twin display of the present embodiment will be described in detail below, so that those skilled in the art can understand the vehicle detection tracking method, device and equipment for quasi-real-time digital twin display of the present application without creative labor.

[0039] Example 1

[0040] As shown in Figure 1 The present embodiment provides a vehicle detection tracking method for quasi-real-time digital twin display, comprising:

[0041] Step S100: vehicle detection classification: collecting a video stream from a traffic camera, using a detection classification neural network to detect a detection frame of a vehicle in each frame of the video stream in real time, and obtaining a vehicle type color classification;

[0042] Step S200: real-time local matching tracking: based on the detection frame information of the vehicle, matching the vehicle positions between different video frames to track the path sequence of the vehicle, and recording a vehicle identification serial number;

[0043] Step S300: global matching of trajectory: based on the internal and external parameters of the camera, projecting and converting the coordinates of the acquired path sequence to a virtual twin space, matching the projected trajectory, and setting a unified vehicle identification serial number;

[0044] Step S400: delay output: smoothing and completing the trajectory in the virtual twin space to output the three-dimensional trajectory of the vehicle, the vehicle type, the color, and the vehicle serial number with constant delay for vehicle animation display in the twin scene.

[0045] The steps S100-S400 of the vehicle detection and tracking method shown by the quasi-real-time digital twin in the embodiment are described in detail as follows.

[0046] Step S100: Vehicle detection and classification: Collect video streams from traffic cameras, and use a detection and classification neural network to detect the bounding boxes of vehicles in each frame of the video stream in real time, while obtaining the vehicle type and color classification.

[0047] In step S100, the detection and classification neural network includes but is not limited to networks such as YOLO and SSD that can be used for vehicle classification and detection. Preferably, the neural network can provide an operation rate of at least 25 frames per second on a hardware device, which is comparable to the shooting rate of a traffic camera and can provide real-time detection services.

[0048] For example, YOLO is selected as the vehicle detection neural network in the embodiment. The original version of YOLO uses one-hot encoding to describe the type of target object, and uses a Sigmoid function to calculate the type output of the network. By changing one-hot encoding to multi-hot encoding, an object can belong to multiple categories at the same time.

[0049] After training on public data sets and supplementary data sets, the detection and classification neural network outputs a number of detection boxes for each frame, each detection box containing its position, height and width, confidence, and score of each vehicle type and color classification.

[0050] In the vehicle detection and classification, the detection and classification neural network detects the detection boxes, target confidence and feature vector of all vehicles appearing in each frame of the video stream in real time; and obtains the semantic information of the vehicle type and color based on the feature vector.

[0051] In step S100, a multi-task deep neural network is trained using traffic camera images and labeled data sets. The trained network can provide real-time vehicle detection and classification tasks.

[0052] One of the functions of the neural network is to predict the position of each possible vehicle in the traffic camera image. When the confidence of the predicted vehicle box exceeds a certain set threshold, it is determined that a vehicle appears in the image.

[0053] The second function of the neural network is to extract the features of the vehicle in the detection box. Each dimension of the obtained vehicle feature vector is a score of the possibility of a certain vehicle type or color. For the same vehicle in the same video, the feature vector output by the neural network has macro consistency.

[0054] After the detection and classification module obtains the image from the traffic camera, it detects and outputs the vehicle detection box, vehicle type and color.

[0055] Specifically, each video frame of the traffic camera is captured in real time and then submitted to the network classification detection, which predicts the detection box, target confidence and feature vector of all vehicles appearing in the traffic camera video picture. The feature vector can be obtained by calculating the semantic information of the vehicle model and color. The neural network is trained by traffic camera images and labeled data sets, and has the ability to provide real-time detection services. For the same vehicle in the same video, the predicted feature vector has macro consistency.

[0056] Step S200: Real-time local matching tracking: based on the detection box information of the vehicle, the vehicle position between different video frames is matched to track the path sequence of the vehicle and record the vehicle identification serial number.

[0057] In this embodiment, in the real-time local matching tracking:

[0058] The detection result of each frame is used as the input of the detection classification neural network, the continuity information between the detection box and the path sequence is used to match the detection box in real time, and it is added to the path sequence;

[0059] The remaining unmatched detection boxes are initialized as a path sequence and are configured with a unique vehicle identification serial number;

[0060] The path sequence that is not matched in local matching is updated in position by speed prediction and vehicle detection box, and the above matching process continues in the next frame.

[0061] Specifically, in this embodiment, the detection classification neural network is used to input the detection result of each frame, the continuity information between the detection box and the path sequence is used to match the detection box in real time, and it is added to the path sequence. The remaining unmatched detection boxes are initialized as a path sequence and are configured with a unique vehicle identification serial number. For the path sequence that is not matched in local matching, the position of the vehicle detection box is updated by speed prediction, and the above matching process continues in the next frame. The continuous prediction of the path sequence will only last for a short period of time. After the continuous prediction is stopped due to timeout, the path sequence is discarded according to its length, or enters the global matching process.

[0062] In this embodiment, the input of the matching algorithm of the S200 step is the detection result output by the detection classification network, and the purpose is to connect the vehicle detection box into a path sequence, which can also be called a local matching algorithm.

[0063] The local matching utilizes the continuity information between the path of the vehicle and the detection frame. By scoring the size, distance and other parameters between the detection frames, and finding the matching result with the optimal score, and adding the path sequence, a continuous path sequence is obtained. The remaining detection frames that are not matched are initialized as a path sequence, and a unique vehicle identification number (hereinafter referred to as vehicle ID) is attached.

[0064] Each frame of the local matching may need to track one or more path sequences at the same time, and the local matching algorithm supports matching multiple paths at the same time.

[0065] When the above path sequence is interrupted due to loss of detection or matching failure, the algorithm predicts the driving direction of the path according to the speed at the time of interruption, and updates the position of the vehicle frame in real time, and continues the above matching process with the next frame data.

[0066] The prediction process will continue for a short time to avoid large errors. If the path sequence stops tracking due to continuous prediction timeout, and the path length exceeds a certain threshold, it is considered to be a real existing path, which enters S400 and is recorded as the first trajectory. Otherwise, discard the path.

[0067] Specifically, in the real-time local matching tracking of the embodiment, DIoU (distance-intersection ratio) is used as an important indicator of the continuity of real-time local matching, as shown in Figure 2 The DIoU calculates the overlap between two detection frames and the difference between the normalized distance squares, and the calculation formula is A and B are two detection frames, and their center points are c A and c B , c represents the diagonal distance of the smallest rectangular region that can contain A and B. By using the greedy algorithm or the Hungarian algorithm, the continuity matching of multiple paths and detection frames can be obtained, and the continuous path sequence is obtained. When the continuity is higher than a certain threshold, it is considered that the detection frame corresponds to a certain path, and it is added to the path sequence.

[0068] After losing tracking, a preferred way is to use linear Kalman filtering to continuously predict the path. Kalman filtering assumes that the center point of the detection frame moves at a constant speed, and uses a linear equation to continuously update the position of the detection frame. Each local matching uses the updated detection frame and the new detection frame of the current frame.

[0069] The predicted path has a lower priority in local matching. The matching process is divided into three stages. The first stage uses local matching for the tracked path and the newly detected bounding boxes in the current frame. The second stage uses local matching for the predicted path and the remaining bounding boxes. The last stage initializes the remaining isolated bounding boxes as new paths and attaches a vehicle identification number (vehicle ID) to them. This is because the predicted path is more unstable and more likely to cause false matching and interruption.

[0070] The prediction process in step S200 continues for about 0.5 seconds. If the length of the terminated path sequence is not greater than a threshold value, it is considered to be an unstable path and is discarded. The remaining paths enter the global matching stage in step S300 and are recorded as the first trajectory.

[0071] Step S300: Global matching of trajectories: Based on the internal and external parameters of the camera, the coordinates of the acquired path sequence are projected and converted to a virtual twin space. The projected trajectories are matched, and a unified vehicle identification number is set.

[0072] In this embodiment, in the global matching of trajectories:

[0073] The path sequence is converted to a virtual twin space using the inverse projection method. Based on the continuity of the trajectories in the virtual twin space coordinate system and the consistency of the vehicle features, each possible path sequence within a certain time after the interruption of the trajectory is detected.

[0074] If a path sequence satisfies the continuity and consistency test, it is considered that the two trajectories belong to the same vehicle and are assigned the same vehicle identification number.

[0075] In the global matching, the path sequence is converted to a virtual twin space using the inverse projection method. Based on the continuity of the trajectories in the virtual twin space coordinate system and the consistency of the vehicle features, each possible path sequence within a certain time after the interruption of the trajectory is checked. If a path sequence satisfies the continuity and consistency test, it is considered that the two trajectories belong to the same vehicle. The same vehicle identification number is assigned. Global matching does not affect the stage in which the second trajectory is located. It will continue to perform local or global matching and be connected as a longer path.

[0076] The purpose of step S300 is to connect two possible path sequences, which can also be called the global matching stage. In addition to using the information of the detected bounding boxes and trajectories of the vehicle in the previous path, the algorithm also uses the features of the vehicle for further discrimination.

[0077] The global matching phase is also performed in real time. For the first trajectory that stops tracking in step S300, global matching will transform the screen trajectory to a twin space coordinate system based on the mapping relationship between traffic cameras and the real world, and check the trajectory continuity and vehicle feature consistency of each possible path sequence within a certain period of time after the trajectory is interrupted.

[0078] If a path sequence satisfies the above continuity consistency check, it is designated as the second trajectory, and these two trajectories are considered to belong to the same vehicle. The vehicle ID of the second trajectory will be modified to match that of the first trajectory. Global matching will not affect the stage of the second trajectory; it will continue to undergo local or global matching and connect to form a longer path.

[0079] Specifically, in the global matching of the trajectory described in this embodiment: within a few seconds after the first trajectory stops tracking, paths whose length reaches a threshold will be included in the calculation of global matching with the first trajectory. Global matching utilizes spatial continuity and consistency information of vehicle features, such as... Figure 3 As shown, spatial continuity and vehicle feature consistency have their own calculation methods, and then their probability product is used as the matching criterion.

[0080] To avoid changes in image speed caused by camera perspective, the path sequence should be inversely projected onto a virtual spatial coordinate system before calculating spatial continuity. The example assumes the camera's 3D projection method is as follows: Figure 4 As shown, the midpoint p′ of the vehicle detection frame is [uv]. T By using the camera's calibration parameters, the inverse projection mapping can be obtained, and then the approximate virtual scene coordinates p = [xyz] of the vehicle can be calculated. T .

[0081] Spatial continuity requires that the ratio of the vehicle's distance to its speed near the break point be approximately equal to the break time. This assumption is based on the premise that the vehicle's speed remains essentially constant. By analyzing the positions of the bounding boxes in the path sequence, the tail velocity and head velocity can be roughly calculated. The tail velocity of the previous trajectory and the head velocity of the next trajectory can be used to predict the position of a point in the path. The spatial continuity of these two trajectories can be represented using the normal distribution probabilities of the predicted two points. Figure 5 As shown.

[0082] Meanwhile, since the position of a single point cannot reflect the directionality of the path, the embodiment uses the dot product d of the velocity directions at both ends as another indicator for direction determination. As its probability distribution density function.

[0083] In addition, the embodiment also uses the normal distribution probability of the Euclidean distance between vehicle feature vectors to calculate the vehicle feature consistency between trajectories. While tracking the vehicle detection frame, the feature vector of each trajectory can be accumulated with the recognition confidence as the weight, and the continuously updated feature vector is used for the Euclidean distance calculation between the path sequences. When considering that two trajectories match, the feature vectors of the two trajectories are further fused according to the recorded weights, and the tracking process continues. The weighted average process cannot be omitted, because the feature vector output by the classification network has obvious noise, and the weighted average process makes the vehicle type and color output by the algorithm relatively stable, and ensures the effect of the quasi-real-time display.

[0084] The second trajectory matched globally is attached with the same vehicle ID as the first trajectory.

[0085] Step S400: Delayed output: smoothing filtering and completion are performed on the trajectories in the virtual twin space, and the three-dimensional trajectories, vehicle types, colors, and vehicle serial numbers of the vehicles are output with constant delay for the vehicle animation display in the twin scene.

[0086] In the embodiment, in the delayed output:

[0087] The delayed output and the matching process are performed simultaneously, the buffer of the path sequence before the longest global matching time is checked, the trajectories and feature vectors in the twin world coordinate system are read, calculated and output;

[0088] The delayed output uses smoothing filtering on the path sequence, and the missing trajectories are completed in an interpolation manner;

[0089] The output feature vector is calculated incrementally in the tracking process.

[0090] That is, in the embodiment, the delayed output and the matching process are performed simultaneously, the algorithm checks the buffer of the path sequence before the longest global matching time, reads, calculates and outputs the trajectories and feature vectors in the twin world coordinate system, and the delayed output makes the matching process invisible when the path is interrupted for the user. The delayed output uses smoothing filtering on the path sequence, and the missing trajectories are completed in an interpolation manner, the output feature vector is calculated incrementally, and the jump of the vehicle type and color in this step is reduced. The above delayed output mode can provide the virtual twin display system with the data required for display in real time, including the three-dimensional twin space coordinates of the vehicle, the vehicle identification serial number, the vehicle type and the color.

[0091] Step S400 is performed simultaneously with the trajectory matching. According to the characteristics of the matching algorithm, the trajectory will have a certain interruption time before being assigned a vehicle identification number, being predicted in real time, or being globally delayed matched. The vehicle trajectory or feature in this part of time is unstable, and therefore cannot be used for twin display. The purpose of the delay output is to mask the instability of the tracking algorithm and achieve the best display effect. According to the size of the set parameters, generally speaking, global delay matching has the longest delay. Delaying the output of the vehicle trajectory by the same time difference can make the user unable to perceive the matching process when the path is interrupted.

[0092] Due to the output requirements of step S400, the results of tracking need to be temporarily stored in a buffer area, and then read, calculated, and output the trajectory and feature vector in the twin world coordinate system by the output module. The output trajectory has been smoothed and interpolated, and is relatively stable. The output feature is calculated incrementally during the tracking process, and is not prone to jump.

[0093] In this embodiment, step S400 is performed simultaneously with steps S200 and S300. Preferably, all trajectory sequences of this embodiment maintain a first-in-first-out buffer queue internally. The output module checks the data at the tail of each buffer queue in each frame, finds a path sequence before the global matching time, and extracts the vehicle position information, vehicle type, color, and vehicle ID after inverse projection, filtering, and interpolation. Since the algorithm ensures that the tracking results of the path are determined after the global matching time ends, the above-mentioned delayed output path and offline processing results should be the same. The semantic features of the vehicle (vehicle type, color) are obtained by incrementally calculating the feature vector, and the possibility of jump in the middle is also small.

[0094] The tracking system of this example communicates with the twin display system in the form of sending TCP packets, and needs to send data packets containing all vehicle information in this time period to the twin system at a constant frequency. The process of sending and receiving data packets will introduce tens to hundreds of milliseconds of delay. However, overall, it can be considered that the output delay of this system is sub-second relative to the real time.

[0095] Example 2

[0096] As shown in Figure 6 The vehicle detection and tracking device 100 of this embodiment is a quasi-real-time digital twin display vehicle detection and tracking device 100, which includes a vehicle detection and classification module 110, a real-time local matching and tracking module 120, a trajectory global matching module 130, and a delay output module 140.

[0097] In this embodiment, the vehicle detection and classification module 110 is configured to collect video streams from traffic cameras, detect the bounding boxes of vehicles in each frame of the video streams in real time using a detection and classification neural network, and obtain the vehicle type and color classification.

[0098] The technical features of the specific implementation of the vehicle detection and classification module 110 are basically the same as those of step S100 in the quasi-real-time digital twin displayed vehicle detection and tracking method in Embodiment 1. The technical content common to the embodiments is not repeated.

[0099] In this embodiment, the real-time local matching and tracking module 120 is configured to match the vehicle positions between different video frames based on the bounding box information of the vehicles, track the path sequence of the vehicles, and record the vehicle identification sequence number.

[0100] The technical features of the specific implementation of the real-time local matching and tracking module 120 are basically the same as those of step S200 in the quasi-real-time digital twin displayed vehicle detection and tracking method in Embodiment 1. The technical content common to the embodiments is not repeated.

[0101] In this embodiment, the trajectory global matching module 130 is configured to project and convert the coordinates of the path sequence obtained to a virtual twin space based on the internal and external parameters of the cameras, match the projected trajectories, and set a unified vehicle identification sequence number.

[0102] The technical features of the specific implementation of the trajectory global matching module 130 are basically the same as those of step S300 in the quasi-real-time digital twin displayed vehicle detection and tracking method in Embodiment 1. The technical content common to the embodiments is not repeated.

[0103] In this embodiment, the delay output module 140 is configured to perform smoothing and completion on the trajectories in the virtual twin space, and output the three-dimensional trajectories, vehicle types, colors, and vehicle sequence numbers of the vehicles with constant delay for vehicle animation display in the twin scene.

[0104] The technical features of the specific implementation of the delay output module 140 are basically the same as those of step S400 in the quasi-real-time digital twin displayed vehicle detection and tracking method in Embodiment 1. The technical content common to the embodiments is not repeated.

[0105] It should be noted that the division of the various modules of the above apparatus is only a logical functional division, and all or part of the modules can be integrated into one physical entity or physically separated in actual implementation. The modules can all be implemented in the form of software invoked by a processing element, all be implemented in the form of hardware, or part of the modules be implemented in the form of software invoked by a processing element and part of the modules be implemented in the form of hardware. For example, the vehicle detection and classification module 110 can be a separately established processing element, or can be integrated in a certain chip of the electronic terminal, and in addition, the vehicle detection and classification module 110 can also be stored in the form of program code in the memory of the terminal and invoked and executed by a certain processing element of the terminal to implement the functions of the above tracking calculation module. The implementation of other modules is similar. In addition, all or part of the modules can be integrated together or independently implemented. The processing element described herein can be an integrated circuit having a signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of hardware in the processing element or the instruction in the form of software.

[0106] For example, the above modules can be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), etc. For another example, when a certain module is implemented in the form of program code invoked by a processing element, the processing element can be a general-purpose processor such as a central processing unit (CPU) or other processor capable of invoking program code. For another example, the modules can be integrated together to implement in the form of a system on a chip (SOC).

[0107] Example 3

[0108] As shown in Figure 4 The electronic device 10 provided in the embodiment includes a processor 1101 and a memory 1102.

[0109] The electronic device 100 can be, for example, a fixed terminal such as a server, a desktop computer, etc., or a mobile terminal such as a notebook computer, a smart phone, a tablet computer, etc., or a vehicle-mounted terminal, etc.

[0110] The memory 1102 is connected with the processor 1101 through the system bus and completes mutual communication, the memory 1102 is used for storing the computer program, the processor 1101 is coupled to the display 1003 and the memory 1002, the processor 1101 is used for running the computer program, to make the electronic equipment 10 execute the vehicle detection tracking method of the quasi real-time digital twin display described in embodiment 1.The quasi real-time digital twin display vehicle detection tracking method has been described in detail in embodiment 1, and will not be repeated here.

[0111] The quasi real-time digital twin display vehicle detection tracking method can be applied to various types of electronic equipment 10.In an exemplary embodiment, the electronic equipment 10 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, cameras or other electronic components for executing the above-mentioned quasi real-time digital twin display vehicle detection tracking method.

[0112] The memory can include random access memory (RAM) and can also include non-volatile memory, such as at least one disk memory.

[0113] The processor 1101 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP) and the like; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0114] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by computer program-related hardware.The aforementioned computer program can be stored in a computer-readable storage medium.The program, when executed, performs steps including the above-mentioned method embodiments; and the aforementioned storage medium includes ROM, RAM, magnetic or optical disk and various storage medium that can store program codes.

[0115] In summary, the application can directly recognize vehicles in traffic monitoring videos, and effectively improve the accuracy of vehicle re-identification.

[0116] The above embodiments only exemplarily illustrate the principles and effects of the application, and are not used to limit the application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought disclosed by the application should be covered by the claims of the application.

Claims

1. A vehicle detection tracking method of quasi-real-time digital twin display, characterized in that: Comprising: Vehicle detection classification: collecting video stream from traffic camera, using detection classification neural network to detect bounding box of vehicle in each frame of the video stream in real time, and obtaining vehicle model and color classification at the same time; Real-time local matching tracking: based on the bounding box information of the vehicle, matching the position of the vehicle between different video frames to track the path sequence of the vehicle and record the vehicle identification number; in the real-time local matching tracking, the distance-intersection ratio is used as the continuity index of real-time local matching; the distance-intersection ratio calculates the coincidence and normalized distance square difference between two bounding boxes; Global matching of trajectory: based on the internal and external parameters of the camera, projecting and converting the coordinates of the obtained path sequence to a virtual twin space, matching the projected trajectories, and setting a unified vehicle identification number; in the global matching of trajectory: using the inverse projection method to convert the path sequence to the virtual twin space, based on the continuity of the trajectory in the virtual twin space coordinate system and the consistency of the vehicle features, detecting each possible path sequence after the interruption of the trajectory; if a path sequence satisfies the continuity and consistency test, it is considered that the two trajectories belong to the same vehicle and are assigned the same vehicle identification number; the product of the spatial continuity and the consistency of the vehicle features is used as the matching standard; the tail speed of the previous trajectory and the head speed of the next trajectory are used to estimate the position of a certain point in the path, and the normal distribution probability of the two estimated points is used to represent the spatial continuity of the two trajectories; wherein, the normal distribution probability of the Euclidean distance between the vehicle feature vectors is used to calculate the consistency of the vehicle features between the trajectories; Delay output: smoothing and completing the trajectories in the virtual twin space to output the three-dimensional trajectory, vehicle model, color, and vehicle number of the vehicle with constant delay for vehicle animation display in the twin scene; in the delay output: the delay output is performed simultaneously with the matching process, the path sequence buffer before the longest global matching time is detected, and the trajectory and feature vector in the twin world coordinate system are read, calculated, and output; the delay output uses smoothing filtering on the path sequence and completes the missing trajectory in an interpolation manner; the output feature is calculated incrementally during the tracking process.

2. The quasi-real-time digital twin display vehicle detection tracking method of claim 1, wherein: In the vehicle detection classification, the detection classification neural network detects the bounding box, target confidence, and feature vector of all vehicles appearing in each frame of the video stream in real time; based on the feature vector, the semantic information of the vehicle model and color is obtained.

3. The quasi-real-time digital twin display vehicle detection tracking method of claim 1 or 2, wherein: In the real-time local matching tracking: The detection results of each frame are used as the input of the detection classification neural network, the continuity information between the detection box and the path sequence is used to match the detection box in real time, and the matched detection box is added to the path sequence; The remaining unmatched detection boxes are initialized as a path sequence and are configured with a unique vehicle identification number; The path sequence that is not matched in the local matching is updated by predicting the speed and updating the position of the vehicle detection box, and the above matching process is continued in the next frame.

4. A vehicle detection tracking apparatus for quasi-real-time digital twin display, characterized by: Comprising: A vehicle detection classification module is configured to collect a video stream from a traffic camera, and use a detection classification neural network to detect a detection frame of a vehicle in each frame of the video stream in real time, and obtain a vehicle model and color classification. A real-time local matching tracking module is configured to match vehicle positions between different video frames based on the detection frame information of the vehicle, track a path sequence of the vehicle, and record a vehicle identification sequence number. In the real-time local matching tracking, a distance-intersection-over-union is used as an index of continuity of real-time local matching. The distance-intersection-over-union calculates a difference between a coincidence degree and a normalized distance square of two detection frames. A global matching module of a trajectory is configured to project and convert coordinates of the path sequence to a virtual twin space based on internal and external parameters of the camera, match the projected trajectory, and set a unified vehicle identification sequence number. In the global matching module of the trajectory, an inverse projection method is used to convert the path sequence to the virtual twin space, a continuity of the trajectory in the virtual twin space coordinate system and a consistency of vehicle features are used to detect each possible path sequence after a break in the trajectory, if a path sequence satisfies the continuity and consistency test, it is considered that the two trajectories belong to the same vehicle, and the same vehicle identification number is assigned, a product of a probability of spatial continuity and a probability of consistency of vehicle features is used as a matching standard, a tail speed of a previous trajectory and a head speed of a next trajectory are used to estimate positions of a certain point in the path, and a normal distribution probability of the two estimated points is used to represent spatial continuity of the two trajectories, and a normal distribution probability of a Euclidean distance between vehicle feature vectors is used to calculate the consistency of vehicle features between the trajectories. A delay output module is configured to perform smoothing filtering and completion on the trajectory in the virtual twin space, output a three-dimensional trajectory, a vehicle model, a color, and a vehicle sequence number of the vehicle with a constant delay, and display a vehicle animation in a twin scene. In the delay output module, the delay output is performed simultaneously with the matching process, a buffer of path sequences before a longest global matching time is detected, trajectories and feature vectors in a twin world coordinate system are read, calculated, and output, the path sequences are smoothed and interpolated to complete the missing trajectories, and the output features are calculated incrementally in the tracking process.

5. The quasi-real-time digital twin display vehicle detection tracking apparatus of claim 4, wherein: In the real-time local matching tracking module, detection results of each frame are used as inputs of the detection classification neural network, the continuity information between the detection frame and the path sequence is used to match the detection frame in real time, and the detection frame is added to the path sequence, the remaining unmatched detection frames are initialized as a path sequence, and a unique vehicle identification sequence number is configured, the unmatched path sequence in the local matching is updated by predicting a speed and updating a position of the vehicle detection frame, and the matching process is continued in the next frame.

6. An electronic device, comprising: A processor and a memory are included, the memory stores program instructions, and the processor runs the program instructions to implement the vehicle detection tracking method of the quasi-real-time digital twin display.

Images