A method and related equipment for vehicle trajectory extraction based on video stitching

Abstract

This invention provides a vehicle trajectory extraction method and related equipment based on video stitching, comprising: Step 1, acquiring multiple traffic video segments within a target traffic area; Step 2, detecting target vehicles in each traffic video segment to obtain multiple detection results, and tracking the target vehicle in each detection result to obtain the trajectory coordinates of the target vehicle in each traffic video segment; Step 3, extracting the trajectory of the target vehicle based on the trajectory coordinates in each traffic video segment to obtain the driving trajectory of the target vehicle in each traffic video segment; Step 4, matching and stitching the driving trajectory in the current traffic video segment with the driving trajectory in the current traffic video segment to obtain the trajectory stitching result of the target vehicle during its driving process. Compared with the prior art, this method improves the accuracy and stability of trajectory extraction and has good scalability.

Description

Technical Field

[0001] This invention relates to the field of vehicle trajectory extraction technology, and in particular to a vehicle trajectory extraction method and related equipment based on video stitching. Background Technology

[0002] Intelligent connected vehicles are a key development direction for the future automotive industry. This year, national authorities, including the Ministry of Industry and Information Technology, the Ministry of Transport, the Ministry of Public Security, and the State Administration for Standardization, have jointly formulated industrial policies for intelligent connected vehicles to accelerate their development and the construction of related infrastructure. Simultaneously, with the maturity of 5G technology, the speed of transmitting large amounts of data in intelligent vehicles has greatly increased, ensuring real-time updates of in-vehicle navigation information. The decentralized, immutable, and transparent characteristics of blockchain, applied to the entire process of vehicle big data processing in the era of intelligent driving, can effectively enhance the overall network security of intelligent connected vehicles. Advanced sensor technology drives the development of ADAS (Advanced Driver Assistance Systems). In short, the rapid advancements in 5G, blockchain, and sensor technologies provide a solid technological foundation and national policy support for the development of intelligent connected vehicles. Therefore, the market size and prospects for intelligent connected vehicles in my country are very promising.

[0003] Despite the promising future of intelligent connected vehicles in addressing traffic issues, their current development and implementation face numerous critical challenges, the most fundamental being the operational safety of intelligent connected vehicles themselves. Current methods for assessing the operational safety of intelligent connected vehicles fall into two main categories: one based on real vehicle experiments, and the other based on simulation-environment-based assessments of intelligent connected vehicle controller models. Real vehicle-based assessments more closely resemble the traffic flow environment, resulting in more reliable results. However, this method is costly, inefficient, and carries inherent safety risks during testing. Simulation-environment-based assessments offer advantages such as safety, efficiency, and low cost, serving as an effective supplement to real vehicle experiments. The reliability of these results depends on the realism and typicality of the simulated vehicle operating environment. Most existing simulation studies provide computer-simulated background traffic environments, using numerical simulations to analyze and assess the operational safety of intelligent connected vehicles. However, the test results are often not entirely convincing. To address this, researchers acquired trajectory data of manually driven vehicles through video capture and processing using plastic tubes. They then applied automatic segmentation and discrimination techniques for complex traffic scenarios to construct a relatively realistic operating environment for the controllers of the intelligent connected vehicles under evaluation, thereby enabling a safety assessment of the vehicles' operational status. However, because the natural vehicle trajectory data used for safety assessment of intelligent connected vehicles has a relatively small spatiotemporal range, it sometimes fails to reflect the vehicle's traffic operation patterns, which can negatively impact the safety assessment test results. Summary of the Invention

[0004] This invention provides a vehicle trajectory extraction method and related equipment based on video stitching, with the aim of improving the accuracy and stability of trajectory extraction.

[0005] To achieve the above objectives, the present invention provides a vehicle trajectory extraction method based on video stitching, comprising:

[0006] Step 1: Collect multiple traffic video clips within the target traffic area;

[0007] Step 2: Detect target vehicles for each traffic video segment to obtain multiple detection results, and track the target vehicle for each detection result to obtain the trajectory coordinates of the target vehicle in each traffic video segment.

[0008] Step 3: Extract the trajectory of the target vehicle based on the trajectory coordinates in each traffic video segment to obtain the driving trajectory of the target vehicle in each traffic video segment.

[0009] Step 4: For each traffic video segment, match and stitch the driving trajectory in the current traffic video segment with the driving trajectory in the adjacent video segment to obtain the trajectory stitching result of the target vehicle during the driving process.

[0010] Furthermore, the process before step 2 also includes:

[0011] For each traffic video segment, the traffic video is broken down into multiple frames.

[0012] Preprocess each frame of the image;

[0013] Each preprocessed frame is synthesized to obtain multiple preprocessed traffic videos.

[0014] Target vehicle detection is performed on multiple pre-processed traffic video segments.

[0015] Furthermore, step 2 includes:

[0016] Multiple preprocessed traffic videos were input into the YOLO4 model for object detection, resulting in multiple detection results.

[0017] The Deep SORT algorithm is used to perform cascaded matching on each detection result to obtain multiple first matching results;

[0018] Perform IOU matching on each first matching result to obtain multiple second matching results;

[0019] Multiple first matching results and multiple second matching results are input into the Kalman update equation for correction;

[0020] The corrected first matching results and the corrected second matching results are input into the Kalman prediction equation for prediction, and the trajectory coordinates of the target vehicle in each traffic video segment are obtained.

[0021] Furthermore, step 3 includes:

[0022] The target vehicles include multiple vehicles;

[0023] For each target vehicle, the trajectory coordinates of the target vehicle in each traffic video segment are transformed to the world coordinate system to obtain the trajectory coordinates in the world coordinate system;

[0024] The trajectory of the target vehicle is extracted based on the trajectory coordinates in the world coordinate system, and the driving trajectory of each target vehicle in each traffic video segment is obtained.

[0025] Furthermore, step 4 includes:

[0026] For each traffic video segment, the driving trajectory of all target vehicles is calculated. Taking the first traffic video segment as the current traffic video segment, the longitudinal distance corresponding to the last time frame of each driving trajectory of all target vehicles is calculated from the current traffic video segment.

[0027] The longitudinal distance of the target vehicle leaving the image area is filtered out from the longitudinal distance corresponding to the last time frame, and the first average value is calculated from the longitudinal distance of the target vehicle leaving the image area.

[0028] For each traffic video segment, the longitudinal distance corresponding to the starting time frame of each target vehicle's trajectory is calculated from the next traffic video segment adjacent to the current traffic video segment.

[0029] The longitudinal distance of the target vehicle entering the image region is selected from the longitudinal distance corresponding to the starting time frame, and the longitudinal distance of the target vehicle entering the image region is calculated to obtain the second average value;

[0030] Based on the first average value and the second average value, the trajectory overlap range between the current traffic video segment and the next traffic video segment is obtained;

[0031] The time frame vectors of the driving trajectory of each target vehicle within the trajectory overlap interval are selected, and the average value of the time frame vectors with respect to the trajectory overlap interval is calculated.

[0032] Add a time frame attribute for trajectory matching to the driving trajectory of each target vehicle, which constitutes the first matching tuple of the current traffic video segment and the second matching tuple of the next traffic video segment;

[0033] The cost matrix is ​​obtained by calculating the first and second matching tuples using Mahalanobis distance;

[0034] Based on the cost matrix, the Hungarian algorithm is used to match the trajectories, and the matched trajectories are obtained.

[0035] The matched trajectories are then stitched together to obtain the trajectories of all target vehicles during their journey.

[0036] Furthermore, the first matching tuple of the current traffic video segment is:

[0037] match_object_pre=(mean_x_pre,mean_y_pre,mean_v_pre,match_frame_pre)

[0038] Where match_frame_pre represents the matching time frame attribute value of the trajectory in the current segment of traffic video, and mean_x_pre, mean_y_pre, and mean_v_pre represent the average value of the trajectory overlap interval in the current segment of traffic video.

[0039] Furthermore, the second matching tuple for the next traffic video segment is:

[0040] match_object_last=(mean_x_last,mean_y_last,mean_v_last,match_frame_last)

[0041] Where match_frame_last represents the matching time frame attribute value of the trajectory in the next traffic video segment, and mean_x_last, mean_y_last, and mean_v_last represent the average values ​​of the overlapping intervals of the trajectories in the next traffic video segment.

[0042] The present invention also provides a vehicle trajectory extraction device based on video stitching, comprising:

[0043] The acquisition module is used to acquire multiple traffic video clips within the target traffic area;

[0044] The detection and tracking module is used to detect target vehicles in each traffic video segment, obtain multiple detection results, and track the target vehicle in each detection result to obtain the trajectory coordinates of the target vehicle in each traffic video segment.

[0045] The trajectory extraction module is used to extract the trajectory of the target vehicle based on the trajectory coordinates in each traffic video segment, thereby obtaining the driving trajectory of the target vehicle in each traffic video segment.

[0046] The matching and stitching module is used to match and stitch the driving trajectory in the current traffic video segment with the driving trajectory in the adjacent video segment to obtain the trajectory stitching result of the target vehicle during the driving process.

[0047] The present invention also provides a computer-readable storage medium for storing a computer program, which, when executed, enables a method for extracting vehicle trajectories based on video stitching.

[0048] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a vehicle trajectory extraction method based on video stitching.

[0049] The above-described solution of the present invention has the following beneficial effects:

[0050] This invention acquires multiple traffic video segments within a target traffic area; detects target vehicles in each segment, obtaining multiple detection results; tracks each detection result to obtain the trajectory coordinates of the target vehicle in each traffic video segment; extracts the target vehicle's trajectory based on the trajectory coordinates in each traffic video segment, obtaining the target vehicle's driving trajectory in each traffic video segment; and matches and stitches the driving trajectory in the current traffic video segment with the driving trajectories in adjacent video segments to obtain the stitched trajectory result of the target vehicle during its journey. Compared with existing technologies, this invention improves the accuracy and stability of trajectory extraction while exhibiting good scalability. By extracting and stitching the target vehicle's trajectory from multiple video segments, the stitched vehicle trajectory has a wider spatial coverage, and the longer vehicle trajectory can effectively reflect the traffic operation patterns of vehicles, making it of significant data support value for traffic flow theory research and intelligent connected vehicle safety assessment.

[0051] Other beneficial effects of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0053] Figure 2 This is a flowchart of the Kalman filtering algorithm in an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the trajectory extraction process in an embodiment of the present invention;

[0055] Figure 4 This is a flowchart illustrating the trajectory matching process in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of trajectory stitching according to an embodiment of the present invention. Detailed Implementation

[0057] To make the technical problems, solutions, and advantages of this invention clearer, a detailed description will be provided below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0058] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0059] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a locking connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0060] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0061] This invention addresses existing problems by providing a vehicle trajectory extraction method and related equipment based on video stitching.

[0062] like Figure 1 As shown, an embodiment of the present invention provides a vehicle trajectory extraction method based on video stitching, including:

[0063] Step 1: Collect multiple traffic video clips within the target traffic area;

[0064] Step 2: Detect target vehicles for each traffic video segment to obtain multiple detection results, and track the target vehicle for each detection result to obtain the trajectory coordinates of the target vehicle in each traffic video segment.

[0065] Step 3: Extract the trajectory of the target vehicle based on the trajectory coordinates in each traffic video segment to obtain the driving trajectory of the target vehicle in each traffic video segment.

[0066] Step 4: For each traffic video segment, match and stitch the driving trajectory in the current traffic video segment with the driving trajectory in the adjacent video segment to obtain the trajectory stitching result of the target vehicle during the driving process.

[0067] In this embodiment of the invention, step 1 specifically includes:

[0068] First, identify the target traffic area for data collection. Prepare multiple DJI Mavic 2 drones. The video acquisition personnel also need to go to the video shooting location to measure the real coordinates of landmarks with obvious characteristics for subsequent pixel coordinate to real coordinate conversion. During the shooting process, multiple people need to operate the drones simultaneously to fly to a certain altitude above the target traffic area and ensure that the areas shot by each drone overlap. Finally, when shooting the video, all drones need to be synchronized in time. By following the above operations, multiple traffic video segments that are synchronized in time can be collected.

[0069] Specifically, the steps preceding step 2 also include:

[0070] For each traffic video segment, the traffic video is broken down into multiple frames.

[0071] Preprocess each frame of the image;

[0072] Each preprocessed frame is synthesized to obtain multiple preprocessed traffic videos.

[0073] Target vehicle detection is performed on multiple pre-processed traffic video segments.

[0074] In this embodiment of the invention, traffic videos are collected by a camera installed on a drone. Although the distortion is not as severe as that of images taken by a fisheye camera, distortion does exist, but it is difficult for the human eye to perceive. If images with distortion are used for target detection and tracking, the accuracy of detection and tracking will be greatly affected. Therefore, the first step in video preprocessing is distortion removal.

[0075] First, camera parameter calibration is required. A commonly used calibration method is the Zhang Zhengyou calibration method. The camera captures multiple images (usually around 20) of the chessboard calibration board from different angles. Then, the OpenCV function `cv2.calibrateCamera(*)` is used to obtain the camera's distortion coefficients. Finally, these distortion coefficients are passed to the function `cv2.undistort(*)` to correct the distortion in the drone images. This yields the drone video after distortion correction. The distorted video then undergoes histogram equalization to further improve video quality, resulting in multiple pre-processed traffic video segments.

[0076] Specifically, step 2 includes:

[0077] Multiple preprocessed traffic videos were input into the YOLO4 model for object detection, resulting in multiple detection results.

[0078] The Deep SORT (Deep Simple Online and Realtime Tracking) real-time tracking algorithm is used to perform cascade matching on each detection result to obtain multiple first matching results;

[0079] For each first match result, perform an IOU (Intersection of Union) match to obtain multiple second match results;

[0080] Multiple first matching results and multiple second matching results are input into the Kalman update equation for correction;

[0081] The corrected first matching results and the corrected second matching results are input into the Kalman prediction equation for prediction, and the trajectory coordinates of the target vehicle in each traffic video segment are obtained.

[0082] In this embodiment of the invention, object detection is defined as identifying the type of a target object in an image and simultaneously determining its location. It is a hot topic in computer vision. In deep learning, object detection is mainly divided into two types: single-stage models and two-stage models. Single-stage models mainly include SSD (Single Shot MultiBox Detector) and the YOLO (You Only Look Once) series of models; two-stage models are mainly represented by RCNN (Region-based Convolutional Neural Networks) series of object detection models. Since the YOLO series of models are end-to-end models with advantages such as simple training, fast detection speed, and high detection accuracy, this embodiment of the invention uses the YOLOv4 model for object detection in drone videos. The YOLOv4 model mainly consists of three parts: the backbone network for feature extraction, the Neck network layer for collecting feature maps at different stages, and the Head detection head for predicting the type and location of the target. The backbone network uses CSPDarknet53, which is mainly composed of five layers of stacked residual units (resblock_body). The input image pixel size is 608*608. The stacked residual units (resblock_body) have dedicated convolution operations to reduce the resolution, with each layer gradually reducing the pixel count by half. Its main function is to extract feature information from the image data. The Neck network layer mainly consists of SPP (Spatial Pyramid Pooling) (from Conv×3 to Concat+Conv×3 in the lower left part) and PANet (Path Aggregation). The network consists of a path aggregation network; SPP's main function is to increase the receptive field; PANet's main function is to convert the extracted feature information into coordinates, categories, etc., and it is mainly composed of upsampling and downsampling; the head detection head continues the detection head of YOLOv3, and its three detection heads have the following main functions: after obtaining the output, compare it with the real data annotation, calculate the loss function (which is mainly composed of three parts: ① localization loss ② confidence loss ③ classification loss), and then reshape the data format as needed, while activating the original grid coordinates accordingly.

[0083] Specifically, multiple preprocessed traffic video segments are input into the YOLO4 model's backbone network for feature extraction. The extracted results are then input into the Neck network layer for feature fusion. The fused features are then input into the Head detection head to predict the type and location of the target, resulting in multiple detection results.

[0084] In this embodiment of the invention, the YOLOv4 model performs target detection on traffic videos. After obtaining the detection results, target tracking is also required. This embodiment uses the Deep SORT algorithm to track each detection result. The Deep SORT algorithm is an improvement on the SORT (Simple Online and Realtime Tracking) algorithm. Compared to the SORT algorithm, the Deep SORT algorithm, in addition to having the core Kalman filter algorithm and Hungarian algorithm, also employs feature re-identification technology to reduce the number of ID exchanges during target tracking. The Deep SORT algorithm adds cascaded matching and new trajectory confirmation compared to the SORT algorithm. Due to the addition of these components, the Deep SORT algorithm has better performance in detecting occluded objects. The main core of the Deep SORT algorithm is the Kalman filter, and the Kalman filter algorithm process is as follows: Figure 2 As shown, this invention provides a brief introduction to the Kalman filtering algorithm, as follows:

[0085] The Kalman time update equation is:

[0086]

[0087]

[0088] The Kalman state update equation is:

[0089]

[0090]

[0091]

[0092] When applying the Kalman filter algorithm, firstly, each detection result and trajectory is cascaded and matched to obtain multiple first matching results, which are respectively matched and confirmed trajectories, unmatched trajectories, and detected objects. Then, each first matching result is subjected to IOU matching to obtain multiple second matching results, which are respectively matched but unconfirmed trajectories, unmatched but confirmed trajectories with fewer than 70 unmatches, and trajectories with more than 70 unmatches. The matched and confirmed trajectories and the matched but unconfirmed trajectories are all input into the Kalman update equation for correction. Then, the corrected matched and confirmed trajectories, the corrected matched but unconfirmed trajectories, and the unmatched but confirmed trajectories with fewer than 70 unmatches are input into the Kalman prediction equation for prediction to obtain the predicted trajectory. The above process is repeated and finally all trajectories are output to obtain the trajectory coordinates of the target vehicle in each traffic video segment.

[0093] Specifically, step 3 includes:

[0094] The target vehicles include multiple vehicles;

[0095] For each target vehicle, the trajectory coordinates of the target vehicle in each traffic video segment are transformed to the world coordinate system to obtain the trajectory coordinates in the world coordinate system;

[0096] The trajectory of the target vehicle is extracted based on the trajectory coordinates in the world coordinate system, and the driving trajectory of each target vehicle in each traffic video segment is obtained.

[0097] In this embodiment of the invention, the trajectory coordinates obtained from the video through target detection and tracking are pixel coordinates. However, to reflect the movement of real objects in the real world, the pixel coordinates need to be converted into real-world coordinates. Camera calibration is a crucial step, determining whether the machine vision system can effectively locate and calculate target objects. Camera calibration can be divided into two types: the first is self-calibration of the camera; the second is a calibration method relying on a calibration reference. The former involves the camera taking pictures of surrounding objects and obtaining camera parameters through digital image processing and related geometric calculations. However, this method has a large calibration error and is not suitable for high-precision scenarios. The latter method calibrates using a reference object, with the camera taking images and calculating the camera's intrinsic and extrinsic parameters through digital image processing and subsequent spatial arithmetic operations. This method has higher calibration accuracy and is suitable for applications requiring high precision. This paper adopts the second calibration method.

[0098] Camera coordinate system (optical center coordinate system) C = [x C ,y C ,z C ] T The optical center O of the camera C The origin is the axis whose axis coincides with the optical axis, and the horizontal direction is the x-axis. C y-axis C Perpendicular to X C O C , Z C The image plane is a plane, and the focal length f is the distance from the optical center to the image plane.

[0099] Aircraft coordinate system B = [xx,y] B ,z B ] T These are the coordinates of the center of the drone's fuselage.

[0100] There is an installation error between the camera coordinate system and the body coordinate system, denoted by E:

[0101]

[0102] Where, [α,β,γ]T It is the installation error angle, [x e ,y e ,z e ] T It is the spatial distance from the camera to the origin of the camera body coordinate system.

[0103] The relationship between the camera coordinate system and the body coordinate system is as follows:

[0104] C=EB

[0105] The world coordinate system is the absolute coordinate system of the objective three-dimensional world, also called the objective coordinate system. For any point P in space... E =(x E ,y E ,z E ), its corresponding camera coordinates (x C ,y C ,z C The attitude angle information of the drone is related to the camera's attitude angle and its location. It can be acquired in real time by an IMU (inertial measurement unit) and used for camera attitude correction.

[0106] The image pixel coordinate system [u,v] has no physical unit. The coordinate point (u1,v1) in the pixel coordinate system represents the row and column position of the image in the storage array. The storage array stores the color or grayscale information of the image.

[0107] The physical coordinate system of the image is I = [x1, y1], where the x1 and y1 axes are perpendicular to the x-axis and y-axis, respectively. C and y C The axes are parallel. Coordinates in the physical coordinate system are in millimeters. If a pixel corresponds to a physical size d in the x1 and y1 directions respectively... x and d y The focal length is f.

[0108] The relationship between pixel coordinates (u1, v1) and physical coordinates (x1, y1) is as follows:

[0109]

[0110] Assume a point P = (x) in the camera coordinate system C ,y C ,z C Connecting the optical center O in the physical coordinate system. C And its projection in the physical coordinate system is P. I1 If the coordinates of the two points are (x1, y1), then the coordinate transformation relationship between these two points is as follows:

[0111]

[0112] The trajectory of the target vehicle is extracted based on the trajectory coordinates in the world coordinate system, resulting in the driving trajectory of each target vehicle in each traffic video segment. The driving trajectory includes curvature, acceleration, velocity, position coordinates, and time frame, such as... Figure 3 As shown.

[0113] Since the coverage of video data acquired by a single drone is sometimes limited and cannot meet researchers' needs for trajectory data, stitching together video trajectories from multiple drones into a longer trajectory has certain theoretical and practical significance.

[0114] The traffic video used for trajectory stitching in this embodiment of the invention has the following characteristics:

[0115] (1) Drone video requires multiple drones to shoot simultaneously and ensure time synchronization;

[0116] (2) The shaking of each drone should be minimized when it hovers in the air to shoot video;

[0117] (3) There are certain overlapping areas in the videos.

[0118] Specifically, such as Figure 4 As shown, step 4 includes:

[0119] For each traffic video segment, taking the first traffic video segment as the current segment, calculate the longitudinal distance y corresponding to the last time frame of each trajectory of all target vehicles within the current traffic video segment. end_frame (i);

[0120] From the vertical distance y corresponding to the last time frame end_frame (i) filters out the longitudinal distance y' of the target vehicle leaving the image area. end_frame (i), and the longitudinal distance y' of the target vehicle leaving the image area. end_frame (i) Perform calculations to obtain the first average value y' end_frame (mean);

[0121] For each traffic video segment, for the driving trajectories of all target vehicles, from the next traffic video segment adjacent to the current segment, calculate the longitudinal distance y corresponding to the starting time frame of each driving trajectory of all target vehicles. start_frame (i);

[0122] The vertical distance y corresponding to the starting time frame start_frame (i) filters out the longitudinal distance y' of the target vehicle entering the image region. start_frame(i), and the longitudinal distance y' of the target vehicle entering the image area. start_frame (i) Perform the calculation to obtain the second average value y' start_frame (mean);

[0123] According to the first average value y' end_frame (mean) and second average y' start_frame (mean), obtain the trajectory overlap interval between the current traffic video segment and the next traffic video segment [y_frame(mean), y′ end_frame (mean)];

[0124] Filter out the overlapping area of ​​the trajectory of each target vehicle in the current traffic video segment and the next traffic video segment [y_frame(mean),y'] end_frame In (mean), x, y, and v are vectors about time frames. The solution is to calculate x, y, and v with respect to the trajectory overlap interval y_frame(mean),y'. end_frame The average values ​​of (maych)]: mean_x, mean_y, and mean_v;

[0125] Add a time frame attribute for trajectory matching to the driving trajectory of each target vehicle, which constitutes the first matching tuple of the current traffic video segment and the second matching tuple of the next traffic video segment;

[0126] The first matching tuple of the current traffic video segment is:

[0127] match_object_pre=(mean_x_pre,mean_y_pre,mean_v_pre,match_frame_pre)

[0128] Where match_frame_pre represents the matching time frame attribute value of the trajectory in the current segment of traffic video, maych_frame_pre = end_frame - len(y) / 2, mean_x_pre, mean_y_pre and mean_v_pre represent the average value of the trajectory overlap interval in the current segment of traffic video;

[0129] The second matching tuple for the next traffic video segment is:

[0130] match_object_last=(mean_x_last,mean_y_last,mean_v_last,match_frame_last)

[0131] Where match_frame_last represents the matching time frame attribute value of the trajectory in the next traffic video segment, match_frame_last = start_frame + len(y) / 2, mean_x_last, mean_y_last and mean_v_last represent the average value of the trajectory overlap interval in the next traffic video segment;

[0132] The cost matrix is ​​obtained by calculating the first matching tuple match_object_pre and the second matching tuple maych_object_last using Mahalanobis distance;

[0133] Based on the cost matrix, the Hungarian algorithm is used to match the trajectories and obtain the matched trajectories;

[0134] The matched trajectories are then stitched together to obtain the stitched trajectory results of all target vehicles during their driving process, such as... Figure 5 As shown.

[0135] If the number of trajectory videos to be stitched is greater than 2, then the trajectory obtained in the first few steps of step 4 will be used as the first trajectory to be stitched, and the process will return to the first step of step 4 to continue; otherwise, the process will stop.

[0136] This invention collects multiple traffic video segments within a target traffic area; detects target vehicles in each segment, obtaining multiple detection results; tracks each detection result to obtain the trajectory coordinates of the target vehicle in each traffic video segment; extracts the target vehicle's trajectory based on the trajectory coordinates in each traffic video segment, obtaining the target vehicle's driving trajectory in each traffic video segment; and matches and stitches the driving trajectory in the current traffic video segment with the driving trajectories in adjacent video segments to obtain the stitched trajectory result of the target vehicle during its journey. Compared with existing technologies, this invention has good scalability, high precision, and high stability. By extracting and stitching the target vehicle's trajectory from multiple video segments, the stitched vehicle trajectory has a wider spatial coverage, and the longer vehicle trajectory can effectively reflect the traffic operation patterns of vehicles, making it of significant data support value for traffic flow theory research and intelligent connected vehicle safety assessment.

[0137] This invention also provides a vehicle trajectory extraction device based on video stitching, comprising:

[0138] The acquisition module is used to acquire multiple traffic video clips within the target traffic area;

[0139] The detection and tracking module is used to detect target vehicles in each traffic video segment, obtain multiple detection results, and track the target vehicle in each detection result to obtain the trajectory coordinates of the target vehicle in each traffic video segment.

[0140] The trajectory extraction module is used to extract the trajectory of the target vehicle based on the trajectory coordinates in each traffic video segment, thereby obtaining the driving trajectory of the target vehicle in each traffic video segment.

[0141] The matching and stitching module is used to match and stitch the driving trajectory in the current traffic video segment with the driving trajectory in the adjacent video segment to obtain the trajectory stitching result of the target vehicle during the driving process.

[0142] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0143] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of the present invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0144] This invention provides a computer-readable storage medium for storing a computer program, which, when executed, is used to implement the aforementioned vehicle trajectory extraction method based on video stitching.

[0145] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the embodiments of the present invention can implement all or part of the processes in the methods described above by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a building device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0146] This invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a vehicle trajectory extraction method based on video stitching.

[0147] It should be noted that the terminal device can be a mobile phone, tablet computer, laptop computer, Ultra-mobile Personal Computer (UMPC), netbook, Personal Digital Assistant (PDA), etc. For example, the terminal device can be a station (ST) in a WLAN, a cellular phone, cordless phone, Session Initiation Protocol (SIP) phone, Wireless Local Loop (WLL) station, PDA, handheld device with wireless communication capabilities, computing device or other processing device connected to a wireless modem, computer, laptop computer, handheld communication device, handheld computing device, satellite wireless device, etc. The embodiments of this invention do not limit the specific type of terminal device.

[0148] The processor referred to can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0149] In some embodiments, the memory may be an internal storage unit of the terminal device, such as a hard drive or RAM. In other embodiments, the memory may be an external storage device of the terminal device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital Card (SD), or Flash Card. Furthermore, the memory may include both internal and external storage units of the terminal device. The memory is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or will be output.

[0150] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0151] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for vehicle trajectory extraction based on video stitching, characterized in that, include: Step 1: Collect multiple traffic video clips within the target traffic area; Step 2: Detect target vehicles for each traffic video segment to obtain multiple detection results, and track the target vehicle for each detection result to obtain the trajectory coordinates of the target vehicle in each traffic video segment. Step 3: Extract the trajectory of the target vehicle based on the trajectory coordinates in each traffic video segment to obtain the driving trajectory of the target vehicle in each traffic video segment. Step 4: For each traffic video segment, match and stitch the driving trajectory in the current traffic video segment with the driving trajectory in the adjacent video segment to obtain the trajectory stitching result of the target vehicle during the driving process. Step 4 specifically includes: For each traffic video segment, the driving trajectory of all target vehicles is calculated. Taking the first traffic video segment as the current traffic video segment, the longitudinal distance corresponding to the last time frame of each driving trajectory of all target vehicles is calculated from the current traffic video segment. The longitudinal distance of the target vehicle leaving the image area is filtered out from the longitudinal distance corresponding to the last time frame, and the longitudinal distance of the target vehicle leaving the image area is calculated to obtain a first average value; For each traffic video segment, the longitudinal distance corresponding to the starting time frame of each target vehicle's trajectory is calculated from the next traffic video segment adjacent to the current traffic video segment. The longitudinal distance of the target vehicle entering the image region is filtered out from the longitudinal distance corresponding to the starting time frame, and the longitudinal distance of the target vehicle entering the image region is calculated to obtain a second average value; Based on the first average value and the second average value, the trajectory overlap range between the current traffic video segment and the next traffic video segment is obtained; The time frame vectors of the driving trajectory of each target vehicle within the trajectory overlap interval are selected, and the average value of the time frame vectors with respect to the trajectory overlap interval is calculated. Add a time frame attribute for trajectory matching to the driving trajectory of each target vehicle, which constitutes the first matching tuple of the current traffic video segment and the second matching tuple of the next traffic video segment; The cost matrix is ​​obtained by calculating the first matching tuple and the second matching tuple using Mahalanobis distance; Based on the cost matrix, the Hungarian algorithm is used to match the trajectory to obtain the matched trajectory; The matched trajectories are then stitched together to obtain the trajectories of all target vehicles during their journey.

2. The vehicle trajectory extraction method based on video stitching according to claim 1, characterized in that, The steps preceding step 2 also include: For each traffic video segment, the traffic video is decomposed into multiple frames of images; Preprocess each frame of the image; Each preprocessed frame is synthesized to obtain multiple preprocessed traffic videos. Target vehicle detection is performed on multiple pre-processed traffic video segments.

3. The vehicle trajectory extraction method based on video stitching according to claim 2, characterized in that, Step 2 specifically includes: The preprocessed traffic videos were input into the YOLO4 model for object detection, resulting in multiple detection results. Each detection result is cascaded and matched based on the Deep SORT algorithm to obtain multiple first matching results; Perform an IOU match on each of the first matching results to obtain multiple second matching results; Multiple first matching results and multiple second matching results are all input into the Kalman update equation for correction; The corrected first matching results and the corrected second matching results are input into the Kalman prediction equation for prediction, so as to obtain the trajectory coordinates of the target vehicle in each traffic video segment.

4. The vehicle trajectory extraction method based on video stitching according to claim 3, characterized in that, Step 3 specifically includes: The target vehicles in each traffic video segment include multiple vehicles; For each target vehicle, the trajectory coordinates of the target vehicle in each traffic video segment are transformed to the world coordinate system to obtain the trajectory coordinates in the world coordinate system. The trajectory of each target vehicle is extracted based on the trajectory coordinates in the world coordinate system, thus obtaining the driving trajectory of each target vehicle in each traffic video segment.

5. The vehicle trajectory extraction method based on video stitching according to claim 4, characterized in that, The first matching tuple of the current segment of traffic video is: ； in, This represents the time frame attribute value of the trajectory matching in the current segment of traffic video. This represents the average value of the overlapping trajectories in the current segment of traffic video.

6. The vehicle trajectory extraction method based on video stitching according to claim 4, characterized in that, The second matching tuple for the next segment of traffic video is: ； in, This indicates the time frame attribute value for matching the trajectory in the next segment of traffic video. This represents the average value of the overlapping trajectories in the next traffic video segment.

7. A vehicle trajectory extraction device based on video stitching, characterized in that, include: The acquisition module is used to acquire multiple traffic video clips within the target traffic area; The detection and tracking module is used to detect target vehicles in each traffic video segment, obtain multiple detection results, and track the target vehicle in each detection result to obtain the trajectory coordinates of the target vehicle in each traffic video segment. The trajectory extraction module is used to extract the trajectory of the target vehicle based on the trajectory coordinates in each traffic video segment, so as to obtain the driving trajectory of the target vehicle in each traffic video segment. The matching and stitching module is used to match and stitch the driving trajectory in the current traffic video segment with the driving trajectory in the adjacent video segment for each segment, so as to obtain the trajectory stitching result of the target vehicle during the driving process. The matching and stitching module is also used to calculate the longitudinal distance corresponding to the last time frame of each driving trajectory of all target vehicles in each traffic video segment, taking the first traffic video segment as the current traffic video segment. The longitudinal distance of the target vehicle leaving the image area is filtered out from the longitudinal distance corresponding to the last time frame, and the longitudinal distance of the target vehicle leaving the image area is calculated to obtain a first average value; For each traffic video segment, the longitudinal distance corresponding to the starting time frame of each target vehicle's trajectory is calculated from the next traffic video segment adjacent to the current traffic video segment. The longitudinal distance of the target vehicle entering the image region is filtered out from the longitudinal distance corresponding to the starting time frame, and the longitudinal distance of the target vehicle entering the image region is calculated to obtain a second average value; Based on the first average value and the second average value, the trajectory overlap range between the current traffic video segment and the next traffic video segment is obtained; The time frame vectors of the driving trajectory of each target vehicle within the trajectory overlap interval are selected, and the average value of the time frame vectors with respect to the trajectory overlap interval is calculated. Add a time frame attribute for trajectory matching to the driving trajectory of each target vehicle, which constitutes the first matching tuple of the current traffic video segment and the second matching tuple of the next traffic video segment; The cost matrix is ​​obtained by calculating the first matching tuple and the second matching tuple using Mahalanobis distance; Based on the cost matrix, the Hungarian algorithm is used to match the trajectory to obtain the matched trajectory; The matched trajectories are then stitched together to obtain the trajectories of all target vehicles during their journey.

8. A computer-readable storage medium for storing a computer program, characterized in that, The computer program is executed to implement the vehicle trajectory extraction method based on video stitching as described in any one of claims 1 to 6.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the vehicle trajectory extraction method based on video stitching as described in any one of claims 1 to 6.

Images