A ground constraint-based roadside camera three-dimensional detection and tracking method and system

Abstract

The application discloses a kind of based on ground restraint roadside camera three-dimensional detection tracking method and system, the method is first in image perspective two-dimensional target detection, utilize the dense output information to difficult target such as pedestrian, based on road end camera stationary, road end scene, first offline obtain ground equation, then utilize ground equation, camera internal parameter and landing site estimation obtain target three-dimensional information;Using two-dimensional tracker obtains the interframe matching relationship of target, with subsequent three-dimensional tracker cooperation completes data association, effectively improves the problem that small target such as pedestrian is due to two-dimensional detection frame unstable and leads to three-dimensional landing site estimation error is larger and disrupts three-dimensional tracking trajectory, the present application can realize high-precision, low-cost real-time detection tracking, the present application eliminates the high cost of obtaining laser radar and other sensors, but detection tracking precision is still guaranteed, can make full use of the space priori of roadside equipment, effectively improves the generalization of method.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of intelligent transportation, and particularly relates to a roadside camera three-dimensional detection tracking method and system based on ground constraints. BACKGROUND

[0002] Since the 21st century, with the rapid development of the world economy, road traffic has become an indispensable part of modern society. However, the rapid growth of road traffic has also brought a series of problems, such as traffic congestion, frequent traffic accidents, etc., which seriously affect people's life and safety. In order to solve these problems, researchers began to apply various high-tech to the transportation system, thus promoting the development of intelligent transportation system (ITS).

[0003] Three-dimensional tracking of moving traffic participants is one of the important tasks in intelligent transportation systems. It refers to continuously tracking the real-world position and motion trajectory of traffic participants (such as vehicles, pedestrians, etc.) in a sensor time sequence. This task is often based on three-dimensional target detection and is one of the core parts of intelligent transportation systems.

[0004] The commonly used detection and tracking paradigm currently relies on high-precision three-dimensional target detection models, but high-precision three-dimensional target detection models often rely on costly lidar to complete detection, which poses a cost challenge to the deployment of detection and tracking models by roadside devices. Camera-based detection and tracking models can effectively solve the above cost problems. Known camera-based three-dimensional target detection models require a large amount of high-quality training data and computing resources to learn a series of parameters of the model, and the performance of the model often has great limitations, can only adapt to new situations very close to known data, and has poor generalization ability. Effective use of prior information of camera static, unique scene can greatly improve the detection accuracy and generalization ability of the detection model. For example, patent No. CN115984766A (hereinafter referred to as "prior art patent") describes a monocular vision three-dimensional detection method based on a two-dimensional target detection model and a target inherent heading angle, which calculates the two-dimensional bounding box bottom center point of the target on the image, and then combines the target inherent heading angle and the prior length and width to calculate the real three-dimensional position of the target. However, this method requires the target to have an inherent heading angle and prior length and width, which is not suitable for high-degree-of-freedom targets at the roadside. Therefore, while controlling the deployment cost, it is necessary to fully utilize the roadside scene to improve the detection and tracking accuracy and generalization of the detection and tracking algorithm. SUMMARY

[0005] In order to solve the problems in the prior art, the present application aims to provide a roadside camera three-dimensional detection tracking method based on ground constraints, which can realize high-precision, low-cost real-time detection tracking based on roadside camera static, roadside scene, and guarantee high generalization.

[0006] To achieve the above object, the technical scheme adopted by the present application is: a roadside camera three-dimensional detection tracking method based on ground constraint, comprising the following steps:

[0007] According to the monitored intersection camera image, the monitored intersection camera internal parameter K and the distortion coefficient, the ground equation is obtained offline and the camera external parameter of the camera system to the ground coordinate system is calibrated;

[0008] The YOLOX detection model is used to detect the camera image, and two-dimensional detection results are obtained respectively, which contain categories, confidence scores and two-dimensional bounding boxes;

[0009] For non-vehicle targets, combined with the confidence score, a non-dense suppression algorithm is used for processing to obtain non-vehicle target two-dimensional detection;

[0010] For vehicle targets, combined with the confidence score, a non-maximum suppression algorithm is used for processing to obtain vehicle target two-dimensional detection result output;

[0011] The Sort algorithm based on Kalman filtering and Hungarian matching is used, the intersection-over-union of the two-dimensional bounding box of the non-vehicle target is calculated, the affinity matrix is constructed with the intersection-over-union as the measurement parameter, and the constant velocity motion model is used for dynamic modeling, so as to track the two-dimensional target of the non-vehicle target, and the front and rear frame matching relationship of the non-vehicle target is obtained;

[0012] The camera image, the two-dimensional bounding box of all targets on the image and the ground equation are used as the input of the roadside monocular three-dimensional target detection model, the image block is cropped according to the two-dimensional bounding box of the target, all image blocks are scaled to a uniform size, the image features of each target are extracted in turn by using the two-dimensional backbone network, the target image landing point, the camera system orientation angle and the actual length and width are estimated after the image features are obtained, the three-dimensional landing point in the target camera coordinate system is calculated combined with the ground plane equation and the camera internal parameter, and is converted to the ground coordinate system, the three-dimensional detection result is obtained combined with the camera system orientation angle and the actual length and width, the three-dimensional detection result includes the ground coordinate system three-dimensional landing point, the orientation angle and the actual length and width;

[0013] Combined with the three-dimensional detection result of the vehicle target, a three-dimensional multi-target tracking method based on Kalman filtering and greedy matching is used, an affinity matrix is constructed with the Euclidean distance between the three-dimensional landing points of the targets as the measurement parameter in the ground coordinate system, and a constant acceleration motion model is used for dynamic modeling to construct a vehicle tracker, so as to obtain the three-dimensional position, size, orientation, speed and historical trajectory information of the vehicle target;

[0014] The front and rear frame matching relationship is used to complete data association of non-vehicle targets, and a constant acceleration motion model based on a Kalman filtering algorithm is used to perform dynamic modeling in a ground coordinate system to construct a non-vehicle tracker to estimate three-dimensional position, size, orientation, speed and historical trajectory information of the non-vehicle target.

[0015] Further, the camera image is obtained from a camera device deployed at the monitored intersection, and there is a clear zebra crossing sign in the camera field of view.

[0016] Further, according to the monitored intersection camera image information, the monitored intersection camera intrinsic parameter K and the distortion coefficient, the ground equation is obtained offline and the camera extrinsic parameter from the camera system to the ground coordinate system is calibrated, including:

[0017] The zebra crossing is fitted to the ground equation, the pixel coordinate system is a two-dimensional coordinate system, the coordinate origin is at the top left corner of the image, the camera coordinate system and the ground coordinate system are three-dimensional coordinate systems, the origin of the camera coordinate system is at the center of the camera, and the origin of the ground coordinate system is at the top left corner of the zebra crossing; solving the ground equation in the camera coordinate system is converted into transforming the [0, 0, 1] vector in the ground coordinate system to the camera coordinate system, and the transformation process needs the transformation matrix T = [R | t] from the camera coordinate system to the ground coordinate system, the camera intrinsic parameter K, the pixel coordinate P uv =[u, v], the camera coordinate P C =[X c ,Y c ,Z c ], the ground coordinate P W =[X W ,Y W ,Z W ]; the point on the zebra crossing is on the ground, Z W = 0, the camera imaging equation Z C P uv = KP C , the coordinate system conversion equation P C = TP W ; the zebra crossing corner points are taken as calibration reference objects, the projection error is minimized, and the transformation matrix T cg from the camera to the ground coordinate system is solved as the camera extrinsic parameter, and the ground equation in the camera system is obtained further after the camera extrinsic parameter T cg is obtained.

[0018] Further, for the non-vehicle target, the target with a confidence lower than a preset threshold thod f is filtered out according to the inference confidence; the non-maximum suppression algorithm is used to select the target with a confidence higher than the threshold thod n , and the target with a confidence higher than the threshold thod nThe number of targets obscured by the target; if the confidence level of the output result is higher than the threshold. n Or, although the confidence level is below the threshold, the number of occluded targets exceeds the threshold. num Then the target will be output as the final reliable two-dimensional detection result.

[0019] Furthermore, the two-dimensional backbone network adopts ResNet50, HRNet, or ShuffleNet.

[0020] Furthermore, during training, the two-dimensional backbone network: for the landing point prediction branch, the training objective is the relative offset of the landing point relative to the bottom center point, and the L1 loss function is used; for the length and width prediction, the training objective is the logarithmic value of the actual length and width compared to the reference length and width, and the loss function is the L1 loss function; for the angle prediction, the MultiBin loss is used.

[0021] Furthermore, utilizing camera extrinsic parameters T cg The specific steps for converting the 3D landing point and orientation angle from the camera coordinate system to the ground coordinate system are as follows:

[0022] The transformation of the three-dimensional landing point coordinate system can be solved using the following formula:

[0023] T cg ×[X c ,Y c Z c ,1] T =[X g ,Y g Z g ,1] T

[0024] Among them, [X c ,Y c Z c [1] represents the three-dimensional coordinates of the landing point in the camera coordinate system, [X... g ,Y g Z g [1] represents the three-dimensional coordinates of the landing point in the ground coordinate system.

[0025] For orientation angular coordinate system transformations, the following formula needs to be used to solve:

[0026]

[0027] Where, θ c Let θ be the angle between the target orientation and the x-axis in the camera coordinate system. g Let be the angle between the target orientation and the x-axis in the ground coordinate system. This is a virtual vector constructed in the camera coordinate system based on the orientation angle. A virtual vector in a ground coordinate system.

[0028] Based on the technical concept of the method, a multi-stage three-dimensional detection and tracking system using ground prior under the perspective of a roadside camera is also provided, comprising a parameter acquisition module, a two-dimensional detection module, a front and rear frame matching relationship acquisition module, a three-dimensional detection module, and a tracking module.

[0029] The parameter acquisition module is used to acquire the ground equation and calibrate the camera extrinsic parameters from the camera coordinate system to the ground coordinate system according to the monitoring intersection camera image, the monitoring intersection camera intrinsic parameter K and the distortion coefficient.

[0030] The two-dimensional detection module uses a YOLOX detection model to detect the camera image to obtain two-dimensional detection results, which include categories, confidence scores and two-dimensional bounding boxes.

[0031] For non-vehicle targets, a non-dense suppression algorithm is used to process the confidence scores to obtain non-vehicle target two-dimensional detection.

[0032] For vehicle targets, a non-maximum suppression algorithm is used to process the confidence scores to obtain vehicle target two-dimensional detection results.

[0033] The front and rear frame matching relationship acquisition module uses a Sort algorithm based on Kalman filtering and Hungarian matching to calculate the intersection-over-union ratio of the two-dimensional bounding boxes of non-vehicle targets, construct an affinity matrix using the intersection-over-union ratio as a measurement parameter, and perform dynamic modeling using a constant velocity motion model to track the two-dimensional targets of non-vehicle targets and obtain the front and rear frame matching relationship of non-vehicle targets.

[0034] The three-dimensional detection module is used to input the camera image, the two-dimensional bounding boxes of all targets on the image, and the ground equation into a roadside monocular three-dimensional target detection model, crop the image blocks according to the two-dimensional bounding boxes of the targets, resize all the image blocks to a uniform size, extract the image features of each target using a two-dimensional backbone network, estimate the target image landing position, camera system orientation angle and actual length and width after obtaining the image features, calculate the three-dimensional landing position in the target camera coordinate system combined with the ground plane equation and the camera intrinsic parameter, and convert it to the ground coordinate system, combine the camera system orientation angle and the actual length and width to obtain the three-dimensional detection results, which include the ground coordinate system three-dimensional landing position, orientation angle, and actual length and width.

[0035] The tracking module is used in combination with the three-dimensional detection result of the vehicle target, a three-dimensional multi-target tracking method based on Kalman filtering and greedy matching is used, in the ground coordinate system, an affinity matrix is constructed by taking the Euclidean distance between the three-dimensional landing positions of the targets as the measurement parameter, a constant acceleration motion model is used for dynamic modeling, a vehicle tracker is constructed, and the three-dimensional position, size, orientation, speed and historical trajectory information of the vehicle target are obtained; the data association of non-vehicle targets is completed by using the front and rear frame matching relationship, and in the ground coordinate system, a constant acceleration motion model based on Kalman filtering algorithm is used for dynamic modeling, a non-vehicle tracker is constructed, and the three-dimensional position, size, orientation, speed and historical trajectory information of the non-vehicle target are estimated.

[0036] The application can also provide a computer device, comprising a processor and a memory, the memory is used for storing a computer executable program, the processor reads the computer executable program from the memory and executes, and the processor can realize the three-dimensional detection and tracking method of the roadside camera based on ground constraints when executing the computer executable program.

[0037] Meanwhile, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program can realize the three-dimensional detection and tracking method of the roadside camera based on ground constraints when being executed by a processor.

[0038] Compared with the prior art, the application has at least the following beneficial effects:

[0039] The application firstly performs two-dimensional target detection in the image perspective, ensures the recall rate of the target, and has good robustness; the dense output information of the difficult-to-detect targets such as pedestrians is used, the detection capability of the two-dimensional detection model for the difficult-to-detect targets is effectively improved; the monocular three-dimensional target detection model is used to estimate the real landing position, orientation angle and length-width of the target on the image, so that the advantages of the roadside camera being stationary and the roadside scene being fixed can be effectively combined, the three-dimensional information of the target is obtained, and the application of the prior art in the high-degree-of-freedom target detection scene is expanded; the application uses two independent three-dimensional trackers to separately process the vehicle targets and the non-vehicle targets, and uses the two-dimensional tracker to obtain the interframe matching relationship of the non-vehicle targets, and cooperates with the subsequent three-dimensional tracker to complete data association, and effectively improves the problem that the three-dimensional landing position estimation error of the non-vehicle targets such as pedestrians is large and the three-dimensional tracking trajectory is disturbed due to the unstable two-dimensional detection frame. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 It is a schematic flow chart of a multi-stage three-dimensional detection and tracking solution under the perspective of a roadside camera.

[0041] Figure 2 It is a calibration diagram of the ground.

[0042] Figure 3 A coordinate system is defined. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to specific embodiments and drawings. It should be noted that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0044] With the development of intelligent transportation, intelligent devices at the road end will be deployed in large quantities, and cameras, as cheap and commonly used road end monitoring devices, can provide effective time sequence video data for three-dimensional target tracking, which is one of the important tasks in the intelligent transportation system. The present application proposes a multi-stage road end three-dimensional detection and tracking method under the camera view, which can realize high-precision, low-cost real-time detection and tracking, and ensure high generalization. The present application eliminates the high cost of obtaining sensors such as laser radars, while the detection and tracking accuracy is still guaranteed, which can fully utilize the spatial prior of the roadside device and effectively improve the generalization of the method.

[0045] The overall schematic flowchart of the embodiment of the present application is shown in Figure 1 . The embodiment of the present application obtains ground information offline, mainly for scenes with typical ground plane signs such as zebra crossings, as shown in Figure 2 . When fitting the ground equation using zebra crossings, it is ensured that there is a zebra crossing in the camera view. If there is no zebra crossing in the camera view, a zebra crossing prepared in advance can be laid on the ground for data collection. The length h, width w1 and distance w2 between any two zebra crossings are measured. When fitting the ground equation using zebra crossings, pixel coordinate system, camera coordinate system and ground coordinate system are involved. The pixel coordinate system is a two-dimensional coordinate system, the coordinate origin is at the top left corner of the image, the camera coordinate system and the ground coordinate system are both three-dimensional coordinate systems, the origin of the camera coordinate system is at the center of the camera, and the origin of the ground coordinate system is at the top left corner of the zebra crossing. The zebra crossing is on the ground, so solving the ground equation in the camera coordinate system is equivalent to transforming the [0, 0, 1] vector in the ground coordinate system to the camera coordinate system. The transformation process needs the transformation matrix T = [R | t] from the camera coordinate system to the ground coordinate system. Given the camera intrinsic parameter K, pixel coordinates P uv = [u, v], camera coordinates P C = [X c ,Y c ,Z c ], ground coordinates P W = [X W ,Y W ,Z W ]; because the points on the zebra crossing are on the ground, Z W = 0, the camera imaging equation Z C Puv = KP C , coordinate system conversion equation P C = TP W . Taking the zebra crossing corner point as a calibration reference object, a PnP problem is constructed, a projection error is minimized, and a transformation matrix T of the camera to the ground coordinate system is solved cg as the camera external parameter, the camera external parameter T cg is obtained

[0046] The embodiment of the application provides a non-dense suppression method for two-dimensional detection, which can include the following steps:

[0047] 1) using a YOLOX detection model to perform real-time detection on a camera image, obtaining a dense target class and a two-dimensional detection result before non-maximum suppression;

[0048] 2) for non-vehicle targets, filtering out targets with a confidence lower than a preset threshold thod f using an inference confidence as a standard; using a non-maximum suppression algorithm to select targets with a confidence higher than a threshold thod n , and counting the number of targets hidden by the targets with a confidence higher than the threshold thod n ; if the confidence of the output result is higher than the threshold thod n , or the confidence is lower than the threshold but the number of hidden targets exceeds the threshold thod num , the target is output as the final credible two-dimensional detection result.

[0049] It should be noted that the above non-dense suppression method is mainly aimed at the categories with poor two-dimensional model detection effect, such as pedestrians and bicycles. The purpose is to improve the detection ability of the model for difficult-to-detect targets such as pedestrians. As an example, thod f ∈[0.01, 0.05], thod n ∈[0.08, 0.12], thod o ∈[0.25, 0.35], thod num ∈[15, 20].

[0050] According to the embodiment of the application, a two-dimensional tracking algorithm based on Kalman filtering and Hungarian matching is used, an intersection over union (IOU) is used as a metric to construct an affinity matrix, and a constant velocity (CV) motion model is used for dynamic modeling; the two-dimensional target tracking of non-vehicle targets including pedestrians is completed, the matching relationship between the front and rear frames is obtained, and is used for the subsequent data association process of the three-dimensional tracking algorithm for non-vehicle targets.

[0051] Embodiments of the present application provide a Bird's Eye View (BEV) detection method based on two-dimensional detection and ground plane equation prior. The camera placement position is fixed, which means that the ground plane equation is fixed in the camera coordinate system, facilitating the use of object ground plane constraints; the deployment of the two-dimensional detection algorithm has been completed, and the category of the object and the position of the object on the image have been determined. Based on the two-dimensional target detection result, the corresponding target in the image is segmented and obtained, and the target is scaled to the same size. The accurate landing position of the target in the image is estimated through a deep learning model, and finally the landing position on the image is back projected to the three-dimensional space by using the ground plane constraint and the camera intrinsic constraint, combined with the orientation estimation and the length-width estimation, to realize the BEV detection. The implementation details in the method are as follows:

[0052] 1) Data preprocessing. The input data is a camera image, two-dimensional bounding boxes of all targets on the image, and a ground plane equation. In the training stage, the two-dimensional labels in the Rope3D dataset are used as two-dimensional bounding boxes to crop the objects from the camera image. In order to facilitate data augmentation, the two-dimensional boxes in the Rope3D dataset are expanded by 25% up, down, left and right. After cropping the image block, all image blocks are uniformly scaled to 224x224 size for subsequent model processing. In addition, the three-dimensional annotation boxes in the Rope3D dataset do not fully meet the ground plane constraint. This means that the three-dimensional annotation boxes provided by the dataset are floating in the air or sinking into the ground, which is unreasonable. Therefore, in order to obtain effective training labels, the three-dimensional boxes are all projected onto the ground surface by using the ground plane equation, so as to strictly meet the ground plane constraint. In the inference stage, the two-dimensional bounding boxes in the two-dimensional detection result are used to complete the cropping, and the same cropping and scaling details as in the training process are adopted to complete the data preprocessing.

[0053] 2) Data augmentation. When training the monocular three-dimensional target detection model, the input image is sequentially subjected to three kinds of data augmentation to reduce overfitting and improve the generalization ability of the model.

[0054] ① Random horizontal flip. It is the most commonly used data augmentation method in visual tasks.

[0055] ② Random cropping. During data preprocessing, a small amount of background is reserved in the cropped image block, and random cropping is performed on the edges of the image block up, down, left and right, which can simulate the case that the two-dimensional detection box is not accurate;

[0056] ③ HSV transformation. A data augmentation method based on YOLOV5, which transforms the pixel value in the HSV color space.

[0057] 3) Single three-dimensional target detection model structure. Multiple image blocks form a batch input into a two-dimensional backbone network, which can have multiple options, such as ResNet50, HRNet, ShuffleNet. The features output by the two-dimensional backbone network are input into three branches after global average pooling, respectively, to estimate the target image landing site, camera system orientation angle, and actual length and width.

[0058] 4) Loss function. For the landing site prediction branch, the training target is the relative offset of the landing site relative to the center point of the bottom, that is, the offset ratio of the image block, and the L1 loss function is used; for length and width prediction, the training target is the logarithmic value after the actual length and width ratio reference length and width, and the loss function is the L1 loss function; for angle prediction, the MultiBin loss is used.

[0059] 5) Data post-processing. Data post-processing includes target three-dimensional landing site solving and coordinate system transformation. The positional relationship between the target landing site [x, y, 1] in the image and the corresponding three-dimensional landing site [X c ,Y c ,Z c ] in the camera system is as follows, wherein K 3×3 is the camera intrinsic parameter matrix, and G 1×4 is the calibrated ground plane equation parameter [A, B, C, D].

[0060]

[0061] The two equations are solved simultaneously to obtain the coordinates [X c ,Y c ,Z c ] of the target landing site in the camera coordinate system, and the solving formula of the camera system point is as follows:

[0062]

[0063] The present embodiment uses two three-dimensional coordinate systems, ground coordinate system and camera coordinate system, which are defined as Figure 3 . The result of the above processing of the model is in the camera coordinate system, but in order to facilitate subsequent three-dimensional target tracking, the three-dimensional landing site and the orientation angle are converted from the camera system to the ground coordinate system using the camera external parameter T cg . For three-dimensional landing site coordinate system transformation, the following formula can be used to solve:

[0064] T cg ×[X c ,Y c ,Z c ,1] T =[X g ,Y g ,Z g ,1]T

[0065] where [X c , Y c , Z c , 1] is the three-dimensional coordinate of the falling point in the camera coordinate system, and [X g , Y g , Z g , 1] is the three-dimensional coordinate of the falling point in the ground coordinate system.

[0066] For the heading angle coordinate system transformation, the following formula is used to solve:

[0067]

[0068] where θ c is the angle between the target heading and the x-axis in the camera coordinate system, θ g is the angle between the target heading and the x-axis in the ground coordinate system, is a virtual vector in the camera coordinate system constructed according to the heading angle, is a virtual vector in the ground coordinate system.

[0069] According to the embodiment of the present application, the three-dimensional target tracking includes the following steps:

[0070] 1) Initial detection and tracker division. In the initial frame, the three-dimensional detection results of all targets in the ground coordinate system are obtained by the three-dimensional target detector, and the targets are divided into two categories: vehicles and non-vehicles, which are used to maintain vehicle trackers and non-vehicle trackers, respectively.

[0071] 2) State estimation. The initial state vector and covariance matrix of each target in the two trackers are set. The initial state vector includes position, velocity and acceleration information, and is in the following form:

[0072]

[0073] 3) Motion prediction. The state vector of each target in the two trackers is predicted based on the Kalman filter using the constant acceleration (CA) motion model, and the state transition matrix F is represented as:

[0074]

[0075] where Δt is the time interval.

[0076] 4) Calculate the Euclidean distance and construct the affinity matrix. Given the three-dimensional falling point coordinate set of the target in the current frame t in the vehicle tracker and the three-dimensional falling point coordinate set of the target after motion prediction in the previous frame t-1 M and N are the target number in the current frame and the previous frame respectively, is the landing coordinates of the i-th target in the current frame t, is the landing coordinates of the j-th target in the previous frame t-1; the landing coordinates of the two detection frames before and after are calculated And the Euclidean distance formula between According to the Euclidean distance, the affinity matrix A is constructed, and wherein σ is a scale parameter for controlling distance attenuation.

[0077] 5) Use two-dimensional target tracking algorithm to obtain the matching relationship between the current frame target and the previous frame target in the non-vehicle tracker.

[0078] 6) Use the greedy matching algorithm to process the affinity matrix to obtain the matching relationship between the current frame target and the previous frame target in the vehicle tracker; find the current maximum value A ij from the affinity matrix, and determine the target pair A ij corresponding to the current maximum value as the matching pair, remove the matched row and column from the affinity matrix; repeat the operation of finding the maximum value and removing until all possible matching pairs are determined.

[0079] 7) State update. For each matched target in the two trackers, use the detection result of the current frame to update the state vector and covariance matrix of the Kalman filter, initialize a new target trajectory for the unmatched detection box, and process the unmatched target trajectory according to the loss strategy.

[0080] 8) Use constant acceleration (CA) motion model for dynamic modeling, and use three-dimensional Kalman filtering algorithm to estimate the three-dimensional position, size, orientation, velocity and historical trajectory of the target in the two trackers, including: three-dimensional position: [x, y, z], size: [l, w], velocity: Historical trajectory: a set of historical position points.

[0081] where the target is on the ground, z = 0,

[0082] Based on the technical concept of the method described in the application, a multi-stage three-dimensional detection and tracking system using ground prior under the perspective of roadside cameras is also provided, which includes a parameter acquisition module, a two-dimensional detection module, a front and rear frame matching relationship acquisition module, a three-dimensional detection module, and a tracking module; The parameter acquisition module is used to acquire the ground equation offline and calibrate the camera external parameter from the camera coordinate system to the ground coordinate system according to the monitoring intersection camera image, the monitoring intersection camera intrinsic parameter K and the distortion coefficient;

[0083] The two-dimensional detection module uses a YOLOX detection model to detect the camera image to obtain a two-dimensional detection result, which includes a category, a confidence score and a two-dimensional bounding box;

[0084] For non-vehicle targets, a non-dense suppression algorithm is used to process the confidence score to obtain a two-dimensional detection of the non-vehicle target.

[0085] For vehicle targets, a non-maximum suppression algorithm is used to process the confidence score to obtain a two-dimensional detection result of the vehicle target.

[0086] The front and rear frame matching relationship acquisition module uses a Sort algorithm based on Kalman filtering and Hungarian matching to calculate an intersection-over-union ratio of the two-dimensional bounding box of the non-vehicle target, construct an affinity matrix using the intersection-over-union ratio as a measurement parameter, and perform dynamic modeling using a constant velocity motion model to track the two-dimensional target of the non-vehicle target and obtain a front and rear frame matching relationship of the non-vehicle target.

[0087] The three-dimensional detection module is used to input the camera image, the two-dimensional bounding box of all targets on the image and the ground equation into a roadside monocular three-dimensional target detection model, cut the image block according to the two-dimensional bounding box of the target, resize all the image blocks to a uniform size, extract the image features of each target using a two-dimensional backbone network, estimate the target image landing position, the camera system orientation angle and the actual length and width after obtaining the image features, calculate the three-dimensional landing position of the target in the camera coordinate system in combination with the ground equation and the camera intrinsic parameter, and convert it to the ground coordinate system, obtain the three-dimensional detection result in combination with the camera system orientation angle and the actual length and width, the three-dimensional detection result including the ground coordinate system three-dimensional landing position, the orientation angle, the actual length and width.

[0088] The tracking module is used to combine the three-dimensional detection result of the vehicle target, use a three-dimensional multi-target tracking method based on Kalman filtering and greedy matching, construct an affinity matrix using the Euclidean distance between the three-dimensional landing positions of the targets as a measurement parameter, and perform dynamic modeling using a constant acceleration motion model to construct a vehicle tracker and obtain the three-dimensional position, size, orientation, speed and historical trajectory information of the vehicle target; the data association of the non-vehicle target is completed using the front and rear frame matching relationship, and a constant acceleration motion model based on Kalman filtering algorithm is used to perform dynamic modeling in the ground coordinate system to construct a non-vehicle tracker and estimate the three-dimensional position, size, orientation, speed and historical trajectory information of the non-vehicle target.

[0089] In another aspect, the application provides a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the roadside camera three-dimensional detection and tracking method based on ground constraints.

[0090] The computer device can be a notebook computer, a desktop computer or a workstation.

[0091] The application can also provide a computer device, comprising a processor and a memory, the memory being used to store a computer executable program, the processor reading the computer executable program from the memory and executing, and the processor executing the computer executable program can realize the ground constraint based roadside camera three-dimensional detection and tracking method.

[0092] The processor can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a ready programmable gate array (FPGA).

[0093] For the memory of the application, it can be an internal storage unit of a notebook computer, a desktop computer or a workstation, such as a memory, a hard disk, and can also use an external storage unit, such as a mobile hard disk, a flash card.

[0094] The computer readable storage medium can include a computer storage medium and a communication medium. The computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. The computer readable storage medium can include read only memory (ROM), random access memory (RAM), solid state disk (SSD) or optical disk, etc. Among them, the random access memory can include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

[0095] The above is only to illustrate the technical idea of the application, and cannot limit the protection scope of the application. Any modification made according to the technical idea of the application on the basis of the technical scheme falls within the protection scope of the claims of the application.

Claims

1. A three-dimensional detection and tracking method for roadside cameras based on ground constraints, characterized in that, Includes the following steps: Based on the images from the monitoring intersection camera, the intrinsic parameters K and distortion coefficients of the monitoring intersection camera, the ground equation is obtained offline and the camera extrinsic parameters from the camera system to the ground coordinate system are calibrated. The YOLOX detection model is used to detect camera images, and two-dimensional detection results are obtained. The two-dimensional detection results include category, confidence score and two-dimensional bounding box. For non-vehicle targets, the non-dense suppression algorithm is used to process the data based on the confidence score to obtain the two-dimensional detection results of non-vehicle targets. For vehicle targets, the non-maximum suppression algorithm is used to process the data based on the confidence score to obtain the two-dimensional detection results of vehicle targets. The Sort algorithm based on Kalman filtering and Hungarian matching is used to calculate the cross-union ratio (CUR) of the two-dimensional bounding boxes of non-vehicle targets. The CUR is used as a metric to construct an affinity matrix. A constant velocity motion model is used for dynamic modeling to perform two-dimensional target tracking of non-vehicle targets and obtain the frame-to-frame matching relationship of non-vehicle targets. The camera image, the 2D bounding boxes of all targets in the image, and the ground equation are used as inputs to the roadside monocular 3D target detection model. Image blocks are cropped sequentially according to the target's 2D bounding box, and all image blocks are scaled to a uniform size. Image features of each target are extracted sequentially using a 2D backbone network. After obtaining the image features, the target image landing point, camera orientation angle, and actual length and width are estimated respectively. Combining the ground plane equation and camera intrinsic parameters, the 3D landing point of the target in the camera coordinate system is calculated and transformed to the ground coordinate system. The 3D detection result is obtained by combining the camera orientation angle and actual length and width. The 3D detection result includes the 3D landing point, orientation angle, and actual length and width in the ground coordinate system. Combining the 3D detection results of vehicle targets, a 3D multi-target tracking method based on Kalman filtering and greedy matching is used. In the ground coordinate system, an affinity matrix is ​​constructed with the Euclidean distance between the 3D landing points of the targets as the metric parameter. A constant acceleration motion model is used for dynamic modeling to construct a vehicle tracker and obtain the 3D position, size, orientation, speed and historical trajectory information of the vehicle targets. The data association of non-vehicle targets is completed by using the matching relationship between the preceding and following frames. In the ground coordinate system, a constant acceleration motion model based on the Kalman filter algorithm is used for dynamic modeling to construct a non-vehicle tracker and estimate the three-dimensional position, size, orientation, velocity and historical trajectory information of the non-vehicle targets.

2. The three-dimensional detection and tracking method for roadside cameras based on ground constraints according to claim 1, characterized in that, The camera images are acquired from camera equipment deployed at the monitored intersection, and the zebra crossing signs are clearly visible in the camera's field of view.

3. The three-dimensional detection and tracking method for roadside cameras based on ground constraints according to claim 1, characterized in that, Based on the image information from the surveillance intersection camera, the camera's intrinsic parameter K, and the distortion coefficient, the ground equation is obtained offline, and the camera's extrinsic parameters from the camera system to the ground coordinate system are calibrated, including: To fit the ground equation using zebra stripes, the pixel coordinate system is a two-dimensional coordinate system with its origin at the top left corner of the image. Both the camera coordinate system and the ground coordinate system are three-dimensional coordinate systems. The origin of the camera coordinate system is at the camera center, and the origin of the ground coordinate system is set at the top left corner of the zebra stripes. Solving the ground equation in the camera coordinate system is transformed into transforming the [0, 0, 1] vector in the ground coordinate system to the camera coordinate system. This transformation requires a transformation matrix T = [R|t] from the camera coordinate system to the ground coordinate system. Given the camera intrinsic parameters K and pixel coordinates P... uv = [u, v], camera coordinate system P C =[X c Y c Z c Ground coordinate system P W =[X W Y W Z W The dots on the zebra crossing are on the ground, Z. W =0, camera imaging equation Z C P uv =KP C Coordinate system transformation equation P C =TP W Using the zebra crossing corners as calibration references, minimize the projection error and solve for the transformation matrix T from the camera to the ground coordinate system. cg As a camera extrinsic parameter, obtain the camera extrinsic parameter T. cg Then, the ground equations in the camera system were further obtained.

4. The three-dimensional detection and tracking method for roadside cameras based on ground constraints according to claim 1, characterized in that, For non-vehicle targets, filtering out those with a confidence level below a preset threshold is based on the inference confidence level. f The objective is to use a non-maximum suppression algorithm to filter out methods with confidence levels higher than a threshold. n The goal is to statistically analyze the methods with confidence scores above a threshold. n The number of targets obscured by the target; if the confidence level of the output result is higher than the threshold. n Or, although the confidence level is below the threshold, the number of occluded targets exceeds the threshold. num Then the target will be output as the final reliable two-dimensional detection result.

5. The three-dimensional detection and tracking method for roadside cameras based on ground constraints according to claim 1, characterized in that, The two-dimensional backbone network uses ResNet50, HRNet, or ShuffleNet.

6. The three-dimensional detection and tracking method for roadside cameras based on ground constraints according to claim 1, characterized in that, During training, the 2D backbone network uses the following methods: for landing point prediction, the training objective is the relative offset of the landing point relative to the bottom center point, and the L1 loss function is used; for length and width prediction, the training objective is the logarithmic value of the actual length and width compared to the baseline length and width, and the L1 loss function is used; for angle prediction, the MultiBin loss function is used.

7. The three-dimensional detection and tracking method for roadside cameras based on ground constraints according to claim 1, characterized in that, Using camera extrinsic parameters T cg The specific steps for converting the 3D landing point and orientation angle from the camera coordinate system to the ground coordinate system are as follows: The transformation of the three-dimensional landing point coordinate system can be solved using the following formula: T cg ×[X c ,Y c ,Z c ,1] T =[X g ,Y g ,Z g ,1] T Among them, [X c Y c Z c [1] represents the three-dimensional coordinates of the landing point in the camera coordinate system, [X] represents the coordinates of the landing point in the camera coordinate system. g Y g Z g [1] represents the three-dimensional coordinates of the landing point in the ground coordinate system. For orientation angular coordinate system transformations, the following formula needs to be used to solve: Where, θ c Let θ be the angle between the target orientation and the x-axis in the camera coordinate system. g Let be the angle between the target orientation and the x-axis in the ground coordinate system. This is a virtual vector constructed in the camera coordinate system based on the orientation angle. It is a virtual vector in the ground coordinate system.

8. A multi-stage three-dimensional detection and tracking system utilizing ground priors from the perspective of a roadside camera, characterized in that, It includes a parameter acquisition module, a two-dimensional detection module, a front-and-back frame matching relationship acquisition module, a three-dimensional detection module, and a tracking module; The parameter acquisition module is used to obtain the ground equation offline and calibrate the camera extrinsic parameters from the camera system to the ground coordinate system based on the images from the monitoring intersection camera, the intrinsic parameter K of the monitoring intersection camera, and the distortion coefficient. The 2D detection module uses the YOLOX detection model to detect camera images and obtains 2D detection results, which include category, confidence score, and 2D bounding box. For non-vehicle targets, the non-dense suppression algorithm is used in combination with the confidence score to obtain the 2D detection results for non-vehicle targets. For vehicle targets, the non-maximum suppression algorithm is used in combination with the confidence score to obtain the 2D detection results for vehicle targets. The module for obtaining the matching relationship between preceding and following frames uses the Sort algorithm based on Kalman filtering and Hungarian matching. It calculates the cross-union ratio (CUR) by combining the two-dimensional bounding boxes of non-vehicle targets, constructs an affinity matrix using the CUR as a metric parameter, and performs dynamic modeling using a constant velocity motion model to track non-vehicle targets in two dimensions, thereby obtaining the matching relationship between preceding and following frames of non-vehicle targets. The 3D detection module takes camera images, 2D bounding boxes of all targets in the images, and ground equations as input to the roadside monocular 3D target detection model. It sequentially crops image blocks according to the target's 2D bounding boxes, scales all image blocks to a uniform size, and extracts image features of each target sequentially using a 2D backbone network. After obtaining the image features, it estimates the target image landing point, camera orientation angle, and actual length and width. Combining the ground plane equations and camera intrinsic parameters, it calculates the 3D landing point of the target in the camera coordinate system and transforms it to the ground coordinate system. Combining the camera orientation angle and actual length and width, it obtains the 3D detection result, which includes the 3D landing point, orientation angle, and actual length and width in the ground coordinate system. The tracking module combines the 3D detection results of vehicle targets and uses a 3D multi-target tracking method based on Kalman filtering and greedy matching. In the ground coordinate system, it constructs an affinity matrix using the Euclidean distance between the 3D landing points of the targets as a metric parameter, and uses a constant acceleration motion model for dynamic modeling to construct a vehicle tracker, acquiring the 3D position, size, orientation, velocity, and historical trajectory information of the vehicle targets. It also uses the preceding and following frame matching relationship to complete the data association of non-vehicle targets, and uses a constant acceleration motion model based on the Kalman filtering algorithm for dynamic modeling in the ground coordinate system to construct a non-vehicle tracker, estimating the 3D position, size, orientation, velocity, and historical trajectory information of non-vehicle targets.

9. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer-executable program, the processor reading the computer-executable program from the memory and executing it, and the processor executing the computer-executable program is able to implement the roadside camera three-dimensional detection and tracking method based on ground constraints as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the roadside camera three-dimensional detection and tracking method based on ground constraints as described in any one of claims 1 to 7.

Images