A method for detecting traffic targets in three dimensions by a road end monitoring camera

Abstract

The application discloses a kind of road end monitoring camera traffic target three-dimensional detection methods, the core of this method is, for the problem that existing three-dimensional target detection method is not good at depth prediction, a kind of static traffic scene three-dimensional model construction method based on laser radar point cloud is designed, on this basis, a kind of high-resolution dynamic traffic target key point detection network is designed, the accurate key point position extraction of traffic target is realized, on the other hand, for the problem that existing algorithm is not enough for different installation conditions Generalization, a kind of traffic target three-dimensional attribute solution method based on two-dimensional key point and three-dimensional point cloud model is designed, this method can restore the three-dimensional space position of target from accurate two-dimensional key point position and three-dimensional point cloud model, to improve the generalization ability of existing system.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation and vehicle-road cooperative technology, and in particular to a method for three-dimensional detection of traffic targets using roadside surveillance cameras. Background Technology

[0002] With the continuous progress of urbanization and the booming development of the automobile industry in my country, the number of urban populations and cars is constantly increasing, leading to increasingly serious road congestion. The ever-increasing number of cars has resulted in a rapid expansion of traffic flow within urban road networks. Especially during peak travel periods, incidents such as increased queuing at intersections, decreased speed, and increased delays occur frequently. The inconvenience caused by congestion has become a significant factor restricting urban development. Specifically, this manifests as: declining road capacity, increasing vehicle delays, more frequent stops, and a severe decline in traffic efficiency. A scientifically sound traffic signal control scheme can effectively shorten vehicle delays at intersections, appropriately increase the capacity of road intersections, improve traffic efficiency on existing roads, and alleviate traffic congestion. Conventional green wave schemes are designed primarily by acquiring specific traffic flow parameters, selecting an appropriate cycle, and then generating a timing scheme based on the intersection traffic flow parameters. Therefore, accurate traffic target trajectory data is the data foundation and prerequisite for traffic signal control.

[0003] Using roadside cameras for 3D target detection is an important method for acquiring target trajectory data. Existing 3D target detection methods applied to roadside cameras primarily rely on depth prediction. The purpose of depth prediction is to obtain the true depth values ​​of pixels in the image, thereby reconstructing the 3D spatial position of the target using pixel locations and depth values. Therefore, accurate depth values ​​are crucial for 3D target detection. However, using pixel prediction for depth calculation suffers from several drawbacks. First, the method lacks real point-by-point data for learning, resulting in poor accuracy. Second, it is susceptible to the influence of factors such as installation location and orientation, making it difficult to generalize to different surveillance cameras and meet the performance requirements of practical applications. Summary of the Invention

[0004] To address the shortcomings of existing methods in terms of detection accuracy and scene generalization, this invention specifies the following technical solution:

[0005] First, a 3D point cloud scan is performed on the specific traffic scene to be observed to obtain a 3D point cloud model of the scene. The positions of key static points in the 3D point cloud and the corresponding static points in the camera image are marked. The spatial extrinsic parameters of the camera and the scene model are solved using the corresponding key points. Second, a high-resolution image target key point detection network is designed, and the key point positions of different traffic types are defined. The key point positions of dynamic traffic targets are inferred in the designed network. Finally, the 2D key points of the target are associated with the scene model using spatial pixel projection operations to obtain the 3D positions of the key points. The 3D parameters of the target are then analyzed to obtain the final detection results.

[0006] The specific steps of the method involved in this invention are as follows:

[0007] Step 1: Construct a 3D point cloud model of the traffic scene using LiDAR, establish the corresponding set of key points of the static scene in the traffic camera and LiDAR point cloud, and solve for the camera extrinsic parameters;

[0008] Sub-step 1: Scan the static traffic scene using a high-resolution LiDAR, preprocess the collected point cloud data, including data filtering, noise reduction, and inter-frame coordinate transformation; mark the time. The collected point cloud is ,in for The first time the lidar outputs One point, This is the maximum number of output points for this type of lidar; set... , , Maximum scan range on the coordinate axis , , and minimum scan range , , ,set up , , Voxel resolution on coordinate axes , , ,but voxel coordinates for:

[0009] (1)

[0010] For the number of points less than the set threshold The voxels are discarded and the points within the voxels are discarded; the coordinate transformation between laser point clouds is performed using the lidar odometry method to unify all coordinates to the initial frame position; the result of superimposing all frame point clouds is the 3D point cloud model of the traffic scene.

[0011] Sub-step 2: Mark the key point numbers of traffic elements on the traffic scene image captured by the camera, with no less than 6 points marked; in the obtained 3D point cloud model, mark the 3D key point positions of the corresponding elements in order of the key point numbers in the image, forming a set of corresponding points arranged in sequence.

[0012] Sub-step 3: Using the marked set of corresponding points, solve for the external parameter information of the 3D point cloud model to the camera; for each pair of pixels in the point set... and three-dimensional laser points There is a system of equations:

[0013] (2)

[0014] in to Let be the external parameters to be solved. The depth value of the 3D laser point in the camera coordinate system is eliminated using equation constraints. We can obtain:

[0015] (3)

[0016] That is, each pair of points in the set can provide 2 constraints. When the number of pairs of points in the set is equal to 6, the external parameters of the equation are solved by a system of linear equations. When the number of pairs of points in the set exceeds 6, the solution of the equation is solved by the least squares method.

[0017] Step 2: Design a high-resolution image target key point detection network and define the key point locations of different types of dynamic traffic targets. Perform key point location inference for dynamic traffic targets in the designed network.

[0018] Sub-step 1: Define the structure and components of the high-resolution target keypoint detection network. This high-resolution keypoint detection network consists of a visual feature extraction backbone, a multi-level feature fusion neck structure, and a keypoint detection head. The components of the visual feature extraction backbone include a multi-resolution convolutional fusion module, a max pooling layer with a stride of 2, and a residual connection module. The purpose of the multi-resolution convolutional fusion module is to fuse visual semantics and low-level information from different receptive fields. The purpose of the max pooling layer is to downsample the input feature map and extract high-level features. The purpose of the residual connection module is to enhance the stability and feature extraction effectiveness of the deep convolutional neural network during training. The components of the multi-level feature fusion neck structure include a deconvolution module and a feature map concatenation module. The deconvolution module upsamples the low-resolution feature map to obtain features to be concatenated. Figure 1The key point detection head is composed of key point classification convolution, key point group embedding convolution and key point location convolution. The key point classification convolution is responsible for obtaining the semantic type of each key point, the key point group embedding convolution is responsible for obtaining the same spatial embedding of the key point and the same target key point, and the key point location convolution is responsible for obtaining the target position of the key point on the feature map.

[0019] Sub-step 2: Define the key point location descriptions for different types of dynamic traffic targets; for vehicle targets, define their key point locations as the vertex locations of the smallest bounding rectangle under the ground projection; for pedestrian targets, define their key point locations as the center locations of the smallest bounding circle under the ground projection; for cyclist targets, define their key point locations as the vertex locations of the smallest bounding rectangle under the ground projection.

[0020] Sub-step 3: Perform key point location reasoning for dynamic traffic targets in the designed network; for traffic scene videos captured by cameras, starting from the first frame of the video, perform image format adjustment, image test data augmentation, model inference, and model key point location extraction operations in sequence; among them, image format adjustment is to adjust the input image format to a unified data encoding method, adjust the image resolution and normalize the data; image test data augmentation is to rotate, translate, mirror, adjust contrast, adjust brightness, and perform local magnification and filling operations on the image to generate multiple augmented data of one image, which are input into the model in parallel; the model performs forward inference operations based on the multiple augmented data input to obtain the output feature map; in the output feature map, key point locations are extracted based on the high confidence positions of the key point location convolution feature map. Key points with close embedding distances in key point groups are considered to be the same target, otherwise they are key points of different targets. The category probability with the highest probability in the key point classification convolution is taken as the semantic type of the key point;

[0021] Step 3: Using spatial pixel projection operation and the 3D scene point cloud model established in Step 1, associate the 2D key points detected in the image with the scene model to obtain the 3D position of the key points in the 3D scene model, and calculate the spatial attributes of the target based on the 3D position to obtain the final detection result.

[0022] Sub-step 1: For the key points of detection Calculate the pixel set of the point cloud model on the image. Given a distance threshold ,when At that time, in the set Select the one with the smallest distance to give by The same depth value is used to realize the data association between the two-dimensional key point and the scene model. The three-dimensional position of the key point in the three-dimensional scene model is calculated using formula (2).

[0023] Sub-step two: Based on the target's multiple 3D locations, analyze the target's 3D geometric properties; for traffic targets defined by the minimum bounding rectangle under ground projection, for points... The formulas for calculating the width and length are as follows:

[0024] (4)

[0025] Point Arrange them in order of length of side, and you will get For this ordered set of points, the formula for calculating the target's angle is as follows:

[0026] (5)

[0027] in and They are respectively the corresponding The depth and lateral positions; for traffic targets defined by the center of the smallest enclosing circle, a fixed circle radius is selected as the width and length of the target, and the angle of the target is not calculated;

[0028] Compared with existing methods, the present invention has the following advantages:

[0029] 1) The traffic target 3D detection method based on 3D scene model and key point target detection disclosed in this invention effectively overcomes the error problem of depth prediction in existing methods, improves the detection accuracy of existing methods in different depth ranges, and has high detection performance.

[0030] 2) The pixel projection target position calculation method disclosed in this invention effectively utilizes the depth prior information of the three-dimensional point cloud model compared with the traditional visual prediction method, which improves the range performance and generalization performance of the detection model in different scenarios. Especially for small targets, this method can use the precise position information of the three-dimensional point cloud model to achieve three-dimensional target detection results with higher accuracy and better generalization ability. Attached Figure Description

[0031] Figure 1 This is a diagram illustrating the overall technical solution of the present invention. Detailed Implementation

[0032] The technical solution of the present invention will be further described below with reference to the accompanying drawings and examples.

[0033] With the continuous progress of urbanization and the booming development of the automobile industry in my country, the number of urban populations and cars is constantly increasing, leading to increasingly serious road congestion. The ever-increasing number of cars has resulted in a rapid expansion of traffic flow within urban road networks. Especially during peak travel periods, incidents such as increased queuing at intersections, decreased speed, and increased delays occur frequently. The inconvenience caused by congestion has become a significant factor restricting urban development. Specifically, this manifests as: declining road capacity, increasing vehicle delays, more frequent stops, and a severe decline in traffic efficiency. A scientifically sound traffic signal control scheme can effectively shorten vehicle delays at intersections, appropriately increase the capacity of road intersections, improve traffic efficiency on existing roads, and alleviate traffic congestion. Conventional green wave schemes are designed primarily by acquiring specific traffic flow parameters, selecting an appropriate cycle, and then generating a timing scheme based on the intersection traffic flow parameters. Therefore, accurate traffic target trajectory data is the data foundation and prerequisite for traffic signal control.

[0034] Using roadside cameras for 3D target detection is an important method for acquiring target trajectory data. Existing 3D target detection methods applied to roadside cameras primarily rely on depth prediction. The purpose of depth prediction is to obtain the true depth values ​​of pixels in the image, thereby reconstructing the 3D spatial position of the target using pixel locations and depth values. Therefore, accurate depth values ​​are crucial for 3D target detection. However, using pixel prediction for depth calculation suffers from several drawbacks. First, the method lacks real point-by-point data for learning, resulting in poor accuracy. Second, it is susceptible to the influence of factors such as installation location and orientation, making it difficult to generalize to different surveillance cameras and meet the performance requirements of practical applications.

[0035] To address the shortcomings of existing methods in terms of detection accuracy and scene generalization, this invention specifies the following technical solution:

[0036] First, a 3D point cloud scan is performed on the specific traffic scene to be observed to obtain a 3D point cloud model of the scene. The positions of key static points in the 3D point cloud and the corresponding static points in the camera image are marked. The spatial extrinsic parameters of the camera and the scene model are solved using the corresponding key points. Second, a high-resolution image target key point detection network is designed, and the key point positions of different traffic types are defined. The key point positions of dynamic traffic targets are inferred in the designed network. Finally, the 2D key points of the target are associated with the scene model using spatial pixel projection operations to obtain the 3D positions of the key points. The 3D parameters of the target are then analyzed to obtain the final detection results.

[0037] Overall plan as follows Figure 1 As shown.

[0038] The specific steps of the method involved in this invention are as follows:

[0039] Step 1: Construct a 3D point cloud model of the traffic scene using LiDAR, establish the corresponding set of key points of the static scene in the traffic camera and LiDAR point cloud, and solve for the camera extrinsic parameters;

[0040] Sub-step 1: Scan the static traffic scene using a high-resolution LiDAR, preprocess the collected point cloud data, including data filtering, noise reduction, and inter-frame coordinate transformation; mark the time. The collected point cloud is ,in for The first time the lidar outputs One point, This represents the maximum number of output points for this type of lidar. Generally, a 64-line lidar has 131,072 output points. , , Maximum scan range on the coordinate axis , , and minimum scan range , , The settings are generally 100, 50, 3 and 0, -50, -1. , , Voxel resolution on coordinate axes , , The values ​​are typically 0.5, 0.5, 0.5. voxel coordinates for:

[0041] (1)

[0042] For the number of points less than the set threshold The voxels are discarded, along with the points within them. This threshold is typically 3. The coordinate transformation between laser point clouds is performed using the lidar odometer calculation method, unifying all coordinates to the initial frame position. The result of overlaying all frame point clouds is the 3D point cloud model of the traffic scene.

[0043] Sub-step 2: Mark the key point numbers of traffic elements on the traffic scene image captured by the camera, with no less than 6 points marked; in the obtained 3D point cloud model, mark the 3D key point positions of the corresponding elements in order of the key point numbers in the image, forming a set of corresponding points arranged in sequence.

[0044] Sub-step 3: Using the marked set of corresponding points, solve for the external parameter information of the 3D point cloud model to the camera; for each pair of pixels in the point set... and three-dimensional laser points There is a system of equations:

[0045] (2)

[0046] in to Let be the external parameters to be solved. The depth value of the 3D laser point in the camera coordinate system is eliminated using equation constraints. We can obtain:

[0047] (3)

[0048] That is, each pair of points in the set can provide 2 constraints. When the number of pairs of points in the set is equal to 6, the external parameters of the equation are solved by a system of linear equations. When the number of pairs of points in the set exceeds 6, the solution of the equation is solved by the least squares method.

[0049] Step 2: Design a high-resolution image target key point detection network and define the key point locations of different types of dynamic traffic targets. Perform key point location inference for dynamic traffic targets in the designed network.

[0050] Sub-step 1: Define the structure and components of the high-resolution target keypoint detection network. This high-resolution keypoint detection network consists of a visual feature extraction backbone, a multi-level feature fusion neck structure, and a keypoint detection head. The components of the visual feature extraction backbone include a multi-resolution convolutional fusion module, a max-pooling layer with a stride of 2, and a residual connection module. The purpose of the multi-resolution convolutional fusion module is to fuse visual semantics and low-level information from different receptive fields. The purpose of the max-pooling layer is to downsample the input feature map and extract high-level features. The purpose of the residual connection module is to enhance the stability and feature extraction effectiveness of the deep convolutional neural network during training. Generally, the multi-resolution convolutional fusion module is InceptionV2, and the residual connection module is ResNet. The components of the multi-level feature fusion neck structure include a deconvolution module and a feature map concatenation module. The deconvolution module upsamples the low-resolution feature map to obtain features to be concatenated. Figure 1 To achieve consistent resolution, different levels of feature stitching are performed. Generally, the feature map stitching module is a feature pyramid network. The components of the keypoint detection head include keypoint classification convolution, keypoint group embedding convolution, and keypoint location convolution. The keypoint classification convolution is responsible for obtaining the semantic type of each keypoint, the keypoint group embedding convolution is responsible for obtaining the same spatial embedding of the keypoint and the same target keypoint, and the keypoint location convolution is responsible for obtaining the target location of the keypoint on the feature map.

[0051] Sub-step 2: Define the key point location descriptions for different types of dynamic traffic targets; for vehicle targets, define their key point locations as the vertex locations of the smallest bounding rectangle under the ground projection; for pedestrian targets, define their key point locations as the center locations of the smallest bounding circle under the ground projection; for cyclist targets, define their key point locations as the vertex locations of the smallest bounding rectangle under the ground projection.

[0052] Sub-step 3: Perform key point location inference for dynamic traffic targets within the designed network; for traffic scene videos captured by cameras, starting from the first frame of the video, perform image format adjustment, image test data augmentation, model inference, and model key point location extraction operations sequentially; among them, image format adjustment involves adjusting the input image format to a unified data encoding method, generally RGB, adjusting the image resolution, and performing data normalization, with the resolution typically set to [value missing]. Image test data augmentation involves rotating, translating, mirroring, adjusting contrast and brightness, and locally magnifying and filling an image to generate multiple augmented data sets for a single image, which are then input into the model in parallel. The model performs forward inference operations based on the multiple input augmented data sets to obtain an output feature map. Keypoint locations are extracted from the output feature map based on the high-confidence locations of the convolutional feature map at the keypoint locations. Typically, 100 keypoints are extracted. Keypoints with close embedding distances are considered to be the same target; otherwise, they are considered to be different target keypoints. The category with the highest probability in the keypoint classification convolution is taken as the semantic type of the keypoint.

[0053] Step 3: Using spatial pixel projection operation and the 3D scene point cloud model established in Step 1, associate the 2D key points detected in the image with the scene model to obtain the 3D position of the key points in the 3D scene model, and calculate the spatial attributes of the target based on the 3D position to obtain the final detection result.

[0054] Sub-step 1: For the key points of detection Calculate the pixel set of the point cloud model on the image. Given a distance threshold ,when At that time, in the set Select the one with the smallest distance to give by The same depth value is used to realize the data association between the two-dimensional key point and the scene model. Generally, the threshold is selected as 2. Using formula (2), the three-dimensional position of the key point in the three-dimensional scene model is calculated.

[0055] Sub-step two: Based on the target's multiple 3D locations, analyze the target's 3D geometric properties; for traffic targets defined by the minimum bounding rectangle under ground projection, for points... The formulas for calculating the width and length are as follows:

[0056] (4)

[0057] Point Arrange them in order of length of side, and you will get For this ordered set of points, the formula for calculating the target's angle is as follows:

[0058] (5)

[0059] in and They are respectively the corresponding The depth and lateral positions; for traffic targets defined by the center position of the smallest enclosing circle, a fixed circle radius is selected as the width and length of the target, and the angle of the target is not calculated.

Claims

1. A method for three-dimensional detection of traffic targets using a roadside surveillance camera, characterized in that: The method includes the following steps: Step 1: Construct a 3D point cloud model of the traffic scene using LiDAR, establish the corresponding set of key points of the static scene in the traffic camera and LiDAR point cloud, and solve for the camera extrinsic parameters; Step 2: Design a high-resolution image target key point detection network and define the key point locations of different types of dynamic traffic targets. Perform key point location inference for dynamic traffic targets in the designed network. Step 3: Using spatial pixel projection operation and the 3D scene point cloud model established in Step 1, associate the 2D key points detected in the image with the scene model to obtain the 3D position of the key points in the 3D scene model, and calculate the spatial attributes of the target based on the 3D position to obtain the final detection result. The detailed steps for Step One are as follows: Sub-step 1: Use high-resolution LiDAR to scan the static traffic scene and preprocess the collected point cloud, including data filtering, noise reduction and inter-frame coordinate transformation. Marking time The collected point cloud is ,in for The first time the lidar outputs One point, This is the maximum number of output points for the lidar; set... , , Maximum scan range on the coordinate axis , , and minimum scan range , , ,set up , , Voxel resolution on coordinate axes , , ,but voxel coordinates for: （1） For the number of points less than the set threshold The voxels are discarded and the points within the voxels are discarded; the coordinate transformation between laser point clouds is performed using the lidar odometry method to unify all coordinates to the initial frame position; the result of superimposing all frame point clouds is the 3D point cloud model of the traffic scene. Sub-step 2: Mark the key point numbers of traffic elements on the traffic scene image captured by the camera, with no less than 6 points marked; in the obtained 3D point cloud model, mark the 3D key point positions of the corresponding elements in order of the key point numbers in the image, forming a set of corresponding points arranged in sequence. Sub-step 3: Using the marked set of corresponding points, solve for the external parameter information of the 3D point cloud model to the camera; for each pair of pixels in the point set... and three-dimensional laser points There is a system of equations: （2） in to Let be the external parameters to be solved. The depth value of the 3D laser point in the camera coordinate system is eliminated using equation constraints. Able to obtain: （3） That is, each pair of points in the set can provide 2 constraints. When the number of pairs of points in the set is equal to 6, the external parameters of the equation are solved by a system of linear equations. When the number of pairs of points in the set exceeds 6, the solution of the equation is solved by the least squares method.

2. The method for three-dimensional detection of traffic targets using a roadside surveillance camera according to claim 1, characterized in that: The detailed steps for step two are as follows: Sub-step 1: Define the structure and components of the high-resolution target keypoint detection network; the high-resolution keypoint detection network consists of a visual feature extraction backbone structure, a multi-level feature fusion neck structure, and a keypoint detection head. The components of the visual feature extraction backbone structure include a multi-resolution convolutional fusion module, a max pooling layer with a stride of 2, and a residual connection module. The purpose of the multi-resolution convolutional fusion module is to fuse visual semantics and low-level information from different receptive fields. The purpose of the max pooling layer is to downsample the input feature map and extract high-level features. The purpose of the residual connection module is to enhance the stability and feature extraction effectiveness of the deep convolutional neural network during training. The components of the multi-level feature fusion neck structure include a deconvolution module and a feature map stitching module. The deconvolution module upsamples the low-resolution feature map to obtain the same resolution as the feature map to be stitched, and performs feature stitching at different levels. The components of the keypoint detection head include keypoint classification convolution, keypoint group embedding convolution, and keypoint location convolution. The keypoint classification convolution is responsible for obtaining the semantic type of each keypoint, the keypoint group embedding convolution is responsible for obtaining the same spatial embedding of the keypoint and the same target keypoint, and the keypoint location convolution is responsible for obtaining the target location of the keypoint on the feature map. Sub-step 2: Define the key point location descriptions for different types of dynamic traffic targets; for vehicle targets, define their key point locations as the vertex locations of the smallest bounding rectangle under the ground projection; for pedestrian targets, define their key point locations as the center locations of the smallest bounding circle under the ground projection; for cyclist targets, define their key point locations as the vertex locations of the smallest bounding rectangle under the ground projection. Sub-step 3: Perform key point location reasoning for dynamic traffic targets within the designed network; for traffic scene videos captured by cameras, starting from the first frame of the video, perform image format adjustment, image test data augmentation, model inference, and model key point location extraction operations sequentially; among them, image format adjustment involves adjusting the input image format to a unified data encoding method, adjusting the image resolution, and performing data normalization; image test data augmentation involves rotating, translating, mirroring, adjusting contrast, adjusting brightness, and locally magnifying and filling the image to generate multiple augmented data for a single image, which are then input into the model in parallel; the model performs forward inference operations based on the multiple input augmented data to obtain the output feature map; In the output feature map, the key point location is extracted based on the high confidence position of the convolutional feature map of the key point location. Key points with close embedding distances in a key point group are considered to be the same target; otherwise, they are key points of different targets. The category with the highest probability in the key point classification convolution is taken as the semantic type of the key point.

3. The method for three-dimensional detection of traffic targets using a roadside surveillance camera according to claim 2, characterized in that: The detailed steps for step three are as follows: Sub-step 1: For the key points of detection Calculate the pixel set of the point cloud model on the image. Given a distance threshold ,when At that time, in the set Select the one with the smallest distance to give by The same depth value is used to realize the data association between the two-dimensional key point and the scene model. The three-dimensional position of the key point in the three-dimensional scene model is calculated using formula (2). Sub-step two: Based on the target's multiple 3D locations, analyze the target's 3D geometric properties; for traffic targets defined by the minimum bounding rectangle under ground projection, for points... The formulas for calculating the width and length are as follows: （4） Point Arrange them in order of length of side, and you will get For this ordered set of points, the formula for calculating the target's angle is as follows: （5） in and They are respectively the corresponding The depth and lateral positions; for traffic targets defined by the center position of the smallest enclosing circle, a fixed circle radius is selected as the width and length of the target, and the angle of the target is not calculated.

Images