Multi-view transformation fusion roadside monocular camera vehicle three-dimensional detection method and system

Abstract

The invention discloses a multi-view transformation and fusion roadside monocular camera vehicle three-dimensional detection method and system. A roadside monocular camera is widely applied to vehicle three-dimensional detection increasingly. Due to the influences of view angle limitation, long-distance target blurring and complex traffic environment, an existing method faces a large challenge in vehicle three-dimensional detection. In order to solve the problem, a BDA data enhancement matrix is firstly constructed to enhance the diversity of training data, so that the generalization ability of the model is effectively improved. Then, a 3D-2D visual transformation module is constructed, two-dimensional image features collected by the monocular camera are efficiently mapped to a bird's-eye view (BEV) feature space, and the spatial expression ability of target detection is significantly improved. Furthermore, the output features of the 2D-3D visual transformation model and the 3D-2D visual transformation module are fused, and a fused BEV feature map is generated, so that the accuracy and robustness of a detection result are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle three-dimensional detection, and relates to a method and system for three-dimensional vehicle detection using a roadside monocular camera with multi-view transformation and fusion. Background Art

[0002] In the field of intelligent transportation, three-dimensional vehicle detection technology based on roadside monocular cameras is one of the core technologies for achieving key functions such as traffic flow monitoring, illegal parking detection, and precise measurement of vehicle position and posture. This technology uses roadside monocular cameras to obtain three-dimensional information about vehicles, enabling real-time monitoring of the spatial relationship between vehicles and road infrastructure, providing important data support for traffic management and autonomous driving. For example, in traffic flow monitoring, this technology can accurately count the number and speed of vehicles in different lanes; in illegal parking detection, it can quickly identify illegally parked vehicles and provide their precise location information. However, existing three-dimensional vehicle detection technology still has many problems and shortcomings in roadside applications that need to be improved.

[0003] Currently, 3D vehicle detection technologies for roadside monocular camera scenes fall into three main categories: The first category involves 3D detectors modified from 2D detectors. These methods achieve 3D detection by adding an additional regression branch to 2D detection. This approach offers a simple framework and fast inference speed, but detection accuracy is limited, making it difficult to accurately perceive vehicle depth and 3D pose. The second category involves 3D detectors for roadside scenes based on the Transformer architecture. These detectors, leveraging their powerful feature extraction capabilities and ability to model long-range dependencies, offer excellent 3D detection accuracy, but they require significant computational resources. The third category involves 3D detectors based on depth prediction. The mainstream approach utilizes a backbone network to extract image features and generate a feature map. The depth or height of each feature point is then predicted. Combining the feature point coordinates with the predicted depth / height, a pseudo-point cloud in 3D space is generated through camera inverse projection. This point cloud is then projected onto a bird's-eye view (BEV) feature map, ultimately performing 3D detection based on the BEV feature map. However, current 3D detector technology based on depth estimation still faces the following challenges in roadside applications: Low detection accuracy for distant targets, prone to missed detections, which limits the detector's reliability in long-distance monitoring scenarios; overfitting of scene information from training samples, resulting in a significant drop in detection accuracy when migrating to unknown scenes; and low prediction accuracy in the mask branch used to suppress background overfitting, which impacts the generation of high-quality BEV feature maps. Furthermore, the large number of parameters in this mask branch increases computational complexity, impacting the overall detector's efficiency. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems in the prior art of three-dimensional vehicle detection, such as poor long-range target detection accuracy, scene overfitting, and poor mask prediction, which lead to the inability to obtain true three-dimensional information of the vehicle. The present invention provides a method and system for three-dimensional vehicle detection using a roadside monocular camera using multi-view transformation and fusion.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for three-dimensional vehicle detection using a roadside monocular camera using multi-view transformation and fusion, comprising the following steps:

[0007] Acquire an image containing vehicle traffic scene information, and obtain a camera-related transformation matrix based on the traffic scene information;

[0008] Enhance the image to obtain the image enhancement matrix; construct the BDA data enhancement matrix; integrate the camera's related transformation matrix, BDA data enhancement matrix and image enhancement matrix into a matrix dictionary;

[0009] A multi-scale feature map is extracted from the image enhancement matrix, and a semantic feature map, a depth probability feature map and a mask probability feature map are obtained based on the multi-scale feature map and the matrix dictionary; a 2D-3D visual transformation model and a 3D-2D visual transformation model are obtained, and the semantic feature map, the depth probability feature map and the mask probability feature map are used as inputs of the model to obtain the first BEV feature map and the second BEV feature map respectively, and the first BEV feature map and the second BEV feature map are fused to obtain a BEV feature map, and the detection head output is obtained based on the BEV feature map.

[0010] A further improvement of the present invention is:

[0011] The vehicle traffic scene information includes:

[0012] An image and a ground depth map, camera projection parameters, camera extrinsics, ground equations in the camera coordinate system, and target annotation information;

[0013] The target annotation information includes the target type, two-dimensional frame, three-dimensional frame bottom center point, three-dimensional frame size and three-dimensional deflection angle parameters;

[0014] The camera's related transformation matrix includes the camera's rotation matrix to the world coordinate system, the translation matrix, and the camera projection matrix;

[0015] The rotation matrix and translation matrix are calculated according to the ground equation in the camera coordinate system;

[0016] The camera projection matrix is ​​constructed according to the camera projection parameters.

[0017] The bottom center point, size and deflection angle of the 3D frame of the target in the original image are converted by the BDA matrix to obtain the converted 3D target frame information.

[0018] The calculation process of the 3D-2D visual transformation model includes:

[0019] Preset the grid shape of the BEV feature map, perform grid division and height stratification on the grid shape, and obtain the three-dimensional reference points of the input image;

[0020] Project the three-dimensional reference point of the image into the image space to obtain the three-dimensional point index of the image;

[0021] Based on the 3D point index of the image, the semantic feature c and the corresponding depth probability p of the corresponding image feature point are obtained through the 3D sampling point index, and the feature of each 3D point is obtained through p×c calculation;

[0022] The three-dimensional point features belonging to the same BEV grid are accumulated to obtain a second BEV feature map with a shape of c×128×128.

[0023] The grid shape of the preset BEV characteristic map is 128×128, the actual physical size corresponding to each grid is 0.8m×0.8m, and the height layer is divided into 13 layers;

[0024] Define the center point of each grid as the x- and y-coordinates of the preset 3D reference point, and set multiple layers of height sampling points along the vertical direction to form a 3D reference point cloud;

[0025] When performing height division, the height range is set to [-5, 3] meters, and dense sampling point areas and sparse sampling point areas are set according to the proportion of vehicles;

[0026] The coordinates of the three-dimensional reference point are normalized and the range is set to [0,1], resulting in:

[0027]

[0028] Project the 3D reference point of the image to the image space to obtain the 3D point index of the image, including:

[0029] Transform the world coordinate system to the image coordinate system through the transformation matrix to obtain its projection coordinates (u, v) and depth value z on the camera image plane;

[0030] The coordinates and depth channel number of the image feature map corresponding to each reference 3D point are obtained by the following formula:

[0031]

[0032] Among them, W = 96 and H = 54 are the width and height of the image feature map, and d = 0.8m is the depth interval.

[0033] The fusing of the first BEV characteristic map and the second BEV characteristic map to obtain the BEV characteristic map includes:

[0034] Concatenate the first BEV feature map and the second BEV feature map to obtain a high-dimensional feature tensor;

[0035] Extract the local attention features and global attention features of the high-dimensional feature tensor, add the local attention features and the global attention features, and generate the channel attention weight channel_factor, whose value range is [0,1];

[0036] According to the channel attention weight channel_factor, the features extracted from the first BEV feature map and the second BEV feature map are fused:

[0037] out=channel_factor×x1+(1-channel_factor)×x2

[0038] Wherein, x1 represents the first BEV feature map obtained by the 2D-3D visual transformation model, and x2 represents the second BEV feature map obtained by the 3D-2D visual transformation model.

[0039] A roadside monocular camera 3D vehicle detection system based on multi-view transformation and fusion, comprising:

[0040] A data acquisition module is used to acquire images containing vehicle traffic scene information and obtain a related transformation matrix of the camera based on the traffic scene information;

[0041] The data enhancement module is used to enhance the image and obtain the image enhancement matrix; construct the BDA data enhancement matrix; integrate the camera's related transformation matrix, BDA data enhancement matrix and image enhancement matrix into a matrix dictionary;

[0042] The detection module is used to extract a multi-scale feature map from the image enhancement matrix, obtain a semantic feature map, a depth probability feature map and a mask probability feature map based on the multi-scale feature map and the matrix dictionary, obtain a 2D-3D visual transformation model and a 3D-2D visual transformation model, use the semantic feature map, the depth probability feature map and the mask probability feature map as inputs of the model, obtain a first BEV feature map and a second BEV feature map respectively, fuse the first BEV feature map and the second BEV feature map to obtain a BEV feature map, and obtain the detection head output based on the BEV feature map.

[0043] A terminal device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0044] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of any method described in the present invention.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention discloses a method for three-dimensional vehicle detection with a roadside monocular camera using multi-view transformation fusion, constructs a BDA data enhancement matrix to enhance the scene conversion capability, constructs a 3D-2D visual transformation model for efficiently converting image feature maps into BEV feature maps to enhance target detection performance in intelligent traffic scenarios, and fuses the outputs of the 2D-3D visual transformation model and the 3D-2D visual transformation model to obtain a transformed BEV feature map. The detection results of the obtained feature map are more accurate. This method improves the three-dimensional detection accuracy of vehicles under the BEV perspective of the roadside monocular camera, especially in the detection of long-range targets. The measurement accuracy is greatly improved compared with the existing technology. After being migrated to a new scene, the model exhibits stronger generalization ability, and the vehicle three-dimensional detection effect is better than the existing model. It can obtain more realistic three-dimensional vehicle information and can be widely used in intelligent traffic monitoring, autonomous driving scene understanding, traffic safety management and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 is a network structure diagram in an embodiment of the present invention;

[0049] Figure 2 is an example of a roadside scene picture in an embodiment of the present invention;

[0050] Figure 3 1 is an example diagram of BDA enhancement in an embodiment of the present invention;

[0051] Figure 4 is an example diagram of mask generation in an embodiment of the present invention;

[0052] Figure 5 1 is a comparison diagram of distant target detection effects in an embodiment of the present invention;

[0053] Figure 6 This is a scene migration detection effect diagram in an embodiment of the present invention;

[0054] Figure 7 3 is a comparison chart of experimental accuracy in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0057] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0058] In the description of the embodiments of the present invention, it should be noted that if the terms "upper," "lower," "horizontal," "inner," etc. appear, the orientation or positional relationship indicated is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the inventive product is typically placed when in use. These terms are merely for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "first," "second," etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0059] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0060] In the description of the embodiments of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0061] The present invention is described in further detail below with reference to the accompanying drawings:

[0062] See also Figures 1 to 7 , an embodiment of the present invention discloses a method for three-dimensional vehicle detection using a roadside monocular camera with multi-view transformation fusion to solve the problems existing in the prior art, such as poor accuracy in long-distance target detection, scene overfitting, and poor mask prediction. First, a trained vehicle three-dimensional detection model is loaded. For the sample to be detected, its image is adjusted to a uniform size of 1080×1920, and the original camera internal and external parameters of the sample are converted into a camera-to-world rotation and translation matrix and a camera projection matrix, and these matrices are stored in a JSON dictionary format. Subsequently, the adjusted image and JSON dictionary are input into the detection network. The network outputs the category information, confidence level, and three-dimensional box information in the real world of each detected vehicle, specifically including the position of the bottom center point of the three-dimensional box, the size of the three-dimensional box, and the yaw angle, specifically including the following steps:

[0063] Step 1: Dataset acquisition and preprocessing

[0064] This paper uses the open source roadside perception target detection dataset Rope3D as the basic data source for model training. The dataset contains 50,009 images collected by multiple cameras distributed in different road sections. The camera collection samples of the dataset are as follows: Figure 2 , covering a rich set of traffic scene information. Each sample includes an image and a ground depth map, camera projection parameters, camera extrinsics, ground equations in the camera coordinate system, and object annotation information. The annotations detail key parameters such as the object type, 2D bounding box, 3D bounding box bottom center point, 3D bounding box size, and 3D deflection angle.

[0065] In order to adapt to the model construction requirements of the present invention, the Rope3D dataset is preprocessed in a targeted manner:

[0066] First, the rotation matrix and translation matrix from the camera to the world coordinate system are calculated according to the ground equation in the camera coordinate system. At the same time, the camera projection matrix is ​​constructed using the camera projection parameters, providing a basis for subsequent coordinate transformation and projection calculation.

[0067] Next, the target annotation information is adjusted, the annotation type is uniformly modified to be consistent with the model training category name, and the target information is simplified, retaining only key parameters such as type, 3D box bottom center point, 3D box size and yaw angle.

[0068] In addition, the mask generation method of the present invention is used to generate corresponding instance masks for all images in the dataset for mask supervision.

[0069] Finally, the dataset is randomly divided into training and validation sets in a ratio of 7:3.

[0070] Furthermore, dataset mask generation includes:

[0071] In order to generate a more accurate mask of the dataset image, the present invention adopts a method that combines semantic segmentation and annotation box information. The process of using the original image to obtain an accurate mask is as follows: Figure 4 , specifically including:

[0072] First, the dataset image is input into the YOLOv8 semantic segmentation model to generate a semantic segmentation mask f1 with the same size as the original image, where the pixel value of the target area is 1 and the background pixel value is 0.

[0073] At the same time, according to the two-dimensional border information annotated by the dataset, a label box mask f2 with the same size as the original image is generated, the pixel value of the target area is also 1, and the background pixel value is 0, and the two-dimensional border coordinates B[x min ,y min ,x max .y max ].

[0074] Next, through the intersection operation of the semantic segmentation mask f1 and the annotation box mask f2, the false detection targets that may be generated by the YOLOv8 model are removed to generate the mask f3.

[0075] Then, using the extracted border coordinates B, on the mask f3, if the pixel values ​​in the border are all 0, all pixel values ​​in the border are set to 1, thus obtaining the final accurate mask f final The specific formula is as follows:

[0076] f3=f1∩f2

[0077]

[0078] in, This represents the operation of setting the pixel values ​​in the area within the border B where the pixel values ​​are 0 to 1 in the mask f3.

[0079] Step 2: Dataset loading

[0080] Perform unified preprocessing operations on the image and its corresponding mask, including image enhancement processing such as scaling and rotation, and simultaneously calculate the corresponding image enhancement matrix. At the same time, for the BEV feature map, a three-dimensional bird's eye view data augmentation (BDA) operation is implemented and the corresponding BDA matrix is ​​generated. The camera projection matrix, camera-to-world coordinate system conversion matrix, image enhancement matrix and BDA matrix of each sample are integrated into a matrix dictionary for subsequent processing and calling

[0081] Furthermore, a Bird's Eye View Data Augmentation (BDA) operation is performed based on the three-dimensional space, and the corresponding BDA matrix is ​​generated, including:

[0082] The present invention migrates and adapts the BEV feature enhancement technology perceived by the vehicle side to the BEV vehicle three-dimensional detection scene of the roadside monocular camera. This method is based on the geometric enhancement method in the world coordinate system of the roadside scene, and the enhancement range is the part x>0. This method enhances the BEV feature map through three operations: global rotation, scale scaling and axisymmetric flipping. Global rotation enhancement is a random rotation around the Z axis of the origin of the world coordinate system, with an angle range of ±22.5°; scale scaling enhancement is based on the origin of the world coordinate system as the reference point, and the scaling ratio is between 0.95 and 1.05; axisymmetric flipping enhancement is based on the X axis as the reference axis, with a probability of 50%. These enhancement operations are matrix multiplied to obtain the overall BDA matrix. The specific formula is as follows:

[0083] T aug =R global ×S scale ×F flip

[0084] Since the detection head of the model is based on the BEV feature map for detection, after the BEV feature map passes through BDA, the label information supervised by the detection head also needs to be converted into enhanced information accordingly. The target information enhancement process of this method is as follows: Figure 3 .

[0085] Furthermore, the center coordinates, orientation angles, and size parameters of the 3D object box must be recalculated according to the BDA transformation rules to ensure that the annotation information is strictly consistent with the enhanced 3D spatial distribution. Specifically, for global rotation, the center coordinates of the 3D object box need to be rotated synchronously; for scaling, the center coordinates and size of the 3D object box need to be scaled; for axisymmetric flipping, the center coordinates of the 3D object box become [-x, y, z], and the yaw angle becomes π-yaw.

[0086] The specific information of the 3D target frame in this step includes the bottom center point of the 3D frame, the 3D frame size, and the 3D deflection angle.

[0087] Step 3: Train the detection model

[0088] The image enhancement matrix in step 2 and the matrix dictionary corresponding to each image are used as input, and the type information of the target vehicle contained in each image and the position and morphology information of each vehicle in the real world are used as output. A multi-view transformation fusion roadside monocular camera bird's-eye view vehicle 3D detection model is trained; the detection model is a 3D detection model based on depth prediction, and the model can be divided into BEV feature map generation and CenterNet detection head detection. The present invention makes improvements to BEV feature map generation that are more suitable for roadside monocular perspective 3D detection.

[0089] The BEV feature map generated by the above detection model:

[0090] c×128×128, c is the semantic feature dimension

[0091] The multi-task detection head output in the above detection model is:

[0092] c×512×512, c is the number of type information;

[0093] 2×512×512, 2 is the horizontal and vertical coordinates of the vehicle center point offset from the BEV perspective;

[0094] 7×512×512, 1 to 3 are the coordinates of the bottom center point of the vehicle in the world coordinate system [x, y, z], 4 to 6 are the length, width and height of the vehicle [l, w, h], and 7 is the yaw angle of the vehicle.

[0095] Furthermore, the BEV characteristic map is obtained by the following steps:

[0096] First, ResNet-50 is used as the feature extraction network to extract multi-scale feature maps from the input image. The feature maps output by the ResNet-50 network are 1024×54×96 and 2048×27×48, where 1024 and 2048 represent the number of channels. These feature maps are fused through the multi-scale feature map operation to output a feature map F with a shape of 512×54×96. fuse .

[0097] Then, the fused feature map F fuse The image matrix dictionary is input to the depth prediction module, which outputs three feature maps: semantic feature map (shape 80×54×96), depth probability feature map (shape 120×54×96) and mask probability feature map (shape 2×54×96).

[0098] These three feature maps are input into the 2D-3D module and the 3D-2D module, respectively. These two modules each output a BEV feature map with a shape of 80 × 128 × 128. Finally, the feature outputs of these two modules are sent to the BEV feature fusion module, which fuses them to generate the final BEV feature map with a shape of 80 × 128 × 128.

[0099] The loss function of the above model training consists of a basic loss function and an enhanced loss function, specifically including:

[0100] The basic loss functions include:

[0101] The vehicle classification loss function, the vehicle offset coordinate regression loss function at the BEV center point, and the vehicle 3D information regression loss function are used to optimize the classification accuracy of BEV features, the accuracy of the target center point position, and the accuracy of the 3D information of key points, respectively.

[0102] The enhanced loss function includes:

[0103] Image feature map target mask regression loss function and BEV feature map target mask regression loss function. The image feature map target mask regression loss function is used to optimize the image instance mask prediction branch, and the BEV feature map target mask regression loss function is used to optimize the BEV target mask prediction branch.

[0104] Furthermore, in this step, the 3D-2D visual transformation model processing process includes:

[0105] This paper proposes an innovative 3D-2D view transformation branch for efficiently converting image feature maps into BEV feature maps to enhance the performance of object detection in intelligent traffic scenarios. The input of this branch is an image feature map of shape c×54×96, where c is the number of semantic channels; the output is a BEV feature map of shape c×128×128, with the same number of channels as the input. The design of this branch includes three key steps: reference point generation, reference point projection, and BEV voxel pooling. The general process of this branch is as follows: Figure 1 3D-2D view transformation module.

[0106] First, the generation of 3D reference points is based on the meshing and height-layered design of the BEV feature map.

[0107] The preset grid shape of the BEV feature map is 128×128, the actual physical size corresponding to each grid is 0.8m×0.8m, and the height layers are divided into 13 layers.

[0108] In the BEV plane, the center point of each grid is defined as the x and y coordinates of the preset three-dimensional reference point, and multiple layers of height sampling points are set along the vertical direction to form a three-dimensional reference point cloud.

[0109] The height layer design adopts a segmented strategy: the height range is set to [-5, 3] meters, dense sampling points (for example, 9 evenly distributed points) are set in the area with a large proportion of vehicles (such as -2m to 3m), and sparse sampling points (such as 4 points) are set in the remaining area. The coordinates of these three-dimensional reference points are normalized to limit their range to [0, 1]. The specific normalization formula is:

[0110]

[0111] Subsequently, the three-dimensional reference point is projected into the image space through the transformation matrix from the world coordinate system to the image coordinate system.

[0112] The specific process includes: converting the normalized coordinates of the reference point cloud into actual world coordinates, and the normalized coordinate xyz range [0,1] corresponds to the world coordinates [0~102.4m, -51,2m~51.2m, -5m~3m]. Multiplying the point cloud coordinates with the transformation matrix to obtain its projection coordinates (u, v) and depth value z on the camera image plane. At the same time, filter out the projection points that are not in the image range [1080,1920] and the depth range [4m,100m]. The projection coordinates (u, v) are normalized, and then the coordinates and depth channel number of the image feature map corresponding to each reference 3D point are obtained by the following formula:

[0113]

[0114] Here, W = 96 and H = 54 are the width and height of the image feature map, and d = 0.8m is the depth interval. The resulting 3D point index is of the form [B, Nq, 3], where B is the number of samples, Nq is the number of reference 3D points, and 3 represents the u and v coordinates of the image feature map and the depth channel number.

[0115] In the voxel pooling stage, Bird's Eye View Pooling (BEV Pooling) is used. The semantic features c and the corresponding depth probability p of the corresponding image feature points are obtained by indexing the 3D sampling points. The features of each 3D point are calculated using the p×c matrix. The 3D point features belonging to the same BEV grid are accumulated to obtain a BEV feature map with a shape of c×128×128. The channel dimension is consistent with the input features.

[0116] Furthermore, the first BEV characteristic map and the second BEV characteristic map are fused to obtain a BEV characteristic map, which specifically includes:

[0117] The feature fusion process of this module is as follows Figure 1The BEV feature fusion module takes as input two BEV feature maps of shape c×128×128, where c represents the number of channels. One feature map comes from the traditional 2D-3D view transformation branch, specifically the 2D to 3D projection module.

[0118] The other comes from the 3D-2D view transformation branch proposed in this paper. The two feature maps are first concatenated in the channel dimension to form a high-dimensional feature tensor with a shape of 2c×128×128.

[0119] The concatenated feature tensor is fed into the multi-scale channel attention module. Local feature extraction is performed by reducing the dimension of the input feature to After passing through a batch normalization (BN) layer and a ReLU activation function, it passes through another convolutional layer with a 1×1 kernel, ultimately outputting local attention features with a shape of c×128×128. Global feature extraction uses a global average pooling (GAP) operation to compress the bird's-eye view shape of the input features to 2c×1×1. After passing through a batch normalization (BN) layer and a ReLU activation function, it passes through a 1×1 convolutional layer with a kernel to generate global attention features with a shape of c×1×1. After adding the local and global features, a sigmoid function is used to generate the channel attention weight channel_factor, which has a value range of [0,1].

[0120] Finally, the features of the two branches are fused using the following formula:

[0121] out=channel_factor×x1+(1-channel_factor)×x2

[0122] where x1 and x2 represent the BEV feature maps from the 2D-3D and 3D-2D view transformation branches, respectively.

[0123] Furthermore, it also includes supervision loss:

[0124] A difference evaluation method between feature map instance masks and mask supervision information is adopted.

[0125] The mask supervision information is the ground-truth mask of the feature map generated by downsampling the original image mask. Each feature point in the feature map corresponds to a 16×16 pixel block in the input image. If there is at least one pixel value of 1 in the pixel block, the corresponding feature point value is marked as 1, indicating that the area contains the target; otherwise, the feature point value is 0, indicating the background. The model's feature map instance mask predicts the probability of the presence of an object at each feature point, compares these predicted probabilities with the ground-truth mask value of the feature map, and calculates the instance mask loss value according to the L1 loss function. The specific formula is as follows:

[0126]

[0127] By integrating multiple view transformation techniques, this invention significantly improves the accuracy of three-dimensional vehicle detection from the perspective of a roadside monocular camera (BEV). This is particularly true for long-range target detection, significantly improving upon existing technologies. Furthermore, the model exhibits enhanced generalization capabilities when transferred to new scenarios, outperforming existing models in three-dimensional vehicle detection. Furthermore, the invention boasts more accurate mask prediction and a more lightweight framework design, making it widely applicable to intelligent traffic monitoring, autonomous driving scenario understanding, and traffic safety management.

[0128] Figure 5 It is the visualization effect after three-dimensional vehicle detection. Compared with the original depth prediction three-dimensional detection network, it can detect distant vehicles more accurately.

[0129] This embodiment discloses a 3D vehicle detection system for roadside monocular cameras that uses multi-view transformation fusion. Roadside monocular cameras are increasingly used in 3D vehicle detection. Existing methods face significant challenges in 3D vehicle detection due to limited viewing angles, blurry objects at long distances, and complex traffic environments. To address these challenges, this paper first constructs a Bird's Eye View Data Augmentation (BDA) data augmentation matrix to increase the diversity of training data, thereby effectively improving the model's generalization capabilities. Subsequently, a 3D-2D visual transformation module is constructed to efficiently map the 2D image features captured by the monocular camera into the bird's-eye view (BEV) feature space, significantly enhancing the spatial representation of object detection. Furthermore, the output features of the 2D-3D visual transformation model and the 3D-2D visual transformation module are fused to generate a fused BEV feature map, thereby improving the accuracy and robustness of detection results. Experimental verification demonstrates that this method significantly outperforms existing methods in 3D vehicle detection accuracy from the perspective of a roadside monocular camera, particularly in detecting objects at long distances. Furthermore, the model exhibits stronger generalization performance in cross-scene transfer, enabling the acquisition of more realistic and accurate 3D vehicle information. This method provides an efficient and practical technical means for fields such as intelligent traffic monitoring, autonomous driving environment perception and traffic safety management.

[0130] A roadside monocular camera 3D vehicle detection system based on multi-view transformation and fusion, comprising:

[0131] A data acquisition module is used to acquire images containing vehicle traffic scene information and obtain a related transformation matrix of the camera based on the traffic scene information;

[0132] The data enhancement module is used to enhance the image and obtain the image enhancement matrix; construct the BDA data enhancement matrix; integrate the camera's related transformation matrix, BDA data enhancement matrix and image enhancement matrix into a matrix dictionary;

[0133] The detection module is used to extract a multi-scale feature map from the image enhancement matrix, obtain a semantic feature map, a depth probability feature map and a mask probability feature map based on the multi-scale feature map and the matrix dictionary, obtain a 2D-3D visual transformation model and a 3D-2D visual transformation model, use the semantic feature map, the depth probability feature map and the mask probability feature map as inputs of the model, obtain a first BEV feature map and a second BEV feature map respectively, fuse the first BEV feature map and the second BEV feature map to obtain a BEV feature map, and obtain the detection head output based on the BEV feature map.

[0134] The present invention has significant advantages in both detection accuracy and generalization capabilities, and can be widely used in fields such as intelligent traffic monitoring, autonomous driving scene understanding, and traffic safety management. The method includes: preprocessing the roadside perception target detection dataset to generate instance masks for mask supervision; designing a 3D-2D view transformation branch and a BEV feature fusion module based on an improved depth prediction network; and generating high-quality BEV feature maps through multi-scale feature map fusion, a depth prediction module, and bird's-eye view feature enhancement technology. By integrating multiple view transformation technologies, the present invention effectively solves the technical problem of insufficient long-distance perception of roadside monocular cameras in the existing technology, significantly improving the three-dimensional detection accuracy of vehicles.

[0135] A schematic diagram of a terminal device provided in one embodiment of the present invention. The terminal device in this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of each of the aforementioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in each of the aforementioned device embodiments are implemented.

[0136] The computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to accomplish the present invention.

[0137] The terminal device may be a computing device such as a desktop computer, a notebook computer, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0138] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0139] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0140] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0141] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

[0142] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A three-dimensional vehicle detection method using a roadside monocular camera based on multi-view transformation and fusion, characterized in that: The following steps are involved: Acquire an image containing vehicle traffic scene information, and obtain a camera-related transformation matrix based on the traffic scene information; Enhance the image to obtain the image enhancement matrix; construct the BDA data enhancement matrix; integrate the camera's related transformation matrix, BDA data enhancement matrix and image enhancement matrix into a matrix dictionary; A multi-scale feature map is extracted from the image enhancement matrix, and a semantic feature map, a depth probability feature map and a mask probability feature map are obtained based on the multi-scale feature map and the matrix dictionary; a 2D-3D visual transformation model and a 3D-2D visual transformation model are obtained, and the semantic feature map, the depth probability feature map and the mask probability feature map are used as inputs of the model to obtain the first BEV feature map and the second BEV feature map respectively, and the first BEV feature map and the second BEV feature map are fused to obtain a BEV feature map, and the detection head output is obtained based on the BEV feature map.

2. The method for three-dimensional vehicle detection using a roadside monocular camera with multi-view transformation and fusion according to claim 1 is characterized in that: The vehicle traffic scene information includes: An image and a ground depth map, camera projection parameters, camera extrinsics, ground equations in the camera coordinate system, and target annotation information; The target annotation information includes the target type, two-dimensional frame, three-dimensional frame bottom center point, three-dimensional frame size and three-dimensional deflection angle parameters; The camera's related transformation matrix includes the camera's rotation matrix to the world coordinate system, the translation matrix, and the camera projection matrix; The rotation matrix and translation matrix are calculated according to the ground equation in the camera coordinate system; The camera projection matrix is ​​constructed according to the camera projection parameters.

3. The method for three-dimensional vehicle detection using a roadside monocular camera with multi-view transformation and fusion according to claim 2 is characterized in that: The bottom center point, size and deflection angle of the 3D frame of the target in the original image are converted by the BDA matrix to obtain the converted 3D target frame information.

4. The method for three-dimensional vehicle detection using a roadside monocular camera with multi-view transformation and fusion according to claim 1 is characterized in that: The calculation process of the 3D-2D visual transformation model includes: Preset the grid shape of the BEV feature map, perform grid division and height stratification on the grid shape, and obtain the three-dimensional reference points of the input image; Project the three-dimensional reference point of the image into the image space to obtain the three-dimensional point index of the image; Based on the 3D point index of the image, the semantic feature c and the corresponding depth probability p of the corresponding image feature point are obtained through the 3D sampling point index, and the feature of each 3D point is obtained through p×c calculation; The three-dimensional point features belonging to the same BEV grid are accumulated to obtain a second BEV feature map with a shape of c×128×128.

5. The method for three-dimensional vehicle detection using a roadside monocular camera with multi-view transformation and fusion according to claim 4 is characterized in that: The grid shape of the preset BEV characteristic map is 128×128, the actual physical size corresponding to each grid is 0.8m×0.8m, and the height layer is divided into 13 layers; Define the center point of each grid as the x- and y-coordinates of the preset 3D reference point, and set multiple layers of height sampling points along the vertical direction to form a 3D reference point cloud; When performing height division, the height range is set to [-5, 3] meters, and dense sampling point areas and sparse sampling point areas are set according to the proportion of vehicles; The coordinates of the three-dimensional reference point are normalized and the range is set to [0,1], resulting in:

6. The method for three-dimensional vehicle detection using a roadside monocular camera with multi-view transformation and fusion according to claim 4 is characterized in that: Project the 3D reference point of the image to the image space to obtain the 3D point index of the image, including: Transform the world coordinate system to the image coordinate system through the transformation matrix to obtain its projection coordinates (u, v) and depth value z on the camera image plane; The coordinates and depth channel number of the image feature map corresponding to each reference 3D point are obtained by the following formula: Among them, W = 96 and H = 54 are the width and height of the image feature map, and d = 0.8m is the depth interval.

7. The method for three-dimensional vehicle detection using a roadside monocular camera with multi-view transformation and fusion according to claim 1 is characterized in that: The fusing of the first BEV characteristic map and the second BEV characteristic map to obtain the BEV characteristic map includes: Concatenate the first BEV feature map and the second BEV feature map to obtain a high-dimensional feature tensor; Extract the local attention features and global attention features of the high-dimensional feature tensor, add the local attention features and the global attention features, and generate the channel attention weight channel_factor, whose value range is [0,1]; According to the channel attention weight channel_factor, the features extracted from the first BEV feature map and the second BEV feature map are fused: out=channel_factor×x1+(1-channel_factor)×x2 Wherein, x1 represents the first BEV feature map obtained by the 2D-3D visual transformation model, and x2 represents the second BEV feature map obtained by the 3D-2D visual transformation model.

8. A roadside monocular camera vehicle 3D detection system based on multi-view transformation and fusion, characterized by: include: A data acquisition module is used to acquire images containing vehicle traffic scene information and obtain a related transformation matrix of the camera based on the traffic scene information; The data enhancement module is used to enhance the image and obtain the image enhancement matrix; Construct the BDA data enhancement matrix; integrate the camera's related transformation matrix, BDA data enhancement matrix and image enhancement matrix into a matrix dictionary; The detection module is used to extract a multi-scale feature map from the image enhancement matrix, obtain a semantic feature map, a depth probability feature map and a mask probability feature map based on the multi-scale feature map and the matrix dictionary, obtain a 2D-3D visual transformation model and a 3D-2D visual transformation model, use the semantic feature map, the depth probability feature map and the mask probability feature map as inputs of the model, obtain a first BEV feature map and a second BEV feature map respectively, fuse the first BEV feature map and the second BEV feature map to obtain a BEV feature map, and obtain the detection head output based on the BEV feature map.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images