A ground constraint-based roadside camera three-dimensional target detection method and system

Abstract

The application discloses a roadside camera three-dimensional target detection method and system based on ground constraints. The method uses the advantages of camera fixed positions on the roadside, offline fits a ground equation as prior semantic information, performs two-dimensional target detection in an image perspective, guarantees the recall rate of the target, and then uses a perspective conversion module based on ground constraints to improve the detection accuracy. Specifically, image data is input into a roadside monocular three-dimensional target detection model, two-dimensional backbone network is used to extract image features, and a two-dimensional target detector is used for two-dimensional target detection to obtain a two-dimensional target detection frame. Depth estimation is performed by using a DepthNet network, and image features are projected by using an LSS method to obtain BEV features of a scene. A two-dimensional target frame is projected to a BEV space by using an offline obtained ground equation, the BEV features of the scene are cropped as proposal frames for three-dimensional target detection, RoI feature with the same shape is obtained by predicting three-dimensional target information, and NMS processing is performed to obtain a final three-dimensional target detection result.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of intelligent transportation, and particularly relates to a roadside camera three-dimensional target detection method and system based on ground constraints. BACKGROUND

[0002] With the continuous development of intelligent transportation, it is necessary to use intelligent roadside devices to realize real-time and accurate perception of traffic conditions. At present, commonly used roadside perception devices include laser radar and RGB camera, etc. Among them, the RGB camera is low in cost and has been widely deployed and applied at the roadside. In the intelligent transportation scene, in order to obtain the specific position, three-dimensional size and other information of pedestrians and vehicles on the road, three-dimensional target detection of the target in the image is needed.

[0003] The camera installation position of the roadside is fixed, has rich semantic information, and is high in installation position, so it has the ability of over-the-horizon perception. The existing three-dimensional target detection method based on the roadside camera mostly follows the method in the vehicle-side camera, and does not fully exploit the prior information of the roadside, so the detection accuracy needs to be improved. SUMMARY

[0004] In order to solve the problems in the prior art, the purpose of the embodiments of the present application is to provide a roadside camera three-dimensional target detection method based on ground constraints, which fully exploits the scene information and also widens the applicable scene of the model.

[0005] In order to achieve the above purpose, the technical scheme adopted by the present application is: a roadside camera three-dimensional target detection method based on ground constraints, comprising the following steps:

[0006] Obtain the original image data of the roadside camera as input, use the trained roadside monocular three-dimensional target detection model for inference, and finally obtain the category of the three-dimensional target and the length, width, height, bottom center point and orientation angle in the camera coordinate system;

[0007] Specifically, after the image data is input into the roadside monocular three-dimensional target detection model, the roadside monocular three-dimensional target detection model extracts image features by using a two-dimensional backbone network, and after the image features are obtained, two branches are formed; branch one uses a two-dimensional target detector to perform two-dimensional target detection to obtain a two-dimensional target detection frame; branch two uses a DepthNet network to perform depth estimation, and uses an LSS method to project the image features to obtain BEV features of a scene; a ground equation is used to project the two-dimensional target frame to a BEV space as a proposal frame for three-dimensional target detection; the proposal frame is used to crop the BEV features of the scene to obtain RoI features with the same shape, and the RoI features are used to predict information of a three-dimensional target, and NMS processing is performed to obtain a three-dimensional target detection result, that is, a category of the three-dimensional target and a length, a width, a height, a bottom center point and an orientation angle of the three-dimensional target in a ground coordinate system, and finally a transformation matrix from the ground coordinate system to the camera coordinate system is used to transform the ground center point and the orientation angle of the three-dimensional target to the camera coordinate system.

[0008] Further, the ground equation comprises:

[0009] The zebra crossing is located on the ground, and solving the ground equation in the camera coordinate system is converted into transforming the [0, 0, 1] vector in the physical coordinate system to the camera coordinate system, and in the transformation process, the transformation matrix is combined , the pixel coordinates , the camera coordinate system , the physical coordinate system , and , the camera imaging equation is , and the conversion equation from the physical coordinate system to the camera coordinate system is ; the zebra crossing corner point is used as a calibration reference to construct a PnP problem, minimize the projection error, solve the transformation matrix from the camera to the physical coordinate system, and then obtain the ground equation in the camera coordinate system ;

[0010] If there is no zebra crossing in the camera field of view, the ground is fitted based on the point cloud data collected by the laser radar, and the external parameter matrix T LC is obtained by calibrating the external parameters of the laser radar and the camera. , and then the ground equation in the camera coordinate system is obtained:

[0011] ;

[0012] When fitting the ground equation in the laser radar, the ground points are first segmented using Patchwork++, and then the RANSAC algorithm is used to fit the ground points to obtain the ground equation.

[0013] Further, the two-dimensional backbone extraction network used is ResNet-50; the two-dimensional target detector used is SSD or other two-dimensional target detector based on convolution features, and the two-dimensional target detector is trained separately to achieve a recall rate of 99.5% for targets in the image; and the depth estimation network used is a DepthNet network, which estimates D discrete depths for each image feature.

[0014] Further, projecting the two-dimensional target frame into the BEV space using the ground equation includes: first, projecting the two-dimensional target frame into the camera coordinate system, and then converting it into the ground coordinate system. The left lower corner point and the right lower corner point of the two-dimensional target frame satisfy the ground equation, and the points on the ground in the camera system satisfy:

[0015]

[0016] The following two equations are obtained by combining the ground equation and the above equation, that is, according to the ground equation and the projection point of the target landing point in the image, the coordinates of the point in the camera coordinate system are obtained ,

[0017] The solving formula of the point in the camera system is:

[0018]

[0019] According to the transformation matrix from the camera coordinate system to the ground coordinate system , the target frame in the camera system is projected into the ground coordinate system;

[0020] Given the ground equation in the camera coordinate system, let the origin of the ground coordinate system be the O point of the camera system, X g O y g If the plane is parallel to the ground, the transformation matrix from the camera system to the ground coordinate system can be obtained Each unit vector in the above equation is calculated using Schmidt orthogonalization as follows:

[0021] .

[0022] Where A, B, and C are the coefficients of the ground equation in the camera coordinate system, x c is the unit vector of the x axis in the camera coordinate system, x cg , y cg , z cg are the representations of the x-axis, y-axis, and z-axis unit vectors in the ground system in the camera system, respectively.

[0023] Further, the RoI features are used to predict information of the three-dimensional target, and NMS processing is performed to obtain a three-dimensional target detection result, including:

[0024] The BEV feature is cropped using the proposal box, the BEV feature region corresponding to each proposal is divided into a grid of pool_w * pool_h, and each part of the grid is subjected to maximum pooling processing; the offset of each proposal box is obtained through the regression head using the obtained proposal feature map, and the proposal box is corrected using the offset.

[0025] The height of the target is predicted using a separate height regression branch, and then the length, width, height and bottom center point information of the target in the ground coordinate system are obtained.

[0026] Further, the RoI features are used to predict information of the three-dimensional target, and NMS processing is performed to obtain a three-dimensional target detection result, including: the orientation angle of the three-dimensional target is obtained through reasoning, and specifically as follows: the orientation angle of the target is predicted in a Muti-bin manner, the surrounding 360-degree space is divided into multiple bins with the bottom center point of the target as the origin, the orientation angle falls in which bin is predicted through classification, and then the offset of the target in the bin is predicted to obtain the orientation angle of the target in the ground coordinate system.

[0027] Further, the loss function of the two-dimensional target detector is:

[0028]

[0029] wherein, N is the number of predicted positive samples, c is the actual class of the target, a softmax loss is used, a SmoothL1 loss is used;

[0030] The loss function for training the roadside monocular three-dimensional target detection model is:

[0031]

[0032] wherein , , is a constant, is the classification loss of the target, is the three-dimensional regression loss of the target, is the orientation loss of the target.

[0033] Based on the above idea, the application further provides a roadside camera three-dimensional target detection system based on ground constraints, comprising a data acquisition module and an inference module.

[0034] The data acquisition module is configured to acquire raw image data of a roadside camera.

[0035] The inference module obtains a three-dimensional target detection result by processing the raw image data based on a roadside monocular three-dimensional target detection model.

[0036] The present application can also provide a computer device comprising a processor and a memory, the memory being configured to store a computer executable program, the processor being configured to read the computer executable program from the memory and execute the computer executable program, and the processor being configured to implement the roadside camera three-dimensional target detection method based on ground constraints when executing the computer executable program.

[0037] The present application can also provide a computer device comprising a processor and a memory, the memory being configured to store a computer executable program, the processor being configured to read the computer executable program from the memory and execute the computer executable program, and the processor being configured to implement the roadside camera three-dimensional target detection method based on ground constraints when executing the computer executable program.

[0038] Compared with the prior art, the present application has at least the following beneficial effects: the present application acquires ground information offline, fully mines scene information, and also widens the applicable scene of the model; firstly, two-dimensional target detection is performed in the image perspective, which guarantees the recall rate of the target and has good robustness; the ground equation is acquired offline by taking advantage of the fact that the roadside camera is stationary and the road end scene is fixed, and the two-dimensional detection frame is accurately projected into the camera system by using the ground equation as a proposal for the three-dimensional detection frame; the present application uses the two-dimensional target detection result as a proposal for the three-dimensional target, which greatly optimizes the decoding of the BEV feature and improves the detection accuracy.

[0039] Furthermore, the projection of the 2D detection box is used as the proposal box. The BEV features are clipped using the proposal box. The BEV feature region corresponding to each proposal is divided into a grid of pool_w * pool_h. Max pooling is then performed on each part of the grid to ensure that the features obtained by clipping all proposal boxes are of size pool_w * pool_h, thus achieving fixed-length feature output. Attached Figure Description

[0040] Figure 1 This is a diagram of several coordinate systems used in this invention: pixel coordinate system, camera coordinate system, and ground coordinate system.

[0041] Figure 2 This is a diagram illustrating the solution of the ground equation based on zebra crossings in this invention.

[0042] Figure 3 This is a framework diagram of the monocular three-dimensional target detection method of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described below with reference to the accompanying drawings. The drawings are for reference and illustration only and do not constitute a limitation on the scope of protection of this invention.

[0044] In the description of this invention, it should be understood that the terms "center", "upper", "lower", "left", "right", "top", "bottom", "upper left", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0045] The first aspect of this invention is the offline acquisition of ground information. There are two approaches: the first is based on zebra crossing fitting, applicable to scenes with zebra crossings; the second is using lidar to fit the ground equation.

[0046] When fitting the ground equation using zebra crossings, it's essential to ensure that the zebra crossings are visible within the camera's field of view. If no zebra crossings are visible, a pre-made zebra cloth can be laid on the ground, and several data points can be collected. During data collection, the length of each zebra crossing needs to be measured. h ,width w 1. and the distance between any two zebra crossings. ,like Figure 2 As shown. When fitting the ground equation using zebra stripes, three coordinate systems are involved: pixel coordinate system, camera coordinate system, and physical coordinate system, as follows. Figure 1The pixel coordinate system is a two-dimensional coordinate system, the coordinate origin is at the top left corner of the image, the camera coordinate system and the physical coordinate system are three-dimensional coordinate systems, the origin of the camera coordinate system is at the center of the camera, and the origin of the physical coordinate system is at the top left corner of the zebra line. Because the zebra line is on the ground, solving the ground equation in the camera coordinate system is converted into transforming the [0, 0, 1] vector in the physical coordinate system to the camera coordinate system, and the transformation process needs the transformation matrix from the camera coordinate system to the physical coordinate system . The camera intrinsic parameters are known , the pixel coordinates are , the camera coordinate system is , the physical coordinate system is . Because the points on the zebra line are on the ground, . The camera imaging equation is , the coordinate system conversion equation is . The zebra line corner is used as a calibration reference to construct a PnP problem, minimize the projection error, and solve the transformation matrix from the camera to the physical coordinate system . After obtaining , the ground equation in the camera coordinate system can be obtained .

[0047] To fit the ground equation using a laser radar, a laser radar is temporarily installed at an intersection, and the external parameter matrix T LC is obtained by calibrating the laser radar and the camera. The ground equation is fitted in the laser radar , and the ground equation in the camera coordinate system can be obtained . Specifically, when fitting the ground equation in the laser radar, the ground points are segmented using Patchwork++, and the RANSAC algorithm is used to fit the ground points to obtain the ground equation. For a relatively rugged road, the ground can be fitted in segments. The detection model of the present application supports inputting segmented ground equations.

[0048] A second aspect of the embodiment of the present application is shown in Figure 3 , and the method comprises the steps of:

[0049] The original image data obtained by the monocular camera is input into a feature extraction network to obtain two-dimensional multi-scale features; a two-dimensional target detector is used for two-dimensional target detection to obtain two-dimensional information of the target; a DepthNet network is used for depth estimation, and an LSS (Lift, Splat, Shoot) method is used to project image features to obtain bird's eye view (BEV, Bird's Eye View) features;

[0050] The 2D target bounding box is projected onto the BEV space using the ground equation as a proposal box for 3D target detection. The BEV features are then clipped using the proposal box to obtain RoI (Region of Interest) features with the same shape. The clipped BEV features are used to predict the information of the 3D target, and non-maximum suppression (NMS) is applied to obtain the final detection result.

[0051] Furthermore, the two-dimensional target detection result includes three parts:

[0052] Predict the category of the target p : An integer between 0 and 10, representing the 10 categories in the dataset;

[0053] Predict the 2D bounding box of the target xywh The coordinates of the center point, width, and height of the 2D bounding box in the pixel coordinate system;

[0054] Confidence score of the predicted target conf : A floating-point number between 0 and 1, representing the credibility of the current target.

[0055] Furthermore, the loss function of the two-dimensional target detector is:

[0056]

[0057] in, N It is the predicted number of positive samples. c It is the actual category of the target. The softmax loss used Use SmoothL1 loss.

[0058] Furthermore, when using the DepthNet network for depth estimation, in terms of image features... Depth estimation is performed using the DepthNet network, estimating the depth for each feature point in the image. D A depth map is obtained by considering discrete depths. Image features are extracted using the LSS method. The projection yields dense BEV features. .

[0059] Furthermore, the step of projecting the 2D target bounding box into the BEV space using the ground equation specifically involves projecting the two bottom corner points of the 2D bounding box into the ground coordinate system. On the plane; first project the two top corner points of the 2D frame onto the ground coordinate system. On the plane, then translate to On the plane, the proposal frame is obtained in the ground coordinate system.x p y p z p w p l p , wherein (x, y) are the coordinates of the center of the proposal box, x p ，y p , z p ) are the coordinates of the center of the proposal box, w p , l p are the width and length of the proposal box. The calculation formula of projecting a point in the pixel coordinate system to the ground coordinate system is derived, given a point in the camera coordinate system , camera intrinsic parameters , a point in the pixel coordinate system , the projection relationship from the camera coordinate system to the pixel coordinate system is as follows:

[0060]

[0061] Since the point on the ground in the camera coordinate system satisfies the ground equation, it is obtained that:

[0062]

[0063] The following two equations are obtained by combining the above two equations, that is, according to the ground equation and the ground point in the image, the coordinates of the point in the camera coordinate system can be obtained .

[0064]

[0065] According to the transformation matrix from the camera coordinate system to the ground coordinate system , the target box in the camera system is projected into the ground coordinate system.

[0066] Further, the ground coordinate system is shown in Figure 2 , the origin of the ground coordinate system is the same as the camera coordinate system, so the ground coordinate system can be obtained by rotating the camera coordinate system, and the solving process of the transformation matrix from the camera system to the ground system is as follows:

[0067] First, take the coefficients of the ground equation in the camera system, and unitize it to obtain , The meaning of z is the coordinate representation of the unit vector of the z-axis of the ground coordinate system in the camera coordinate system.

[0068]

[0069] The unit vector of the x-axis of the camera system , Schmidt orthogonalization is performed, and the coordinate expression of the unit vector of the z-axis of the ground coordinate system in the camera coordinate system is obtained x

[0070]

[0071] The unit vector of the x-axis of the camera system is unitized, and according to the orthogonal property of the three axes of the ground coordinate system, the coordinate expression of the unit vector of the z-axis of the ground coordinate system in the camera coordinate system is obtained y

[0072] Therefore, the transformation matrix from the camera coordinate system to the ground coordinate system is obtained

[0073]

[0074] .

[0075] Further, when the BEV feature is cropped using the proposal box to obtain a RoI (Region of Interest) feature with the same shape, the BEV feature is cropped according to the proposal box, the BEV feature region corresponding to each proposal is divided into a grid of pool_w*pool_h, and each part of the grid is subjected to maximum pooling processing, so that the features cropped by all proposal boxes are of pool_w*pool_h size, and fixed-length output is realized.

[0076] Further, the BEV feature cropped is used to predict the information of the three-dimensional target, and NMS processing is performed to obtain the final detection result, which is: using the obtained proposal feature map, the offset of each proposal box is obtained through the regression head, which is used to regress a more accurate target detection box; the height of the target is predicted by the height regression branch, for each proposal feature map, the BEV frame information of the cropped BEV feature is known, that is, in the ground coordinate system, the information of the length and width of the proposal box and the center point of the bottom surface. After obtaining the offset using the regression head, the final length, width and center point of the bottom surface can be obtained, in addition, the height of the target is predicted by a separate height regression branch, and the length, width and center point of the target in the ground coordinate system are calculated according to the offset, and the formula is as follows:

[0077]

[0078]

[0079]

[0080] ​​​

[0081] in, , These are the length and width of the target in the ground coordinate system. , It is the offset of the length and width predicted by the detection head; , It is the center point of the target's bottom surface in the ground coordinate system. coordinates and coordinate, , This is the offset of the target's center point predicted by the detector. The formulas for the target's height and the z-value of the ground center point in the ground coordinate system are as follows:

[0082]

[0083]

[0084] in, It is the height of the target in the ground coordinate system. It is a target height benchmark set according to different categories. It is the offset of the target height predicted by the height branch. It is the center point of the target's bottom surface in the ground coordinate system. coordinate, It is the offset of the center point of the target's bottom surface predicted by the height branch. It is the minimum height of the center point of the target's bottom surface in the defined ground coordinate system. The formula for calculating the regression branch loss function is as follows:

[0085] ;

[0086] Furthermore, the target's orientation angle is predicted using a multi-bin approach. Taking the center point of the car's underside as the origin, the surrounding 360-degree space is divided into multiple bins. First, the orientation angle is predicted to belong to a specific bin through classification. Then, the target's offset within that bin is predicted. The aim is to decompose a complex regression problem into a simpler classification and regression problem. The loss function is...

[0087]

[0088]

[0089]

[0090] in, It is the category of the predicted global azimuth. It is the truth value of which bin the rot angle belongs to. It's from a predictive perspective. is the true value of the global azimuth angle.

[0091] Further, the predicted three-dimensional target is subjected to Non-Maximum Suppression (NMS) processing, and a final three-dimensional target frame is screened out, and the three-dimensional detection result comprises:

[0092] a category of the target p : an integer between 0 and 10, representing 10 categories on an existing data set;

[0093] a center point coordinate of the target : a position of the center point of the target in a camera coordinate system, in meters;

[0094] a length-width-height of the target : size information of the target, in meters;

[0095] a global azimuth angle of the target rot : representing an included angle between a vehicle body coordinate system and an x-axis of the camera coordinate system, in radians, for determining an orientation of the target frame.

[0096] a confidence score of the predicted target conf : a floating point number between 0 and 1, representing a degree of credibility of the current target.

[0097] Further preferably, the loss function for training the roadside monocular three-dimensional target detection model of the present application is:

[0098]

[0099] wherein , , is a constant, is a classification loss of the target, is a three-dimensional regression loss of the target, is an orientation loss of the target.

[0100] The present application also provides a roadside camera three-dimensional target detection system based on ground constraints, comprising a data acquisition module and an inference module.

[0101] The data acquisition module is used for acquiring original image data of the roadside camera.

[0102] The inference module processes the original image data based on the roadside monocular 3D target detection model to obtain 3D target detection results. Specifically: after the image data is input into the roadside monocular 3D target detection model, the model uses a 2D backbone network to extract image features. After obtaining the image features, it splits into two branches: branch one uses a 2D target detector to perform 2D target detection, obtaining 2D target detection boxes; branch two uses a DepthNet network for depth estimation, and uses LSS to project the image features to obtain the scene's BEV features; the ground equation is used to project the 2D target boxes onto the BEV space as proposal boxes for 3D target detection; the BEV features of the scene are cropped using the proposal boxes to obtain RoI features with the same shape; the RoI features are used to predict the information of the 3D target, and NMS processing is performed to obtain the 3D target detection results, namely the category of the 3D target and its length, width, height, bottom center point, and orientation angle in the ground coordinate system. Finally, the ground coordinates are used to obtain the 3D target detection results.

[0103] The transformation matrix, which is linked to the camera coordinate system, transforms the ground center point and orientation angle of the 3D target to the camera coordinate system. On the other hand, this invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the roadside camera 3D target detection method based on ground constraints described in this invention.

[0104] The computer equipment may be a laptop, desktop computer, workstation, or vehicle-mounted computer.

[0105] The present invention can also provide a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and the processor can implement the roadside camera three-dimensional target detection method based on ground constraints described in the present invention when executing the computer executable program.

[0106] The processor can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or an off-the-shelf programmable gate array (FPGA).

[0107] The memory described in this invention can be an internal storage unit of a laptop, desktop computer, workstation, or vehicle-mounted computer, such as memory or hard disk; or it can be an external storage unit, such as a portable hard disk or flash memory card.

[0108] The computer-readable storage medium can include a computer storage medium and a communication medium. The computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. The computer-readable storage medium can include read-only memory (ROM), random access memory (RAM), solid state disk (SSD), optical disk, etc. Among them, the random access memory can include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

[0109] The above is only to illustrate the technical idea of the present application, and cannot limit the protection scope of the present application. Any modification made according to the technical idea of the present application on the basis of the technical scheme falls within the protection scope of the claims of the present application.

Claims

1. A ground constraint based roadside camera 3D target detection method, characterized in that, The method comprises the following steps: Obtain the original image data of the roadside camera as input, use the trained roadside monocular three-dimensional target detection model for inference, and finally obtain the category of the three-dimensional target and the length, width, height, bottom center point and orientation angle in the camera coordinate system; Specifically, after the image data is input into the roadside monocular three-dimensional target detection model, the roadside monocular three-dimensional target detection model extracts image features using a two-dimensional backbone network, and after the image features are obtained, they are divided into two branches; branch one uses a two-dimensional target detector for two-dimensional target detection to obtain a two-dimensional target detection frame; branch two uses a DepthNet network for depth estimation, and uses an LSS method to project the image features to obtain the BEV features of the scene; The two-dimensional target frame is projected to the BEV space using an offline obtained ground equation as a proposal frame for three-dimensional target detection; the BEV features of the scene are cropped using the proposal frame to obtain RoI features with the same shape, and the information of the three-dimensional target is predicted using the RoI features, and NMS processing is performed to obtain the three-dimensional target detection result, i.e., the category of the three-dimensional target and the length, width, height, bottom center point and orientation angle in the ground coordinate system; Finally, the ground center point and orientation angle of the three-dimensional target are transformed into the camera coordinate system using a transformation matrix from the ground coordinate system to the camera coordinate system. The ground equation is obtained by: The zebra line is located on the ground, and the ground equation in the camera coordinate system is converted into transforming the [0, 0, 1] vector in the physical coordinate system to the camera coordinate system, and in the transformation process, the transformation matrix is combined Convert the camera coordinate system to the physical coordinate system, and the camera intrinsic parameter is known , pixel coordinates , camera coordinate system , physical coordinate system , and , the camera imaging equation is The conversion equation from the physical coordinate system to the camera coordinate system is ; With the corner point of the zebra crossing as the calibration reference, a PnP problem is constructed to minimize the projection error and solve the transformation matrix from the camera to the physical coordinate system , and then the ground equation in the camera system is obtained ; If there is no zebra crossing in the camera field of view, the ground is fitted based on the point cloud data collected by the laser radar, and the external parameter matrix of the laser radar and the camera is calibrated T LC The ground equation is fitted in the laser radar , and the ground equation in the camera system is obtained ； When fitting the ground equation in the laser radar, the ground points are first segmented using Patchwork++, and then the ground equation is fitted using the RANSAC algorithm.

2. The ground constraint based roadside camera 3D target detection method of claim 1, wherein, The two-dimensional backbone extraction network used is ResNet-50; the two-dimensional target detector used is SSD or other two-dimensional target detector based on convolution features, and the two-dimensional target detector is separately trained to achieve a recall rate of 99.5% for the targets in the image; the depth estimation network used is the DepthNet network, which estimates D discrete depths for each image feature.

3. The ground constraint based roadside camera 3D target detection method of claim 1, wherein, Projecting the two-dimensional target frame to the BEV space using the ground equation comprises: first projecting the two-dimensional target frame to the camera coordinate system, and then converting it to the ground coordinate system, wherein the left lower corner point and the right lower corner point of the two-dimensional target frame satisfy the ground equation, and the points on the ground in the camera system satisfy: Solving the ground equation and the above equation simultaneously gives the following two equations, i.e. according to the ground equation, and the projection point of the target's landing site in the image, the coordinates of the point in the camera coordinate system are obtained , The solving formula of the point in the camera system is: According to the transformation matrix from the camera coordinate system to the ground coordinate system projecting the target frame in the camera system into the ground coordinate system; Given the ground equation in the camera coordinate system, let the origin of the ground coordinate system be the O point of the camera system, X g O y g If the plane is parallel to the ground, the transformation matrix from the camera system to the ground coordinate system can be obtained Each of the unit vectors is calculated by Schmidt orthogonalization as follows: where A, B, C are the coefficients of the ground equation in the camera coordinate system, x c is the unit vector of the z-axis in the camera coordinate system, x is the unit vector of the z-axis in the camera coordinate system, x cg , y cg , z cg are the representations of the x-axis, y-axis, z-axis unit vectors in the ground coordinate system in the camera system, respectively.

4. The ground constraint based roadside camera 3D target detection method of claim 1, wherein, Predicting the information of the three-dimensional target using the RoI features and performing NMS processing to obtain the final three-dimensional target detection result comprises: Crop the BEV features using the proposal frame, divide the BEV feature region corresponding to each proposal into a grid of pool_w*pool_h, and then perform maximum pooling processing on each part of the grid; use the obtained proposal feature map to obtain the offset of each proposal frame through the regression head, and modify the proposal frame using the offset; Use a separate height regression branch to predict the height of the target, and then obtain the length, width, height and bottom center point information of the target in the ground coordinate system.

5. The ground constraint based roadside camera 3D target detection method of claim 1, wherein, The RoI feature is used to predict information of the three-dimensional target, and NMS processing is performed to obtain a final three-dimensional target detection result, including an inference of an orientation angle of the three-dimensional target, and the specific process is as follows: the orientation angle of the target is predicted in a Muti-bin manner, the surrounding 360-degree space is divided into multiple bins with the center point of the bottom surface of the target as the origin, the orientation angle is first predicted to fall in which bin, and then the offset of the target in the bin is predicted to obtain the orientation angle of the target in the ground coordinate system.

6. The ground constraint based roadside camera 3D target detection method of claim 1, wherein, The loss function of the two-dimensional target detector is: wherein, N is the predicted number of positive samples, c is the target actual class, softmax loss is used, SmoothL1 loss is used; The loss function of the roadside monocular three-dimensional target detection model is: wherein , , is a constant, is a classification loss for the target, is a three-dimensional regression loss for the target, is an orientation loss for the target.

7. A ground constraint based roadside camera 3D target detection system, characterized in that, The roadside monocular three-dimensional target detection model comprises a data acquisition module and an inference module. The data acquisition module is configured to acquire original image data of a roadside camera. The inference module is configured to process the original image data based on the roadside monocular three-dimensional target detection model to obtain a three-dimensional target detection result, and the specific process is as follows: after the image data is input into the roadside monocular three-dimensional target detection model, the roadside monocular three-dimensional target detection model extracts image features by using a two-dimensional backbone network, and after the image features are obtained, the image features are divided into two branches; the first branch is used for two-dimensional target detection by using a two-dimensional target detector to obtain a two-dimensional target detection frame; and the second branch is used for depth estimation by using a DepthNet network, and a BEV feature of a scene is projected by using an LSS manner to obtain the BEV feature; The two-dimensional target frame is projected to a BEV space by using a ground equation to obtain a proposal frame for three-dimensional target detection; the BEV feature of the scene is cropped by using the proposal frame to obtain an RoI feature with the same shape, the RoI feature is used to predict information of the three-dimensional target, and NMS processing is performed to obtain a three-dimensional target detection result, that is, a category of the three-dimensional target and a length, a width, a height, a bottom center point and an orientation angle of the three-dimensional target in a ground coordinate system, and finally, the ground coordinate system is converted into a camera coordinate system by using a conversion matrix, and the ground center point and the orientation angle of the three-dimensional target are converted into the camera coordinate system. The ground equation is obtained by: When the ground equation is fitted in a laser radar, ground points are segmented by using Patchwork++, and the ground equation is fitted by using a RANSAC algorithm. The zebra line is located on the ground, and the ground equation in the camera coordinate system is converted into the transformation of the [0, 0, 1] vector in the physical coordinate system to the camera coordinate system. In the transformation process, the transformation matrix is combined The camera coordinate system is converted to the physical coordinate system, and the camera intrinsic parameters are known , pixel coordinates , camera coordinate system , physical coordinate system , and The camera imaging equation is The conversion equation from the physical coordinate system to the camera coordinate system is ; the zebra line corner is taken as the calibration reference, a PnP problem is constructed, the projection error is minimized, the transformation matrix of the camera to the physical coordinate system is solved , and the ground equation in the camera system is obtained ; If there is no zebra crossing in the camera field of view, the ground is fitted based on the point cloud data collected by the laser radar, and the external parameter matrix of the laser radar and the camera is calibrated T LC The ground equation is fitted in the laser radar , and the ground equation in the camera system is obtained ； The roadside camera three-dimensional target detection method based on ground constraint comprises a processor and a memory, the memory is configured to store a computer executable program, the processor is configured to read the computer executable program from the memory and execute the computer executable program, and the processor is configured to implement the roadside camera three-dimensional target detection method based on ground constraint according to any one of claims 1 to 6 when the computer executable program is executed.

8. A computer device, comprising: A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the roadside camera three-dimensional target detection method based on ground constraint according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, ​

Images