Vehicle three-dimensional shape detection method and device under roadside view angle of improved Center Net

Abstract

The invention belongs to the technical field of intelligent traffic, and discloses a vehicle three-dimensional shape detection method and device under the roadside view angle of an improved Center Net, and the method comprises the steps: carrying out the shooting of a roadside camera through the roadside camera, obtaining the camera parameters of the roadside camera and a ground plane equation under the camera coordinates, and obtaining a ground plane equation under the camera coordinates; and inputting the shot picture into the vehicle three-dimensional detection network model to obtain a thermodynamic diagram, center point offset, key point offset and vanishing points, and converting the thermodynamic diagram, the center point offset, the key point offset and the vanishing points into three-dimensional coordinates under a camera coordinate system to obtain a vehicle three-dimensional form, so that the three-dimensional form detection of the vehicle can be applied in a real-time scene, the detection speed is improved, and the detection efficiency is improved. The rapid response capability is provided for an intelligent traffic system, and the problems that an existing vehicle detection method usually depends on template matching, multi-sensor fusion and a complex deep learning network, certain limitations exist in cost effectiveness, precision and real-time performance, and most detection technologies acquire information through a vehicle-mounted camera, and the detection efficiency is poor are solved. And compared with a roadside camera, the sensing range is relatively small.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and in particular relates to a method and device for detecting three-dimensional vehicle shape under a roadside perspective of an improved CenterNet. Background Art

[0002] The development of intelligent transportation systems has become a key focus in today's urban traffic management and vehicle safety fields. With the development of technology, the demand for vehicle 3D morphology detection methods is increasing. The 3D morphology information of a vehicle includes key parameters such as precise physical position, size, and deflection angle. This information is essential for building intelligent transportation systems. By obtaining the 3D morphology information of a vehicle, the intelligent transportation system can more accurately understand the actual position and orientation of the vehicle on the road, thereby achieving accurate vehicle positioning and path planning.

[0003] The vehicle's size information can avoid collisions, optimize lane utilization, and improve traffic flow efficiency; the vehicle's deflection angle can provide driving status information to the traffic management system, which helps predict the vehicle's driving trajectory and the dynamic adjustment of traffic flow.

[0004] However, existing vehicle detection methods usually rely on template matching, multi-sensor fusion and complex deep learning networks, which have certain limitations in cost-effectiveness, accuracy and real-time performance. In addition, most detection technologies obtain information through on-board cameras, which have a smaller perception range than roadside cameras. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method and device for detecting the three-dimensional form of vehicles from the roadside perspective of an improved CenterNet, which obtains the three-dimensional form of vehicles in traffic scenes through a roadside monocular camera, thereby solving the technical problems that the existing vehicle detection methods usually rely on template matching, multi-sensor fusion and complex deep learning networks, have certain limitations in cost-effectiveness, accuracy and real-time performance, and most detection technologies obtain information through on-board cameras, which have a smaller perception range than roadside cameras.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] The present invention provides a vehicle three-dimensional form detection method under the roadside perspective of an improved CenterNet, comprising: using a roadside camera to shoot a traffic scene, obtaining camera parameters of the roadside camera and a ground plane equation in a camera coordinate system; inputting the picture shot by the roadside camera into a vehicle three-dimensional detection network model to obtain a vehicle heat map, a center point offset, a key point offset and a vanishing point; converting the vehicle heat map, the center point offset, the key point offset and the vanishing point into three-dimensional coordinates in a camera coordinate system to obtain the vehicle three-dimensional form.

[0008] Furthermore, the vehicle three-dimensional detection network model includes a backbone feature extraction network and multi-scale feature fusion.

[0009] Furthermore, the multi-scale feature fusion performs weighted feature fusion on the high-resolution feature map or the low-resolution feature map of the backbone feature extraction network according to the multi-task detection head to obtain a fused feature map containing multi-scale information of the vehicle.

[0010] Furthermore, the multi-task detection head includes a vehicle type classification branch, a vehicle center point regression branch, a vehicle three-dimensional box vertex regression branch and a vehicle vanishing point regression branch; the vehicle type classification branch, the vehicle center point regression branch, the vehicle three-dimensional box vertex regression branch and the vehicle vanishing point regression branch further improve the vehicle's three-dimensional spatial positioning according to the loss function.

[0011] Furthermore, the three-dimensional form of the vehicle includes the vehicle type, the vehicle physical position, the vehicle deflection angle and the vehicle three-dimensional size.

[0012] Furthermore, the camera parameters of the roadside camera are obtained through Zhang Zhengyou calibration and VLH, where V represents the vehicle vanishing point, L represents the road line length, and H represents the camera height.

[0013] Furthermore, the ground plane equation is Ax+By+Cz+D=0, wherein A, B, C, and D are known constants and are not zero, and x, y, and z are specific coordinates of any point on the road in the camera coordinate system.

[0014] The present invention also provides a vehicle three-dimensional shape detection device under the roadside perspective of an improved CenterNet, comprising: an acquisition module: using a roadside camera to shoot a traffic scene, and obtaining camera parameters of the roadside camera and a ground plane equation in a camera coordinate system;

[0015] Operation module: Input the pictures taken by the roadside camera into the vehicle 3D detection network model to obtain the vehicle heat map, center point offset, key point offset and vanishing point;

[0016] Conversion module: used to convert the vehicle heat map, center point offset, key point offset and vanishing point into three-dimensional coordinates in the camera coordinate system to obtain the three-dimensional shape of the vehicle.

[0017] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer device is special in that when the processor executes the computer program, the computer device implements the steps of any one of the above-mentioned methods for detecting three-dimensional vehicle shapes from a roadside perspective of an improved CenterNet.

[0018] The present invention also provides a computer-readable storage medium, which stores a computer program. The special feature of the computer program is that when the computer program is executed by a processor, the steps of any one of the above-mentioned methods for improving the vehicle three-dimensional form detection from the roadside perspective of CenterNet are implemented.

[0019] Compared with the prior art, the present invention has the following beneficial technical effects:

[0020] The present invention provides a vehicle three-dimensional shape detection method under the roadside perspective of an improved CenterNet. A roadside camera shoots a traffic scene according to physical data under a monocular perspective, and the shot picture is input into a vehicle three-dimensional detection network model to obtain a vehicle heat map, a center point offset, a key point offset and a vanishing point. According to the connection between a ground plane equation and a vanishing point, the vehicle three-dimensional shape is obtained, thereby significantly improving the detection precision and accuracy.

[0021] The present invention provides a vehicle three-dimensional morphology detection method under the roadside perspective of an improved CenterNet. The multi-scale feature fusion of the three-dimensional detection network model can perform weighted feature fusion on the high-resolution features or low-resolution features extracted by the backbone features to obtain a fused feature map containing multi-scale information of the vehicle, so as to achieve more accurate feature fusion.

[0022] The present invention provides a vehicle three-dimensional shape detection method under the roadside perspective of an improved CenterNet, which helps to improve the detection speed, so that the three-dimensional shape detection of the vehicle can be applied in real-time scenes, and provides a rapid response capability for the intelligent transportation system.

[0023] The present invention provides a vehicle three-dimensional morphology detection method from a roadside perspective based on an improved CenterNet. Compared with the traditional multi-sensor fusion method or complex deep learning network, the present invention can reduce costs. Only a single roadside camera is required, and the accuracy and speed required by the intelligent transportation system can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a method for detecting three-dimensional vehicle shape from a roadside perspective of an improved CenterNet of the present invention;

[0025] Figure 2 A schematic diagram of a three-dimensional structure of a vehicle in an embodiment of the present invention;

[0026] Figure 3 is the camera space model in the embodiment of the present invention, where a is the three-dimensional coordinate system and b is the image coordinate system;

[0027] Figure 4 A schematic diagram of a network model in an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of weighted feature fusion in an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of space constraints in an embodiment of the present invention;

[0030] Figure 7 This is a visualization result diagram of a data set in an embodiment of the present invention;

[0031] Figure 8 It is a visual structure diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] like Figure 1 As shown, a vehicle 3D shape detection method from a roadside perspective of an improved CenterNet includes using a roadside camera to shoot a traffic scene, obtaining the camera parameters of the roadside camera and the ground plane equation in the camera coordinate system; inputting the picture taken by the roadside camera into a vehicle 3D detection network model to obtain a vehicle heat map, center point offset, key point offset and vanishing point; converting the vehicle heat map, center point offset, key point offset and vanishing point into 3D coordinates in the camera coordinate system to obtain the vehicle 3D shape.

[0034] Use a roadside camera to shoot the traffic scene, obtain the camera parameters of the roadside camera and the ground plane equation in the camera coordinate system, specifically: in general road scenes, the roadside camera and the road markings have fixed standard physical data, and the roadside camera shoots the traffic scene according to the physical data, wherein the physical data includes the installation position of the roadside camera and the road markings, the road markings have fixed standards, and whether the traffic scene contains vehicles or not, preferably the traffic scene without vehicles, the physical data is combined with the camera self-calibration to obtain the camera parameters of the roadside camera and the ground plane equation in the camera coordinate system, wherein the camera parameters are obtained through Zhang Zhengyou calibration and VLH, the main purpose of Zhang Zhengyou calibration is to obtain the intrinsic and extrinsic matrix of the roadside camera, as well as the selection and translation matrix of each calibration image, correct the captured image, and obtain a relatively small distortion image, only need to use a printed chessboard, compared with self-calibration, the accuracy is improved and the operation is easy. VLH obtains the camera parameters and calculates the camera parameters through the vehicle trajectory in the video stream, the road line length and the assumed camera height.

[0035] The ground plane equation is obtained based on the camera parameters of the roadside camera, road surface information, and the pictures taken by the roadside camera. The ground plane equation is set to Ax+By+Cz+D=0. When z≠0, it is obtained by linear change Multiple constitute

[0036]

[0037] Among them, x i ,y i 、z i is the coordinate position of the point on the ground in the camera coordinate system, and m is n is p is The matrices formed by them are represented as R, E, and Z respectively, and R, E, and Z are three different matrices. If R×V=Z, it satisfies (m 2 +n 2 +1)·C 2 = 1, according to V = (R T ,R) -1 ·R T ·Y, the ground plane equation ax+By+Cz+D=0 can be obtained, where A, B, C, and D are known constants and are not zero, x, y, and z are the specific coordinates of any point on the road in the camera coordinate system, A=-mc, B=-nc, and D=-pc. A, B, C, and D are all constants, but the expressions are different depending on the scene and the image obtained. Where T is the transpose matrix of the matrix.

[0038] The images taken by the roadside camera are input into the vehicle 3D detection network model to obtain the vehicle heat map, center point offset, key point offset and vanishing point. Specifically, the images taken by the roadside camera are input into the vehicle 3D detection network model. The 3D detection network model includes a backbone feature extraction network and multi-scale feature fusion, such as Figure 4 As shown in , the backbone feature extraction network extracts the pictures taken by the roadside camera to obtain high-resolution feature maps or low-resolution feature maps. The high-resolution feature map retains more accurate local features and is suitable for detecting small targets. The low-resolution feature map contains higher-level semantic information and is suitable for detecting large targets. Figure 5As shown in the figure, multi-scale feature fusion uses a multi-task detection head to perform weighted feature fusion on high-resolution feature maps or low-resolution feature maps to obtain a fused feature map containing multi-scale information of vehicles. Weighted feature fusion is due to the fact that in the intersection scene, the roadside camera is distributed at different positions on the road, and the sizes in the captured pictures are also different. In order to make the backbone feature extraction network better adapt to the detection of vehicles of different sizes, in addition, weighted feature fusion can not only obtain feature maps containing multi-scale information, but also reduce certain calculations and improve overall efficiency. The fused feature map containing multi-scale information of vehicles includes vehicle heat map, center point offset, key point offset and vanishing point information. Figure 2 As shown, the vehicle 3D detection network model constrains the predicted target 3D box through the reprojection constraints of the camera calibration prior, and strengthens the constraints through the vehicle vanishing point, making the prediction results of the vehicle 3D detection network model more robust.

[0039] The multi-task detection head includes a vehicle type classification branch, a vehicle center point offset regression branch, a vehicle 3D box vertex offset regression branch, and a vehicle vanishing point regression branch. The loss function is used to further improve the accuracy of the vehicle's 3D spatial positioning, such as Figure 6 As shown in Figure 2, the loss function is used to solve the problem of imbalanced positive and negative ratios in the vehicle detection process.

[0040] Convert the vehicle heat map, center point offset, key point offset and vanishing point to three-dimensional coordinates in the camera coordinate system, such as Figure 3 As shown in a in the figure, the three-dimensional shape of the vehicle is obtained. Specifically, the corresponding relationship between the vehicle deflection angle and the vanishing point is determined, and then the corresponding relationship between the vanishing point and the horizon is determined. First, a camera space model is constructed. In the camera space model, the image coordinates of the vehicle vanishing point are {v1=(u0,v00}, and the homogeneous coordinates corresponding to this two-dimensional coordinate in the camera coordinate system are {v w =[-tanθ,1,0,0]}, the corresponding relationship between them is v I =KRTv W , where R is the rotation matrix of the camera. Since the establishment of the world coordinate system only requires the pitch angle φ and the roll angle, the roll angle in the road scene is consistent with the horizon tangent value of the camera's picture.

[0041] in.

[0042]

[0043]

[0044] like Figure 3 As shown in b, the second coordinates corresponding to the vehicle vanishing point {v w =[-tanθ,1,0,0]}Substituting into the above formula we get:

[0045]

[0046] Then the relationship between the vanishing point coordinates and the vehicle deflection angle is:

[0047]

[0048] The three-dimensional shape of the vehicle includes the vehicle type, the vehicle physical position, the vehicle deflection angle and the vehicle three-dimensional size. The vehicle type can be directly obtained through the thermal map of the vehicle. The physical position of the vehicle assumes that the scale factor of the center point of the vehicle is D, D = μ z +D offset *σ z , where μz is the mean of the scale factor of the vehicle center point and is a fixed constant, D offset is the offset of the scale factor of the vehicle center point obtained through the network, σ z The standard deviation of the scale factor of the vehicle center point is a fixed constant, the spatial position is {cen3D=(Xp,Yp,Zp)}, and the transformation matrix from the camera coordinate system to the image coordinate system is Calculate the physical position of the vehicle in the camera coordinate system, where K is the camera intrinsic parameter matrix, which is used to correspond the points on the image to the points in space.

[0049] Vehicle deflection angle in f can be obtained by camera calibration x 、f y Camera focal length; x0, y0 camera principal point coordinates; the coordinates of the vehicle vanishing point in the image obtained through the network are {v I =(u0,v0)}. The three-dimensional dimensions of the vehicle, the deflection angle R obtained by calculation y , the vehicle center point predicted by the network and the pre-calculated mean vehicle size, we can get the vehicle's reference 3D frame. Map it to the image to get the 2D reference envelope, and then calculate the final 2D envelope with the offset value output by the network, denoted as {cor_i=(x i ,y i )|i=0,1,…,7}. The vehicle is located on the ground plane, so the four bottom vertices in the envelope are recorded as cor0, cor1, cor2, and cor3. The four-point coordinates are converted to the camera coordinate system, and the length L and width W of the vehicle are calculated according to the distance formula. The distance from the center point to the ground plane is multiplied by two to obtain the vehicle height H.

[0050] like Figure 7As shown, the visualization effect in the actual traffic scene is shown in Table 1. The vehicle 3D morphology detection method under the roadside perspective of an improved CenterNet and the existing algorithms are shown in Table 1. The average precision AP of M3D-RPN, MonoDLE, and MonoFlex is used to measure the comprehensive performance of the model under different recall rates and Rope score Among them, Rope score =0.8*AP+0.2*(ACS+AOS+AAS+AGS) / 4, ACS represents the average ground center similarity, which is obtained by calculating the distance between the predicted vehicle center point and the actual center point; AOS represents the average orientation similarity, which is obtained by calculating the difference between the predicted vehicle deflection angle and the actual deflection angle; AAS represents the average area similarity, which is obtained by calculating the difference between the predicted vehicle footprint and the actual footprint; AGS represents the average distance and similarity of the four ground points, and the test results under the two indicators are obtained by calculating the difference between the predicted vehicle three-dimensional frame bottom surface vertex and the actual bottom surface vertex. Figure 8 The structure is visualized.

[0051] Table 1

[0052]

[0053] Among them, a means using the enhanced loss function, and b means not using the enhanced loss function.

[0054] The present invention also provides a vehicle three-dimensional shape detection device under the roadside perspective of an improved CenterNet, comprising an acquisition module: using a roadside camera to shoot a traffic scene, obtaining camera parameters of the roadside camera and a ground plane equation in a camera coordinate system;

[0055] Operation module: Input the pictures taken by the roadside camera into the vehicle 3D detection network model to obtain the vehicle heat map, center point offset, key point offset and vanishing point;

[0056] Conversion module: used to convert the vehicle heat map, center point offset, key point offset and vanishing point into three-dimensional coordinates in the camera coordinate system to obtain the three-dimensional shape of the vehicle.

[0057] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The special feature of the computer device is that when the processor executes the computer program, the steps of an improved vehicle three-dimensional form detection method from a roadside perspective of CenterNet as described in any one of the above embodiments are implemented.

[0058] In another embodiment of the present invention, a terminal device is provided, the terminal device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding functions; the processor described in the embodiment of the present invention can improve the operation of the vehicle three-dimensional morphology detection method under the roadside perspective of CenterNet.

[0059] The present invention also provides a computer-readable storage medium, which stores a computer program, characterized in that when the computer program is executed by a processor, the computer program implements the steps of an improved vehicle three-dimensional form detection method from a roadside perspective of CenterNet as described in any one of the above embodiments.

[0060] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the vehicle three-dimensional morphology detection method under the roadside perspective of CenterNet in the above embodiment.

[0061] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0062] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

Claims

1. A vehicle three-dimensional shape detection method from the roadside perspective based on an improved CenterNet, characterized in that: include: Use a roadside camera to shoot the traffic scene, and obtain the camera parameters of the roadside camera and the ground plane equation in the camera coordinate system; The images taken by the roadside camera are input into the vehicle 3D detection network model to obtain the vehicle heat map, center point offset, key point offset and vanishing point; The vehicle heat map, center point offset, key point offset and vanishing point are converted into three-dimensional coordinates in the camera coordinate system to obtain the three-dimensional shape of the vehicle.

2. According to claim 1, the vehicle three-dimensional shape detection method under the roadside perspective of the improved CenterNet is characterized by: The vehicle three-dimensional detection network model includes a backbone feature extraction network and multi-scale feature fusion.

3. According to claim 2, the method for detecting three-dimensional vehicle shape from the roadside perspective of an improved CenterNet is characterized by: The multi-scale feature fusion performs weighted feature fusion on the high-resolution feature map or the low-resolution feature map of the backbone feature extraction network according to the multi-task detection head to obtain a fused feature map containing multi-scale information of the vehicle.

4. The vehicle 3D shape detection method under the roadside perspective of the improved CenterNet according to claim 3 is characterized by: The multi-task detection head includes a vehicle type classification branch, a vehicle center point regression branch, a vehicle three-dimensional box vertex regression branch and a vehicle vanishing point regression branch; The vehicle type classification branch, the vehicle center point regression branch, the vehicle three-dimensional box vertex regression branch and the vehicle vanishing point regression branch further improve the vehicle three-dimensional space positioning according to the loss function.

5. The vehicle 3D shape detection method under the roadside perspective of the improved CenterNet according to claim 1 is characterized by: The three-dimensional shape of the vehicle includes the vehicle type, the physical position of the vehicle, the vehicle deflection angle and the three-dimensional size of the vehicle.

6. The vehicle 3D shape detection method from the roadside perspective of an improved CenterNet according to claim 1 is characterized by: The camera parameters of the roadside camera are obtained through Zhang Zhengyou calibration and VLH; Among them, V represents the vanishing point of the vehicle, L represents the length of the road line, and H represents the height of the camera.

7. The vehicle 3D shape detection method from the roadside perspective of an improved CenterNet according to claim 1 is characterized by: The ground plane equation is Ax+By+Cz+D=0; Among them, A, B, C, and D are known constants and are not zero. x, y, and z are the specific coordinates of any point on the road in the camera coordinate system.

8. A vehicle three-dimensional shape detection device from a roadside perspective based on an improved CenterNet, characterized in that: include: Acquisition module: used to shoot traffic scenes using roadside cameras, and obtain camera parameters of the roadside cameras and ground plane equations in the camera coordinate system; Operation module: Input the pictures taken by the roadside camera into the vehicle 3D detection network model to obtain the vehicle heat map, center point offset, key point offset and vanishing point; Conversion module: used to convert the vehicle heat map, center point offset, key point offset and vanishing point into three-dimensional coordinates in the camera coordinate system to obtain the three-dimensional shape of the vehicle.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of an improved vehicle three-dimensional form detection method from a roadside perspective of CenterNet are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of an improved vehicle three-dimensional form detection method from a roadside perspective of CenterNet are implemented as described in any one of claims 1 to 7.

Images