Vehicle size identification method, system and device and storage medium

Abstract

The invention discloses a vehicle size identification method, system and device, and a storage medium. The method comprises the following steps: reading a radar point cloud of a radar device and an image of a camera device; obtaining a 3D surrounding frame under a relative coordinate system according to the obtained 2D surrounding frames and altitude lines of all the vehicles in the image; screening radar points of the target vehicle in the radar point cloud; obtaining the absolute distance of the radar point; and identifying the size of the target vehicle according to the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance. The 3D bounding box of the target vehicle in the relative coordinate system is acquired based on the image of the camera device, the absolute distance of the radar point of the target vehicle is acquired based on the radar point cloud, and the size of the target vehicle is identified according to the 3D bounding box and the absolute distance in the relative coordinate system. According to the invention, accurate identification of the vehicle size under the condition of low cost is completed based on the radar equipment and the camera equipment, and the accuracy of vehicle size identification is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle identification, and particularly relates to a method, a system, a device and a storage medium for identifying the size of a vehicle. Background Art

[0002] Vehicle size identification is applied to the field of road transport management. The current conventional method is to perform object recognition and vehicle size prediction based on the dense lidar point cloud obtained by lidar devices from multiple perspectives. However, this method not only requires multiple lidar devices, but also the obtained point cloud is very dense. The method for performing object recognition and vehicle size prediction on the dense point cloud has a huge amount of calculation and requires high-performance hardware devices to perform real-time processing, resulting in a very high hardware cost for such methods.

[0003] A millimeter-wave radar refers to a radar device operating in the frequency domain of 30 - 300 GHz. It can obtain the horizontal angle, linear distance, and Doppler velocity of sparse points. Since it uses short-wavelength electromagnetic waves, theoretically, it can detect the movement of objects at the millimeter level, and the maximum distance is far, reaching more than 300 meters. In addition, its stability and robustness in bad weather are better. The 4D millimeter-wave radar further adds the detection of the angle in the vertical direction on the basis of the ordinary millimeter-wave radar, and can obtain a denser radar point cloud than the ordinary millimeter-wave radar.

[0004] Compared with lidar, the millimeter-wave radar not only has a longer maximum distance, but also can additionally give additional information such as the Doppler velocity and scattering cross-section area of the object, and the estimation of the absolute distance is more accurate. However, the radar point cloud obtained by the millimeter-wave radar is much sparser than that of the lidar. In the case of only using the radar point cloud, the objects therein cannot be accurately identified, and due to the sparsity of the radar point cloud, the size of the objects cannot be accurately predicted.

[0005] A camera is the most commonly used visual perception device and is widely used in various recognition and detection algorithms. However, since its imaging principle depends on the imaging of visible light on a CMOS (image sensor), mapping from three-dimensional space to a two-dimensional plane, the obtained information does not have the dimension of the object distance. Therefore, the distance of the object cannot be accurately estimated based on the image. Therefore, the current methods based on cameras are limited to tasks that are not related to the scale in the real world. Vehicle size identification is a task related to the real-world scale. Therefore, there is no pure camera-based vehicle size detection method. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the defect of low recognition accuracy in the existing vehicle size identification method, and provide a method, a system, a device and a storage medium for identifying the size of a vehicle.

[0007] The present invention solves the above technical problem through the following technical solutions:

[0008] The first aspect of the present invention provides a method for identifying the size of a vehicle, and the identification method includes:

[0009] Reading the radar point cloud of the radar device and the image of the camera device;

[0010] Obtaining the 2D bounding boxes and height lines of all vehicles in the above image;

[0011] Obtaining the 3D bounding box in the relative coordinate system according to the 2D bounding box and the height line;

[0012] Filtering the radar points of the target vehicle in the radar point cloud;

[0013] Obtaining the absolute distance of the radar points of the target vehicle;

[0014] Identifying the size of the target vehicle according to the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle.

[0015] Preferably, after the step of reading the radar point cloud of the radar device and the image of the camera device, the identification method further includes:

[0016] Performing a perspective transformation on the image to obtain the perspective-transformed image;

[0017] The step of obtaining the 2D bounding boxes and height lines of all vehicles in the above image includes:

[0018] Obtaining the 2D bounding boxes and height lines of all vehicles in the perspective-transformed image above;

[0019] And / or

[0020] The step of obtaining the 3D bounding box in the relative coordinate system according to the 2D bounding box and the height line includes:

[0021] Obtaining a pre-configured vanishing point;

[0022] Obtaining the projection image of the 3D bounding box on the 2D pixel plane according to the vanishing point, the 2D bounding box and the height line;

[0023] Converting to obtain the 3D bounding box in the relative coordinate system according to the perspective principle and the size ratio of the front and rear sides in the projection image.

[0024] Preferably, the step of filtering the radar points of the target vehicle in the radar point cloud includes:

[0025] Selecting the projection of the front side of the 3D bounding box in the relative coordinate system on the 2D pixel plane;

[0026] Filter the radar points of the target vehicle from the above-mentioned radar point cloud in the projection area.

[0027] Preferably, the step of obtaining the absolute distance of the radar points of the target vehicle includes:

[0028] Perform density clustering on the filtered radar points of the target vehicle, and average the positions of the radar points in the category with the most radar points to obtain an average position point;

[0029] Perform a modulo operation on the average position point to obtain the absolute distance of the radar points of the target vehicle;

[0030] And / or,

[0031] The step of identifying the size of the target vehicle according to the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle includes:

[0032] Convert the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle to obtain a 3D bounding box in the absolute coordinate system;

[0033] Identify the size of the target vehicle according to the 3D bounding box in the absolute coordinate system.

[0034] The second aspect of the present invention provides a vehicle size identification system, and the identification system includes:

[0035] A reading module, configured to read the radar point cloud of the radar device and the image of the camera device;

[0036] A first acquisition module, configured to acquire the 2D bounding box and height line of all vehicles in the above-mentioned image;

[0037] A second acquisition module, configured to obtain a 3D bounding box in the relative coordinate system according to the 2D bounding box and the height line;

[0038] A screening module, configured to screen the radar points of the target vehicle from the radar point cloud;

[0039] A third acquisition module, configured to acquire the absolute distance of the radar points of the target vehicle;

[0040] An identification module, configured to identify the size of the target vehicle according to the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle.

[0041] Preferably, the identification system further includes:

[0042] A perspective transformation module for performing perspective transformation on the image to obtain a perspective-transformed image;

[0043] A second acquisition module for acquiring 2D bounding boxes and height lines of all vehicles in the perspective-transformed image;

[0044] And / or

[0045] The second acquisition module includes:

[0046] A first acquisition unit for acquiring a pre-configured vanishing point;

[0047] A second acquisition unit for obtaining a projected image of the 3D bounding box on the 2D pixel plane according to the vanishing point, the 2D bounding box, and the height line;

[0048] A third acquisition unit for calculating and obtaining the 3D bounding box in the relative coordinate system according to the perspective principle and the size ratio of the front and rear sides in the projected image.

[0049] Preferably, the screening module includes:

[0050] A selection unit for selecting the projection of the front side of the 3D bounding box in the relative coordinate system on the 2D pixel plane;

[0051] A screening unit for screening the radar points of the target vehicle from the radar point cloud in the above projection area.

[0052] Preferably, the third acquisition module includes:

[0053] A fourth acquisition unit for performing density clustering on the screened radar points of the target vehicle, and performing an average operation on the positions of the radar points in the category with the most radar points to obtain an average position point;

[0054] A fifth acquisition unit for performing a modulo operation on the average position point to obtain the absolute distance of the radar points of the target vehicle;

[0055] And / or

[0056] The recognition module includes:

[0057] A sixth acquisition unit for calculating and obtaining the 3D bounding box in the absolute coordinate system according to the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle;

[0058] A recognition unit for recognizing the size of the target vehicle according to the 3D bounding box in the absolute coordinate system.

[0059] The third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and configured to run on the processor. When the processor executes the computer program, the method for identifying the vehicle size as described in the first aspect is implemented.

[0060] The fourth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for identifying the vehicle size as described in the first aspect is implemented.

[0061] On the basis of conforming to the common knowledge in the art, the above preferred conditions can be combined arbitrarily to obtain various preferred embodiments of the present invention.

[0062] The positive and progressive effects of the present invention are as follows:

[0063] Based on the image of the camera device, the present invention obtains the 3D bounding box of the target vehicle in the relative coordinate system, and based on the radar point cloud of the radar device, obtains the absolute distance of the radar points of the target vehicle. The size of the target vehicle is identified according to the 3D bounding box in the relative coordinate system and the absolute distance. It realizes the accurate identification of the vehicle size at low cost based on the radar device and the camera device, and improves the accuracy of vehicle size identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is the first flowchart of the method for identifying the vehicle size in Embodiment 1 of the present invention.

[0065] Figure 2 It is the second flowchart of the method for identifying the vehicle size in Embodiment 1 of the present invention.

[0066] Figure 3 It is a schematic diagram of the projection image of the 3D bounding box in the 2D pixel plane in Embodiments 1 and 2 of the present invention.

[0067] Figure 4 It is a schematic diagram of the modules of the control device of the vehicle size identification system in Embodiment 2 of the present invention.

[0068] Figure 5 It is a schematic diagram of the structure of the electronic device for implementing the method for identifying the vehicle size in Embodiment 3 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] The present invention will be further described below by way of embodiments, but the present invention is not limited to the scope of the described embodiments.

[0070] Embodiment 1

[0071] A method for identifying the vehicle size provided in this embodiment is as Figure 1 shown, and the identification method includes:

[0072] Step 101: Read the radar point cloud of the radar device and the image of the camera device;

[0073] In this embodiment, the read radar point cloud includes the radar point cloud of the current environment collected by the radar device; the read image includes the current environment image collected by the camera device; it should be noted that the radar point cloud of the current environment may include, but is not limited to, the radar points of the vehicle and the radar points of other objects; the current environment image may include, but is not limited to, the vehicle image and the images of other objects.

[0074] In the specific implementation process, the camera device and the radar device simultaneously collect the image and the radar point cloud at the same moment, and perform time alignment and coordinate system mapping.

[0075] In addition, the radar device may be a millimeter-wave radar, for example, a 4D millimeter-wave radar; the camera device may be a camera.

[0076] Step 102: Obtain the 2D bounding boxes and height lines of all vehicles in the above image;

[0077] In this embodiment, a neural network is used to identify all vehicles in the image to obtain the 2D bounding boxes of all vehicles in the image, and a regression method is used to predict the height lines. Specifically, the above image is used as the input of the neural network, and the neural network outputs the 2D bounding boxes and height lines of all vehicles in the above image.

[0078] It should be noted that the height line refers to the relative height in the pixel plane after the horizontal projection of the highest point of the vehicle onto the front plane of the vehicle head.

[0079] Step 103: Obtain the 3D bounding box in the relative coordinate system according to the 2D bounding box and the height line;

[0080] Step 104: Filter the radar points of the target vehicle in the radar point cloud;

[0081] Step 105: Obtain the absolute distance of the radar points of the target vehicle;

[0082] Step 106: Identify the size of the target vehicle according to the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle.

[0083] In this embodiment, it is necessary to calibrate the coordinate systems of both a radar device (such as a millimeter-wave radar) and a camera device (such as a camera), as well as calibrate the camera device to obtain the internal parameters of the camera device, and establish a transformation relationship from the radar point cloud to the pixel plane of the camera device (i.e., project the radar point cloud onto the pixel coordinate system of the camera device) for subsequent fusion of the real-time data acquired by the radar device and the camera device. The radar device and the camera device are used in combination, and the complementarity of the information of the two is utilized to achieve accurate vehicle size recognition based on the radar device and the camera device.

[0084] In an alternative embodiment, as Figure 2 shown, after step 101, the recognition method further includes:

[0085] Step 101-1: Perform a perspective transformation on the image to obtain the image after the perspective transformation;

[0086] As Figure 2 shown, step 102 includes:

[0087] Step 102-1: Obtain the 2D bounding boxes and height lines of all vehicles in the image after the above perspective transformation;

[0088] In this embodiment, a perspective transformation is performed on the image read by the camera device to correct the deviation in the roll angle direction of the camera device.

[0089] In an alternative embodiment, as Figure 2 shown, step 103 includes:

[0090] Step 103-1: Obtain the pre-configured vanishing point;

[0091] In this embodiment, during use, it is necessary to manually configure the vanishing point at infinity of the road relied on in this embodiment in the camera device screen.

[0092] Step 103-2: Obtain the projected image of the 3D bounding box on the 2D pixel plane according to the vanishing point, 2D bounding boxes, and height lines;

[0093] Step 103-3: Calculate and obtain the 3D bounding box in the relative coordinate system according to the perspective principle and the size ratio of the front and rear sides in the projected image.

[0094] In an alternative embodiment, as Figure 2 shown, step 104 includes:

[0095] Step 104-1: Select the projection of the front side of the 3D bounding box in the relative coordinate system on the 2D pixel plane;

[0096] Step 104-2: Filter the radar points of the target vehicle from the radar point cloud in the above projection area.

[0097] In this embodiment, the projection of the front side of the 3D bounding box on the 2D pixel plane is selected, and according to the projection of the radar point cloud on the pixel plane, the radar points that may belong to the target vehicle are filtered.

[0098] In an alternative embodiment, as Figure 2 shown, step 105 includes:

[0099] Step 105-1: Perform density clustering on the radar points of the target vehicle that have been filtered, and calculate the average of the positions of the radar points in the category with the most radar points to obtain the average position point;

[0100] In this embodiment, density clustering is performed on the radar points of the target vehicle that have been filtered in the case where the xyz position and Doppler value are used as the observation dimensions.

[0101] Step 105-2: Perform a modulo operation on the average position point to obtain the absolute distance of the radar points of the target vehicle;

[0102] In an alternative embodiment, as Figure 2 shown, step 106 includes:

[0103] Step 106-1: Convert the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle to obtain the 3D bounding box in the absolute coordinate system;

[0104] Step 106-2: Identify the size of the target vehicle according to the 3D bounding box in the absolute coordinate system.

[0105] In this embodiment, the 3D bounding box in the absolute coordinate system is converted according to the 3D object bounding box of the target vehicle in the relative coordinate system and the absolute distance; the size prediction results of the same vehicle in multiple frames are jointly used by the tracker, and weighted averaging is performed to obtain the final absolute size of the target vehicle.

[0106] In this embodiment, the radar device and the camera device are complementary in the dimension of perception information. The radar device can supplement the distance information and speed information that the camera device cannot perceive, and the camera device can supplement the information missing due to the sparsity of the millimeter-wave point cloud. Therefore, this embodiment combines the radar device and the camera device, and can complete the function of accurate vehicle size recognition at low cost. In addition, since the radar device (such as a 4D millimeter-wave radar) can itself obtain the Doppler speed of the vehicle, the function of measuring the speed of the vehicle can also be realized at the same time. Since the cost of the 4D millimeter-wave radar is lower than that of the lidar, the cost of the acquisition device will be lower. Moreover, the method of using images as input for neural network model prediction is very suitable for GPU acceleration, and the computational cost is also lower than the method using radar point cloud as input. Therefore, the vehicle size recognition method in this embodiment has a much lower cost than the existing vehicle size recognition method using laser point cloud.

[0107] In the specific implementation process, for example, taking one of the vehicles as an example, assume that the vehicle is driving along a straight road, and the size of the vehicle is the minimum size of the 3D bounding box that can accommodate the specified vehicle along the road direction. Assume that the vehicle bounding box is Box (the top endpoints are B1 and B2, and the bottom endpoints are B3 and B4), the height line value is u, the infinitely far vanishing point of the road in the configuration is vp1, the infinitely far vanishing point of the road side is vp2, and the infinitely far vanishing point perpendicular to the ground direction is vp3.

[0108] S11. Calculate the points A1 and A2 where the height line intersects the left and right side edges of the bounding box, where A1 is the point closer to the outer side of the picture and A2 is the point closer to the inner side of the picture;

[0109] In this embodiment, if the point A1 closer to the outer side of the picture and the point A2 closer to the inner side of the picture appear on both sides of the vanishing point respectively, then perform S12 - S14; if the point A1 closer to the outer side of the picture and the point A2 closer to the inner side of the picture are on the same side of the vanishing point, then perform S15 - S20.

[0110] S12. Calculate the points C1 and C2 where the connection lines between the infinitely far vanishing point vp1 of the road and the point A1 closer to the outer side of the picture and the point A2 closer to the inner side of the picture intersect the top edge of the bounding box;

[0111] S13. At the points C1 and C2, draw perpendicular lines L1 and L2, and calculate the intersection points of the connection lines between the infinitely far vanishing point vp1 of the road and the bottom endpoints (B3 and B4) of the bounding box and the perpendicular lines, denoted as D1 and D2;

[0112] S14. At this time, the figure formed by A1, A2, C1, C2, B3, B4, D1, and D2 is the projection image of the 3D bounding box on the 2D pixel plane, as Figure 3 (b);

[0113] S15. Calculate the point C where the connection line between the infinitely far vanishing point vp1 of the road and the point A1 closer to the outer side of the picture intersects the top edge of the bounding box. Assume that the point on the top of the bounding box closer to the infinitely far vanishing point vp1 of the road is B1

[0114] S16. Assume that the point on the bottom of the bounding box far from the infinitely far vanishing point vp1 of the road is B3. Draw a perpendicular line at point C, and the intersection point of the perpendicular line and the connection line between the infinitely far vanishing point vp1 of the road and B3 is D;

[0115] S17. Draw a horizontal line at point D, and the intersection point of the horizontal line and the connection line between B1 and B4 is E;

[0116] S18. Calculate the intersection point F of the connection line between E and the infinitely far vanishing point vp1 of the road and the bottom of the bounding box;

[0117] S19. Draw a perpendicular line at point F, and the intersection point of the perpendicular line and the line connecting the point A1 close to the outer side of the screen and the point A2 close to the inner side of the screen is G.

[0118] S20. At this time, the figure formed by the points A1, C, B1, D, E, F, G, B3 close to the outer side of the screen is the projection image of the 3D bounding box on the 2D pixel plane, as Figure 3 (a)

[0119] S21. According to the ratio of the mutual distances of the four front points and the mutual distances of the four rear points of the projection image of the 3D bounding box, combined with the internal parameters of the imaging device of the imaging device, calculate the relative dimensions (w, h, l) after back-projecting to the 3D bounding box;

[0120] S22. Perform an inverse perspective transformation on the projection image of the 3D bounding box, and filter the radar points of the target vehicle in the radar point cloud according to the area formed by the four front points;

[0121] S23. Perform density clustering on the filtered radar points of the vehicle. At the same time, considering the information of the radar point position and speed dimensions, average the positions of the points under the category with the most points to obtain the average position point P. Perform a modulo operation on the average position point P to obtain the absolute distance N of the radar points of the vehicle. At the same time, the average value V of the Doppler speed of the vehicle can also be obtained;

[0122] S24. Use the absolute distance N to convert the relative dimensions (w, h, l) of the vehicle to obtain the absolute vehicle dimensions (W, H, L);

[0123] S25. Use the tracker to combine the size prediction results of the same vehicle in multiple frames and perform weighted averaging to obtain the final absolute vehicle size.

[0124] In this embodiment, in order to avoid accidental errors, the tracker is also used to combine the size prediction results of the same vehicle in multiple frames and perform weighted averaging to obtain the final absolute vehicle size.

[0125] Based on the image of the imaging device, this embodiment obtains the 3D bounding box of the target vehicle in the relative coordinate system, obtains the absolute distance of the radar points of the target vehicle based on the radar point cloud of the radar device, and identifies the size of the target vehicle according to the 3D bounding box in the relative coordinate system and the absolute distance. It realizes the accurate identification of the vehicle size at low cost based on the radar device and the imaging device, and improves the accuracy of vehicle size identification.

[0126] Embodiment 2

[0127] A vehicle size recognition system provided in this embodiment, as Figure 4As shown in the figure, the recognition system includes: a reading module 21, a first acquisition module 22, a second acquisition module 23, a screening module 24, a third acquisition module 25, and an identification module 26;

[0128] The reading module 21 is configured to read the radar point cloud of the radar device and the image of the camera device;

[0129] In this embodiment, the read radar point cloud includes the radar point cloud of the current environment collected by the radar device; the read image includes the current environment image collected by the camera device. It should be noted that the radar point cloud of the current environment may include, but is not limited to, the radar points of the vehicle and the radar points of other objects; the current environment image may include, but is not limited to, the vehicle image and the images of other objects.

[0130] In the specific implementation process, the camera device and the radar device simultaneously collect the image and the radar point cloud at the same moment, and perform time alignment and coordinate system mapping.

[0131] In addition, the radar device may be a millimeter-wave radar, for example, a 4D millimeter-wave radar; the camera device may be a camera.

[0132] The first acquisition module 22 is configured to acquire the 2D bounding boxes and height lines of all vehicles in the above image;

[0133] In this embodiment, a neural network is used to identify all vehicles in the image to obtain the 2D bounding boxes of all vehicles in the image, and a regression method is used to predict the height lines. Specifically, the above image is used as the input of the neural network, and the neural network outputs the 2D bounding boxes and height lines of all vehicles in the above image.

[0134] It should be noted that the height line refers to the relative height of the horizontal projection of the highest point of the vehicle onto the front plane in the pixel plane.

[0135] The second acquisition module 23 is configured to acquire the 3D bounding box in the relative coordinate system according to the 2D bounding box and the height line;

[0136] The screening module 24 is configured to screen the radar points of the target vehicle in the radar point cloud;

[0137] The third acquisition module 25 is configured to acquire the absolute distance of the radar points of the target vehicle;

[0138] The identification module 26 is configured to identify the size of the target vehicle according to the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle.

[0139] In this embodiment, it is necessary to calibrate the coordinate systems of both a radar device (such as a millimeter-wave radar) and a camera device (such as a camera), as well as calibrate the camera device to obtain the internal parameters of the camera device, and establish a transformation relationship from the radar point cloud to the pixel plane of the camera device (that is, project the radar point cloud onto the pixel coordinate system of the camera device) for subsequent fusion of the real-time data acquired by the radar device and the camera device. The radar device and the camera device are used in combination, and the complementarity of the information of the two is utilized to achieve accurate vehicle size recognition based on the radar device and the camera device.

[0140] In an alternative embodiment, as Figure 4 shown, the recognition system further includes: a perspective transformation module 27;

[0141] The perspective transformation module 27 is configured to perform a perspective transformation on an image to obtain a perspective-transformed image;

[0142] A second acquisition module 23 is configured to acquire 2D bounding boxes and height lines of all vehicles in the perspective-transformed image described above;

[0143] In this embodiment, the perspective transformation is performed on the image read by the camera device to correct the deviation in the roll angle direction of the camera device.

[0144] In an alternative embodiment, as Figure 4 shown, the second acquisition module 23 includes: a first acquisition unit 231, a second acquisition unit 232, and a third acquisition unit 233;

[0145] The first acquisition unit 231 is configured to acquire a pre-configured vanishing point;

[0146] In this embodiment, during use, it is necessary to manually configure the vanishing point at infinity of the road on which this embodiment depends in the camera device screen.

[0147] The second acquisition unit 232 is configured to obtain a projected image of the 3D bounding box on the 2D pixel plane according to the vanishing point, the 2D bounding box, and the height line;

[0148] The third acquisition unit 233 is configured to calculate and obtain the 3D bounding box in the relative coordinate system according to the perspective principle and the size ratio of the front and rear sides in the projected image.

[0149] In an alternative embodiment, as Figure 4 shown, the screening module 24 includes: a selection unit 241, a screening unit 242;

[0150] The selection unit 241 is configured to select the projection of the front side of the 3D bounding box in the relative coordinate system on the 2D pixel plane;

[0151] The screening unit 242 is configured to screen the radar points of the target vehicle from the radar point cloud in the above projection area.

[0152] In this embodiment, the projection of the front side of the 3D bounding box on the 2D pixel plane is selected, and according to the projection of the radar point cloud on the pixel plane, the radar points that may belong to the target vehicle are filtered.

[0153] In an alternative embodiment, as Figure 4 shown, the third acquisition module 25 includes: a fourth acquisition unit 251 and a fifth acquisition unit 252;

[0154] The fourth acquisition unit 251 is configured to perform density clustering on the radar points of the filtered target vehicle, and perform an averaging operation on the positions of the radar points in the category with the most radar points to obtain an average position point;

[0155] In this embodiment, density clustering is performed on the radar points of the filtered target vehicle in the case where the xyz position and the Doppler value are used as the observation dimensions.

[0156] The fifth acquisition unit 252 is configured to perform a modulo operation on the average position point to obtain the absolute distance of the radar points of the target vehicle;

[0157] In an alternative embodiment, as Figure 4 shown, the recognition module 26 includes: a sixth acquisition unit 261 and a recognition unit 262;

[0158] The sixth acquisition unit 261 is configured to convert the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle to obtain the 3D bounding box in the absolute coordinate system;

[0159] The recognition unit 262 is configured to recognize the size of the target vehicle according to the 3D bounding box in the absolute coordinate system.

[0160] In this embodiment, the 3D bounding box in the absolute coordinate system is converted according to the 3D object bounding box of the target vehicle in the relative coordinate system and the absolute distance; the size prediction results of the same vehicle in multiple frames are jointly used by the tracker, and weighted averaging is performed to obtain the final absolute size of the target vehicle.

[0161] In this embodiment, the radar device and the camera device are complementary in the dimension of perception information. The radar device can supplement the distance information and speed information that the camera device cannot perceive, and the camera device can supplement the information loss caused by the sparsity of millimeter-wave point clouds. Therefore, this embodiment combines the radar device and the camera device, and can complete the function of accurate vehicle size recognition at low cost. In addition, since the radar device (such as a 4D millimeter-wave radar) can obtain the Doppler speed of the vehicle itself, the function of measuring the speed of the vehicle can also be realized simultaneously. Since the cost of the 4D millimeter-wave radar is lower than that of the lidar, the cost of the acquisition device will be lower. Moreover, the method of using images as input for neural network model prediction is very suitable for acceleration by a graphics card, and the computational cost is also lower than the method using radar point clouds as input. Therefore, the vehicle size recognition method in this embodiment has a much lower cost than the existing vehicle size recognition method using laser point clouds.

[0162] In the specific implementation process, for example, taking one vehicle as an example, assume that the vehicle is driving along a straight road, and the size of the vehicle is the minimum size of the 3D bounding box that can accommodate the specified vehicle in the road direction. Assume that the vehicle bounding box is Box (the top endpoints are B1 and B2, and the bottom endpoints are B3 and B4), the height line value is u, the infinitely far vanishing point of the road in the configuration is vp1, the infinitely far vanishing point of the road side is vp2, and the infinitely far vanishing point perpendicular to the ground direction is vp3.

[0163] S11. Calculate the points A1 and A2 where the height line intersects the left and right side edges of the bounding box, where A1 is the point closer to the outer side of the picture and A2 is the point closer to the inner side of the picture;

[0164] In this embodiment, if the point A1 closer to the outer side of the picture and the point A2 closer to the inner side of the picture appear on both sides of the vanishing point respectively, then perform S12 - S14; if the point A1 closer to the outer side of the picture and the point A2 closer to the inner side of the picture are on the same side of the vanishing point, then perform S15 - S20.

[0165] S12. Calculate the points C1 and C2 where the line connecting the infinitely far vanishing point vp1 of the road and the points A1 closer to the outer side of the picture and A2 closer to the inner side of the picture intersects the top edge of the bounding box;

[0166] S13. Draw perpendicular lines L1 and L2 at points C1 and C2, and calculate the intersection points of the line connecting the infinitely far vanishing point vp1 of the road and the bottom endpoints (B3 and B4) of the bounding box with the perpendicular lines, denoted as D1 and D2;

[0167] S14. At this time, the figure formed by A1, A2, C1, C2, B3, B4, D1, and D2 is the projection image of the 3D bounding box on the 2D pixel plane, as Figure 3 (b);

[0168] S15. Calculate the intersection point C of the line connecting the infinitely distant vanishing point vp1 of the road and the point A1 near the outer side of the screen with the top edge of the bounding box. Assume that the point on the top of the bounding box close to the infinitely distant vanishing point vp1 of the road is B1.

[0169] S16. Assume that the point B3 at the bottom of the bounding box far from the infinitely distant vanishing point vp1 of the road. Draw a perpendicular line at point C, and the intersection point of the perpendicular line and the line connecting the infinitely distant vanishing point vp1 of the road and B3 is D.

[0170] S17. Draw a horizontal line at point D, and the intersection point of the horizontal line and the line connecting B1 and B4 is E.

[0171] S18. Calculate the intersection point F of the line connecting E and the infinitely distant vanishing point vp1 of the road with the bottom of the bounding box.

[0172] S19. Draw a perpendicular line at point F, and the intersection point with the line connecting the point A1 near the outer side of the screen and the point A2 near the inner side of the screen is G.

[0173] S20. At this time, the figure formed by the point A1 near the outer side of the screen, C, B1, D, E, F, G, and B3 is the projection image of the 3D bounding box on the 2D pixel plane, as Figure 3 (a)

[0174] S21. According to the ratio of the mutual distances of the four front points and the mutual distances of the four rear points of the projection image of the 3D bounding box, combined with the internal parameters of the camera device, calculate the relative dimensions (w, h, l) after back-projecting to the 3D bounding box.

[0175] S22. Perform an inverse perspective transformation on the projection image of the 3D bounding box. According to the area formed by the four front points, filter the radar points of the target vehicle in the radar point cloud.

[0176] S23. Perform density clustering on the filtered radar points of the vehicle. At the same time, considering the information of the radar point position and speed dimensions, average the positions of the points under the category with the most points to obtain the average position point P. Perform a modulo operation on the average position point P to obtain the absolute distance N of the radar points of the vehicle. At the same time, the average value V of the Doppler speed of the vehicle can also be obtained.

[0177] S24. Use the absolute distance N to convert the relative dimensions (w, h, l) of the vehicle to obtain the absolute vehicle dimensions (W, H, L).

[0178] S25. Use the tracker to combine the size prediction results of the same vehicle in multiple frames and perform weighted averaging to obtain the final absolute size of the vehicle.

[0179] In this embodiment, in order to avoid accidental errors, the tracker is also used to combine the size prediction results of the same vehicle in multiple frames and perform weighted averaging to obtain the final absolute size of the vehicle.

[0180] In this embodiment, the 3D bounding box of the target vehicle in the relative coordinate system is obtained based on the images captured by the imaging device, and the absolute distance of the radar points of the target vehicle is obtained based on the radar point cloud of the radar device. The size of the target vehicle is identified according to the 3D bounding box in the relative coordinate system and the absolute distance. It realizes the accurate identification of the vehicle size at low cost based on the radar device and the imaging device, and improves the accuracy of vehicle size identification.

[0181] Embodiment 3

[0182] Figure 5 It is a schematic structural diagram of an electronic device provided in Embodiment 3 of the present invention. The electronic device includes a memory, a processor, and a computer program stored on the memory and configured to run on the processor. When the processor executes the program, it implements the method for identifying the vehicle size in Embodiment 1. Figure 5 The shown electronic device 30 is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present invention.

[0183] As Figure 5 shown, the electronic device 30 may be presented in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 30 may include, but are not limited to: the at least one processor 31 mentioned above, the at least one memory 32 mentioned above, and a bus 33 connecting different system components (including the memory 32 and the processor 31).

[0184] The bus 33 includes a data bus, an address bus, and a control bus.

[0185] The memory 32 may include volatile memory, such as a random access memory (RAM) 321 and / or a cache memory 322, and may further include a read-only memory (ROM) 323.

[0186] The memory 32 may further include a program / utilities 325 having a set (at least one) of program modules 324. Such program modules 324 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0187] The processor 31 executes various functional applications and data processing by running the computer program stored in the memory 32, such as the method for identifying the vehicle size in Embodiment 1 of the present invention.

[0188] The electronic device 30 can also communicate with one or more external devices 34 (such as a keyboard, a pointing device, etc.). Such communication can be carried out through the input / output (I / O) interface 35. Moreover, the model generation device 30 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through the network adapter 36. As Figure 5 shown, the network adapter 36 communicates with other modules of the model generation device 30 through the bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in combination with the model generation device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (redundant array of independent disks) systems, tape drives, and data backup storage systems, etc.

[0189] It should be noted that, although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present invention, the features and functions of two or more of the above-described units / modules can be embodied in one unit / modules. Conversely, the features and functions of one unit / modules described above can be further divided and embodied by multiple units / modules.

[0190] Embodiment 4

[0191] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the vehicle size recognition method provided in Embodiment 1.

[0192] Among them, the more specific computer-readable storage medium can include but not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0193] In a possible implementation manner, the present invention can also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to enable the terminal device to execute the vehicle size recognition method described in Embodiment 1.

[0194] Among them, the program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed completely on the user device, partially on the user device, executed as an independent software package, partially on the user device and partially on a remote device, or completely on a remote device.

[0195] Although the specific embodiments of the present invention have been described above, those skilled in the art should understand that this is only an example, and the protection scope of the present invention is defined by the appended claims. Without departing from the principle and essence of the present invention, those skilled in the art can make various changes or modifications to these embodiments, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A method for identifying the size of a vehicle, characterized in that, the identification method includes: reading the radar point cloud of the radar device and the image of the camera device; obtaining the 2D bounding box and height line of all vehicles in the above image; obtaining a 3D bounding box in the relative coordinate system according to the 2D bounding box and the height line; screening the radar points of the target vehicle in the radar point cloud; obtaining the absolute distance of the radar points of the target vehicle; identifying the size of the target vehicle according to the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle.

2. The method for identifying the size of a vehicle according to claim 1, characterized in that, after the step of reading the radar point cloud of the radar device and the image of the camera device, the identification method further includes: performing a perspective transformation on the image to obtain a perspective-transformed image; the step of obtaining the 2D bounding box and height line of all vehicles in the above image includes: obtaining the 2D bounding box and height line of all vehicles in the perspective-transformed image; and / or, the step of obtaining a 3D bounding box in the relative coordinate system according to the 2D bounding box and the height line includes: obtaining a pre-configured vanishing point; obtaining a projection image of the 3D bounding box on the 2D pixel plane according to the vanishing point, the 2D bounding box and the height line; converting to obtain the 3D bounding box in the relative coordinate system according to the perspective principle and the size ratio of the front and rear sides in the projection image.

3. The method for identifying the size of a vehicle according to claim 1, characterized in that, the step of screening the radar points of the target vehicle in the radar point cloud includes: selecting the projection of the front side of the 3D bounding box in the relative coordinate system on the 2D pixel plane; screening the radar points of the target vehicle from the radar point cloud in the above projection area.

4. The method for identifying the size of a vehicle according to claim 1, characterized in that, the step of obtaining the absolute distance of the radar points of the target vehicle includes: performing density clustering on the screened radar points of the target vehicle, averaging the positions of the radar points in the category with the most radar points to obtain an average position point; performing a modulo operation on the average position point to obtain the absolute distance of the radar points of the target vehicle; and / or, the step of identifying the size of the target vehicle according to the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle includes: converting to obtain a 3D bounding box in the absolute coordinate system according to the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle; identifying the size of the target vehicle according to the 3D bounding box in the absolute coordinate system.

5. A vehicle size identification system, characterized in that, the identification system includes: a reading module for reading the radar point cloud of the radar device and the image of the camera device; a first acquisition module for obtaining the 2D bounding box and height line of all vehicles in the above image; a second acquisition module for obtaining a 3D bounding box in the relative coordinate system according to the 2D bounding box and the height line; A screening module for screening radar points of a target vehicle in the radar point cloud; A third acquisition module for acquiring the absolute distance of the radar points of the target vehicle; An identification module for identifying the size of the target vehicle based on the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle.

6. The vehicle size identification system according to claim 5, wherein, the identification system further includes: A perspective transformation module for performing perspective transformation on the image to obtain a perspective-transformed image; A second acquisition module for acquiring the 2D bounding box and height lines of all vehicles in the perspective-transformed image; and / or, the second acquisition module includes: A first acquisition unit for acquiring a pre-configured vanishing point; A second acquisition unit for obtaining a projection image of the 3D bounding box on the 2D pixel plane based on the vanishing point, the 2D bounding box, and the height lines; A third acquisition unit for converting, according to the perspective principle and the size ratio of the front and rear sides in the projection image, to obtain the 3D bounding box in the relative coordinate system.

7. The vehicle size identification system according to claim 5, wherein, the screening module includes: A selection unit for selecting the projection of the front side of the 3D bounding box in the relative coordinate system on the 2D pixel plane; A screening unit for screening radar points of the target vehicle from the radar point cloud in the above projection area.

8. The vehicle size identification system according to claim 5, wherein, the third acquisition module includes: A fourth acquisition unit for performing density clustering on the screened radar points of the target vehicle, averaging the positions of the radar points in the category with the most radar points to obtain an average position point; A fifth acquisition unit for performing a modulo operation on the average position point to obtain the absolute distance of the radar points of the target vehicle; and / or, the identification module includes: A sixth acquisition unit for converting, according to the 3D bounding box of the target vehicle in the relative coordinate system and the absolute distance of the radar points of the target vehicle, to obtain the 3D bounding box in the absolute coordinate system; An identification unit for identifying the size of the target vehicle based on the 3D bounding box in the absolute coordinate system.

9. An electronic device, including a memory, a processor, and a computer program stored on the memory and used to run on the processor, wherein, when the processor executes the computer program, it implements the vehicle size identification method according to any one of claims 1-4.

10. A computer-readable storage medium, on which a computer program is stored, wherein, when the computer program is executed by a processor, it implements the vehicle size identification method according to any one of claims 1-4.

Images