A method for generating vehicle oriented bounding box based on multivariate data fusion

Abstract

The present application discloses a method for generating a vehicle directional bounding box based on multivariate data fusion, which relates to the field of intelligent transportation technology, and comprises: step 1: performing target posture recognition on a target vehicle in a monocular image, and performing information fusion on a binocular recognition result in the same scene, and obtaining updated target posture data and projection results; step 2: obtaining point cloud data of the scene according to a front-end device, and extracting corresponding target-related point clouds, performing OBB reconstruction of the target vehicle in the image, and obtaining the geometric size, position information and heading angle of the target vehicle; step 3: performing information fusion on the updated target posture data and projection results and the OBB reconstruction results corresponding to the target-related point clouds, and optimizing the target size estimation value; the accuracy of target detection can be achieved, and a more reliable vehicle directional bounding box can be produced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a method for generating a vehicle directional bounding box based on multivariate data fusion. Background Art

[0002] In the field of intelligent roadside parking monitoring, there are usually perception and recognition errors caused by serious target occlusion, such as inaccurate detection or missed detection, which causes serious troubles to the overall data management of the system and correspondingly increases the labor cost in the management process. Therefore, in related tasks, perception enhancement for occluded targets is an important issue that needs to be solved urgently. Currently, with the continuous iteration and update of front-end equipment and technology, other perception results can be used to fuse the original results to improve the accuracy of the final information obtained.

[0003] Currently, a commonly used recognition data in parking management is the oriented bounding box (OBB) of the vehicle in the image. Due to the limitations of the types of front-end acquisition equipment and the existing annotation data types, monocular target posture detection solutions are mostly used to implement OBB construction. However, when there are obvious obstructions in front of the corresponding target (such as the vehicle in the front of the field of view in the parking area), there will be deviations in the prediction results. In particular, the deviation in the target's height information will cause a significant position offset in the projection of the vehicle's bottom surface, further resulting in a decrease in the accuracy of vehicle parking space matching in parking management, and causing subsequent data management logic deviations. Summary of the invention

[0004] The embodiment of the present application solves the problem of position offset of the obtained position information due to the influence of obstacles in the prior art by providing a method for generating a vehicle directional bounding box based on multivariate data fusion, thereby achieving the accuracy of vehicle position calculation.

[0005] The embodiment of the present application provides a method for generating a vehicle directional bounding box based on multivariate data fusion, comprising:

[0006] Step 1: Perform target posture recognition on the target vehicle in the monocular image, and perform information fusion on the binocular recognition results in the same scene to obtain updated target posture data and projection results;

[0007] Step 2: Obtain the point cloud data of the scene based on the front-end equipment, extract the corresponding target-related point cloud, reconstruct the OBB of the target vehicle in the image, and obtain the geometric size, position information and heading angle of the target vehicle;

[0008] Step 3: Information fusion is performed on the updated target pose data and projection results with the OBB reconstruction results corresponding to the target-related point cloud to optimize the target size estimate.

[0009] The specific implementation of step 1 is as follows:

[0010] Step 101, obtaining a monocular image corresponding to the camera, performing image detection on the target vehicle in the monocular image, and obtaining the target coordinates of the original image coordinate system;

[0011] Step 102: If the front-end device is a binocular camera, the same operation as step 101 is performed on another monocular camera to obtain the target coordinates of the target vehicle in the original image coordinate system in the other monocular camera, and the average of the relative scale information of the monocular camera and the prediction is used as the currently updated relative scale information;

[0012] Step 103, using the target detection result of a single picture, the target occlusion rate of each target vehicle is calculated through position correlation.

[0013] The specific implementation of step 101 includes:

[0014] Step 1011, performing target detection on the monocular image, cropping the target area, and estimating the target posture of the target area to obtain target posture data;

[0015] Step 1012, performing edge detection on the target area, extracting the bounding box of the target image after projection, and extracting relative scale information corresponding to the target area;

[0016] Step 1013, according to the target posture data and the relative scale information, the target coordinates of the local area of ​​the monocular image after segmentation are calculated, and the target coordinates of the local area are projected into the original image coordinate system to obtain the target coordinates of the original image coordinate system.

[0017] The specific implementation methods of step 2 include:

[0018] Step 201, extracting point cloud data in the scene through the device and performing preprocessing, such as filtering, removing interference such as ground reflection and surrounding invalid static objects;

[0019] Step 202, performing target recognition on the point cloud data, extracting the point cloud data group corresponding to each target vehicle, and further extracting interference data around the target point cloud;

[0020] Step 203, using the three-dimensional information of the target point cloud and the target characteristic size and structure features, calculate the three-dimensional coordinates of the target vehicle, calculate the projection coordinates in the image based on the camera extrinsic parameters that have been registered, and obtain the absolute physical size information corresponding to the target vehicle.

[0021] The specific implementation methods of step 3 include:

[0022] Step 301, obtaining the point-by-point mean and point-by-point average variance of the target coordinates and the projection coordinates of the original image coordinate system;

[0023] Step 302 , according to the point-by-point average variance and the target occlusion rate, the OBB coordinates corresponding to the target vehicle are updated, and the updated OBB coordinates are output as the target size estimation value.

[0024] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0025] By recognizing the target posture in the monocular image and fusing the binocular recognition results, the target vehicle's posture data and projection results can be obtained more accurately. This fusion method can reduce the error caused by a single data source and improve the stability and accuracy of target detection.

[0026] By combining point cloud data with the target vehicle to reconstruct the OBB, the geometric size, position information and heading angle of the target vehicle can be obtained. Point cloud data provides rich three-dimensional information, which helps to reconstruct the shape and posture of the target vehicle more accurately. At the same time, through information fusion, the target size estimation value can be optimized and the robustness of OBB reconstruction can be improved.

[0027] By calculating the target occlusion rate, the degree to which the target vehicle is occluded can be more accurately assessed. This helps to consider the influence of occlusion factors in subsequent data processing and OBB reconstruction, thereby improving the accuracy of target vehicle recognition.

[0028] The information from multiple data sources, such as monocular images, binocular recognition results, and point cloud data, is fully utilized to optimize the target size estimation value through information fusion. This multi-source data fusion method can improve the comprehensiveness and accuracy of data processing and provide more reliable support for the generation of vehicle oriented bounding boxes. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of steps 1-3 of a method for generating a vehicle directional bounding box based on multivariate data fusion according to the present invention;

[0030] Figure 2 It is a flowchart diagram of steps 101-103 of a method for generating a vehicle directional bounding box based on multivariate data fusion according to the present invention;

[0031] Figure 3 It is a flowchart diagram of steps 1011-1013 of a method for generating a vehicle directional bounding box based on multivariate data fusion according to the present invention;

[0032] Figure 4 It is a flowchart diagram of steps 201-203 of a method for generating a vehicle directional bounding box based on multivariate data fusion according to the present invention;

[0033] Figure 5It is a flowchart diagram of steps 301 - 302 of a method for generating a vehicle directional bounding box based on multivariate data fusion according to the present invention. DETAILED DESCRIPTION

[0034] To facilitate the understanding of the present invention, the present application will be described more comprehensively below with reference to the relevant drawings; the drawings show preferred embodiments of the present invention, but the present invention can be implemented in many different forms and is not limited to the embodiments described herein; on the contrary, the purpose of providing these embodiments is to enable a more thorough and comprehensive understanding of the disclosed content of the present invention.

[0035] It should be noted that the terms “vertical”, “horizontal”, “up”, “down”, “left”, “right” and similar expressions used in this document are only for illustrative purposes and do not represent the only implementation method.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs; the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the term "and / or" used herein includes any and all combinations of one or more related listed items.

[0037] At present, the bounding box generation method mainly has the following problems: the bounding box calculation method considering the direction is highly complex, the operation time is too long, and the hardware load is high; the present invention uses the existing basic generation algorithm and the available target point cloud data to correct the target posture data to improve the OBB reconstruction accuracy of the occluded target in the scene and assist in extracting the accuracy of other berth management algorithms; the first step is to calculate the occlusion rate of the target after obtaining the projection of the directed bounding box corresponding to the target to assist in the evaluation of the subsequent fusion method; the second step is to obtain the point cloud data of the scene, and extract the corresponding target-related point cloud, reconstruct the OBB of the target in the image, and obtain the geometric size, position information, etc. of the target; the third step is to fuse the two sets of OBB reconstruction results in the image, optimize the target size estimation value, and improve the accuracy of the projection position.

[0038] Embodiment 1

[0039] like Figure 1 As shown, the present invention provides a method for generating a vehicle directional bounding box based on multivariate data fusion, including:

[0040] Step 1: Perform target posture recognition on the target vehicle in the monocular image, and perform information fusion on the binocular recognition results in the same scene to obtain updated target posture data and projection results;

[0041] Step 2: Obtain the point cloud data of the scene based on the front-end equipment, extract the corresponding target-related point cloud, reconstruct the OBB of the target vehicle in the image, and obtain the geometric size, position information and heading angle of the target vehicle;

[0042] Step 3: Information fusion is performed on the updated target pose data and projection results with the OBB reconstruction results corresponding to the target-related point cloud to optimize the target size estimate.

[0043] When the target size estimate is more accurate, the OBB bounding box can more closely fit the actual shape and size of the target object; this means that when performing collision detection, the possibility of misjudgment and missed detection can be reduced, and the accuracy of detection can be improved; a more accurate target size estimate means that the OBB bounding box can more accurately reflect the occupancy of the target object in three-dimensional space. This helps to use space more efficiently and avoid unnecessary collisions and interference in applications such as path planning and scene understanding.

[0044] In the present invention, step 1 is used to identify the target posture of the target vehicle, obtain the projection result of the directed bounding box, and calculate the occlusion rate of the target. Step 2 is used to obtain the geometric size, position information and heading angle of the target vehicle through point cloud data. Here, the OBB calculation results obtained in steps 1 and 2 are fused in step 3 to optimize the estimated value of the target size. The first step calculates the occlusion rate of the target, and this information can be used to evaluate the reliability and weight of the two sets of OBB results. If the target is severely occluded in the image, the OBB result obtained by relying solely on monocular image recognition may not be accurate enough. At this time, more reliance should be placed on the results obtained from point cloud data. On the contrary, if the occlusion rate is low, the result of monocular image recognition may be more reliable.

[0045] In the present invention, the monocular image required for step 1 is obtained by setting up an image acquisition device such as a camera or a vehicle-mounted camera, and the coordinates and position posture of the corresponding position are obtained after the monocular image is recognized.

[0046] When using the OBB algorithm, the following steps are generally used:

[0047] Step 1: Decompose the xyz components of the point set, that is, put the x, y, and z values ​​of all points into separate arrays

[0048] Step 2: Calculate the covariance matrix for the three random variables x, y, and z (one-dimensional array)

[0049] Step ③: Solve the eigenvalues ​​and eigenvectors of the covariance matrix in step ②, and construct the column vector matrix M from the eigenvectors. Step ④: Translate the geometric center of the point set to the origin of the coordinate system and multiply them all by the M matrix for rotation transformation.

[0050] Step 5: Calculate xMax, xMin, yMax, yMin, zMax, and zMin of the coordinates of the point after rotation transformation, and then calculate the obb center coordinates and obb half-length

[0051] Step ⑥: Multiply the obb center coordinate by the inverse of M to get the coordinate value of this center coordinate in the original coordinate system

[0052] Step 7: Translate the obb center coordinates in the original coordinate system obtained in step 6 back to the original position (the translation vector is exactly the opposite vector to the translation vector in step 4)

[0053] At this point, the processing and analysis of the coordinates in the obtained image is completed. When the present invention uses the OBB algorithm to adjust and identify the coordinates, the occlusion rate of the obtained coordinates and the projected data is calculated, and the relative size that needs to be considered for the corresponding coordinates when there is an occlusion rate is calculated to determine the size range between each vehicle and the surrounding obstructions when a larger obstruction appears, thereby assisting parking.

[0054] Embodiment 2

[0055] In order to get a clear bounding box, such as Figure 2 As shown, the specific implementation of step 1 is as follows:

[0056] Step 101, obtain a monocular image corresponding to the camera, perform image detection on the target vehicle in the monocular image, and obtain the target coordinates in the original image coordinate system.

[0057] Specifically, Figure 3 As shown, the specific implementation of step 101 includes:

[0058] Step 1011, performing target detection on the monocular image, cropping the target area, and estimating the target posture of the target area to obtain target posture data;

[0059] Step 1012, performing edge detection on the target area, extracting the bounding box of the target image after projection, and extracting relative scale information corresponding to the target area;

[0060] Step 1013, according to the target posture data and the relative scale information, the target coordinates of the local area of ​​the monocular image after segmentation are calculated, and the target coordinates of the local area are projected into the original image coordinate system to obtain the target coordinates of the original image coordinate system.

[0061] Specifically, the current processing method is: perform target detection on the monocular image, estimate the target posture after cropping the target area, obtain the target posture data, perform edge detection on the monocular image, identify the projection result of the target image, and extract the relative scale information. According to the posture parameters corresponding to the above-mentioned calculated target posture data, the target coordinates kp8_0 of the local area are obtained, and further according to the coordinates of the target area in the original image, the target coordinates of the local area are projected back to the original image coordinate system to obtain the target coordinates kp8_1a corresponding to the original image coordinate system.

[0062] The target posture data includes the posture angle and relative position of the target vehicle; the relative scale information is expressed as size1 = (wr, 1, hr) = (wo / lo, 1, ho / lo).

[0063] Wherein, wr: the relative width of the target vehicle in the cropped image, hr: the relative height of the target vehicle in the cropped image, wo: the width of the target vehicle in the original image, ho: the height of the target vehicle in the original image, lo: a reference length, which can be the actual length of the target vehicle, a fixed length in the image, or a characteristic length of the target vehicle.

[0064] Then, the calculation of wr and hr can be expressed as:

[0065] wr=wo / lo: the relative width of the target vehicle, that is, the width of the target vehicle in the original image divided by the reference length.

[0066] hr = ho / lo: relative height of the target vehicle, that is, the height of the target vehicle in the original image divided by the reference length.

[0067] Preferably, the attitude angles in the target attitude data include the yaw angle, heading angle and pitch angle of the target vehicle; the yaw angle (or yaw angle) refers to the angle of rotation of the vehicle around the vertical axis, that is, the angular difference between the vehicle's forward direction and the vehicle's longitudinal axis, the heading angle refers to the angle between the vehicle's front wheel steering angle and the vehicle's driving direction, that is, the angular difference between the vehicle's forward direction and the vehicle's longitudinal axis, the pitch angle refers to the angle of the vehicle's front and rear tilt, this angle describes the vehicle's front and rear stability when facing an undulating road surface or accelerating and decelerating; the target attitude data obtained at this time is to determine the posture that the vehicle may move when the vehicle is moving, and to identify whether the current vehicle will collide based on the projection results corresponding to this part of the target attitude data; and when an obstruction appears during the movement of the vehicle, accurately identify the corresponding position of the vehicle.

[0068] Preferably, the relative position in the above-mentioned target posture data is obtained by calculating the distance between the target vehicles.

[0069] Preferably, when acquiring the attitude angle of the target vehicle, the coordinates of the target are calculated and converted to obtain the value of the attitude angle, and the relative position and relative scale information between each target vehicle at the current scale are determined to determine the relative positions between different target vehicles.

[0070] Preferably, after acquiring a monocular image from a camera, an edge detection algorithm (such as Canny edge detection) is used to identify the edge of the target vehicle in the image; based on the similarity of color, texture or other image features, the image is segmented into different regions, and the target vehicle is usually segmented into one or more regions, so as to determine its position and shape in the image; one or more template images are predefined, and these templates represent the possible appearance of the target, and the projection of the target in the image can be obtained by sliding the template and finding the best matching position in the image; deep learning models such as convolutional neural networks (CNN) can learn to extract features from images and directly output bounding boxes of the target in the image, and these bounding boxes are the projection results of the target; at this time, the relative scale information can be directly obtained from the projection result, and the modification of the relative scale information is the update of the projection result.

[0071] Step 102, if the front-end device is a binocular camera, the same operation as step 101 is performed on another monocular camera to obtain the target coordinates of the target vehicle in the original image coordinate system in the other monocular camera, and the average of the relative scale information of the monocular camera and the prediction is used as the currently updated relative scale information.

[0072] Specifically, the OBB coordinates kp8_1b and relative scale information size1b of the target vehicle in the original image coordinate system in another monocular camera are obtained, and at the same time, the relative scale information size1=(size1+size1b) / 2 is updated according to the relative scale information predicted by the model.

[0073] In this step, if the camera used is a binocular camera, the data collected by the binocular camera is matched with the target coordinates of the original image coordinate system in the manner of step 1.1, and the mean of the monocular camera and the binocular camera is obtained as the average value of the target coordinates of the original image coordinate system to update the relative scale information.

[0074] Step 103, using the target detection result of a single image, the target occlusion rate bioa of each target vehicle is calculated through position correlation, which is used as reference data for subsequent information fusion.

[0075] In this step, the relative position and target posture data collected by the monocular camera and the binocular camera are calculated to determine the target occlusion rate between the target vehicles appearing in each single picture. When calculating the target occlusion rate, the overlap rate of the projection area of ​​each detected target vehicle and the corresponding overlapping position coordinates are calculated to obtain it.

[0076] Specifically, the implementation method of calculating the target occlusion rate of each target vehicle includes:

[0077] Use target detection algorithms such as Faster R-CNN and YOLO to detect targets in a single image and determine the bounding box of each target vehicle;

[0078] Extract the pixel value of the bounding box of the target vehicle, and when the pixel value in the bounding box is greater than a threshold, mark the corresponding area as an occluded area; determine the convex hull and contour area of ​​the occluded area, and determine the target pixel point of the occluded area based on the convex hull and contour area of ​​the occluded area, and the target pixel point is used to judge the size and range of the current occluded area to determine the accuracy of the current occluded area;

[0079] Use image processing technology to determine the position of the overlapping part of the target vehicle in the occluded area, extract the first pixel point and the second pixel point with the largest slope among the target pixel points, and determine the third pixel point that is farthest from the first pixel point and the second pixel point. Draw a fitting curve based on the first pixel point, the second pixel point, and the third pixel point. The target pixel point closest to the fitting curve is regarded as a boundary point, and the area of ​​the occluded area within the corresponding boundary point is determined. The ratio of the area of ​​the occluded area to the area of ​​the target vehicle after calculation is used as the target occlusion rate of each target vehicle.

[0080] Preferably, when processing monocular and binocular images, multi-target tracking technology is used to predict the movement of multiple targets in a specified area, and to extract target features such as points, lines, curves, reference points, etc. to achieve target tracking.

[0081] Embodiment 3

[0082] Preferably, in order to obtain detailed geometric information of the target vehicle in the three-dimensional space, and determine the rotation and translation changes of the vehicle in the three-dimensional space through the detailed geometric information, such as Figure 4 As shown, step 2 is also implemented in the following way:

[0083] Step 201, extracting point cloud data in the scene through the device and performing preprocessing, such as filtering, removing interference such as ground reflection and surrounding invalid static objects;

[0084] Step 202, target recognition is performed on the point cloud data, a point cloud data group corresponding to each target vehicle is extracted, and interference data around the target point cloud is further extracted;

[0085] Step 203, using the three-dimensional information of the target point cloud and the target characteristic size and structure characteristics, calculate the three-dimensional coordinates of the target vehicle (such as using the point cloud bounding box calculation method provided by open3D), and calculate the projection coordinates kp8_2 in the image based on the camera extrinsic parameters that have been registered, and obtain the absolute physical size information size2 corresponding to the target vehicle.

[0086] This step is to convert the image data in a single picture into point cloud data, and to extract the interference part of the area in the point cloud data to determine the display of the current image in three-dimensional space, and complete the two stages of object detection and object recognition through the point cloud target recognition algorithm. In the object detection stage, the algorithm will identify potential targets in the point cloud data; in the object recognition stage, the algorithm will classify the detected targets.

[0087] Preferably, the method for acquiring the target characteristic size and structural features includes:

[0088] According to the target recognition results of the image, we can use deep neural networks based on deep learning methods to extract the features of point cloud data and classify them to obtain the point cloud data corresponding to each target vehicle. The point cloud data after filtering interference is used as the target point cloud, and the target feature size and structural features of the vehicle corresponding to the target point cloud are determined; the target feature size and structural features include vehicle type and model, vehicle size, vehicle structure, vehicle status and usage.

[0089] Vehicle Type and Model: Different types of vehicles (such as sedans, SUVs, trucks, etc.) and specific models can be distinguished by characteristic dimensions and structural features. For example, certain models may have a unique body length, width or height, or specific body lines and contours.

[0090] Vehicle size: Characteristic dimensions are directly related to the overall size of the vehicle, such as body length, width and height, which helps to understand the vehicle's occupancy in space and is important for parking, driving route planning, etc.

[0091] Vehicle structure: Structural features describe the internal structure and external form of the vehicle, such as the number of doors, window locations, wheel styles, etc. These features provide a deeper understanding of the vehicle's design and functionality.

[0092] Vehicle status and usage: By analyzing the changes in characteristic dimensions and structural features, the vehicle's usage status can be inferred, such as whether it has been modified, whether it is damaged or worn, etc. This has practical value for vehicle evaluation, maintenance and insurance.

[0093] Preferably, in order to obtain the OBB projection coordinates, based on the calculated OBB three-dimensional coordinates, a transformation matrix is ​​used to project the OBB three-dimensional coordinates to the projection coordinate system to obtain projection coordinates of the projection transformation. For the projection coordinates used in step 203, based on the built-in projection matrix of the camera, the points in the three-dimensional space are converted to the clipping space, and then the position of the object on the screen is calculated by perspective division.

[0094] Specifically, the projection matrix transforms the vertices in camera space to clip space, and the coordinates of the points x, y, and z in clip space are all mapped to [-1, 1]. The points in clip space are still homogeneous coordinates and need further transformation to become Cartesian coordinates. This transformation process from homogeneous coordinates to Cartesian coordinates includes perspective division.

[0095] In addition, the projection matrix is ​​not only used to calculate the screen position, but also for clipping. That is, only the xy coordinates of the camera coordinate point multiplied by the projection matrix within a limited XYZ size range will be rendered on the screen, and the rest will be discarded. This range also constitutes the so-called clipping space.

[0096] Therefore, through a series of mathematical transformations and calculations, the projection coordinates of the object on the screen are finally obtained. This process involves steps such as coordinate conversion, clipping, and perspective division, and finally the OBB projection coordinates kp8_2 are calculated.

[0097] Preferably, after obtaining the absolute physical size information of the target vehicle, the method for obtaining the geometric size, position information and heading angle of the target vehicle includes the following implementation methods:

[0098] Extract the feature information corresponding to the boundary and corner points of the target vehicle from the target point cloud, and calculate the geometric dimensions of the target vehicle according to the three-dimensional coordinates of the target vehicle;

[0099] According to the target point cloud, the center of mass of the target vehicle is calculated, and the calculated center of mass is used as the position information of the target vehicle;

[0100] The covariance matrix of the point cloud data corresponding to the target vehicle is calculated, the eigenvector corresponding to the maximum eigenvalue is used as the heading of the target vehicle, and the heading angle of the target vehicle is output according to the heading of the target vehicle.

[0101] Once you have the point cloud data of the target vehicle, you can calculate its center of mass or center of gravity as the location of the vehicle. The center of mass is the average location of all the points in the point cloud data and is calculated as follows:

[0102] Python code

[0103] import numpy as np

[0104] #Assume that points is an Nx3 numpy array, where N is the number of points and 3 represents the (x, y, z) coordinates

[0105] points = np.array(...) #Get the point cloud data of the target vehicle from the point cloud data

[0106] # Calculate the centroid

[0107] centroid=np.mean(points,axis=0)

[0108] x_centroid,y_centroid,z_centroid=centroid

[0109] The centroid here is the position coordinates of the target vehicle in three-dimensional space.

[0110] At this point, the processing of the corresponding size information, position, and heading of the target vehicle in the three-dimensional space is completed. By calculating the target vehicle in the three-dimensional space separately, the size information of the target vehicle in the three-dimensional space can be clearly obtained. Combining the information processed by this part of the point cloud data with the information in image recognition can reduce the problem of image recognition being affected by large obstacles and improve the accuracy of positioning.

[0111] Embodiment 4

[0112] In order to make the projection result and the coordinates of the 3D reconstructed target vehicle more accurate and optimize the target size estimation, such as Figure 5 As shown, the implementation of step 3 includes:

[0113] Step 301, obtaining the point-by-point mean and point-by-point average variance of the target coordinates and the projection coordinates of the original image coordinate system;

[0114] Step 302 , according to the point-by-point average variance and the target occlusion rate, the OBB coordinates corresponding to the target vehicle are updated, and the updated OBB coordinates are output as the target size estimation value.

[0115] First, calculate the point-wise mean kp8_mean=(kp8_1+kp8_2) / 2 and the point-wise average variance kp8_var=sum((kp8_1-kp8_mean) / kp8_mean) / 8 of the OBB reconstruction results of the two groups of sources;

[0116] And obtain the target occlusion rate calculated in step 1. In the solution of the present invention, the target occlusion rate is obtained by checking the consistency of the occluded pixels and the unoccluded pixels, and then obtaining the corresponding ratio.

[0117] Preferably, when updating the OBB coordinates corresponding to the target vehicle, the following conditions are followed:

[0118] The point-by-point average variance value kp8_var and target occlusion rate bioa calculated in the above steps are discussed in different situations;

[0119] If the point-by-point average variance value kp8_var and the target occlusion rate bioa are both less than the threshold, the surface target is not obviously occluded and the estimated value is relatively stable. Then the OBB coordinates are simply updated by average. The updated OBB coordinate value is the point-by-point average of the target coordinates and the projection coordinates of the original image coordinate system.

[0120] If the point-by-point average variance value kp8_var is greater than the threshold, but the target occlusion rate bioa is less than the threshold, the surface target is not obviously occluded, but the data error causes the prediction result to be unstable. In this case, two schemes are pre-evaluated according to the scene, and one of them is selected as the current OBB coordinate value; or a secondary judgment is made based on the predicted size information. If the value of one of the schemes is seriously inconsistent with the existing target geometric size characteristics (the length, width and height ratio is out of balance, and the size prediction exceeds the conventional size too much), the result of the scheme is abandoned, and the value of the other scheme is used as the current OBB coordinate value;

[0121] If the point-by-point average variance value kp8_var is less than the threshold, but the target occlusion rate bioa is greater than the threshold, it means that although the target is severely occluded, the two sets of prediction results are highly consistent. At this time, it is necessary to further judge by verifying the rationality of the physical size information, such as whether the length, width and height ratio and the absolute size estimation meet the reasonable range. If so, the OBB coordinates are simply updated. The updated OBB coordinate value is the point-by-point mean of the target coordinates and the projection coordinates of the original image coordinate system. Otherwise, it is necessary to find a set of regular size ratios / target sizes according to the database and bring them into other posture parameters estimated by the model to recalculate the OBB coordinate value;

[0122] If both the point-by-point average variance value kp8_var and the target occlusion rate bioa are greater than the threshold, the surface target is severely occluded and the consistency of the two estimation results is low. According to the basic principle that the reliabilities of the reconstructed target sizes of the two data sources are different (the length-to-width ratio predicted by the model is more accurate but the height is generally smaller, and the height reconstructed by the point cloud is more accurate due to its accurate three-dimensional information), the height value estimated by the point cloud data is used, and the height information in the model size information predicted by the model is updated, and the other posture parameters estimated by the model are brought into the recalculation of the OBB coordinate value.

[0123] After obtaining the updated OBB coordinates, the relative scale information of the target vehicle is recalculated according to the OBB coordinates, and the calculated result is output as the target size estimation value.

[0124] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages:

[0125] The bounding box output by the target recognition algorithm (such as the target detection model in deep learning) can quickly and accurately locate the position of the target vehicle in the image. This helps to quickly filter and extract point cloud data related to the target vehicle and reduce the amount of data for subsequent processing.

[0126] After knowing the position and scale of the target in the image, we can more specifically extract the point cloud subset of the target vehicle from the point cloud data for 3D reconstruction. This can speed up the processing and reduce the consumption of computing resources.

[0127] Combining bounding box information and point cloud data allows for a more precise fit of the target vehicle’s 3D shape, resulting in a more accurate calculation of the geometry.

[0128] The precise position of the target vehicle in the two-dimensional image space can be obtained by using the bounding box position output by the target recognition algorithm. Further combined with the point cloud data, the precise position of the target in the three-dimensional space can be calculated.

[0129] This is more reliable than simply relying on point cloud data to calculate the location, because the target recognition algorithm has already extracted and classified the features of the image and can eliminate interference from background noise and other objects.

[0130] After knowing the position and posture of the target vehicle in the image, the heading angle of the target vehicle can be calculated more accurately by combining the direction information in the point cloud data.

[0131] This is very important for applications such as autonomous driving and traffic monitoring, as the heading angle is a key parameter for assessing the vehicle's driving status and direction.

[0132] By quickly locating the target through the target recognition algorithm and combining the point cloud data for 3D reconstruction and parameter calculation, the entire process can be completed more efficiently, which is crucial for real-time applications.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating a vehicle directional bounding box based on multivariate data fusion, characterized in that: The method comprises: Step 1: performing target posture recognition on the target vehicle in the monocular image, and performing information fusion on the binocular recognition results in the same scene to obtain updated target posture data and projection results; Step 2: Obtain the point cloud data of the scene based on the front-end equipment, extract the corresponding target-related point cloud, reconstruct the OBB of the target vehicle in the image, and obtain the geometric size, position information and heading angle of the target vehicle; Step 3: Information fusion is performed on the updated target pose data and projection results with the OBB reconstruction results corresponding to the target-related point cloud to optimize the target size estimate.

2. The method for generating a vehicle directional bounding box based on multivariate data fusion according to claim 1, characterized in that: The specific implementation of step 1 is as follows: Step 101, obtaining a monocular image corresponding to the camera, performing image detection on the target vehicle in the monocular image, and obtaining the target coordinates of the original image coordinate system; Step 102: If the front-end device is a binocular camera, the same operation as step 101 is performed on another monocular camera to obtain the target coordinates of the target vehicle in the original image coordinate system in the other monocular camera, and the average of the relative scale information of the monocular camera and the prediction is used as the currently updated relative scale information; Step 103, using the target detection result of a single picture, the target occlusion rate of each target vehicle is calculated through position correlation.

3. The method for generating a vehicle directional bounding box based on multivariate data fusion according to claim 2, characterized in that: The specific implementation of step 101 includes: Step 1011, performing target detection on the monocular image, cropping the target area, and estimating the target posture of the target area to obtain target posture data; Step 1012, performing edge detection on the target area, extracting the bounding box of the target image after projection, and extracting relative scale information corresponding to the target area; Step 1013, according to the target posture data and the relative scale information, the target coordinates of the local area of ​​the monocular image after segmentation are calculated, and the target coordinates of the local area are projected into the original image coordinate system to obtain the target coordinates of the original image coordinate system.

4. The method for generating a vehicle directional bounding box based on multivariate data fusion according to claim 2, characterized in that: The target posture data includes obtaining the posture angle and relative position of the target vehicle; the relative scale information is expressed as size1=(wr,1,hr)=(wo / lo,1,ho / lo); Among them, wr: the relative width of the target vehicle in the cropped image, hr: the relative height of the target vehicle in the cropped image, wo: the width of the target vehicle in the original image, ho: the height of the target vehicle in the original image, lo: a reference length.

5. The method for generating a vehicle directional bounding box based on multivariate data fusion according to claim 3, characterized in that: The implementation methods for calculating the target occlusion rate of each target vehicle include: Extract the pixel value of the bounding box of the target vehicle, and when the pixel value in the bounding box is greater than a threshold, mark the corresponding area as an occluded area; determine the convex hull and contour area of ​​the occluded area, and determine the target pixel point of the occluded area based on the convex hull and contour area of ​​the occluded area; The first pixel point and the second pixel point with the largest slope among the target pixel points are extracted, and the third pixel point farthest from the first pixel point and the second pixel point is determined. A fitting curve is drawn according to the first pixel point, the second pixel point, and the third pixel point. The target pixel point closest to the fitting curve is regarded as a boundary point, and the area of ​​the occluded region within the corresponding boundary point is determined. The ratio of the area of ​​the occluded region to the area of ​​the target vehicle after calculation is used as the target occlusion rate of each target vehicle.

6. The method for generating a vehicle directional bounding box based on multivariate data fusion according to claim 3, characterized in that: The specific implementation of step 2 includes: Step 201, extracting point cloud data in the scene through the device and performing preprocessing, such as filtering, removing interference such as ground reflection and surrounding invalid static objects; Step 202, performing target recognition on the point cloud data, extracting the point cloud data group corresponding to each target vehicle, and further extracting interference data around the target point cloud; Step 203, using the three-dimensional information of the target point cloud and the target characteristic size and structure features, calculate the three-dimensional coordinates of the target vehicle, calculate the projection coordinates in the image based on the camera extrinsic parameters that have been registered, and obtain the absolute physical size information corresponding to the target vehicle.

7. The method for generating a vehicle directional bounding box based on multivariate data fusion according to claim 6, characterized in that: The target characteristic size and structure characteristics include vehicle type and model, vehicle size, vehicle structure, vehicle status and usage.

8. The method for generating a vehicle directional bounding box based on multivariate data fusion according to claim 6, characterized in that: The acquisition methods of the geometric dimensions, position information and heading angle of the target vehicle include the following implementation methods: Extract the feature information corresponding to the boundary and corner points of the target vehicle from the target point cloud, and calculate the geometric dimensions of the target vehicle according to the three-dimensional coordinates of the target vehicle; According to the target point cloud, the center of mass of the target vehicle is calculated, and the calculated center of mass is used as the position information of the target vehicle; The covariance matrix of the point cloud data corresponding to the target vehicle is calculated, the eigenvector corresponding to the maximum eigenvalue is used as the heading of the target vehicle, and the heading angle of the target vehicle is output according to the heading of the target vehicle.

9. The method for generating a vehicle directional bounding box based on multivariate data fusion according to claim 6, characterized in that: The specific implementation of step 3 includes: Step 301, obtaining the point-by-point mean and point-by-point average variance of the target coordinates and the projection coordinates of the original image coordinate system; Step 302 , according to the point-by-point average variance and the target occlusion rate, the OBB coordinates corresponding to the target vehicle are updated, and the updated OBB coordinates are output as the target size estimation value.

10. The method for generating a vehicle directional bounding box based on multivariate data fusion according to claim 9, characterized in that: When updating the OBB coordinates corresponding to the target vehicle, the following conditions are followed: If the point-by-point average variance value and the target occlusion rate are both less than the threshold, the updated OBB coordinate value is the point-by-point average of the target coordinate and the projection coordinate of the original image coordinate system; If the point-by-point average variance value is greater than the threshold, but the target occlusion rate is less than the threshold, two schemes are pre-evaluated according to the scene, and one of the schemes is selected as the current OBB coordinate value. If the value of one of the schemes is seriously inconsistent with the existing target geometric size characteristics, the result of the scheme is abandoned and the value of the other scheme is used as the current OBB coordinate value; If the point-by-point average variance value is less than the threshold, but the target occlusion rate is greater than the threshold, check the rationality of the physical size information. If it is satisfied, the updated OBB coordinate value is the point-by-point mean of the target coordinate and the projection coordinate in the original image coordinate system. Otherwise, find a set of regular size ratios according to the database, bring them into other posture parameters estimated by the model to recalculate the OBB coordinate value. If the point-by-point average variance value and the target occlusion rate are both greater than the threshold, the height value estimated by the point cloud data is used, and the height information in the model size information predicted by the model is updated, and the other posture parameters estimated by the model are brought into the recalculation of the OBB coordinate value.

Images