Course angle detection method and device, electronic equipment and medium
By using multiple fisheye cameras in the automatic parking system to acquire image data, and combining fisheye model and inverse perspective mapping model, the problem of insufficient heading angle detection accuracy in automatic parking is solved, and higher heading angle detection accuracy is achieved.
Patent Information
- Application Number
- CN202510050750.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-13
- Publication Date
- 2025-04-15
AI Technical Summary
In the automatic parking function, it is crucial to accurately detect the heading angle of surrounding vehicles, but it is difficult for the prior art to achieve high accuracy detection.
By setting up multiple fisheye cameras on the first vehicle, obtaining image data and inputting the fisheye model and inverse perspective mapping model, the first corner point coordinate set and the second corner point coordinate set of the vehicle are obtained respectively, and the third corner point coordinate set of the target vehicle is determined based on these coordinate sets, and the target heading angle of the target vehicle is finally calculated.
This method improves the accuracy of heading angle detection by integrating the output results of the fisheye model and perspective mapping model, and can more accurately output the heading angle of surrounding vehicles.
Smart Images
Figure CN119832081A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of vehicle heading angle detection, and in particular to a heading angle detection method, a heading angle detection device, an electronic device and a computer-readable storage medium. Background Art
[0002] With the development of my country's automobile industry, the rate of autonomous driving in vehicles has increased year by year. As an important function of autonomous driving, automatic parking is used by more and more consumers in daily life. In the automatic parking function, it is necessary to determine the heading angle of surrounding vehicles in order to formulate subsequent automatic parking strategies. It can be seen that the accuracy of the heading angle detection of surrounding vehicles is crucial for the automatic parking function. Therefore, how to accurately determine the heading angle of surrounding vehicles has become one of the problems that need to be solved in the current automatic parking. Summary of the invention
[0003] In view of the above problems, a method for detecting a heading angle, a device for detecting a heading angle, an electronic device and a computer-readable storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:
[0004] A method for detecting a heading angle is applied to a first vehicle, wherein the first vehicle is provided with a plurality of fisheye cameras, and the method comprises:
[0005] Acquire first image data captured by the plurality of fisheye cameras, and input the first image data into a preset fisheye model to obtain a first corner point coordinate set for the second vehicle;
[0006] splicing the first image data collected by each fisheye camera into second image data, and inputting the second image data into a preset inverse perspective mapping model to obtain a second corner point coordinate set for the second vehicle;
[0007] Determine a third corner point coordinate set of the target vehicle according to the first corner point coordinate set and the second corner point coordinate set;
[0008] A target heading angle of the target vehicle is determined according to the third corner point coordinate set.
[0009] Optionally, determining a third corner point coordinate set of the target vehicle according to the first corner point coordinate set and the second corner point coordinate set includes:
[0010] Converting the first corner point coordinate set into a fourth corner point coordinate set in a camera coordinate system;
[0011] Converting the fourth corner point coordinate set into a fifth corner point coordinate set in a world coordinate system, and converting the second corner point coordinate set into a sixth corner point coordinate set in a world coordinate system;
[0012] A third corner point coordinate set of the target vehicle is determined based on the fifth corner point coordinate set and the sixth corner point coordinate set.
[0013] Optionally, converting the first corner point coordinate set into a fourth corner point coordinate set in a camera coordinate system includes:
[0014] The first corner point coordinate set is projected to a top-down perspective to obtain the fourth corner point coordinate set.
[0015] Optionally, the fifth corner point coordinate set includes a plurality of sets, the sixth corner point coordinate set includes a plurality of sets, and determining the third corner point coordinate set of the target vehicle according to the fifth corner point coordinate set and the sixth corner point coordinate set includes:
[0016] Determining a seventh corner point coordinate set for the target vehicle according to the plurality of fifth corner point coordinate sets;
[0017] and, determining an eighth corner point coordinate set for the target vehicle from the plurality of sixth corner point coordinate sets;
[0018] The third corner point coordinate set is determined according to the seventh corner point coordinate set and the eighth corner point coordinate set.
[0019] Optionally, determining a seventh corner point coordinate set for the target vehicle according to the plurality of fifth corner point coordinate sets comprises:
[0020] Calculate the first intersection-and-union correlation between the coordinate sets of the fifth corner points;
[0021] The seventh corner point coordinate set is determined according to the first intersection-over-union ratio association degree and a plurality of fifth corner point coordinate sets.
[0022] Optionally, determining an eighth corner point coordinate set for the target vehicle from a plurality of sixth corner point coordinate sets includes:
[0023] Determining, from the plurality of sixth corner point coordinate sets, a target sixth corner point coordinate set representing the same vehicle as the seventh corner point coordinate set;
[0024] Based on Kalman wave filtering, the eighth corner point coordinate set is determined according to the sixth corner point coordinate set of the target.
[0025] Optionally, determining, from a plurality of sixth corner point coordinate sets, a target sixth corner point coordinate set representing the same vehicle as the seventh corner point coordinate set comprises:
[0026] Based on the preset rectangular geometric constraints, each sixth corner point coordinate set is supplemented;
[0027] From the supplemented sixth corner point coordinate set, a target sixth corner point coordinate set representing the same vehicle as the seventh corner point coordinate set is determined.
[0028] Optionally, determining the third corner point coordinate set according to the seventh corner point coordinate set and the eighth corner point coordinate set includes:
[0029] Based on a preset rectangular geometric constraint, supplementing the seventh corner point coordinate set;
[0030] The third corner point coordinate set is determined according to the supplemented seventh corner point coordinate set and the eighth corner point coordinate set.
[0031] Optionally, determining the third corner point coordinate set according to the supplemented seventh corner point coordinate set and the eighth corner point coordinate set includes:
[0032] The eighth corner point coordinate set is calibrated according to the supplemented seventh corner point coordinate set to obtain the third corner point coordinate set.
[0033] Optionally, calibrating the eighth corner point coordinate set according to the supplemented seventh corner point coordinate set to obtain the third corner point coordinate set includes:
[0034] Determining a width ratio according to the supplemented seventh corner point coordinate set and the eighth corner point coordinate set;
[0035] When the width ratio is greater than a first preset value, using the eighth corner point coordinate set as the third corner point coordinate set;
[0036] When the width ratio is not greater than the first preset value, the direction of the vehicle head represented by the eighth corner point coordinate set is adjusted to obtain the third corner point coordinate set.
[0037] Optionally, the method further comprises:
[0038] When the distance between the target vehicle and the vehicle is less than a preset distance threshold, and / or the target vehicle output by the inverse perspective mapping model is lost, the supplemented seventh corner point coordinate set is used as the third corner point coordinate set.
[0039] The present invention also provides a heading angle detection device, which is applied to a first vehicle, wherein the first vehicle is provided with a plurality of fisheye cameras, and the device comprises:
[0040] A first analysis module, configured to obtain first image data collected by the plurality of fisheye cameras, and input the first image data into a preset fisheye model to obtain a first corner point coordinate set for a second vehicle;
[0041] A second analysis module, configured to stitch the first image data collected by each fisheye camera into second image data, and input the second image data into a preset inverse perspective mapping model to obtain a second corner point coordinate set for the second vehicle;
[0042] A corner point coordinate determination module, used for determining a third corner point coordinate set of the target vehicle according to the first corner point coordinate set and the second corner point coordinate set;
[0043] The heading angle determination module is used to determine the target heading angle of the target vehicle according to the third corner point coordinate set.
[0044] Optionally, the corner point coordinate determination module is used to convert the first corner point coordinate set into a fourth corner point coordinate set in the camera coordinate system; convert the fourth corner point coordinate set into a fifth corner point coordinate set in the world coordinate system, and convert the second corner point coordinate set into a sixth corner point coordinate set in the world coordinate system; and determine the third corner point coordinate set of the target vehicle based on the fifth corner point coordinate set and the sixth corner point coordinate set.
[0045] Optionally, the corner point coordinate determination module is used to project the first corner point coordinate set to a top-down perspective to obtain the fourth corner point coordinate set.
[0046] Optionally, the fifth corner point coordinate set includes multiple, the sixth corner point coordinate set includes multiple, and the corner point coordinate determination module is used to determine the seventh corner point coordinate set for the target vehicle based on multiple fifth corner point coordinate sets; and, to determine the eighth corner point coordinate set for the target vehicle from multiple sixth corner point coordinate sets; and to determine the third corner point coordinate set based on the seventh corner point coordinate set and the eighth corner point coordinate set.
[0047] Optionally, the corner point coordinate determination module is used to calculate a first intersection-and-union ratio correlation degree between any two fifth corner point coordinate sets; and determine the seventh corner point coordinate set based on the first intersection-and-union ratio correlation degree and multiple fifth corner point coordinate sets.
[0048] Optionally, the corner point coordinate determination module is used to determine a target sixth corner point coordinate set representing the same vehicle as the seventh corner point coordinate set from multiple sixth corner point coordinate sets; and determine the eighth corner point coordinate set based on the target sixth corner point coordinate set based on Kalman wave filtering.
[0049] Optionally, the corner point coordinate determination module is used to supplement each sixth corner point coordinate set based on a preset rectangular geometric constraint; and determine a target sixth corner point coordinate set representing the same vehicle as the seventh corner point coordinate set from the supplemented sixth corner point coordinate set.
[0050] Optionally, the corner point coordinate determination module is used to supplement the seventh corner point coordinate set based on a preset rectangular geometric constraint; and determine the third corner point coordinate set according to the supplemented seventh corner point coordinate set and the eighth corner point coordinate set.
[0051] Optionally, the corner point coordinate determination module is used to calibrate the eighth corner point coordinate set according to the supplemented seventh corner point coordinate set to obtain the third corner point coordinate set.
[0052] Optionally, the corner point coordinate determination module is used to determine the width ratio based on the supplemented seventh corner point coordinate set and the eighth corner point coordinate set; when the width ratio is greater than a first preset value, the eighth corner point coordinate set is used as the third corner point coordinate set; when the width ratio is not greater than the first preset value, the direction of the front of the vehicle represented by the eighth corner point coordinate set is adjusted to obtain the third corner point coordinate set.
[0053] Optionally, the corner point coordinate determination module is also used to use the supplemented seventh corner point coordinate set as the third corner point coordinate set when the distance between the target vehicle and the vehicle is less than a preset distance threshold, and / or the target vehicle output by the inverse perspective mapping model is lost.
[0054] The present invention also provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned vehicle intention recognition method when executed by the processor.
[0055] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for identifying vehicle intention as described above is implemented.
[0056] Beneficial effects of the present invention:
[0057] In the embodiment of the present invention, the first image data collected by multiple fisheye cameras can be obtained first, and the first image data can be input into a preset fisheye model to obtain a first corner point coordinate set for the second vehicle; the first image data collected by each fisheye camera can be spliced into second image data, and the second image data can be input into a preset inverse perspective mapping model to obtain a second corner point coordinate set for the second vehicle; then, according to the first corner point coordinate set and the second corner point coordinate set, the third corner point coordinate set of the target vehicle can be determined; thereafter, the target heading angle of the target vehicle can be determined according to the third corner point coordinate set. Through the embodiment of the present invention, the heading angle can be output by combining the accurate judgment of the front parking space of the fisheye model and the accurate detection of the coordinates by the perspective mapping model, thereby improving the accuracy of heading angle detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1a is a flowchart of a method for detecting a heading angle according to an embodiment of the present invention;
[0059] Figure 1b is a schematic diagram of a partial structure of a vehicle according to an embodiment of the present invention;
[0060] Figure 1c is a schematic diagram of a corner point in an embodiment of the present invention;
[0061] Figure 1d is a schematic diagram of another corner point in an embodiment of the present invention;
[0062] Figure 2 is a flowchart of another method for detecting a heading angle according to an embodiment of the present invention;
[0063] Figure 3 is a flow chart of steps for determining a heading angle according to an embodiment of the present invention;
[0064] Figure 4 It is a structural schematic diagram of a heading angle detection device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0065] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0066] In order to accurately detect the heading angle of the surrounding vehicles, an embodiment of the present invention provides a heading angle detection method, which can combine the output results of the fisheye model and the output results of the perspective mapping model to determine the heading angle of the surrounding vehicles, thereby combining the accurate judgment of the front parking space of the fisheye model and the accurate detection of the coordinates by the perspective mapping model, so that the heading angle of the surrounding vehicles can be accurately output. For details, please refer to Figure 1a , Figure 1a The flowchart of a method for detecting a heading angle according to an embodiment of the present invention is shown; the method can be applied to a first vehicle, i.e., the vehicle; a plurality of fisheye cameras can be arranged on the first vehicle; illustratively, the fisheye cameras can be arranged at four positions, front, back, left, and right, of the first vehicle, respectively, to collect image data in the four directions of the first vehicle; Figure 1bAs shown, the plurality of fisheye cameras 10 can be connected and interacted with the automatic parking controller 20 based on LVDS (Low Voltage Differential Signaling), so that the automatic parking controller 20 can output the heading angle of the vehicle.
[0067] like Figure 1a As shown, the method may include the following steps:
[0068] Step 101: Acquire first image data collected by multiple fisheye cameras, and input the first image data into a preset fisheye model to obtain a first corner point coordinate set for a second vehicle.
[0069] The fisheye camera is a wide-angle camera whose lens design enables it to capture a wider field of view. In practical applications, multiple fisheye cameras can be set on the first vehicle, and different fisheye cameras can face different directions to capture panoramic images of the area around the first vehicle.
[0070] The second vehicle may refer to the vehicles around the first vehicle. In practical applications, the first vehicle may determine the heading angle for each second vehicle. Specifically, each time the heading angle of a second vehicle is determined, the second vehicle whose heading angle is currently being determined may be taken as the target vehicle, and the target heading angle of the target vehicle may be determined.
[0071] In an embodiment of the present invention, when it is necessary to determine the heading angle of a second vehicle around a first vehicle, multiple fisheye cameras deployed on the first vehicle can be called to capture images to obtain one or more first image data. One first image data can be captured and obtained by one fisheye camera.
[0072] After obtaining a plurality of first image data, the first image data can be input into a pre-trained fisheye model to obtain a first corner point coordinate set for a plurality of second vehicles around the first vehicle; wherein the fisheye model can be pre-trained based on the labeled image data. A first corner point coordinate set of a second vehicle can include coordinate information of at least three corner points of the second vehicle output by the fisheye model; wherein the corner points can be numbered, for example, "0, 1, 2, 3"; the numbers can be in the order of Figure 1c Arrangement shown.
[0073] Corner points refer to the points on the ground projected from the smallest rectangle formed by fitting the top view of the vehicle. They are used to replace the wheel contact points to solve the problem that the wheels cannot be seen at certain angles and the correct heading angle cannot be output.
[0074] like Figure 1dAs shown, due to the position of the vehicle, the right rear wheel of the vehicle 30 cannot be identified; therefore, it can be identified based on the corner points; wherein the position of the corner point numbered "3" can be inferred based on the positions of the corner points numbered "0", "1" and "2".
[0075] Step 102: splice the first image data collected by each fisheye camera into second image data, and input the second image data into a preset inverse perspective mapping model to obtain a second corner point coordinate set for the second vehicle.
[0076] On the other hand, the first image data collected by each fisheye camera can be stitched into second image data; specifically, a plurality of first image data can be stitched into second image data through a preset stitching algorithm, and the second image data can be a BEV (Bird's Eye View) stitched image.
[0077] After obtaining the second image data, the second image data can be input into a pre-trained inverse perspective mapping model to obtain a second corner point coordinate set for multiple second vehicles around the first vehicle; wherein the inverse perspective mapping model can be pre-trained based on the labeled image data. A second corner point coordinate set of a second vehicle can include coordinate information of at least three corner points of the second vehicle output by the inverse perspective mapping model.
[0078] For example, the inverse perspective mapping model may refer to an IPM (Inverse Perspective Mapping) model, which may be obtained by training based on a large number of labeled mosaic images in advance.
[0079] Step 103: Determine a third corner point coordinate set of the target vehicle according to the first corner point coordinate set and the second corner point coordinate set.
[0080] After obtaining the first corner point coordinate set output by the fisheye model and the second corner point coordinate set output by the inverse perspective mapping model, a second vehicle can be taken as the target vehicle, and based on the first corner point coordinate set and the second corner point coordinate set of the target vehicle, a third corner point coordinate set of the target vehicle can be obtained by fusing; the third corner point coordinate set can be used as the final target detection frame of the target vehicle.
[0081] Step 104: Determine the target heading angle of the target vehicle according to the third corner point coordinate set.
[0082] After the third corner point coordinate set of the target vehicle is obtained, the target heading angle of the target vehicle can be calculated according to the coordinate information in the third corner point coordinate set.
[0083] For example, the third corner point coordinate set can be represented by (x i ,yi ,u i , v i ), the target heading angle a can be calculated by the following formula:
[0084]
[0085] After obtaining the target heading angle of the target vehicle, the heading angle of another second vehicle can be calculated; then, a strategy for subsequent automatic parking and other operations is formulated based on the heading angles of each second vehicle around the first vehicle, which is not limited in this embodiment of the present invention.
[0086] In an embodiment of the present invention, the first image data collected by multiple fisheye cameras can be obtained first, and the first image data can be input into a preset fisheye model to obtain a first corner point coordinate set for the second vehicle; the first image data collected by each fisheye camera can be spliced into second image data, and the second image data can be input into a preset inverse perspective mapping model to obtain a second corner point coordinate set for the second vehicle; then, according to the first corner point coordinate set and the second corner point coordinate set, the third corner point coordinate set of the target vehicle can be determined; thereafter, according to the third corner point coordinate set, the target heading angle of the target vehicle can be determined. Through the embodiment of the present invention, the accurate judgment of the front parking space of the fisheye model and the accurate detection of the coordinates by the perspective mapping model can be combined to output the heading angle, thereby improving the accuracy of heading angle detection.
[0087] Reference Figure 2 , shows a flowchart of another method for detecting a heading angle according to an embodiment of the present invention, which may include the following steps:
[0088] Step 201: Acquire first image data collected by multiple fisheye cameras, and input the first image data into a preset fisheye model to obtain a first corner point coordinate set for a second vehicle.
[0089] When the heading angle of the second vehicle around the first vehicle needs to be determined, multiple fisheye cameras deployed on the first vehicle can be called to collect images to obtain one or more first image data.
[0090] After obtaining a plurality of first image data, the first image data may be input into a pre-trained fisheye model to obtain a first corner point coordinate set for a plurality of second vehicles around the first vehicle.
[0091] Step 202: splice the first image data collected by each fisheye camera into second image data, and input the second image data into a preset inverse perspective mapping model to obtain a second corner point coordinate set for the second vehicle.
[0092] On the other hand, the first image data collected by each fisheye camera may be stitched together into the second image data; specifically, a plurality of first image data may be stitched together into the second image data through a preset stitching algorithm.
[0093] After the second image data is obtained, the second image data may be input into a pre-trained inverse perspective mapping model to obtain a second corner point coordinate set for a plurality of second vehicles around the first vehicle.
[0094] Step 203: convert the first corner point coordinate set into a fourth corner point coordinate set in the camera coordinate system.
[0095] In practical applications, the first corner point coordinate set is actually a set of coordinates of the corner points of the second vehicle in the fisheye image; in order to facilitate subsequent processing, the first corner point coordinate set can be first converted into a fourth corner point coordinate set in the camera coordinate system. The specific conversion process can be performed based on a preset conversion matrix, which is not limited in the embodiment of the present invention.
[0096] In one embodiment of the present invention, step 203 may be implemented by the following sub-steps:
[0097] Sub-step 11: Project the first corner point coordinate set to a top-down perspective to obtain a fourth corner point coordinate set.
[0098] In some feasible embodiments, the corner points in the first corner point coordinate set may be projected into a view from a top-down perspective to obtain the coordinates of each corner point in the first corner point coordinate set in the camera coordinate system, that is, to obtain a fourth corner point coordinate set.
[0099] Step 204: convert the fourth corner point coordinate set into a fifth corner point coordinate set in the world coordinate system, and convert the second corner point coordinate set into a sixth corner point coordinate set in the world coordinate system.
[0100] In some feasible embodiments, after obtaining the fourth corner point coordinate set in the camera coordinate system, the fourth corner point coordinate set in the camera coordinate system can be further converted into a fifth corner point coordinate set in the world coordinate system; and the second corner point coordinate set can also be converted into a sixth corner point coordinate set in the world coordinate system.
[0101] Exemplarily, the coordinates of the corner points in the camera coordinate system may be converted to the world coordinate system by presetting a rotation matrix and a translation matrix, thereby obtaining a fifth corner point coordinate set and a sixth corner point coordinate set.
[0102] Step 205: Determine a third corner point coordinate set of the target vehicle according to the fifth corner point coordinate set and the sixth corner point coordinate set.
[0103] After obtaining the fifth corner point coordinate set and the sixth corner point coordinate set, the fifth corner point coordinate set of the target vehicle and the sixth corner point coordinate set of the target vehicle may be fused to obtain the third corner point coordinate set that is finally output for the target vehicle.
[0104] In one embodiment of the present invention, the fifth corner point coordinate set may include multiple sets, and one fifth corner point coordinate set may correspond to one second vehicle; the sixth corner point coordinate set may also include multiple sets, and one sixth corner point coordinate set may also correspond to one second vehicle. In practical applications, step 205 may be implemented by the following sub-steps:
[0105] Sub-step 21: Determine a seventh corner point coordinate set for the target vehicle based on multiple fifth corner point coordinate sets.
[0106] In practical applications, one of the second vehicles may be taken as the target vehicle first; and then, the seventh corner point coordinate set of the target vehicle may be determined from a plurality of fifth corner point coordinate sets.
[0107] Sub-step 22: Determine an eighth corner point coordinate set for the target vehicle from the plurality of sixth corner point coordinate sets.
[0108] In addition, an eighth corner point coordinate set for the target vehicle may also be determined from the plurality of sixth corner point coordinate sets.
[0109] Sub-step 23: Determine a third corner point coordinate set based on the seventh corner point coordinate set and the eighth corner point coordinate set.
[0110] Then, the seventh corner point coordinate set and the eighth corner point coordinate set may be fused to output the final third corner point coordinate set of the target vehicle.
[0111] In one embodiment of the present invention, sub-step 21 may be implemented in the following manner:
[0112] The first intersection-and-union ratio correlation degree between each pair of the fifth corner point coordinate sets is calculated; and the seventh corner point coordinate set is determined according to the first intersection-and-union ratio correlation degree and a plurality of fifth corner point coordinate sets.
[0113] In some feasible embodiments, since the FOV (Field of View) of the fisheye camera is 180°, the camera field of view has an overlapping area, and the same vehicle target may appear in different first image data; therefore, the first intersection-and-union correlation between each second corner point coordinate set can be calculated respectively; illustratively, A represents the target frame area of the fifth corner point coordinate set of the second vehicle detected in the fisheye camera 1; B represents the target frame area of the fifth corner point coordinate set of the second vehicle detected in the fisheye camera 2, and the first intersection-and-union correlation d is calculated as follows:
[0114]
[0115] The threshold is set to 0.7. When d is greater than 0.7, the vehicle targets appearing in the two fisheye cameras are considered to be the same vehicle.
[0116] Based on the above settings, the seventh corner point coordinate set of the target vehicle can be determined according to the first intersection-and-union ratio correlation between any two fifth corner point coordinate sets and multiple fifth corner point coordinate sets.
[0117] In one embodiment of the present invention, sub-step 22 may be implemented in the following manner:
[0118] From the plurality of sixth corner point coordinate sets, a target sixth corner point coordinate set representing the same vehicle as the seventh corner point coordinate set is determined; and based on Kalman wave filtering, an eighth corner point coordinate set is determined according to the target sixth corner point coordinate set.
[0119] In some feasible embodiments, a target sixth corner point coordinate set representing the same vehicle as the seventh corner point coordinate set may be determined from a plurality of sixth corner point coordinate sets based on the intersection-over-union correlation.
[0120] Then, based on the Kalman wave filter and according to the sixth corner point coordinate set of the target, the eighth corner point coordinate set of the target vehicle can be calculated.
[0121] For example, an 8-dimensional state vector can be established To characterize any target vehicle detected at any time k, where x i ,y i ,u i , v i are the horizontal and vertical coordinates of the corner points on the same side of the two vehicles selected at that moment, It is the speed information corresponding to the first four elements in the image space. The speed information can be calculated based on the coordinates of the corner points observed in two frames.
[0122] Then, the Kalman filter is used to predict and update the coordinates. The Kalman filter adopts a uniform speed model and a linear observation model, and its observation vector is the new corner point coordinates, that is, the eighth corner point coordinate set (x, y, u, v).
[0123] The Kalman filter algorithm process is as follows:
[0124] Prediction status:
[0125]
[0126] Forecast estimate covariance matrix:
[0127]
[0128] Calculate the Kalman gain:
[0129]
[0130] Update the state estimate:
[0131]
[0132] Update the covariance estimate:
[0133] P k|k =(IK k H k ) k|k-1 .
[0134] in:
[0135] F k is the linear system state transfer matrix; is the optimal state estimate at time k-1; is the optimal state estimate at time kk; is the predicted state at time k based on the state estimate at time k-1; P k|k-1 is the predicted state covariance at time k based on the state covariance at time k-1; P k-1|k-1 is the optimal state covariance at time k-1; P k|k is the optimal state covariance at time k; Yes F k The transposed matrix of k is the covariance matrix; K k is the Kalman gain at time k; H k is the observation matrix; Yes H k The transposed matrix of k is the observation covariance; z k is the observation vector at time k; I is the identity matrix.
[0136] In one embodiment of the present invention, the target sixth corner point coordinate set can be determined in the following specific manner:
[0137] Based on the preset rectangular geometric constraint, the seventh corner point coordinate set is supplemented; and the third corner point coordinate set is determined according to the supplemented seventh corner point coordinate set and the eighth corner point coordinate set.
[0138] In some feasible embodiments, due to the occlusion of the second vehicle, the corner point collection for one vehicle may be incomplete, that is, only three corner points can be collected; in this case, the position of the front and rear of the vehicle can be determined by the number of the corner points; wherein, corner points 0 and 1 correspond to the front direction of the vehicle, and corner points 2 and 3 correspond to the rear direction of the vehicle. Based on the existing three corner points, according to the preset rectangular geometric constraints, the position of the occluded corner point is calculated, so as to supplement the seventh corner point coordinate set, and then obtain the seventh corner point coordinate set with four corner points.
[0139] After the seventh corner point coordinate set is supplemented, the third corner point coordinate set may be determined according to the supplemented seventh corner point coordinate set and the eighth corner point coordinate set.
[0140] In some feasible embodiments, the eighth corner point coordinate set may be a complete coordinate set with four corner points; illustratively, before the coordinates are updated by using Kalman filtering to predict the coordinates, the coordinate set may be supplemented and then the coordinates may be updated, which is not limited in this embodiment of the present invention.
[0141] In one embodiment of the present invention, when determining the third corner point coordinate set according to the supplemented seventh corner point coordinate set and the eighth corner point coordinate set, the third corner point coordinate set may be determined in the following specific manner:
[0142] The eighth corner point coordinate set is calibrated according to the supplemented seventh corner point coordinate set to obtain the third corner point coordinate set.
[0143] In some feasible embodiments, after the seventh corner point coordinate set is supplemented, the eighth corner point coordinate set may be calibrated in direction according to the supplemented seventh corner point coordinate set, thereby obtaining the third corner point coordinate set.
[0144] As an example, calibration can be performed as follows:
[0145] The width ratio is determined based on the supplemented seventh corner point coordinate set and the eighth corner point coordinate set; when the width ratio is greater than the first preset value, the eighth corner point coordinate set is used as the third corner point coordinate set; when the width ratio is not greater than the first preset value, the direction of the front of the vehicle represented by the eighth corner point coordinate set is adjusted to obtain the third corner point coordinate set.
[0146] In some feasible embodiments, the supplemented seventh corner point coordinate set and the eighth corner point coordinate set are for the same second vehicle, that is, the target vehicle; specifically, the intersection and union ratio correlation degree can be used to determine whether the two coordinate sets are for the same second vehicle. The specific judgment method can refer to the method for the fisheye model mentioned above, and the embodiment of the present invention is not limited to this.
[0147] After obtaining the supplemented seventh corner point coordinate set and the eighth corner point coordinate set for the target vehicle, the width ratio of the supplemented seventh corner point coordinate set and the eighth corner point coordinate set can be calculated. Specifically, the supplemented seventh corner point coordinate set is (x i|ef ,y i|ef ,u i|ef , v i|ef ), the coordinate set of the eighth corner point is (x i|ipm ,y i|ipm ,u i|ipm , v i|ipm ). The width ratio e can be calculated by the following formula:
[0148]
[0149] If the width ratio e is greater than the first preset value, the eighth corner point coordinate set can be directly used as the third corner point coordinate set; if the width ratio e is not greater than the first preset value, it means that the coordinate order output by the IPM model may need to be adjusted. At this time, the direction of the vehicle head represented by the eighth corner point coordinate set can be adjusted based on the direction of the vehicle head represented by the supplemented seventh corner point coordinate set to obtain the third corner point coordinate set. The first preset value can be set according to actual conditions, for example, the first preset value can be set to 0, which is not limited in the embodiment of the present invention.
[0150] Specifically, the coordinate values of the corner points in the eighth corner point coordinate set obtained based on the IPM model may be exchanged to obtain the third corner point coordinate set.
[0151] In an embodiment of the present invention, the above method may further include the following steps:
[0152] When the distance between the target vehicle and the vehicle is less than a preset distance threshold, and / or the target vehicle output by the inverse perspective mapping model is lost, the supplemented seventh corner point coordinate set is used as the third corner point coordinate set.
[0153] In some feasible embodiments, when the distance between the target vehicle and the vehicle is less than a preset distance threshold, and / or the target vehicle output by the inverse perspective mapping model is lost, the supplemented seventh corner point coordinate set can be directly used as the third corner point coordinate set.
[0154] Step 206: Determine the target heading angle of the target vehicle according to the third corner point coordinate set.
[0155] In some feasible embodiments, after the third corner point coordinate set is obtained, the target heading angle of the target vehicle may be calculated according to the third corner point coordinate set.
[0156] For example, Figure 3As shown, after obtaining the first image data, on the one hand, the first image data can be input into the fisheye model, and IOU (Intersection over Union) association can be performed for the target vehicle; specifically, the first IOU correlation between each fifth corner point coordinate set can be calculated, and then based on the first IOU correlation and multiple fifth corner point coordinate sets, the seventh corner point coordinate set of the target vehicle can be determined.
[0157] On the other hand, the first image data may be spliced to obtain the second image data of the top view; then the second image data may be input into the IPM model, and the eighth corner point coordinate set of the target vehicle may be obtained based on the output of the model.
[0158] Then, after supplementing the seventh corner point coordinate set, IOU association is performed on the target vehicle; specifically, the intersection-and-union ratio correlation degree of the supplemented seventh corner point coordinate set and the eighth corner point coordinate set can be calculated. If both represent the target vehicle, the eighth corner point coordinate set can be first used as the vehicle target corner point.
[0159] Next, it can be determined whether the width ratio e is greater than 0. If it is greater than 0, the vehicle target corner point is directly output; if it is not greater than 0, the coordinate values of the vehicle target corner point are exchanged based on the directions of the front and rear of the vehicle in the supplemented seventh corner point coordinate set, and the vehicle target corner point after the exchanged coordinate values is output.
[0160] Then, the heading angle of the target vehicle can be calculated based on the output vehicle target corner point. In addition, if the distance between the target vehicle and the vehicle is less than a preset distance threshold, and / or the target vehicle output by the inverse perspective mapping model is lost, the supplemented seventh corner point coordinate set is maintained as the vehicle target corner point to calculate the heading angle of the target vehicle.
[0161] In an embodiment of the present invention, first image data collected by multiple fisheye cameras are obtained, and the first image data is input into a preset fisheye model to obtain a first corner point coordinate set for a second vehicle; the first image data collected by each fisheye camera is spliced into second image data, and the second image data is input into a preset inverse perspective mapping model to obtain a second corner point coordinate set for the second vehicle; the first corner point coordinate set is converted into a fourth corner point coordinate set in a camera coordinate system; the fourth corner point coordinate set is converted into a fifth corner point coordinate set in a world coordinate system, and the second corner point coordinate set is converted into a sixth corner point coordinate set in a world coordinate system; the third corner point coordinate set of the target vehicle is determined according to the fifth corner point coordinate set and the sixth corner point coordinate set; the target heading angle of the target vehicle is determined according to the third corner point coordinate set. Through the embodiment of the present invention, the heading angle can be output by combining the accurate judgment of the front parking space of the fisheye model and the accurate detection of the coordinates by the perspective mapping model, thereby improving the accuracy of heading angle detection.
[0162] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0163] Reference Figure 4 , shows a schematic structural diagram of a heading angle detection device according to an embodiment of the present invention, which is applied to a first vehicle, and the first vehicle is provided with a plurality of fisheye cameras; Figure 4 As shown, the device may include the following modules:
[0164] A first analysis module 401 is used to obtain first image data collected by multiple fisheye cameras, and input the first image data into a preset fisheye model to obtain a first corner point coordinate set for the second vehicle;
[0165] A second analysis module 402 is used to stitch the first image data collected by each fisheye camera into second image data, and input the second image data into a preset inverse perspective mapping model to obtain a second corner point coordinate set for the second vehicle;
[0166] A corner point coordinate determination module 403, configured to determine a third corner point coordinate set of the target vehicle based on the first corner point coordinate set and the second corner point coordinate set;
[0167] The heading angle determination module 404 is used to determine the target heading angle of the target vehicle according to the third corner point coordinate set.
[0168] In one embodiment of the present invention, the corner point coordinate determination module 403 is used to convert the first corner point coordinate set into a fourth corner point coordinate set in the camera coordinate system; convert the fourth corner point coordinate set into a fifth corner point coordinate set in the world coordinate system, and convert the second corner point coordinate set into a sixth corner point coordinate set in the world coordinate system; determine the third corner point coordinate set of the target vehicle based on the fifth corner point coordinate set and the sixth corner point coordinate set.
[0169] In an embodiment of the present invention, the corner point coordinate determination module 403 is used to project the first corner point coordinate set to a top-down perspective to obtain a fourth corner point coordinate set.
[0170] In one embodiment of the present invention, the fifth corner point coordinate set includes multiple, the sixth corner point coordinate set includes multiple, and the corner point coordinate determination module 403 is used to determine the seventh corner point coordinate set for the target vehicle based on multiple fifth corner point coordinate sets; and, determine the eighth corner point coordinate set for the target vehicle from multiple sixth corner point coordinate sets; determine the third corner point coordinate set based on the seventh corner point coordinate set and the eighth corner point coordinate set.
[0171] In one embodiment of the present invention, the corner point coordinate determination module 403 is used to calculate the first intersection-and-union correlation between any two fifth corner point coordinate sets; and determine the seventh corner point coordinate set based on the first intersection-and-union correlation and multiple fifth corner point coordinate sets.
[0172] In one embodiment of the present invention, the corner point coordinate determination module 403 is used to determine a target sixth corner point coordinate set representing the same vehicle as the seventh corner point coordinate set from multiple sixth corner point coordinate sets; based on Kalman wave filtering, determine the eighth corner point coordinate set according to the target sixth corner point coordinate set.
[0173] In one embodiment of the present invention, the corner point coordinate determination module 403 is used to supplement each sixth corner point coordinate set based on a preset rectangular geometric constraint; and determine a target sixth corner point coordinate set representing the same vehicle as the seventh corner point coordinate set from the supplemented sixth corner point coordinate set.
[0174] In one embodiment of the present invention, the corner point coordinate determination module 403 is used to supplement the seventh corner point coordinate set based on a preset rectangular geometric constraint; and determine the third corner point coordinate set according to the supplemented seventh corner point coordinate set and the eighth corner point coordinate set.
[0175] In one embodiment of the present invention, the corner point coordinate determination module 403 is used to calibrate the eighth corner point coordinate set according to the supplemented seventh corner point coordinate set to obtain the third corner point coordinate set.
[0176] In one embodiment of the present invention, the corner point coordinate determination module 403 is used to determine the width ratio based on the supplemented seventh corner point coordinate set and the eighth corner point coordinate set; when the width ratio is greater than the first preset value, the eighth corner point coordinate set is used as the third corner point coordinate set; when the width ratio is not greater than the first preset value, the direction of the front of the vehicle represented by the eighth corner point coordinate set is adjusted to obtain the third corner point coordinate set.
[0177] In one embodiment of the present invention, the corner point coordinate determination module 403 is also used to use the supplemented seventh corner point coordinate set as the third corner point coordinate set when the distance between the target vehicle and the vehicle is less than a preset distance threshold, and / or the target vehicle output by the inverse perspective mapping model is lost.
[0178] In an embodiment of the present invention, the first image data collected by multiple fisheye cameras can be obtained first, and the first image data can be input into a preset fisheye model to obtain a first corner point coordinate set for the second vehicle; the first image data collected by each fisheye camera can be spliced into second image data, and the second image data can be input into a preset inverse perspective mapping model to obtain a second corner point coordinate set for the second vehicle; then, according to the first corner point coordinate set and the second corner point coordinate set, the third corner point coordinate set of the target vehicle can be determined; thereafter, according to the third corner point coordinate set, the target heading angle of the target vehicle can be determined. Through the embodiment of the present invention, the accurate judgment of the front parking space of the fisheye model and the accurate detection of the coordinates by the perspective mapping model can be combined to output the heading angle, thereby improving the accuracy of heading angle detection.
[0179] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the control method for a hybrid vehicle as described above is implemented.
[0180] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the control method for a hybrid vehicle as described above is implemented.
[0181] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0182] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0183] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0184] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.
[0185] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0186] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0187] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0188] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0189] The above provides a detailed introduction to a heading angle detection method, a heading angle detection device, an electronic device and a computer-readable storage medium. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a general technician in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A method for detecting a heading angle, characterized in that: Applied to a first vehicle, the first vehicle is provided with a plurality of fisheye cameras, the method comprising: Acquire first image data collected by the plurality of fisheye cameras, and input the first image data into a preset fisheye model to obtain a first corner point coordinate set for the second vehicle; splicing the first image data collected by each fisheye camera into second image data, and inputting the second image data into a preset inverse perspective mapping model to obtain a second corner point coordinate set for the second vehicle; Determine a third corner point coordinate set of the target vehicle according to the first corner point coordinate set and the second corner point coordinate set; A target heading angle of the target vehicle is determined according to the third corner point coordinate set.
2. The method according to claim 1, characterized in that Determining a third corner point coordinate set of the target vehicle according to the first corner point coordinate set and the second corner point coordinate set includes: Converting the first corner point coordinate set into a fourth corner point coordinate set in a camera coordinate system; Converting the fourth corner point coordinate set into a fifth corner point coordinate set in a world coordinate system, and converting the second corner point coordinate set into a sixth corner point coordinate set in a world coordinate system; A third corner point coordinate set of the target vehicle is determined based on the fifth corner point coordinate set and the sixth corner point coordinate set.
3. The method according to claim 2, characterized in that The converting the first corner point coordinate set into a fourth corner point coordinate set in a camera coordinate system comprises: The first corner point coordinate set is projected to a top-down perspective to obtain the fourth corner point coordinate set.
4. The method according to claim 2, characterized in that: The fifth corner point coordinate set includes a plurality of sets, the sixth corner point coordinate set includes a plurality of sets, and determining the third corner point coordinate set of the target vehicle according to the fifth corner point coordinate set and the sixth corner point coordinate set includes: Determining a seventh corner point coordinate set for the target vehicle according to the plurality of fifth corner point coordinate sets; and, determining an eighth corner point coordinate set for the target vehicle from the plurality of sixth corner point coordinate sets; The third corner point coordinate set is determined according to the seventh corner point coordinate set and the eighth corner point coordinate set.
5. The method according to claim 4, characterized in that The step of determining a seventh corner point coordinate set for the target vehicle according to the plurality of fifth corner point coordinate sets comprises: Calculate the first intersection-and-union correlation between the coordinate sets of the fifth corner points; The seventh corner point coordinate set is determined according to the first intersection-over-union ratio association degree and a plurality of fifth corner point coordinate sets.
6. The method according to claim 4, characterized in that The step of determining an eighth corner point coordinate set for the target vehicle from the plurality of sixth corner point coordinate sets comprises: Determining, from the plurality of sixth corner point coordinate sets, a target sixth corner point coordinate set representing the same vehicle as the seventh corner point coordinate set; Based on Kalman wave filtering, the eighth corner point coordinate set is determined according to the sixth corner point coordinate set of the target.
7. The method according to claim 6, characterized in that The step of determining, from a plurality of sixth corner point coordinate sets, a target sixth corner point coordinate set representing the same vehicle as the seventh corner point coordinate set comprises: Based on the preset rectangular geometric constraints, each sixth corner point coordinate set is supplemented; From the supplemented sixth corner point coordinate set, a target sixth corner point coordinate set representing the same vehicle as the seventh corner point coordinate set is determined.
8. The method according to claim 4, characterized in that The step of determining the third corner point coordinate set according to the seventh corner point coordinate set and the eighth corner point coordinate set comprises: Based on a preset rectangular geometric constraint, supplementing the seventh corner point coordinate set; The third corner point coordinate set is determined according to the supplemented seventh corner point coordinate set and the eighth corner point coordinate set.
9. The method according to claim 8, characterized in that The step of determining the third corner point coordinate set according to the supplemented seventh corner point coordinate set and the eighth corner point coordinate set comprises: The eighth corner point coordinate set is calibrated according to the supplemented seventh corner point coordinate set to obtain the third corner point coordinate set.
10. The method according to claim 9, characterized in that The step of calibrating the eighth corner point coordinate set according to the supplemented seventh corner point coordinate set to obtain the third corner point coordinate set includes: Determining a width ratio according to the supplemented seventh corner point coordinate set and the eighth corner point coordinate set; When the width ratio is greater than a first preset value, using the eighth corner point coordinate set as the third corner point coordinate set; When the width ratio is not greater than the first preset value, the direction of the vehicle head represented by the eighth corner point coordinate set is adjusted to obtain the third corner point coordinate set.
11. The method according to claim 9, characterized in that The method further comprises: When the distance between the target vehicle and the vehicle is less than a preset distance threshold, and / or the target vehicle output by the inverse perspective mapping model is lost, the supplemented seventh corner point coordinate set is used as the third corner point coordinate set.
12. A device for detecting a heading angle, characterized in that: Applied to a first vehicle, the first vehicle is provided with a plurality of fisheye cameras, and the device comprises: A first analysis module, configured to obtain first image data collected by the plurality of fisheye cameras, and input the first image data into a preset fisheye model to obtain a first corner point coordinate set for a second vehicle; A second analysis module, configured to stitch the first image data collected by each fisheye camera into second image data, and input the second image data into a preset inverse perspective mapping model to obtain a second corner point coordinate set for the second vehicle; A corner point coordinate determination module, used for determining a third corner point coordinate set of the target vehicle according to the first corner point coordinate set and the second corner point coordinate set; The heading angle determination module is used to determine the target heading angle of the target vehicle according to the third corner point coordinate set.
13. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for detecting the heading angle according to any one of claims 1 to 11 is implemented.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for detecting a heading angle according to any one of claims 1 to 11 is implemented.