[0034] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described The embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.
[0035] Participate figure 1 As shown, the vehicle-mounted 360-degree panoramic stitching method of the present invention includes the following steps:
[0036] S1. Fit the fisheye camera model with the hyperboloid model, and calculate the fisheye camera model parameters;
[0037] S2, determine the mapping relationship between the picture coordinate system and the camera coordinate system;
[0038] S3. Determine the homography matrix from the camera coordinate system to the ground coordinate system;
[0039] S4. Generate a mapping lookup table from the picture coordinate system to the ground coordinate system;
[0040] S5. Combine ground pictures acquired by the camera according to the mapping lookup table to obtain a final bird's-eye view.
[0041] Combine figure 1 , image 3 As shown, the specific steps in the present invention are:
[0042] S1. Fit the fisheye camera model with the hyperboloid model, and calculate the fisheye camera model parameters.
[0043] Participate figure 2 As shown, in this embodiment, a hyperboloid model symmetric about the Z axis is used to fit the lens system of the fisheye camera. figure 2 The middle arc represents the cross-sectional view of one branch of the hyperboloid, and the fitting of the hyperboloid adopts Taylor expansion, expressed as:
[0044] z = f ( ρ ) = a 0 + a 1 ρ + a 2 ρ 2 + a 3 ρ 3 + · · · ,
[0045] When the hyperboloid is symmetrical u and u are the points (u, v) in the picture coordinate system, so the fisheye camera model parameters are the coefficients of the Taylor expansion. This part belongs to the description of the camera characteristics, so it can be considered that the Taylor coefficients are a representation of the internal parameters of the camera. The internal parameters can be obtained by calibrating the camera.
[0046] S2, determine the mapping relationship between the picture coordinate system and the camera coordinate system
[0047] according to figure 2 Can get: the vector in the camera coordinate system With vector Proportional relationship, while The end point of the vector is on the hyperboloid, so the vector With vector Satisfy:
[0048] OA → - u v f ( ρ ) - δ OB → - δ X C Y C Z C And δ<1,
[0049] According to the above corresponding relationship, the mapping relationship between the picture coordinate system and the camera coordinate system can be established.
[0050] S3. Determine the homography matrix from the camera coordinate system to the ground coordinate system.
[0051] Determine the homography matrix from the camera coordinate system to the ground coordinate system (world coordinate system) on the basis of step S1. First, it is necessary to predefine the display range of each field of view image, and specify the predetermined starting point in the upper right corner of each camera field of view Position, front, back, left and right cameras are respectively marked as Front_SP, Rear_SP, Left_SP, Right_SP;
[0052] Then determine the corresponding position points of the four starting points in the final bird's-eye view. The position points and position points are counted with the side length of the checkerboard as a unit, and the four starting points can be respectively in the corresponding field of view;
[0053] Then, in different fields of view (here, front, back, left, and right), place the checkerboard with Front_SP, Rear_SP, Left_SP, and Right_SP as the upper right vertices. Finally, through step S1, place the ground coordinate system on the checkerboard The point coordinates correspond to the coordinates of the corresponding field of view, and the transformed homography matrix is calculated.
[0054] S4. Generate a mapping lookup table from the picture coordinate system to the ground coordinate system.
[0055] Combining steps S1, S2, and S3 to establish the correspondence between the points in the picture coordinate system and the points on the bird's-eye view under the ground coordinate system, and map the four scenes with Front_SP, Rear_SP, Left_SP, and Right_SP as the starting point in the upper right corner to the bird's-eye view On the map, fine-tune the positions of the four vertices, and the four field of view pictures are spliced onto the bird’s-eye view through mapping;
[0056] Then, according to the pre-determined field of view, the display range is delineated on the final bird's-eye view, a mask template is made, and the coordinate position of the mask boundary in the image coordinate system is inversely calculated to obtain the final four field of view pictures to the final bird's-eye view. Mapping relations.
[0057] S5. Combine ground pictures acquired by the camera according to the mapping lookup table to obtain a final bird's-eye view.
[0058] In a preferred embodiment of the present invention, the solution for determining the field of view of the final bird's-eye view is specifically as follows:
[0059] 1. Determine the world coordinate system: X axis is parallel to the vehicle width direction, Y axis is parallel to the vehicle length direction, Z axis is parallel to the vehicle height direction, the minimum unit is the side length of the checkerboard, and the origin position is fixed at the upper right corner of the vehicle body , In the X-axis direction away from the car body 10-20 checkerboard side length, Y-axis direction away from the car body 10-20 checkerboard side length.
[0060] 2. Determine the upper right starting points of the four fields of view Front_SP, Rear_SP, Left_SP, Right_SP: these four starting points are also the first inner grid point from the upper right of the checkerboard under the four fields of view, and adjust the four fields under the four fields of view. A starting point, so that the distance between the four points Front_SP, Rear_SP, Left_SP, and Right_SP in the world coordinate system can be expressed as a whole point by the side length of the checkerboard. This is to improve the accuracy of splicing. The scale here is to use the checkerboard. The side length is one unit.
[0061] 3. Layout of checkerboards: in four fields of view, the four checkerboards are placed as shown Figure 4.
[0062] 4. In a similar way to calibrating a fisheye camera, the mapping relationship between the four field of view checkerboard points and the plane perpendicular to the optical axis in the camera coordinate system is obtained, and the pictures in the four fields of view are mapped to the bird's-eye view according to the mapping relationship , Fine-tune the four starting points to get a spliced bird's-eye view.
[0063] 5. According to the requirements of the field of view, the area size is delineated on the bird's-eye view and a mask template is generated, and the mask boundary is calculated inversely to determine the final bird's-eye view boundary.
[0064] The present invention is an algorithm designed for vehicle-mounted surround view splicing, which has real-time and robustness, Figure 5 Shown are the indoor splicing test results using the vehicle-mounted 360-degree panoramic splicing method of the present invention, which has small errors and high accuracy. Compared with the existing technology, it has the following advantages:
[0065] 1. Avoid the use of feature extraction schemes and avoid uncertainty. The present invention replaces corner point matching, splicing line optimization and other matching methods, uses position mapping and coordinate transformation to establish the set point of the world coordinate system and the original image coordinate system The mapping relationship of the next point is finally accurately matched on the bird's-eye view. For position mapping, as long as the camera's internal and external parameters are fixed, it will not be affected by the environment, so the algorithm has higher reliability;
[0066] 2. Compared with the general scheme, there are two steps of correction and mapping. According to the relationship between the ground and the camera coordinate system in the world coordinate system, this algorithm puts the bird’s-eye view in the camera coordinate system perpendicular to the optical axis to establish a mapping with the original image coordinate system. Complete the mapping in one step, which can reduce the interference caused by the truncation and correction errors of floating-point bits during calculation, and further improve the accuracy of splicing;
[0067] 3. This solution uses Taylor's fitting symmetrical fisheye camera lens, which is versatile and does not require different models for different cameras. Experiments have proved that this fitting is good for cameras with wide-angle symmetrical lenses below 180 degrees;
[0068] 4. In order to enhance the real-time performance, the algorithm uses a look-up table method to generate the mapping relationship, and the operation only needs to look up the table, which speeds up the algorithm and has better portability for embedded systems.
[0069] It can be seen from the above embodiments that the vehicle-mounted 360-degree panoramic stitching method of the present invention aims at the complexity of the images in the four fields of view in the vehicle environment, and proposes to use the mapping relationship to establish a mapping lookup table from the original field of view to the bird’s-eye view. Matching, this method only depends on the coordinate relationship, and the coordinates are relatively fixed when the camera is fixed, so compared to matching, this method is more reliable; in addition, it directly maps the original image to the camera coordinate system, omitting the camera coordinates The mapping from the system to the world coordinate system reduces errors and has higher accuracy.
[0070] For those skilled in the art, it is obvious that the present invention is not limited to the details of the foregoing exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description, and therefore it is intended to fall within the claims. All changes within the meaning and scope of equivalent elements of are included in the present invention. Any reference signs in the claims should not be regarded as limiting the claims involved.
[0071] In addition, it should be understood that although this specification is described in accordance with the embodiments, not every embodiment only includes an independent technical solution. This narration in the specification is only for clarity, and those skilled in the art should consider the specification as a whole The technical solutions in the embodiments can also be appropriately combined to form other implementations that can be understood by those skilled in the art.
