Vehicle-mounted 360-degree panorama mosaic method

A picture and coordinate system technology, applied in image data processing, instrumentation, calculation, etc., can solve the problems of poor robustness, noise sensitivity, inability to achieve real-time and accuracy, and achieve high accuracy and reduce errors.

Inactive Publication Date: 2013-06-05
KUNSHAN BRANCH INST OF MICROELECTRONICS OF CHINESE ACADEMY OF SCI
3 Cites 37 Cited by

AI-Extracted Technical Summary

Problems solved by technology

[0003] In the prior art, the patent application with the application number 201110199571.9 mainly uses shif feature point matching and RANSAC matching of the optimal stitching line to determine the stitching result. The disadvantage of this solution is that the extraction of shif feature points is very real-time and the environment is complex It is difficult to accurately determine the feature points in the vehicle-mounted surround view system, and the intersecting areas under the other four fields of view cannot guarantee enough feature points for matching, so this solution often fails to meet the requirements of re...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Abstract

The invention discloses a vehicle-mounted 360-degree panorama mosaic method which includes utilizing a double-curve model to match a fisheye camera model, and calculating fisheye camera model parameters; confirming a mapping relation between a picture coordinate system and a camera coordinate system; confirming a homography matrix from the camera coordinate system to a ground coordinate system; generating a mapping lookup table from the picture coordinate system to the ground coordinate system; and enabling ground pictures obtained by a camera to be matched into a final aerial view according to the mapping lookup table. According to the vehicle-mounted 360-degree panorama matching method, for complexity of images of four view fields of the vehicle-mounted environment, the mapping lookup table from an original view field to the aerial view is built by utilizing the mapping relation. Compared with feature point match, the method only depends on the coordinate relation, a coordinate is relatively fixed when the camera is fixed, so compared with matching, the method is reliable. The original pictures are directly mapped to positions below the camera coordinate system, mapping from the camera coordinate system to a world coordinate system is omitted, errors are reduced, and accuracy is high.

Application Domain

Technology Topic

Feature point matchingPoint match +7

Image

  • Vehicle-mounted 360-degree panorama mosaic method
  • Vehicle-mounted 360-degree panorama mosaic method
  • Vehicle-mounted 360-degree panorama mosaic method

Examples

  • Experimental program(1)

Example Embodiment

[0034] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described The embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.
[0035] Participate figure 1 As shown, the vehicle-mounted 360-degree panoramic stitching method of the present invention includes the following steps:
[0036] S1. Fit the fisheye camera model with the hyperboloid model, and calculate the fisheye camera model parameters;
[0037] S2, determine the mapping relationship between the picture coordinate system and the camera coordinate system;
[0038] S3. Determine the homography matrix from the camera coordinate system to the ground coordinate system;
[0039] S4. Generate a mapping lookup table from the picture coordinate system to the ground coordinate system;
[0040] S5. Combine ground pictures acquired by the camera according to the mapping lookup table to obtain a final bird's-eye view.
[0041] Combine figure 1 , image 3 As shown, the specific steps in the present invention are:
[0042] S1. Fit the fisheye camera model with the hyperboloid model, and calculate the fisheye camera model parameters.
[0043] Participate figure 2 As shown, in this embodiment, a hyperboloid model symmetric about the Z axis is used to fit the lens system of the fisheye camera. figure 2 The middle arc represents the cross-sectional view of one branch of the hyperboloid, and the fitting of the hyperboloid adopts Taylor expansion, expressed as:
[0044] z = f ( ρ ) = a 0 + a 1 ρ + a 2 ρ 2 + a 3 ρ 3 + · · · ,
[0045] When the hyperboloid is symmetrical u and u are the points (u, v) in the picture coordinate system, so the fisheye camera model parameters are the coefficients of the Taylor expansion. This part belongs to the description of the camera characteristics, so it can be considered that the Taylor coefficients are a representation of the internal parameters of the camera. The internal parameters can be obtained by calibrating the camera.
[0046] S2, determine the mapping relationship between the picture coordinate system and the camera coordinate system
[0047] according to figure 2 Can get: the vector in the camera coordinate system With vector Proportional relationship, while The end point of the vector is on the hyperboloid, so the vector With vector Satisfy:
[0048] OA → - u v f ( ρ ) - δ OB → - δ X C Y C Z C And δ<1,
[0049] According to the above corresponding relationship, the mapping relationship between the picture coordinate system and the camera coordinate system can be established.
[0050] S3. Determine the homography matrix from the camera coordinate system to the ground coordinate system.
[0051] Determine the homography matrix from the camera coordinate system to the ground coordinate system (world coordinate system) on the basis of step S1. First, it is necessary to predefine the display range of each field of view image, and specify the predetermined starting point in the upper right corner of each camera field of view Position, front, back, left and right cameras are respectively marked as Front_SP, Rear_SP, Left_SP, Right_SP;
[0052] Then determine the corresponding position points of the four starting points in the final bird's-eye view. The position points and position points are counted with the side length of the checkerboard as a unit, and the four starting points can be respectively in the corresponding field of view;
[0053] Then, in different fields of view (here, front, back, left, and right), place the checkerboard with Front_SP, Rear_SP, Left_SP, and Right_SP as the upper right vertices. Finally, through step S1, place the ground coordinate system on the checkerboard The point coordinates correspond to the coordinates of the corresponding field of view, and the transformed homography matrix is ​​calculated.
[0054] S4. Generate a mapping lookup table from the picture coordinate system to the ground coordinate system.
[0055] Combining steps S1, S2, and S3 to establish the correspondence between the points in the picture coordinate system and the points on the bird's-eye view under the ground coordinate system, and map the four scenes with Front_SP, Rear_SP, Left_SP, and Right_SP as the starting point in the upper right corner to the bird's-eye view On the map, fine-tune the positions of the four vertices, and the four field of view pictures are spliced ​​onto the bird’s-eye view through mapping;
[0056] Then, according to the pre-determined field of view, the display range is delineated on the final bird's-eye view, a mask template is made, and the coordinate position of the mask boundary in the image coordinate system is inversely calculated to obtain the final four field of view pictures to the final bird's-eye view. Mapping relations.
[0057] S5. Combine ground pictures acquired by the camera according to the mapping lookup table to obtain a final bird's-eye view.
[0058] In a preferred embodiment of the present invention, the solution for determining the field of view of the final bird's-eye view is specifically as follows:
[0059] 1. Determine the world coordinate system: X axis is parallel to the vehicle width direction, Y axis is parallel to the vehicle length direction, Z axis is parallel to the vehicle height direction, the minimum unit is the side length of the checkerboard, and the origin position is fixed at the upper right corner of the vehicle body , In the X-axis direction away from the car body 10-20 checkerboard side length, Y-axis direction away from the car body 10-20 checkerboard side length.
[0060] 2. Determine the upper right starting points of the four fields of view Front_SP, Rear_SP, Left_SP, Right_SP: these four starting points are also the first inner grid point from the upper right of the checkerboard under the four fields of view, and adjust the four fields under the four fields of view. A starting point, so that the distance between the four points Front_SP, Rear_SP, Left_SP, and Right_SP in the world coordinate system can be expressed as a whole point by the side length of the checkerboard. This is to improve the accuracy of splicing. The scale here is to use the checkerboard. The side length is one unit.
[0061] 3. Layout of checkerboards: in four fields of view, the four checkerboards are placed as shown Figure 4.
[0062] 4. In a similar way to calibrating a fisheye camera, the mapping relationship between the four field of view checkerboard points and the plane perpendicular to the optical axis in the camera coordinate system is obtained, and the pictures in the four fields of view are mapped to the bird's-eye view according to the mapping relationship , Fine-tune the four starting points to get a spliced ​​bird's-eye view.
[0063] 5. According to the requirements of the field of view, the area size is delineated on the bird's-eye view and a mask template is generated, and the mask boundary is calculated inversely to determine the final bird's-eye view boundary.
[0064] The present invention is an algorithm designed for vehicle-mounted surround view splicing, which has real-time and robustness, Figure 5 Shown are the indoor splicing test results using the vehicle-mounted 360-degree panoramic splicing method of the present invention, which has small errors and high accuracy. Compared with the existing technology, it has the following advantages:
[0065] 1. Avoid the use of feature extraction schemes and avoid uncertainty. The present invention replaces corner point matching, splicing line optimization and other matching methods, uses position mapping and coordinate transformation to establish the set point of the world coordinate system and the original image coordinate system The mapping relationship of the next point is finally accurately matched on the bird's-eye view. For position mapping, as long as the camera's internal and external parameters are fixed, it will not be affected by the environment, so the algorithm has higher reliability;
[0066] 2. Compared with the general scheme, there are two steps of correction and mapping. According to the relationship between the ground and the camera coordinate system in the world coordinate system, this algorithm puts the bird’s-eye view in the camera coordinate system perpendicular to the optical axis to establish a mapping with the original image coordinate system. Complete the mapping in one step, which can reduce the interference caused by the truncation and correction errors of floating-point bits during calculation, and further improve the accuracy of splicing;
[0067] 3. This solution uses Taylor's fitting symmetrical fisheye camera lens, which is versatile and does not require different models for different cameras. Experiments have proved that this fitting is good for cameras with wide-angle symmetrical lenses below 180 degrees;
[0068] 4. In order to enhance the real-time performance, the algorithm uses a look-up table method to generate the mapping relationship, and the operation only needs to look up the table, which speeds up the algorithm and has better portability for embedded systems.
[0069] It can be seen from the above embodiments that the vehicle-mounted 360-degree panoramic stitching method of the present invention aims at the complexity of the images in the four fields of view in the vehicle environment, and proposes to use the mapping relationship to establish a mapping lookup table from the original field of view to the bird’s-eye view. Matching, this method only depends on the coordinate relationship, and the coordinates are relatively fixed when the camera is fixed, so compared to matching, this method is more reliable; in addition, it directly maps the original image to the camera coordinate system, omitting the camera coordinates The mapping from the system to the world coordinate system reduces errors and has higher accuracy.
[0070] For those skilled in the art, it is obvious that the present invention is not limited to the details of the foregoing exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description, and therefore it is intended to fall within the claims. All changes within the meaning and scope of equivalent elements of are included in the present invention. Any reference signs in the claims should not be regarded as limiting the claims involved.
[0071] In addition, it should be understood that although this specification is described in accordance with the embodiments, not every embodiment only includes an independent technical solution. This narration in the specification is only for clarity, and those skilled in the art should consider the specification as a whole The technical solutions in the embodiments can also be appropriately combined to form other implementations that can be understood by those skilled in the art.
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

no PUM

Description & Claims & Application Information

We can also present the details of the Description, Claims and Application information to help users get a comprehensive understanding of the technical details of the patent, such as background art, summary of invention, brief description of drawings, description of embodiments, and other original content. On the other hand, users can also determine the specific scope of protection of the technology through the list of claims; as well as understand the changes in the life cycle of the technology with the presentation of the patent timeline. Login to view more.
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Similar technology patents

Dual-CPU based circuit breaker intelligent controller

Owner:WUHAN NARI LIABILITY OF STATE GRID ELECTRIC POWER RES INST

Classification and recommendation of technical efficacy words

  • Improve accuracy
  • Reduce mistakes

Cassette-based dialysis medical fluid therapy systems, apparatuses and methods

InactiveUS20050209563A1Improvement for dialysisImprove accuracyMedical devicesPeritoneal dialysisAccuracy improvementDialysis
Owner:BAXTER INT INC +1

Terminal positioning track fitting method based on Bezier curve

ActiveCN102998684AReduce mistakesSatellite radio beaconingSatellite navigationFitting methods
Owner:XIAMEN YAXON NETWORKS CO LTD

Multifunctional road soil roadbed freezing and thawing circulating test device

ActiveCN101923085ASolving Elasticity ProblemsReduce mistakesRoadwaysEarth material testingTest measurementFrost
Owner:CCCC SECOND HIGHWAY CONSULTANTS CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products