Bird-eye view image splicing method for cross-camera non-overlapping road scene

By constructing the coordinate system and camera imaging model of traffic scenes, and using lane lines to generate cross-scene BEV images, the problem of camera images in different directions of the same monitoring rod cannot be spliced, and high-precision and smooth road panoramic image stitching is achieved, suitable for high-speed Highway monitoring and other fields.

CN118941446APending Publication Date: 2024-11-12CHANGAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411138224.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art cannot stitch between camera images taken in different directions on the same monitoring lever, resulting in the inability to acquire panoramic images.

Method used

By constructing the coordinate system and camera imaging model of the traffic scene, using the lane lines of the expressway to construct a unified world coordinate system, select a rectangular traffic scene area and generate cross-scene BEV images to achieve stitching of images in different directions.

Benefits of technology

Overcome the limitations of conventional image stitching, improve the accuracy and smoothness of image stitching, provide clear and accurate road information, broaden the application scope of image stitching, and is especially suitable for highway monitoring and other fields.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941446A_ABST
    Figure CN118941446A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of image splicing, and discloses a bird's-eye view image splicing method for a cross-camera non-overlapping road scene, and the method achieves the splicing of two images shot by a camera without an overlapping region through the lane line of an expressway. Firstly, images of different scenes are selected, and a coordinate system and a camera imaging model are constructed; secondly, constructing a unified world coordinate system; the method comprises the following steps: indirectly constructing four pairs of feature points by using lane lines on an expressway, respectively obtaining representations of four three-dimensional points of another scene in a scene coordinate system and a reference coordinate system by taking a world coordinate system corresponding to one image as a reference, and unifying the world coordinate system of the scene to the reference coordinate system by adopting the two representations; and finally, through scene area selection and coordinate system unification, a cross-scene BEV image is generated, which can visually reflect the full view and the surrounding environment of the road, and is helpful for a user to better understand and analyze the road condition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of image stitching and relates to a method for stitching bird's-eye view images of non-overlapping road scenes across cameras. Background Art

[0002] On the same single-sided highway, the pictures taken by different surveillance cameras along the way are divided into different scenes. Conventional image stitching stitches the overlapping images from different times, different perspectives, and different sensors into a seamless panorama or high-resolution image. Image matching in image stitching is to find the corresponding position of the template or feature points in the stitched image in the reference image, and then determine the transformation relationship between the images. Image fusion is to fuse the overlapping areas of the stitched images based on image matching to obtain a smooth and seamless panoramic image reconstructed by stitching.

[0003] If two cameras are on the same monitoring pole but in opposite directions, the two pictures taken will not overlap and a panoramic image cannot be obtained. Summary of the invention

[0004] The purpose of the present invention is to solve the technical problem that the existing cameras take pictures from different directions of the same monitoring pole without overlapping parts and cannot obtain panoramic images, and to provide a bird's-eye view image stitching method for non-overlapping road scenes across cameras.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] The present invention provides a method for stitching bird's-eye view images of non-overlapping road scenes across cameras, comprising step 1, inputting a first image of a traffic scene 1 and a second image of a traffic scene 2; step 2, constructing a coordinate system and a camera imaging model of the traffic scene 1, constructing a coordinate system and a camera imaging model of the traffic scene 2, and obtaining a camera 1 calibration matrix corresponding to the traffic scene 1 and a camera 2 calibration matrix corresponding to the traffic scene 2; step 3, determining image areas and corresponding points of the first image and the second image according to the camera 1 calibration matrix and the camera 2 calibration matrix, and judging the image areas and corresponding points in the two images: if both the first image and the second image include image areas and corresponding points, a cross-scene BEV image is generated; otherwise, returning to step 1; and step 4, interactively verifying the accuracy of any point in the area range of the first image and the second image with the corresponding point of the cross-scene BEV image.

[0007] Furthermore, the cross-scene BEV image includes a bird's-eye view of traffic scene 1, a blind spot, and a bird's-eye view of traffic scene 2; the height h of the bird's-eye view of traffic scene 1 is 150 pixels; the width w of the bird's-eye view of traffic scene 1 is Among them, alp is the physical length of the scene specified by the user, and awp is the physical width of the scene specified by the user; the calculation of the size of the bird's-eye view of the traffic scene 2 is the same as the calculation of the size of the bird's-eye view of the traffic scene 1.

[0008] Furthermore, the coordinate system includes a camera coordinate system, a world coordinate system, an image coordinate system and a pixel coordinate system; the coordinates of any point in the world coordinate system are L', L'=(x', y', z', 1); the coordinates in the pixel coordinate system are p'; ​​the projection equations of the world coordinate system and the pixel coordinate system are p'=M'L'=K'R'TL'; wherein M' is a camera calibration matrix, K' is an intrinsic parameter matrix, R' is a rotation matrix, T is a translation matrix, and M' includes a camera 1 calibration matrix and a camera 2 calibration matrix.

[0009] Furthermore, the R'=R x R z

[0010]

[0011]

[0012] Among them, R x is the rotation matrix around the x-axis in the coordinate system transformation, R z It is the rotation matrix around the z-axis in the coordinate system transformation.

[0013] Furthermore, the image area of ​​the first image and the second image includes selecting a pixel point in the first image and the second image as a starting point, calculating the three-dimensional point corresponding to the pixel point according to the calibration matrix of camera 1 and the calibration matrix of camera 2, and obtaining the other three three-dimensional endpoints of the rectangular area from the three-dimensional point according to alp and awp to form a three-dimensional area range, and mapping it to the two-dimensional image.

[0014] Furthermore, the judgment of the image areas and corresponding points in the two images includes a unified coordinate system, selecting a corresponding plurality of pixel points in the first image and the second image according to the lane lines on the road, obtaining four sets of three-dimensional coordinates using the calibration matrix of camera 1 and the calibration matrix of camera 2, taking the world coordinate system of traffic scene 1 as the reference coordinate system, constructing four sets of three-dimensional corresponding points using the three-dimensional lane lines, and converting the world coordinate system of traffic scene 2 into the reference coordinate system.

[0015] Furthermore, the multiple pixel points obtain different groups of pixel points according to different lane lines.

[0016] Furthermore, the image width of the blind area is

[0017] The ratio is the ratio of the width of the 3D blind area to the length of the left 3D scene area, w img1 is the width of the bird’s-eye view of traffic scene 1.

[0018] Furthermore, the interactive verification of the accuracy of any point in the first image and the second image area corresponding to the cross-scene BEV image includes:

[0019] Select a 2D point p(u,v) on a certain image, calculate the 3D point P(x,y,z) based on the corresponding camera calibration result, and then express it in a unified coordinate system to obtain P'=(x',y',z'). The corresponding point of P' in the cross-scene BEV image is (u ans ,v ans ):

[0020]

[0021] Among them, w BEV is the width of the cross-scene BEV image, ratio y is the ratio of y' to the cross-scene 3D range of y represented by the cross-scene BEV image, h BEV is the height of the cross-scene BEV image, ratio x is the ratio of x' to the cross-scene 3D range of x represented by the cross-scene BEV image.

[0022] Further, the method of obtaining the three-dimensional point corresponding to the pixel point according to the calibration matrix of camera 1 and the calibration matrix of camera 2 includes: using the calibration matrix of camera 1 and the calibration matrix of camera 2 to obtain the corresponding three-dimensional coordinates i∈[0,1,2,3], forming four sets of three-dimensional corresponding point pairs i∈[0,1,2,3], according to

[0023]

[0024] Calculate the distance temp-Δy along the y-axis between corresponding points when the coordinate system is not unified, according to Calculate Δy;

[0025] Wherein, temp-Δy is the distance along the y-axis between the corresponding points when the coordinate system is not unified, and Δy is the actual distance along the y-axis between the corresponding points after the coordinate system is unified;

[0026] Compared with the prior art, the present invention has the following beneficial technical effects:

[0027] The present invention discloses a method for stitching bird's-eye view images of non-overlapping road scenes across cameras, which stitches images taken by cameras in different directions and without overlap through lane lines of expressways, overcomes the limitations of conventional image stitching, broadens the application scope of image stitching, and has high practical value, especially in fields requiring panoramic view such as expressway monitoring.

[0028] The present invention discloses a method for stitching images of non-overlapping road scenes from a bird's-eye view across cameras. By selecting a rectangular traffic scene and constructing a unified world coordinate system, the road images in different scenes can be accurately unified into the same coordinate system, thereby achieving effective acquisition of full-scene information of the road. The precision of image stitching is effectively improved, and the smoothness and seamlessness of the stitched image are also guaranteed, providing users with clearer and more accurate road information.

[0029] The present invention discloses a method for stitching bird's-eye view images of non-overlapping road scenes across cameras, which uses cross-scene BE bird's-eye view technology to display road images in different scenes from a bird's-eye view perspective. This display method can intuitively reflect the overall view of the road and the surrounding environment, which helps users better understand and analyze road conditions. At the same time, the cross-scene BEV image also has high flexibility and scalability, and the scene area range and coordinate system settings can be adjusted as needed to meet the needs of different application scenarios.

[0030] The present invention provides a bird's-eye view image stitching method for non-overlapping road scenes across cameras, which has the advantages of high precision, high efficiency, high flexibility, etc., can provide comprehensive road information support for fields such as highway monitoring, and has important practical significance and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flowchart of a method for stitching bird's-eye view images of non-overlapping road scenes across cameras according to the present invention;

[0032] Figure 2 Schematic diagram of the coordinate system of the camera in the embodiment of the present invention; wherein a is a side view of the world coordinate system of the camera, and b is a top view of the camera coordinate system of the camera;

[0033] Figure 3 Schematic diagram of the calibration conditions in the coordinate system in the embodiment of the present invention; wherein a is a schematic diagram of the calibration conditions in the world coordinate system, and b is a schematic diagram of the calibration conditions in the image coordinate system;

[0034] Figure 4 : is an example diagram of cross-camera scene area range selection used in an embodiment of the present invention; wherein a is a first image of traffic scene 1, and b is a second image of traffic scene 2;

[0035] Figure 5 for Figure 4 Schematic diagram of Figure 4 a Schematic diagram of the first image of traffic scene 1, b Figure 4 b Schematic diagram of the second image of traffic scene 2;

[0036] Figure 6 1 is an example diagram of corresponding point selection of the unified coordinate system in an embodiment of the present invention; the left side is an example diagram of the selection of the first image of traffic scene 1, and the right side is an example diagram of the selection of the second image of traffic scene 2;

[0037] Figure 7 A bird's-eye view generated for traffic scene 1 in an embodiment of the present invention;

[0038] Figure 8 A bird's-eye view generated for traffic scene 2 in an embodiment of the present invention;

[0039] Fig. 9 This is an example diagram of generating a cross-scenario BEV result diagram in an embodiment of the present invention;

[0040] Fig.10 A schematic diagram of generating a cross-scenario BEV result graph in an embodiment of the present invention;

[0041] Fig.11 An example diagram of interactive verification in an embodiment of the present invention;

[0042] Fig.12 A schematic diagram of interactive verification in an embodiment of the present invention;

[0043] Fig.13 It is an interactive verification diagram of any point in the area of ​​the first image and the second image and the cross-scene BEV image in an embodiment of the present invention;

[0044] Fig.14 It is a specific flow chart in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0046] The present invention provides a method for stitching bird's-eye view images of non-overlapping road scenes across cameras. A scheme is proposed for stitching images without overlapping parts. First, a rectangular scene area range is selected for two images of different scenes of the same road, such as a length of 100 meters and a width of 15 meters. Secondly, a unified world coordinate system is constructed. Using the prior knowledge of the highway, such as the construction standard of the highway, the length of the lane line on the road, the length between the line segments, and the width of a single lane, four pairs of feature points are indirectly constructed, that is, the world coordinate system corresponding to one of the images is used as a reference, and the four three-dimensional points of the other scene are respectively obtained in the scene coordinate system and the reference coordinate system. The two representations are used to unify the world coordinate system corresponding to the other image to the reference coordinate system. Through the scene area selection and the unification of the coordinate system, a rectangular cross-scene BEV bird's-eye view containing two selected scene ranges and the blind spot between them can be generated, so as to realize the bird's-eye view image stitching of non-overlapping road scenes across cameras.

[0047] like Figure 1 As shown, step 1, input the first image of traffic scene 1 and the second image of traffic scene 2; step 2, construct the coordinate system and camera imaging model of traffic scene 1, construct the coordinate system and camera imaging model of traffic scene 2, and obtain the camera 1 calibration matrix corresponding to traffic scene 1 and the camera 2 calibration matrix corresponding to traffic scene 2; step 3, determine the image area and corresponding points of the first image and the second image according to the camera 1 calibration matrix and the camera 2 calibration matrix, and judge the image area and corresponding points in the two images; if both the first image and the second image include image areas and corresponding points, a cross-scene BEV image is generated; otherwise, return to step 1; step 4, interactively verify the accuracy of any point in the area range of the first image and the second image with the corresponding point of the cross-scene BEV image.

[0048] like Fig.14 As shown, in step 1, a first image pic1 of traffic scene 1 and a second image pic2 of traffic scene 2 are input, and the first image pic1 and the second image pic2 are images that do not contain vehicle targets.

[0049] The first image pic1: is taken by using camera 1 to photograph traffic scene 1, and the taken picture is the first image pic1. The second image pic2: is taken by using camera 2 to photograph traffic scene 2, and the taken picture is the second image pic2. It should be noted that camera 1 and camera 2 are on the same monitoring pole, but one photographs the traffic scene on the left and the other photographs the traffic scene on the right, and there is no overlapping part in the pictures taken of the two traffic scenes.

[0050] Step 2, construct the coordinate system and camera imaging model, specifically: construct the coordinate system and camera imaging model of traffic scene 1, construct the coordinate system and camera imaging model of traffic scene 2, and then calibrate camera 1 of traffic scene 1 and camera 2 of traffic scene 2 to obtain the calibration matrix M1' of camera 1 and the calibration matrix M2' of camera 2. The camera calibration method refers to the paper "ATaxonomy and Analysis of Camera Calibration Methods for Traffic Monitoring Applications".

[0051] The coordinate system includes the camera coordinate system OX C Y C Z C , world coordinate system O-XYZ, image coordinate system OX P Y P And the pixel coordinate system O-UV. Figure 2 As shown in a, the camera coordinate system takes the camera principal point as the origin, X C Axis and Y C The axes are parallel to the length and width of the camera, respectively, X C The Y axis points to the right of the camera. C The axis points downward from the camera. According to the right-hand coordinate system, the Z axis is vertically upward, where the camera's tilt angle Z C The origin of the world coordinate system is the point where the center of the camera coordinate system is projected onto the ground. The X axis is parallel to the X C Axis, the Y axis is perpendicular to the X axis and parallel to the Z axis C The projection direction of the axis on the ground, according to the right-hand coordinate system, the Z axis is vertically upward. Among them, the camera swing angle θ is the angle between the Y axis and the road traffic flow, such as Figure 2 b. The image coordinate system takes the center of the image as its origin, and X P The axis is horizontal to the right, Y P The pixel coordinate system takes the center of the image as its origin, with pixels as its unit, the U axis pointing horizontally to the right, and the V axis pointing horizontally downward.

[0052] Construct the mapping relationship between the world coordinate system containing unknown camera parameters and the image coordinate system. The coordinates of the world coordinate system are L = (x, y, z, 1) T , the corresponding coordinates in the pixel coordinate system are p = (αu, αv, α) T ,α≠0; where x, y, and z are the values ​​of the corresponding coordinate axes in the world coordinate system, u and v are the values ​​of the corresponding coordinate axes in the image coordinate system, and α is a component of the homogeneous coordinate in the pixel coordinate system.

[0053] Then the projection equation from world coordinates to pixel coordinates is:

[0054] p=ML=KRTL

[0055] Among them, K is the intrinsic parameter matrix, R is the rotation matrix, and T is the translation matrix.

[0056]

[0057]

[0058]

[0059] Substituting any point (x, y, 0) on the road surface into p = ML = KRTL, we can get the expression for V. The parameter to be solved is f. θ and h.

[0060]

[0061] Camera 1 of traffic scene 1 and camera 2 of traffic scene 2 use the VLH method to calculate the calibration results and optimize the calibration results. The coordinates of the infinite point in the road extension direction in the world coordinate system are x0 = (-tanθ, 1, 0, 0) T , the vertical coordinates of the infinite point are x1=(1,tanθ,0,0) T , according to p = ML = KRTL, we can calculate the vanishing points (u0, v0) and (u1, v1), then v0 and v1 are

[0062]

[0063] Let v0 equal v1, then

[0064] Further,

[0065] like Figure 3 As shown in Fig. 3a and 3b, if a vanishing point V, a length L of a section running in the traffic flow, and a camera height h of at least one redundant sign on the road surface are known, the camera calibration result can be calculated. The ordinates of the world coordinates of the two end points of the length L and the v coordinates of their image coordinates are: y b and f 、v b and v f ,according to Available

[0066] Since y and u are independent, we can get y based on the length L. b =y f +lcosθ.

[0067]

[0068]

[0069] τ=(v f -v0)(v b -v0) / (v f -v b )

[0070] Will Square both sides and substitute We can get the fourth-order equation about f:

[0071]

[0072] in, For each group f, according to Solve φ and θ, use the redundant identification line segments in the image coordinate system to calculate the corresponding spatial distance in the world coordinate system, and select the best fitting solution to eliminate ambiguity; then combined with the known camera height h, the camera calibration work is completed.

[0073] The camera 1 calibration of traffic scene 1 and the camera 2 calibration of traffic scene 2 of the present invention use the [f, φ, θ, h] result of the general camera calibration. In order to simplify the subsequent steps of unifying the coordinate system, enhance the understanding of the continuity of the traffic flow direction of the two scenes and maintain the consistency of the three-dimensional coordinates of the vehicle across the scenes, some changes are made to the world coordinate system on this basis: the origin and Z axis of the world coordinate system remain unchanged, and the X axis is perpendicular to the road traffic flow direction and points to the inside of the road. Figure 4 As shown in Figures 4a and 4b, the Y axes of the left and right scenes are parallel and point to the direction of traffic flow, that is, the Y axis is determined by the left-handed coordinate system. Then the transformation relationship between any point L'=(x',y',z',1) in the world coordinate system and its corresponding point p'=(u',v') on the image is p'=M'L'=K'R'TL', the translation matrix T does not change, the intrinsic parameter matrix K' and the rotation matrix R' both change, and the final calibration matrix M' is obtained.

[0074]

[0075] Among them, pic u and pic v is the center point of image pic.

[0076] R'=R x R z

[0077]

[0078]

[0079] Since the shooting direction of the camera is opposite to the direction of the traffic flow, specifically the angle between them is close to 180°, then θ=θ+π, so that the angle between the two directions is a relatively small acute angle.

[0080] Determining the image areas of the first image pic1 and the second image pic2 includes selecting a starting point, a length and a width of the scene area, four three-dimensional endpoints of the scene area, and saving the area points.

[0081] Select the starting point: Taking the first image pic1 of traffic scene 1 as an example, select a 3D point with relatively small x and y values ​​according to the world coordinate system corresponding to traffic scene 1. The corresponding 2D image point (u,v) is used as the starting point According to the camera 1 calibration matrix M1', we can get

[0082] Length and width of the scene area: Enter the set length alp and width awp of the three-dimensional road area in the identification physical size input window, in meters. Usually, the effective range of road camera monitoring is 100 meters to 150 meters, so the recommended value of the area length is 100 meters under the premise of a straight road. For the camera on one side of the road, it is recommended to only pay attention to the traffic information on the current side, not the traffic information on the opposite side, so the area width is recommended to be set to the width of the lane on the current side.

[0083] The four 3D endpoints of the scene area: As a starting point, the other three 3D endpoints are

[0084]

[0085] According to p=ML=KRTL, Generates a region mask based on the endpoints of the image's range.

[0086] Preservation of regional points: Sort the range endpoints of the two-dimensional image and the three-dimensional range endpoints so that in the three-dimensional range points, Indicates the upper left corner of the region. Indicates the upper right corner of the area. Indicates the lower right corner of the area. Indicates the lower left corner of the region, and the range points of the two-dimensional image are saved in the same order as the three-dimensional range points. 2D _area and P 3D _area, select scene areas for the two scenes respectively. The actual application diagram of traffic scene 1 is as follows Figure 4 As shown in a, the actual application diagram of traffic scene 2 is as follows Figure 4 b. The schematic diagram of traffic scene 1 is shown in Figure 5As shown in a, the schematic diagram of traffic scene 2 is as follows Figure 5 As shown in b.

[0087] Since the world coordinate systems of traffic scene 1 and traffic scene 2 do not overlap, while the world coordinate system in the bird's-eye view result is unified, the two world coordinate systems need to be unified. In actual situations, the two road images are taken by two road cameras located on the same monitoring pole and in opposite directions. There is no overlap between the images, and conventional image stitching cannot be performed based on feature point matching of the overlapping parts. However, there are lane lines on the highway, which are 6 meters long and the distance between two adjacent lane lines is 9 meters, that is, the distance from a point on a line segment to the point at the same position on the next line segment is 15 meters.

[0088] Select four pixels in the first image pic1 and the second image pic2 according to the lane line i∈[0,1,2,3], use the camera 1 calibration matrix M1' and the camera 2 calibration matrix M2' to obtain the corresponding three-dimensional coordinates i∈[0,1,2,3], forming four sets of three-dimensional corresponding point pairs i∈[0,1,2,3]. Each set of pixel pairs selects the same position of the lane line on the same lane line in the road, such as the lower left corner and the lower right corner, so that the three-dimensional distance between the corresponding points is a multiple of 15 meters. Different sets of pixel pairs can select different lane lines, such as Figure 6 shown.

[0089] Taking the world coordinate system of traffic scene 2 as the reference, the 3D corresponding points in traffic scene 2 are used to obtain the representation of the 3D points in scene 1 in world coordinate system 2. Specifically, due to i∈[0,1,2,3] is based on different world coordinate systems with very close origins, according to

[0090]

[0091] Calculate the distance temp-Δy along the y-axis between corresponding points when the coordinate system is not unified, according to Calculate Δy;

[0092] Among them, temp-Δy is the distance along the y axis when the coordinate system is not unified between the corresponding points, and Δy is the actual distance along the y axis after the coordinate system is unified between the corresponding points. Since the two cameras are on the same monitoring pole, the distance between the optical centers of the two cameras along the traffic flow direction (the y axis of the world coordinate system) is actually very small, so the y values ​​of the world coordinate system corresponding to the two cameras can be subtracted to get a rough estimate of temp-Δy, and then the precise Δy can be determined by combining the prior knowledge that the distance between a set of line segments on the highway is 15 meters.

[0093] according to

[0094]

[0095] get Since each set of pixel pairs is selected along the same lane line on the road plane, The x and z values ​​are same.

[0096]

[0097]

[0098] Get the complete representation of the 3D points in traffic scene 1 in world coordinate system 2

[0099] The representation of the points of traffic scene 1 in world coordinate system 1 is mapped to the representation in world coordinate system 2 through a rigid body transformation matrix to be obtained. Since the transformation from world coordinate system 1 to world coordinate system 2 only involves translation T and rotation R, the coordinate system can be unified by using the transformation matrix M, which is:

[0100]

[0101] Since both R and T have three degrees of freedom, three sets of corresponding points can be used to solve R and T, and one set of corresponding points can be used to optimize the result.

[0102] The cross-scenario BEV image includes a bird's-eye view of traffic scene 1, a blind spot, and a bird's-eye view of traffic scene 2. The bird's-eye view of traffic scene 1 is as follows: Figure 7 As shown, the bird's-eye view of traffic scene 2 is as follows Figure 8 As shown, the height of the cross-scene BEV image is 150 pixels.

[0103] Generation of a bird's-eye view of traffic scene 1. Using perspective transformation, the 2D quadrilateral area of ​​the left scene is transformed The new rectangular image img1 is mapped, and the height h of img1 is 150 pixels, and the width w is determined by the physical length alp of the user-specified scene, the physical width awp of the user-specified scene, and the height h.

[0104] Generation of the bird's-eye view of traffic scene 2. Using perspective transformation, the 2D quadrilateral area of ​​the right scene is transformed The new rectangular image img2 is mapped, and the height h of img2 is 150 pixels, and the width w is determined by the physical length alp of the user-specified scene, the physical width awp of the user-specified scene, and the height h.

[0105]

[0106] After the width and height of img1 and img2 are determined, the selected range boundary points of the left scene are matched one by one with the endpoints of img1, and img1 is generated using perspective transformation. The selected range boundary points of the right scene are matched one by one with the endpoints of img2, and img2 is generated using perspective transformation, generating a bird's-eye view of the two scenes.

[0107] Blind area, the height of the blind area image is h = 150 pixels, and the width of the blind area image is w img1 Determined by the width of image img1, The distance between the right border of the range selected for the left 3D scene and the left border of the right 3D scene, specifically the distance between the y values ​​of the 2 points in the left scene range and the y values ​​of the 3 points in the right scene range after the unified coordinate system, such as Fig.10 shown.

[0108]

[0109] alp1 is the length of the left 3D scene area. After obtaining the width and height of the blind area image, fill it with black, such as Fig. 9 shown.

[0110] The image img1, the blind area, and the image img1 are concatenated column by column to obtain the cross-scene BEV image.

[0111] Interactive verification of the accuracy of any point in the area of ​​the first image pic1 and the second image pic2 with the corresponding point in the cross-scene BEV image. Interactive verification is to click a point in the selected range of a scene image, and a corresponding point appears in the cross-scene BEV image, such as Fig.11 shown.

[0112] like Fig.12 As shown, a 2D point p(u,v) is selected, and a 3D point P(x,y,z) is calculated according to the corresponding camera calibration result, and then P'=(x',y',z') is obtained by expressing it in a unified coordinate system.

[0113] Find the ratio of P' in the entire 3D range across the scene x 、ratio y .x min ,x max : They are represented by the x-values ​​of the points on the lower and upper border lines of the entire range, such as the x-values ​​of the 2 border points of traffic scene 1 and the x-values ​​of the 1 border point of traffic scene 2, that is, y min ,y max: It is represented by the y value of the points on the left and right border lines of the entire range, such as the y value of the 0 border point of traffic scene 1 and the y value of the 1 border point of traffic scene 2, that is,

[0114] high

[0115] width

[0116]

[0117] In the cross-scene 3D range of the cross-scene BEV image representation, x min is the minimum value of x in the range, x range is the length of the range x, y min is the minimum y value of the range, y range is the y length of the range.

[0118] Find the point where P' is displayed on the BEV result graph (u ans ,v ans ),

[0119]

[0120] Among them, w BEV is the width of the cross-scene BEV image, ratio y is the ratio of y' to the cross-scene 3D range of y represented by the cross-scene BEV image, h BEV is the height of the cross-scene BEV image, ratio x is the ratio of x' to the cross-scene 3D range of x represented by the cross-scene BEV image.

[0121] like Fig.13 As shown, the accuracy of the cross-scene BEV image is verified through interactive verification. Six groups of corresponding points in the left and right scenes are selected, and the calculated distance between two points is compared with the true distance, and the accuracy reaches 99%.

[0122] Table 1

[0123] Point selection Calculated distance (mm) True distance(mm) Accuracy (%) 1 point left and 1' point right 59,932 60.000 99.887 2 points left and 2' points right 89,612 90.000 99.569 Left 3' and Right 3' 119.300 120.000 99.417 Left 4 o'clock and right 4' o'clock 149,039 150.000 99.359 Left 5 o'clock and right 5' o'clock 118,955 120.000 99.129 Left 6 o'clock and right 6' o'clock 208,668 210.000 99.367

[0124] The present invention discloses a method for stitching images of a cross-camera non-overlapping road scene from a bird's-eye view. The lane lines of the highway are used to stitch images taken by two cameras with non-overlapping areas. First, images of different scenes are selected to construct a coordinate system and a camera imaging model. Secondly, a unified world coordinate system is constructed. The lane lines on the highway are used to indirectly construct four pairs of feature points. The world coordinate system corresponding to one of the images is used as a reference to obtain the representations of the four three-dimensional points of another scene in the scene coordinate system and the reference coordinate system. The world coordinate system of the scene is unified to the reference coordinate system using these two representations. Finally, after the scene area selection and the unification of the coordinate system, a cross-scene BEV image is generated, which can intuitively reflect the overall picture of the road and the surrounding environment, and help users better understand and analyze road conditions.

[0125] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

Claims

1. A method for stitching bird's-eye view images of non-overlapping road scenes across cameras, characterized by: Step 1, inputting a first image of traffic scene 1 and a second image of traffic scene 2; Step 2, construct the coordinate system and camera imaging model of traffic scene 1, construct the coordinate system and camera imaging model of traffic scene 2, and obtain the camera 1 calibration matrix corresponding to traffic scene 1 and the camera 2 calibration matrix corresponding to traffic scene 2; Step 3, according to the calibration matrix of camera 1 and the calibration matrix of camera 2, determine the image area and corresponding points of the first image and the second image, and judge the image area and corresponding points in the two images: if both the first image and the second image include the image area and corresponding points, generate a cross-scene BEV image; otherwise, return to step 1; Step 4: interactively verify the accuracy of any point in the first image and the second image area with the corresponding point in the cross-scene BEV image.

2. The method for stitching bird's-eye view images of non-overlapping road scenes across cameras according to claim 1, characterized in that: The cross-scenario BEV image includes a traffic scene 1 bird's-eye view, a blind spot, and a traffic scene 2 bird's-eye view; The height h of the bird's-eye view of the traffic scene 1 is 150 pixels; The width w of the bird's-eye view of the traffic scene 1 img1 for, Among them, alp is the physical length of the scene specified by the user, and awp is the physical width of the scene specified by the user; The calculation of the size of the bird's-eye view of the traffic scene 2 is the same as the calculation of the size of the bird's-eye view of the traffic scene 1.

3. The method for stitching bird's-eye view images of non-overlapping road scenes across cameras according to claim 2, characterized in that: The coordinate system includes a camera coordinate system, a world coordinate system, an image coordinate system and a pixel coordinate system; The coordinates of any point in the world coordinate system are L', L'=(x', y', z', 1); the coordinates in the pixel coordinate system are p'; The projection equation of the world coordinate system and the pixel coordinate system is p'=M'L'=K'R'TL'; Among them, M' is the camera calibration matrix, K' is the intrinsic parameter matrix, R' is the rotation matrix, and T is the translation matrix; M' includes the camera 1 calibration matrix and the camera 2 calibration matrix.

4. The method for stitching bird's-eye view images of non-overlapping road scenes across cameras according to claim 3, characterized in that: R'=R x R z Among them, R x is the rotation matrix around the x-axis in the coordinate system transformation, R z It is the rotation matrix around the z-axis in the coordinate system transformation.

5. The method for stitching bird's-eye view images of non-overlapping road scenes across cameras according to claim 4, characterized in that: The image area of ​​the first image and the second image includes selecting a pixel point in the first image and the second image as the starting point, calculating the three-dimensional point corresponding to the pixel point according to the calibration matrix of camera 1 and the calibration matrix of camera 2, and obtaining the other three three-dimensional endpoints of the rectangular area from the three-dimensional point according to alp and awp to form a three-dimensional area range, and mapping it to the two-dimensional image.

6. The method for stitching bird's-eye view images of non-overlapping road scenes across cameras according to claim 1, characterized in that: Before judging the image areas and corresponding points in the two images, a unified coordinate system is included. According to the lane lines on the road, a corresponding plurality of pixel points are selected in the first image and the second image, and four sets of three-dimensional coordinates are obtained using the calibration matrix of camera 1 and the calibration matrix of camera 2. The world coordinate system of traffic scene 1 is used as the reference coordinate system, and four sets of three-dimensional corresponding points are constructed using the three-dimensional lane lines, and the world coordinate system of traffic scene 2 is converted to the reference coordinate system.

7. The method for stitching bird's-eye view images of non-overlapping road scenes across cameras according to claim 6, characterized in that: The multiple pixel points obtain different groups of pixel points according to different lane lines.

8. The method for stitching bird's-eye view images of non-overlapping road scenes across cameras according to claim 7, characterized in that: The image width of the blind area is The ratio is the ratio of the width of the 3D blind area to the length of the left 3D scene area, w img1 is the width of the bird’s-eye view of traffic scene 1.

9. The method for stitching bird's-eye view images of non-overlapping road scenes across cameras according to claim 8, characterized in that: The interactive verification of the accuracy of any point in the first image and the second image area corresponding to the cross-scene BEV image includes: Select a 2D point p(u,v) on a certain image, calculate the 3D point P(x,y,z) according to the corresponding camera calibration result, and then express it in a unified coordinate system to obtain P'=(x',y',z'). The corresponding point of P' in the cross-scene BEV image is (u ans ,v ans ): Among them, w BEV is the width of the cross-scene BEV image, ratio y is the ratio of y' to the cross-scene 3D range of y represented by the cross-scene BEV image, h BEV is the height of the cross-scene BEV image, ratio x is the ratio of x' to the cross-scene 3D range of x represented by the cross-scene BEV image.

10. The method for stitching bird's-eye view images of non-overlapping road scenes across cameras according to claim 5, characterized in that: The method of obtaining the three-dimensional point corresponding to the pixel point according to the calibration matrix of camera 1 and the calibration matrix of camera 2 includes: Use the calibration matrix of camera 1 and camera 2 to obtain the corresponding 3D coordinates Construct four sets of three-dimensional corresponding point pairs according to Calculate the distance temp_Δy along the y-axis between corresponding points when the coordinate system is not unified, according to Calculate Δy; Wherein, temp_Δy is the distance along the y-axis between the corresponding points when the coordinate system is not unified, and Δy is the actual distance along the y-axis between the corresponding points after the coordinate system is unified.