Monocular image self-calibration inverse perspective mapping method and system

Abstract

The present application relates to the technical field of computer vision, and especially relates to a monocular image self-calibration inverse perspective mapping method and system; the method comprises the following steps: acquiring a monocular road image containing at least one road plane marker, and pixel coordinates and real physical dimensions of a plurality of vertexes of the road plane marker; determining a forward direction vanishing point in the road image, and constructing a basic inverse perspective transformation matrix; mapping the vertex pixel coordinates of the road plane marker to an intermediate transformation space through the basic inverse perspective transformation matrix, performing scale self-correction on the basic inverse perspective transformation matrix, and obtaining a corrected homography matrix; determining a pixel-physical length proportionality coefficient based on a ratio of pixel sizes of the road plane marker under the corrected homography matrix to real physical lengths; performing coordinate transformation on the road image by using the corrected homography matrix, and generating an aerial view with a physical scale; the present application does not need to pre-calibrate camera external parameters, and can realize a complete scale self-correction process only by relying on known sizes of road plane markers.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, intelligent transportation, and image processing technology, and particularly to a method and system for self-calibrating inverse perspective mapping of monocular images. Background Technology

[0002] Inverse Perspective Mapping (IPM) is a core technology for transforming road front view images into bird's-eye view (BEV) images, and it is widely used in scenarios such as autonomous driving assistance, traffic monitoring, and road distance measurement. Existing road distance measurement methods mainly cover the following categories.

[0003] The first type is the geometric back-projection method based on camera extrinsic parameter calibration. This method pre-measures and fixes extrinsic parameters such as the camera's installation height and pitch angle, and uses perspective geometry to map the image coordinates to the ground coordinate system, thereby achieving distance measurement. However, in actual driving, factors such as vibration, collision, or loose installation can cause camera attitude deviation, resulting in the failure of extrinsic parameters. Recalibration must be completed in a controlled environment with specialized tools, which is costly and difficult to dynamically update after deployment.

[0004] The second category is monocular depth estimation methods based on deep learning, which predict dense depth maps from a single image by training a neural network without prior manual calibration. However, existing depth estimation networks typically output relative depth rather than absolute physical distance, resulting in unstable generalization performance in scenarios outside the training set distribution. Furthermore, the high computational load limits their real-time deployment capability on embedded platforms.

[0005] The third category is 3D measurement methods based on binocular vision or LiDAR. These methods have high accuracy but require high-end hardware and have high deployment costs, making them unsuitable for low-cost deployment scenarios that use ordinary monocular cameras as sensors.

[0006] Furthermore, road diamond markings, as road surface markings with known geometric proportions, can serve as natural scale constraint benchmarks. However, existing technologies have not yet established a complete and systematic method centered on vanishing point derivation, aspect ratio self-correction, and physical scale recovery. This makes it impossible to achieve bird's-eye view generation and accurate distance measurement without relying on extrinsic parameters based solely on monocular images and the known dimensions of road markings.

[0007] Therefore, there is an urgent need for a self-calibrated inverse perspective mapping and ranging method that does not require pre-calibration of camera extrinsic parameters and only relies on the known dimensions of road surface markings to overcome the above-mentioned shortcomings. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for self-calibration of monocular images and inverse perspective mapping, so as to solve the technical problems in the prior art, such as high cost of camera extrinsic parameter calibration, difficulty in online correction after extrinsic parameter offset, and lack of technical problems in realizing a complete scale self-calibration process using road surface markings.

[0009] To achieve this objective, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for self-calibrating inverse perspective mapping of monocular images, which includes the following steps: S1. Obtain a monocular road image containing at least one road surface plane mark, and the pixel coordinates and actual physical dimensions of multiple vertices of the road surface plane mark; S2. Determine the vanishing point of the forward direction in the road image; S3. Construct a basic inverse perspective transformation matrix based on the vanishing point of the forward direction; S4. Map the vertex pixel coordinates of the road surface plane mark to the intermediate transformation space through the basic inverse perspective transformation matrix. Use the proportional relationship between the pixel geometry of the road surface plane mark after mapping and the actual physical size to perform scale self-correction on the basic inverse perspective transformation matrix to obtain the corrected homography matrix. S5. Determine the pixel-physical length ratio coefficient based on the ratio of the pixel size of the road surface plane mark to the corresponding actual physical length under the corrected homography matrix; S6. Perform coordinate transformation on the road image using the corrected homography matrix to generate a bird's-eye view with a physical scale.

[0010] In some preferred embodiments, when there are no available lane boundary lines in the road image, step S2 specifically includes: The four vertices of the road surface marking are arranged in a clockwise or counterclockwise direction and marked as the first to the fourth vertex in sequence, and the coordinates of each vertex are converted into three-dimensional homogeneous coordinate vectors; The homogeneous line vector of the first side is obtained by performing a cross product of the three-dimensional homogeneous coordinate vectors of the first vertex and the second vertex. The homogeneous line vector of the third side is obtained by performing a cross product of the three-dimensional homogeneous coordinate vectors of the third vertex and the fourth vertex. The homogeneous line vector of the first side and the third side is then cross producted again and normalized to obtain the three-dimensional homogeneous coordinate vector of the vanishing point of the first direction. The homogeneous line vector of the second side is obtained by performing a cross product of the three-dimensional homogeneous coordinate vectors of the second vertex and the third vertex. The homogeneous line vector of the fourth side is obtained by performing a cross product of the three-dimensional homogeneous coordinate vectors of the fourth vertex and the first vertex. The homogeneous line vector of the second side and the fourth side is then cross producted again and normalized to obtain the three-dimensional homogeneous coordinate vector of the vanishing point of the second direction. The homogeneous coordinate vectors of the vanishing points in the first and second directions are cross-producted to obtain the homogeneous straight line vector of the horizon.

[0011] Preferably, determining the vanishing point of the forward direction in the road image further includes: Perform a cross product of the three-dimensional homogeneous coordinate vectors of the first vertex and the third vertex to obtain the homogeneous line vector of the first diagonal. Perform a cross product of the homogeneous line vector of the first diagonal and the homogeneous line vector of the horizon and normalize it to obtain the first candidate vanishing point. Perform a cross product of the three-dimensional homogeneous coordinate vectors of the second vertex and the fourth vertex to obtain the homogeneous line vector of the second diagonal. Perform a cross product of the homogeneous line vector of the second diagonal and the homogeneous line vector of the horizon and normalize it to obtain the second candidate vanishing point. The vanishing point of the forward direction is selected from the first candidate vanishing point and the second candidate vanishing point, and the one that is located within the image coordinate range of the road image is taken as the vanishing point of the forward direction.

[0012] Specifically, when both the first candidate vanishing point and the second candidate vanishing point are located within the image coordinate range of the road image, a weighted average is performed on the first candidate vanishing point and the second candidate vanishing point, and the weighted average result is used as the vanishing point in the forward direction.

[0013] In some preferred embodiments, when lane boundary lines exist in the road image, step S2 specifically includes: Extract the pixel coordinates of multiple endpoints of the left lane boundary line and the right lane boundary line; The endpoints of abnormal line segments are removed by using a random sampling consistency algorithm on the pixel coordinates of each endpoint. The remaining endpoints of the left lane boundary line and the remaining endpoints of the right lane boundary line are respectively fitted with least squares straight lines to obtain the homogeneous straight line vectors of the left lane boundary line and the right lane boundary line. The coordinates of the vanishing point of the forward direction are obtained by performing a cross product and normalizing the homogeneous straight line vectors of the left lane boundary line and the right lane boundary line. A preset effective area constraint is applied to the vanishing point coordinates of the forward direction. When the vanishing point coordinates of the forward direction exceed the preset effective area, the vanishing point estimation is retried.

[0014] In some preferred embodiments, when both lane boundary lines and road surface markings exist simultaneously in the road image, step S2 specifically includes: The estimated vanishing point of the first forward direction is calculated based on the lane boundary line, and the estimated vanishing point of the second forward direction is calculated based on the road surface markings. The consistency between the estimated first vanishing point of the forward direction and the estimated second vanishing point of the forward direction is verified, and the weighted fusion result is used as the vanishing point of the forward direction. When the road image contains only lane boundary lines or only road surface markings, determining the vanishing point of the forward direction in the road image specifically includes: The vanishing point calculated from a single source is used as the vanishing point for the forward direction.

[0015] Preferably, step S3 specifically includes: S31. Construct a three-row, three-column transformation matrix using the pixel coordinates of the vanishing point of the forward direction as parameters. The three-row, three-column transformation matrix maps the vanishing point of the forward direction to infinity. S32. Based on the expected coverage of the bird's-eye view and the resolution of the road image, the scale parameters in the three-row, three-column transformation matrix are calculated and determined in real time during the execution of the monocular image self-calibration inverse perspective mapping method.

[0016] Preferably, step S4 specifically includes: S41. Calculate the horizontal and vertical dimensions of the road surface marking pixels in the intermediate transformation space, wherein the horizontal dimension of the pixels is the absolute value of the difference between the horizontal coordinates of the left and right vertices of the road surface marking in the intermediate transformation space, and the vertical dimension of the pixels is the absolute value of the difference between the vertical coordinates of the top and bottom vertices of the road surface marking in the intermediate transformation space. S42. Calculate the true aspect ratio of the road surface marking based on the ratio of the physical length to the physical width in the true physical dimensions; S43. Divide the actual aspect ratio by the ratio of the vertical dimension of the pixel to the horizontal dimension of the pixel to obtain the vertical scaling correction coefficient; S44. Construct a diagonal scaling matrix with the vertical scaling correction coefficient as the vertical scaling factor and the horizontal scaling factor as 1. Multiply the diagonal scaling matrix with the basic inverse perspective transformation matrix to obtain the corrected homography matrix.

[0017] Preferably, step S5 specifically includes: S51. Recalculate the pixel lateral dimension of the road surface marking under the correction homography matrix; S52. The ratio of the horizontal dimension of the pixel to the physical width in the actual physical dimension is used as the pixel-physical length ratio coefficient.

[0018] Specifically, when the road image contains multiple road surface markings, the pixel lateral dimension of each road surface marking is calculated under the correction homography matrix. The pixel-physical length ratio coefficient corresponding to each road surface marking is obtained by the ratio of the pixel lateral dimension of each road surface marking to the physical width in the corresponding real physical dimension. The arithmetic mean of all the pixel-physical length ratio coefficients is taken, and the arithmetic mean is used as the final pixel-physical length ratio coefficient.

[0019] In some preferred embodiments, the coordinate transformation of the bird's-eye view is generated by inverse mapping. For each target pixel in the bird's-eye view, the source image coordinates corresponding to the target pixel are calculated by the inverse matrix of the correction homography matrix. Then, a positive depth component constraint is applied to the source image coordinates to exclude back-side mapping areas before pixel sampling.

[0020] Specifically, applying a positive depth component constraint to the source image coordinates to exclude backface mapping regions includes: For each target pixel in the bird's-eye view, the product of the inverse of the correction homography matrix and the homogeneous coordinate vector of the target pixel is calculated to obtain the homogeneous coordinate vector of the source image position corresponding to the target pixel. When the depth component of the homogeneous coordinate vector of the source image location is greater than zero, pixel sampling is performed on the corresponding location in the road image and the sampled value is assigned to the target pixel; When the depth component of the homogeneous coordinate vector of the source image location is not greater than zero, the target pixel is marked as an invalid region.

[0021] In some preferred embodiments, the monocular image self-calibration inverse perspective mapping method further includes: In the bird's-eye view, determine the first and second points to be measured; Obtain the pixel coordinates of the first point and the second point; Calculate the pixel Euclidean distance between the first point and the second point; Divide the pixel Euclidean distance by the pixel-physical length scaling factor to obtain the actual physical distance between the first point and the second point.

[0022] In some preferred embodiments, the pixel coordinates of multiple vertices of the road surface marking are obtained in any of the following ways: The user manually marks each vertex of the road surface plane mark in the road image to obtain the pixel coordinates of each vertex; Alternatively, the target detection model can infer from the road image to automatically output the pixel coordinates of each vertex of the road surface marking.

[0023] In some preferred embodiments, the road surface plane marking is a road diamond mark, and the four vertices of the road diamond mark are arranged in a clockwise or counterclockwise direction. The actual physical size is determined according to the road standard specifications of the area where the road diamond mark is deployed and is input as a configurable parameter into the monocular image self-calibration inverse perspective mapping method.

[0024] In some preferred embodiments, the monocular image self-calibration inverse perspective mapping method continuously processes the input road image in units of image frames. If the road surface plane mark is detected when processing each frame of the road image, the recalculation of the correction homography matrix and the pixel-physical length ratio coefficient is automatically triggered.

[0025] Specifically, the vanishing point coordinates of the forward direction and the pixel-to-physical length ratio coefficient calculated in multiple consecutive road images are subjected to temporal filtering and fusion, and the vanishing point coordinates of the forward direction and the pixel-to-physical length ratio coefficient after temporal filtering and fusion are used as the calibration parameters of the current frame.

[0026] Furthermore, the continuous image stream acquired by the forward-looking monocular camera deployed on the vehicle computing platform is used as the input road image, and the monocular image self-calibration inverse perspective mapping method outputs a bird's-eye view with a physical scale in real time after processing each frame of road image.

[0027] In some preferred embodiments, the road images are derived from image sequences captured by a fixed monocular camera or from image frames from historical videos, and the monocular image self-calibration inverse perspective mapping method sequentially performs scale self-correction and bird's-eye view generation on the road images in batches.

[0028] In a second aspect, the present invention provides a monocular image self-calibration inverse perspective mapping system, comprising: The feature input module is used to receive a monocular road image containing at least one road surface plane mark, as well as the pixel coordinates and actual physical size of multiple vertices of the road surface plane mark; The vanishing point calculation module is used to determine the vanishing point of the forward direction in the road image. The vanishing point calculation module supports the independent operation of the vanishing point derivation path based on the opposite edge of the road surface marking and the vanishing point derivation path based on the lane boundary line, and performs weighted fusion when both derivation paths are available. The homography matrix construction module is used to construct a basic inverse perspective transformation matrix based on the vanishing point of the forward direction. The scale self-correction module is used to map the vertex pixel coordinates of the road surface plane mark to the intermediate transformation space through the basic inverse perspective transformation matrix, and to perform scale self-correction on the basic inverse perspective transformation matrix by using the proportional relationship between the pixel geometry of the road surface plane mark after mapping and the actual physical size to obtain the corrected homography matrix. The physical scale calibration module is used to determine the pixel-physical length ratio coefficient based on the ratio of the pixel size of the road surface plane mark to the corresponding actual physical length under the correction homography matrix; The bird's-eye view generation module is used to perform coordinate transformation on the road image using the correction homography matrix, and output a bird's-eye view with a physical scale.

[0029] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the various steps of the monocular image self-calibration inverse perspective mapping method as described above.

[0030] Compared with the prior art, the present invention has the following beneficial effects: This invention, on the one hand, uses the known physical dimensions of the road surface markings as the sole prior calibration input. Through image geometric operations, it automatically determines the vanishing point of the forward direction and constructs an inverse perspective transformation matrix, completely eliminating dependence on external parameters such as camera mounting height and pitch angle. It requires no dedicated calibration equipment or controlled calibration environment, and can automatically complete recalibration when the camera's mounting posture changes. On the other hand, after the basic inverse perspective transformation, a scale self-correction step based on the aspect ratio of the road surface markings is introduced. A longitudinal scaling correction coefficient α compensates for the basic transformation matrix, effectively eliminating the longitudinal scale error remaining after perspective elimination, resulting in a bird's-eye view with an accurate physical scale. Furthermore, based on the ratio of the marker pixel size to the corresponding real physical length under the corrected homography matrix, a pixel-physical length scaling coefficient is established, giving the bird's-eye view a quantifiable physical scale. The entire process relies solely on a standard monocular camera and the known dimensions of the road surface markings, resulting in low deployment costs and wide applicability.

[0031] The present invention has other features and advantages, which will be apparent from or will be set forth in detail in the accompanying drawings and the following detailed description, which together serve to explain the particular principles of the invention. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating the monocular image self-calibration inverse perspective mapping method described in Embodiment 1 of the present invention.

[0034] Figure 2 This is a schematic diagram of the original monocular road image collected in Embodiment 1 of the present invention.

[0035] Figure 3 This is a close-up schematic diagram of marking the coordinates of the vertex of the road surface plane mark on a road image in Embodiment 1 of the present invention.

[0036] Figure 4 This is a distant view of the coordinates of the vertex of the road surface plane mark on the road image in Embodiment 1 of the present invention.

[0037] Figure 5 This corresponds to Embodiment 1 of the present invention. Figure 4 A schematic diagram of the generated bird's-eye view.

[0038] Figure 6 This is a schematic diagram of a road image used for physical scale calibration in Embodiment 1 of the present invention.

[0039] Figure 7 This corresponds to Embodiment 1 of the present invention. Figure 6 The generated bird's-eye view.

[0040] Figure 8 This corresponds to Embodiment 1 of the present invention. Figure 6 A schematic diagram of the generated distance measurement results.

[0041] Figure 9 This is a schematic diagram showing the positional relationship between the road surface plane markings and the lane boundary lines on a road image in Embodiment 1 of the present invention.

[0042] Figure 10 This is a structural block diagram of the monocular image self-calibration inverse perspective mapping system described in Embodiment 2 of the present invention. Detailed Implementation

[0043] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0044] Example 1 Please see Figures 1 to 9 The monocular image self-calibration inverse perspective mapping method of this embodiment is described using a road diamond mark as the road surface plane mark 100. The following will fully demonstrate the entire process of the method of this embodiment from image acquisition to bird's-eye view generation and distance measurement, and give the mathematical formulas and typical numerical results of each step.

[0045] S1. Obtain a monocular road image containing at least one road surface marker, as well as the pixel coordinates and actual physical dimensions of multiple vertices of the road surface marker 100.

[0046] like Figure 2 As shown, a front view image of the road is captured by a monocular camera. The road surface marking 100 is a diamond-shaped road marking, with its four vertices arranged in a clockwise or counterclockwise direction, and labeled as vertex 1, vertex 2, vertex 3, and vertex 4 respectively. Figure 3 and Figure 4 As shown.

[0047] To facilitate the division of left and right vertices and top and bottom vertices, this embodiment stipulates that all vertices are arranged in a clockwise direction. Specifically, the first vertex 1 is located above the rhombus, the second vertex 2 is located to the right, the third vertex 3 is located below, and the fourth vertex 4 is located to the left.

[0048] In this embodiment, the actual physical size of the road surface diamond-shaped markings is set as: physical length. =3 meters, physical width =1.5 meters. In reality, different countries and regions can use the actual physical dimensions as a configurable parameter input according to local road standards and specifications. Furthermore, vertex pixel coordinates can be manually obtained by the user from the road image, or automatically output by the object detection model through inference on the road image; both methods are applicable. In scenarios where lane boundary lines 200 exist in the road image, multiple endpoint pixel coordinates of the left and right lane boundary lines can also be extracted simultaneously for use in step S2.

[0049] S2. Determine the vanishing point of the forward direction in the road image.

[0050] This step uses a homogeneous coordinate geometry method to solve for the vanishing point. The basic calculation rules are as follows: Image point representation: pixels in an image The three-dimensional homogeneous coordinate vector is represented as: .

[0051] Cross product for a line: Two o'clock and The homogeneous representation of line l is: .

[0052] Find the intersection point of two lines: The intersection point v of two homogeneous lines l1 and l2 is obtained by normalizing the cross product: , After normalization, the vanishing point coordinates are obtained.

[0053] Considering that actual roads may have combinations of lane boundary lines and road surface markings, this embodiment will discuss three cases based on these combinations.

[0054] Scenario 1: When lane boundary lines exist in the road image but there are no available road surface markings.

[0055] Pixel coordinates of multiple endpoints of the left lane boundary line 200 and the right lane boundary line 200 are extracted. Random Sample Consensus (RANSAC) algorithm is applied to these endpoints to remove those belonging to abnormal line segments. Then, least-squares line fitting is performed on the remaining endpoints of the left and right lane boundary lines to obtain the homogeneous line vector of the left boundary line. Homogeneous straight line vectors of the right boundary line .

[0056] Find the two boundary lines: , .

[0057] Find the vanishing point of the forward direction: .

[0058] right After normalization, we get: .

[0059] Furthermore, a preset effective region constraint is applied to the obtained VP, and estimation is retried when its coordinates exceed the preset effective region.

[0060] The above scenario 1 applies to situations where identifiable lane boundary lines exist in the road image. When no usable lane boundary lines exist in the road image, this embodiment changes to relying solely on the road surface plane marker 100 to derive the vanishing point, i.e., proceeding to scenario 2.

[0061] Scenario 2: When there are no available lane boundary lines in the road image (i.e., relying only on road surface markings).

[0062] The four vertices arranged clockwise in this embodiment For example, among which, For the first vertex 1 (above), For the second vertex 2 (to the right), For the third vertex 3 (below), For the fourth vertex 4 (on the left), solve for the vanishing point of the forward direction using the following steps. : 1. Find the vanishing point in the first direction from the first pair of opposite edges: First, perform a cross product of the three-dimensional homogeneous coordinate vectors of the first vertex 1 and the second vertex 2 to obtain the homogeneous line vector of the first side. Taking the cross product of the three-dimensional homogeneous coordinate vectors of the third vertex 3 and the fourth vertex 4, we obtain the homogeneous line vector of the third side: Next, perform a cross product on the homogeneous line vectors of the first and third sides. Based on the intersection point rule for two lines described in step S2, normalize the cross product result to obtain the vanishing point of the first direction. Coordinates: .

[0063] 2. Find the vanishing point in the second direction from the opposite sides of the second group: Similarly, in the same way, the cross product of the three-dimensional homogeneous coordinate vectors of the second vertex 2 and the third vertex 3 is performed to obtain the homogeneous line vector of the second side. Taking the cross product of the three-dimensional homogeneous coordinate vectors of the fourth vertex 4 and the first vertex 1, we obtain the homogeneous line vector of the fourth side: Then, take the cross product of the homogeneous line vectors of the second and fourth sides and normalize them to obtain the vanishing point of the second direction. Coordinates: 。

[0064] 3. Determine the horizon : , The horizon represents the set of all vanishing points on the ground and is a key geometric constraint in projective geometry.

[0065] 4. Determine the vanishing point of the forward direction using the intersection of the rhombus's diagonal and the horizon. In this embodiment, the diagonal of the rhombus's major axis is used. Intersection with the horizon As the vanishing point of the first forward direction, the vanishing point of the first forward direction After normalization, the first candidate vanishing point is obtained. .

[0066] Similarly, with the diagonal of the minor axis of the rhombus Intersection with the horizon As the vanishing point of the second forward direction, the vanishing point of the second forward direction After normalization, the second candidate vanishing point is obtained. .

[0067] 5. Take , One of the points located within the image coordinate range of the road image is used as the vanishing point of the forward direction. ;like and If all are within the image coordinate range, then for and A weighted average is used to improve the stability of the vanishing point estimate: , in, and All are weights. and Equal rights can be obtained.

[0068] The scenarios described in Case 1 and Case 2 cover situations where a single feature source is available. When both lane boundary lines and road surface markings exist in the road image, both types of sources can provide vanishing point estimation. This embodiment fuses the two to further improve accuracy, i.e., Case 3.

[0069] Scenario 3: When lane boundary lines and road surface markings coexist in the road image.

[0070] Specifically, the first vanishing point estimate for the forward direction is obtained from case one, and the second vanishing point estimate is obtained from case two. The consistency of the two estimates is verified, and the weighted fusion result is used as the final vanishing point for the forward direction. When the road image contains only lane boundary lines or only road surface markings, the vanishing point calculation result from a single source is used as the vanishing point for the forward direction, achieving complementary redundancy between the two feature sources, such as... Figure 9 As shown.

[0071] S3. Construct the basic inverse perspective transformation matrix based on the vanishing point of the forward direction. .

[0072] The vanishing point of the forward direction determined in step S2 Using the core parameters, construct a 3x3 basic inverse perspective transformation matrix. : , in, The coordinates are the vanishing point coordinates, and k is the scale parameter, which is calculated and determined in real time during the method execution based on the expected coverage of the bird's-eye view and the resolution of the road image. Vanishing point The homogeneous depth component after transformation is equal to zero, thus mapping the vanishing point to infinity, making the road's direction of travel appear as a parallel direction in the intermediate BEV space.

[0073] S4. Map the vertex pixel coordinates of the road surface markings to the intermediate transformation space through the basic inverse perspective transformation matrix. Use the proportional relationship between the pixel geometry of the road surface markings after mapping and the actual physical size to perform scale self-correction on the basic inverse perspective transformation matrix to obtain the corrected homography matrix.

[0074] Step S4 specifically includes the following steps: S41. Transform the four vertices of the rhombus using the basic inverse perspective transformation matrix. Mapped to the intermediate BEV space and normalized, the following calculations are performed: Pixel horizontal size : ; in, This represents the absolute value of the difference between the x-coordinates of the left and right vertices.

[0075] Pixel vertical size : ; in, This represents the absolute value of the difference between the ordinates of the upper and lower vertices.

[0076] S42, Calculate the truth Real ratio example Due to the aforementioned setting of physical length =3 meters, physical width =1.5 meters, so Calculate the pixel aspect ratio in the intermediate BEV space. .

[0077] S43. Calculate the longitudinal scaling correction factor. : in true proportion Divide by pixel aspect ratio ,get .

[0078] Understandable, Only the perspective convergence direction is eliminated; compression or stretching errors may remain in the longitudinal direction. The error is quantified and used for compensation in subsequent step S44.

[0079] S44. Construct the vertical scaling matrix S, and combine the vertical scaling matrix S with the basic inverse perspective transformation matrix. Multiplying them yields the corrected homography matrix. : , .

[0080] After this step, the road surface marking 100 is corrected in the homography matrix. The aspect ratio of the mapped pixels is consistent with the actual physical aspect ratio, thus eliminating the vertical scaling error.

[0081] As an experimental verification, this embodiment uses the actual dimensions of a rhombus (3 meters long and 1.5 meters wide) as input, and calculates the corrected homography matrix in real time. The values ​​are as follows: .

[0082] S5. Determine the pixel-to-physical-length ratio coefficient based on the ratio of the pixel size of the road surface marking to the corresponding actual physical length under the corrected homography matrix.

[0083] Step S5 specifically includes the following steps: S51, In correcting the homography matrix The pixel lateral dimensions of the road surface markings are recalculated to obtain dx', which gives the corrected pixel lateral dimensions. .

[0084] S52. Calculate the pixel / meter ratio (ppm) using the following formula (unit: pixels / meter): .

[0085] As an experimental verification, the ppm obtained in this embodiment is 40 (pixels / meter), meaning that every 40 pixels in the bird's-eye view corresponds to 1 meter in the real world, which is consistent with the above-mentioned correction homography matrix. Together, these constitute the calibration output of this method, and the numerical results are shown at the end of step S4. Figure 6 As shown in the figure, the road surface marker 100 used for physical scale calibration and the measurement range of its pixel lateral dimensions are illustrated. When multiple road surface markers exist in the road image, the arithmetic mean of multiple ppm values ​​is taken as the final pixel-to-physical length scaling factor.

[0086] S6. Perform coordinate transformation on the road image using the corrected homography matrix to generate a bird's-eye view with a physical scale.

[0087] Specifically, this embodiment uses an inverse mapping method: For each target pixel in the BEV image Calculate the coordinates of the corresponding source image: , Add homogeneous depth sign consistency constraints only when homogeneous depth components At that time, for the source image coordinates Sample the data and assign it to the target pixel; otherwise, mark the target pixel as an invalid region and do not sample it, thus avoiding backface mapping. Figure 5 As shown Figure 4 The corresponding bird's-eye view generation result, such as Figure 7 As shown Figure 6 The corresponding bird's-eye view generated result.

[0088] Determine the first point to be measured (coordinates) in the bird's-eye view. ) and the second point (coordinates) ),like Figure 8 As shown, the actual physical distance is: (Unit: meters).

[0089] Example 2 Please see Figure 10 The monocular image self-calibration inverse perspective mapping system of this embodiment implements all the steps of the method described in Embodiment 1. The system includes: The feature input module 10 is used to receive a monocular road image containing at least one road surface plane mark 100, as well as the pixel coordinates and actual physical dimensions of multiple vertices of the road surface plane mark 100, and to transmit the above data to the following module.

[0090] The vanishing point calculation module 20, connected to the feature input module 10, is used to determine the vanishing point of the forward direction in the road image. It supports the independent operation of the vanishing point derivation path based on the opposite side of the road surface mark 100 (using the cross product of the opposite side to calculate the vanishing points of the two directions, and then determining the forward direction by the horizon and the diagonal) and the vanishing point derivation path based on the lane boundary line 200 (such as least squares fitting after RANSAC removal). When both derivation paths are available, they are weighted and fused to achieve complementary redundancy of the two types of feature sources.

[0091] The homography matrix construction module 30, connected to the vanishing point calculation module 20, is used to construct the basic inverse perspective transformation matrix based on the vanishing point in the forward direction. The scale parameters are calculated and determined in real time during the method execution.

[0092] The scale self-correction module 40, connected to the homography matrix construction module 30, is used to transform the vertex pixel coordinates of the road surface plane mark 100 through the basic inverse perspective transformation matrix. Mapping to the intermediate transformation space, calculate the vertical scaling correction coefficient α based on the horizontal dimension dx and vertical dimension dy of the mapped labeled pixels, and construct the diagonal scaling matrix. After the fundamental inverse perspective transformation matrix Multiplication yields the corrected homography matrix. .

[0093] The physical scale calibration module 50, connected to the scale self-calibration module 40, is used to calibrate the homography matrix. The horizontal pixel dimension dx' of the road surface marking 100 is recalculated, in dx' / W real As a pixel-to-physical length ratio factor, ppm.

[0094] The bird's-eye view generation module 60 is connected to the physical scale calibration module 50 and is used to correct the homography matrix. Perform an inverse mapping transformation on the road image (with an additional positive depth component constraint) to output a bird's-eye view with a physical scale.

[0095] Example 3 This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the various steps of the monocular image self-calibration inverse perspective mapping method described in Embodiment 1.

[0096] Combination Figures 1-10 This invention, on the one hand, uses the known physical dimensions of the road surface markings as the sole prior calibration input. Through image geometric operations, it automatically determines the vanishing point of the forward direction and constructs an inverse perspective transformation matrix, completely eliminating dependence on external parameters such as camera mounting height and pitch angle. It requires no dedicated calibration equipment or controlled calibration environment, and can automatically complete recalibration when the camera's mounting posture changes. On the other hand, after the basic inverse perspective transformation, a scale self-correction step based on the aspect ratio of the road surface markings is introduced. A longitudinal scaling correction coefficient α compensates for the basic transformation matrix, effectively eliminating the longitudinal scale error remaining after perspective elimination, resulting in a bird's-eye view with an accurate physical scale. Furthermore, based on the ratio of the marker pixel size to the corresponding real physical length under the corrected homography matrix, a pixel-physical length scaling coefficient is established, giving the bird's-eye view a quantifiable physical scale. The entire process relies solely on a standard monocular camera and the known dimensions of the road surface markings, resulting in low deployment costs and wide applicability.

[0097] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for self-calibrating inverse perspective mapping of monocular images, characterized in that, Includes the following steps: Acquire a monocular road image containing at least one road surface plane marker, and the pixel coordinates and actual physical dimensions of multiple vertices of the road surface plane marker; Determine the vanishing point of the forward direction in the road image; Based on the vanishing point of the forward direction, construct the basic inverse perspective transformation matrix; The vertex pixel coordinates of the road surface plane mark are mapped to the intermediate transformation space through the basic inverse perspective transformation matrix. The basic inverse perspective transformation matrix is ​​then scaled using the proportional relationship between the pixel geometry of the road surface plane mark after mapping and the actual physical size to obtain the corrected homography matrix. The pixel-to-physical-length ratio coefficient is determined based on the ratio of the pixel size of the road surface plane mark to the corresponding actual physical length under the corrected homography matrix. The road image is transformed using the corrected homography matrix to generate a bird's-eye view with a physical scale.

2. The monocular image self-calibration inverse perspective mapping method as described in claim 1, characterized in that, When there are no available lane boundary lines in the road image, determining the vanishing point of the forward direction in the road image specifically includes: The four vertices of the road surface marking are arranged in a clockwise or counterclockwise direction and marked as the first to the fourth vertex in sequence, and the coordinates of each vertex are converted into three-dimensional homogeneous coordinate vectors; The homogeneous line vector of the first side is obtained by performing a cross product of the three-dimensional homogeneous coordinate vectors of the first vertex and the second vertex. The homogeneous line vector of the third side is obtained by performing a cross product of the three-dimensional homogeneous coordinate vectors of the third vertex and the fourth vertex. The homogeneous line vector of the first side and the third side is then cross producted again and normalized to obtain the three-dimensional homogeneous coordinate vector of the vanishing point of the first direction. The homogeneous line vector of the second side is obtained by performing a cross product of the three-dimensional homogeneous coordinate vectors of the second vertex and the third vertex. The homogeneous line vector of the fourth side is obtained by performing a cross product of the three-dimensional homogeneous coordinate vectors of the fourth vertex and the first vertex. The homogeneous line vector of the second side and the fourth side is then cross producted again and normalized to obtain the three-dimensional homogeneous coordinate vector of the vanishing point of the second direction. The homogeneous coordinate vectors of the vanishing points in the first and second directions are cross-producted to obtain the homogeneous straight line vector of the horizon.

3. The monocular image self-calibration inverse perspective mapping method as described in claim 2, characterized in that, Determining the vanishing point of the forward direction in the road image further includes: Perform a cross product of the three-dimensional homogeneous coordinate vectors of the first vertex and the third vertex to obtain the homogeneous line vector of the first diagonal. Perform a cross product of the homogeneous line vector of the first diagonal and the homogeneous line vector of the horizon and normalize it to obtain the first candidate vanishing point. Perform a cross product of the three-dimensional homogeneous coordinate vectors of the second vertex and the fourth vertex to obtain the homogeneous line vector of the second diagonal. Perform a cross product of the homogeneous line vector of the second diagonal and the homogeneous line vector of the horizon and normalize it to obtain the second candidate vanishing point. The vanishing point of the forward direction is selected from the first candidate vanishing point and the second candidate vanishing point, and the one that is located within the image coordinate range of the road image is taken as the vanishing point of the forward direction.

4. The monocular image self-calibration inverse perspective mapping method as described in claim 3, characterized in that, When both the first candidate vanishing point and the second candidate vanishing point are within the image coordinate range of the road image, a weighted average is calculated on the first candidate vanishing point and the second candidate vanishing point, and the weighted average result is taken as the vanishing point in the forward direction.

5. The monocular image self-calibration inverse perspective mapping method as described in claim 1, characterized in that, When lane boundary lines exist in the road image, determining the vanishing point of the forward direction in the road image specifically includes: Extract the pixel coordinates of multiple endpoints of the left lane boundary line and the right lane boundary line; The endpoints of abnormal line segments are removed by using a random sampling consistency algorithm on the pixel coordinates of each endpoint. The remaining endpoints of the left lane boundary line and the remaining endpoints of the right lane boundary line are respectively fitted with least squares straight lines to obtain the homogeneous straight line vectors of the left lane boundary line and the right lane boundary line. The coordinates of the vanishing point of the forward direction are obtained by performing a cross product and normalizing the homogeneous straight line vectors of the left lane boundary line and the right lane boundary line. A preset effective area constraint is applied to the vanishing point coordinates of the forward direction. When the vanishing point coordinates of the forward direction exceed the preset effective area, the vanishing point estimation is retried.

6. The monocular image self-calibration inverse perspective mapping method as described in claim 1, characterized in that, When both lane boundary lines and road surface markings exist simultaneously in the road image, determining the vanishing point of the forward direction in the road image specifically includes: The estimated vanishing point of the first forward direction is calculated based on the lane boundary line, and the estimated vanishing point of the second forward direction is calculated based on the road surface markings. The consistency between the estimated first vanishing point of the forward direction and the estimated second vanishing point of the forward direction is verified, and the weighted fusion result is used as the vanishing point of the forward direction. When the road image contains only lane boundary lines or only road surface markings, determining the vanishing point of the forward direction in the road image specifically includes: The vanishing point calculated from a single source is used as the vanishing point for the forward direction.

7. The monocular image self-calibration inverse perspective mapping method as described in claim 1, characterized in that, The construction of the basic inverse perspective transformation matrix, based on the vanishing point of the forward direction, specifically includes: A three-row, three-column transformation matrix is ​​constructed using the pixel coordinates of the vanishing point of the forward direction as parameters. The three-row, three-column transformation matrix maps the vanishing point of the forward direction to infinity. Based on the expected coverage of the bird's-eye view and the resolution of the road image, the scale parameters in the three-row, three-column transformation matrix are calculated and determined in real time during the execution of the monocular image self-calibration inverse perspective mapping method.

8. The monocular image self-calibration inverse perspective mapping method as described in claim 1, characterized in that, The process of mapping the vertex pixel coordinates of the road surface plane mark to an intermediate transformation space using the basic inverse perspective transformation matrix, and then performing scale self-correction on the basic inverse perspective transformation matrix using the proportional relationship between the pixel geometry of the road surface plane mark after mapping and the actual physical size, to obtain a corrected homography matrix, specifically includes: Calculate the horizontal and vertical dimensions of the road surface plane mark in the intermediate transformation space, wherein the horizontal dimension of the pixel is the absolute value of the difference between the horizontal coordinates of the left and right vertices of the road surface plane mark in the intermediate transformation space, and the vertical dimension of the pixel is the absolute value of the difference between the vertical coordinates of the top and bottom vertices of the road surface plane mark in the intermediate transformation space. The true aspect ratio of the road surface marking is calculated using the ratio of the physical length to the physical width in the actual physical dimensions. The vertical scaling correction factor is obtained by dividing the actual aspect ratio by the ratio of the vertical dimension of the pixel to the horizontal dimension of the pixel. Construct a diagonal scaling matrix with the vertical scaling correction coefficient as the vertical scaling factor and the horizontal scaling factor as 1. Multiply the diagonal scaling matrix with the basic inverse perspective transformation matrix to obtain the corrected homography matrix.

9. The monocular image self-calibration inverse perspective mapping method as described in claim 1, characterized in that, The determination of the pixel-physical length ratio coefficient based on the ratio of the pixel size of the road surface plane mark under the corrected homography matrix to the corresponding actual physical length specifically includes: The pixel lateral dimensions of the road surface markings are recalculated under the corrected homography matrix; The ratio of the pixel's horizontal dimension to the physical width in the actual physical dimension is used as the pixel-physical length ratio coefficient.

10. The monocular image self-calibration inverse perspective mapping method as described in claim 9, characterized in that, When the road image contains multiple road surface markings, the pixel lateral dimension of each road surface marking is calculated under the correction homography matrix. The pixel-physical length ratio coefficient corresponding to each road surface marking is obtained by the ratio of the pixel lateral dimension of each road surface marking to the physical width in the corresponding real physical dimension. The arithmetic mean of all the pixel-physical length ratio coefficients is taken, and the arithmetic mean is used as the final pixel-physical length ratio coefficient.

11. The monocular image self-calibration inverse perspective mapping method as described in claim 1, characterized in that, The coordinate transformation of the bird's-eye view is generated by inverse mapping. For each target pixel in the bird's-eye view, the source image coordinates corresponding to the target pixel are calculated by the inverse matrix of the correction homography matrix. Then, a positive depth component constraint is applied to the source image coordinates to exclude the back-side mapping area before pixel sampling.

12. The monocular image self-calibration inverse perspective mapping method as described in claim 11, characterized in that, The step of applying a positive depth component constraint to the source image coordinates to exclude backface mapping regions specifically includes: For each target pixel in the bird's-eye view, the product of the inverse of the correction homography matrix and the homogeneous coordinate vector of the target pixel is calculated to obtain the homogeneous coordinate vector of the source image position corresponding to the target pixel. When the depth component of the homogeneous coordinate vector of the source image location is greater than zero, pixel sampling is performed on the corresponding location in the road image and the sampled value is assigned to the target pixel; When the depth component of the homogeneous coordinate vector of the source image location is not greater than zero, the target pixel is marked as an invalid region.

13. The monocular image self-calibration inverse perspective mapping method as described in claim 1, characterized in that, Also includes: In the bird's-eye view, determine the first and second points to be measured; Obtain the pixel coordinates of the first point and the second point; Calculate the pixel Euclidean distance between the first point and the second point; Divide the pixel Euclidean distance by the pixel-physical length scaling factor to obtain the actual physical distance between the first point and the second point.

14. The monocular image self-calibration inverse perspective mapping method as described in claim 1, characterized in that, The pixel coordinates of multiple vertices of the road surface marking can be obtained using any of the following methods: The user manually marks each vertex of the road surface plane mark in the road image to obtain the pixel coordinates of each vertex; Alternatively, the target detection model can infer from the road image to automatically output the pixel coordinates of each vertex of the road surface marking.

15. The monocular image self-calibration inverse perspective mapping method as described in claim 1, characterized in that, The road surface plane marking is a road diamond mark, and the four vertices of the road diamond mark are arranged in a clockwise or counterclockwise direction. The actual physical size is determined according to the road standard specifications of the area where the road diamond mark is deployed and is input as a configurable parameter into the monocular image self-calibration inverse perspective mapping method.

16. The monocular image self-calibration inverse perspective mapping method as described in claim 1, characterized in that, The monocular image self-calibration inverse perspective mapping method continuously processes the input road image in units of image frames. If the road surface plane mark is detected when processing each frame of the road image, the recalculation of the correction homography matrix and the pixel-physical length ratio coefficient is automatically triggered.

17. The monocular image self-calibration inverse perspective mapping method as described in claim 16, characterized in that, The vanishing point coordinates of the forward direction and the pixel-to-physical length ratio coefficient calculated in multiple consecutive road images are subjected to temporal filtering and fusion. The vanishing point coordinates of the forward direction and the pixel-to-physical length ratio coefficient after temporal filtering and fusion are used as the calibration parameters of the current frame.

18. The monocular image self-calibration inverse perspective mapping method as described in claim 16, characterized in that, The continuous image stream acquired by the forward-looking monocular camera deployed on the vehicle computing platform is used as the input road image. The monocular image self-calibration inverse perspective mapping method outputs a bird's-eye view with a physical scale in real time after processing each frame of road image.

19. The monocular image self-calibration inverse perspective mapping method as described in claim 1, characterized in that, The road images are derived from image sequences captured by a fixed monocular camera or from image frames of historical videos. The monocular image self-calibration inverse perspective mapping method sequentially performs scale self-correction and generates the bird's-eye view on the road images in batches.

20. A monocular image self-calibration inverse perspective mapping system, characterized in that, include: The feature input module is used to receive a monocular road image containing at least one road surface plane mark, as well as the pixel coordinates and actual physical size of multiple vertices of the road surface plane mark; The vanishing point calculation module is used to determine the vanishing point of the forward direction in the road image. The vanishing point calculation module supports the independent operation of the vanishing point derivation path based on the opposite edge of the road surface marking and the vanishing point derivation path based on the lane boundary line, and performs weighted fusion when both derivation paths are available. The homography matrix construction module is used to construct a basic inverse perspective transformation matrix based on the vanishing point of the forward direction. The scale self-correction module is used to map the vertex pixel coordinates of the road surface plane mark to the intermediate transformation space through the basic inverse perspective transformation matrix, and to perform scale self-correction on the basic inverse perspective transformation matrix by using the proportional relationship between the pixel geometry of the road surface plane mark after mapping and the actual physical size to obtain the corrected homography matrix. The physical scale calibration module is used to determine the pixel-physical length ratio coefficient based on the ratio of the pixel size of the road surface plane mark to the corresponding actual physical length under the correction homography matrix; The bird's-eye view generation module is used to perform coordinate transformation on the road image using the correction homography matrix, and output a bird's-eye view with a physical scale.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements each step of the monocular image self-calibration inverse perspective mapping method according to any one of claims 1 to 19.

Images