A method for producing conjugate lines of large images by using conjugate line pairs on the image plane

Abstract

A method for generating conjugate epipolar lines of large images based on conjugate epipolar line pairs on the image plane. First, divide a large-size remote sensing image into multiple small images of the same size. Then, taking the center point of the small image as the starting point, project the left-image points to the ground surface with the maximum and minimum elevations using a rational function model, and then calculate the corresponding two right-image conjugate epipolar line points by inverse calculation. Repeat the process until the corresponding points exceed the image range. At this time, the conjugate epipolar line pair of the current small image based on the starting point of the left image is obtained. By dividing and processing the large-size remote sensing image and generating conjugate epipolar lines one by one with precise refinement, the method can better complete the generation of conjugate epipolar lines for large-size images, obtain an approximate conjugate epipolar line image pair with sub-pixel error, and further provide support for subsequent processing such as remote sensing image matching and 3D modeling.

Description

Technical Field

[0001] The present invention relates to the technical field of aerospace remote sensing mapping, and specifically to a method for making block-based epipolar lines of large images based on image-plane epipolar line pairs. Background Art

[0002] With the development and progress of aerospace technology, high-resolution and wide-swath satellite remote sensing images have become the main data source for obtaining Earth space information. The three-dimensional information extracted from high-resolution linear array satellite stereo images can be used for digital surface model generation, target three-dimensional modeling, digital cities, etc., with a wide range of applications and high economic value. The strategic significance of high-resolution remote sensing satellites has become increasingly prominent, becoming a technological high ground that countries around the world are competing to occupy.

[0003] Currently, most remote sensing satellites use linear array pushbroom imaging. In order to increase the imaging swath, some also use the method of splicing multiple CCDs for imaging, resulting in an increase in the size of the obtained remote sensing images. Currently, epipolar line generation is an essential step in extracting three-dimensional information from satellite stereo image pairs. The concept of epipolar lines originated from aerial photogrammetry and refers to the intersection line of the epipolar plane and the image plane. The epipolar line theory transforms the matching of homologous points from two-dimensional search to one-dimensional search. Existing matching algorithms basically rely on epipolar line constraints to improve calculation efficiency and reliability. Different from traditional frame images, due to its special imaging mechanism, each scan line of satellite linear array images has an independent projection center, and the projections between lines are approximately parallel. Therefore, linear array images do not have strict epipolar lines like frame images.

[0004] A large number of studies have shown that the epipolar lines of linear array images are hyperbolic and can be regarded as straight lines within a local range of the image. It should be noted that all current epipolar line making methods rely on the Rational Function Model (RFM). Almost all satellite providers use RFM as an alternative model to the strict geometric model and use Rational Polynomial Coefficients (RPC) as orientation parameters to carry out positioning calculations. The main current methods for making epipolar lines include the projection trajectory method, the method based on object space longitude and latitude, the method based on the projection reference plane, etc. However, as the size of satellite images is getting larger and larger, it is necessary to explore more efficient methods for making epipolar lines of large sizes. The present invention aims at large-size remote sensing images and realizes epipolar line making by adopting a block processing idea. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for making block-based epipolar lines of large images based on image-plane epipolar line pairs, which can effectively solve the problems raised in the above background art.

[0006] To solve the above problems, the technical solution adopted by the present invention is: a method for making conjugate epipolar lines for large images based on conjugate epipolar pairs on the image plane, comprising the following steps:

[0007] Step 1: Define the original image as the left image, divide the left image into blocks, and make the starting points of conjugate epipolar lines.

[0008] Step 2: Make a projection plane according to the image, and make sets of conjugate epipolar points for the left and right images through the starting points of conjugate epipolar lines and the projection plane.

[0009] Step 3: Fit a straight line to the conjugate epipolar points of the left and right images, count the distances from each point to the straight line, and make conjugate epipolar lines.

[0010] Step 4: Evaluate the conjugate epipolar lines, and judge whether the processing results are satisfied according to the evaluation results.

[0011] Step 5: Make a DSM in blocks and obtain the three-dimensional coordinates of ground points.

[0012] Step 6: After the block image processing is completed, process each block image of the left image according to the above steps until all block images are processed.

[0013] As a further preferred solution of the present invention, in Step 1, after the left image is divided into blocks, the left image obtains n×n small images, and the center points of the small images are used as the starting points of conjugate epipolar lines.

[0014] As a further preferred solution of the present invention, Step 2 includes defining the starting point as p, extracting the actual maximum and minimum values of the original image to establish a projection plane, the maximum value is Hmax, the minimum value is Hmin, and calculating the coordinates of point p on the Hmax and Hmin projection planes through the inverse calculation formula of the left image RPC model by using the rational function model; then calculating the corresponding two image points through the forward calculation formula of the right image RPC model; passing each image point through the Hmax and Hmin projection planes without repetition, and looping until the corresponding image point exceeds the image range, and obtaining a set of conjugate epipolar points on the left and right images respectively.

[0015] As a further preferred solution of the present invention, Step 3 includes respectively fitting a straight line to the conjugate epipolar points of the left and right images, and counting the distance errors from each point to the fitted straight line. When the maximum error of the straight line is within 1 pixel, rotate the current block image by the corresponding angle to make conjugate epipolar lines; when the maximum error of the straight line exceeds 1 pixel, convert to the conventional projection trajectory method to collect conjugate epipolar lines one by one.

[0016] As a further preferred embodiment of the present invention, step four specifically uses a feature point matching method to evaluate the epipolar line performance, checks the row coordinate difference. If the row coordinate difference meets the conditions, it is considered that the epipolar line image performance is excellent and meets the processing results; if it does not meet the conditions, it goes back to analyze the reasons and checks the positioning model parameters and the epipolar line straight line fitting error; when the above parameters are all correct but the epipolar line row coordinate performance still does not meet the indicators, it goes back to use the projection trajectory method for processing.

[0017] As a further preferred embodiment of the present invention, in step five, by segmenting the epipolar line image, the dense matching algorithm SGM is used to obtain the corresponding point disparity, and at the same time, the "trailing" phenomenon in the disparity map is eliminated by this algorithm; the three-dimensional coordinates of the ground points are obtained.

[0018] As a further preferred embodiment of the present invention, the image pixel is projected onto the ground maximum elevation and minimum elevation planes using a rational function model, and the pixel coordinates (r n , c n ) are described as a polynomial ratio with the corresponding ground coordinates (X n , Y n , Z n ) as independent variables, and the formula is:

[0019]

[0020] Equation (1) is the forward calculation formula for calculating the pixel coordinates from the ground points; in the formula, a ijk , b ijk , c ijk , d ijk represent rational function parameters (RPCs), n represents the number of target points, i + j + k is not greater than 3, representing the degree of the rational function model, that is, i + j + k ≤ 3; taking the numerator P1(X, Y, Z) on the right side of r n as an example, that is Its form is:

[0021]

[0022] The inverse calculation formula for calculating the ground points from the pixel coordinates can be obtained by RPC forward intersection. In order to increase the reliability of the numerical solution and avoid too large parameter differences, translation and scaling parameters are usually introduced to standardize the original object and image points, and the standardized values are between (-1.0 ~ +1.0); when performing RFM stereo positioning, Equation (1) is transformed into:

[0023]

[0024] In the formula, F(X n , Y n , Z n ), G(X n,Y n ,Z n ) represent the right sides of the equations corresponding to r n and c n in Equation (2) respectively. By performing a Taylor expansion on Equation (3), the error equation is obtained as follows:

[0025]

[0026] For a single target point, if the image coordinates of the corresponding points in the left and right images (r l , c l ) and (r r , c r ) are known, then the following four error equations can be listed:

[0027]

[0028] If the above error equations are written as V = AΔ - l, then the least squares solution of the coordinate correction Δ is:

[0029] Δ = [ΔX ΔY ΔZ] T = (A T A) -1 A T l (6),

[0030] The above (3)-(6) are the inverse calculation formulas for calculating object coordinates from image points.

[0031] As a further preferred embodiment of the present invention, the straight line form of the straight line fitting can be expressed as y = ax + b or ax + by + c = 0. The former is suitable for cases where the angle is not too large. When the straight line is close to vertical, the latter is used.

[0032] As a further preferred embodiment of the present invention, the epipolar line performance evaluation specifically includes evaluating the epipolar line performance by using a feature point matching method. The SIFT operator is used to extract feature points in the left and right block epipolar line images. Check the difference in the row coordinates of the feature points. If the mean value of the coordinate differences is within 1 pixel and the maximum difference does not exceed 2 pixels, it is considered that the epipolar line image performance is excellent and meets the processing results; if the above conditions are not met, the reason is analyzed retrospectively, and the positioning model parameters and the epipolar line straight line fitting error are checked. If the above parameters are correct but the epipolar line row coordinate performance still does not meet the index, then the projection trajectory method is used for processing.

[0033] As a further preferred embodiment of the present invention, obtaining the three-dimensional coordinates of ground points includes using the block epipolar line images and obtaining the corresponding point disparities by using the dense matching algorithm SGM (Semi-Global Matching). The SGM algorithm is generally divided into three stages;

[0034] The first stage is pixel matching cost calculation: The matching cost defined by SGM based on mutual information is as follows:

[0035] C MI (p, d) = -h l (p) - h r (p, d) + h l,r (p, d) (7),

[0036]

[0037]

[0038] Wherein is the mutual information of the image given based on the information entropy, I bp is the gray value of the image point p of the reference image, I mn is the gray value of the image point q at the corresponding position to p in the matching image, q = e bm (p, d), e bm (p, d) is the epipolar line corresponding to the point p of the image to be matched, that is, e bm (p, d) = [p x + d, p y ; P I 、 are respectively the gray probability distribution and the joint probability distribution of the image to be matched; is the Gaussian convolution, and n is the total number of pixels;

[0039] The second stage is to calculate the disparity; Based on the matching cost of the above pixels, the SGM algorithm defines the energy function as follows:

[0040]

[0041] Wherein, C MI (p, d p ) is the matching cost in (7), p represents any image point, N p is the neighborhood of this image point, P1 and P2 represent the penalty coefficients of the disparity change, D is the dense disparity map of the image to be matched, and T is a Boolean function; To solve the global minimization problem, a multi-path dynamic programming algorithm is used to optimize the disparity:

[0042]

[0043]

[0044] The third stage is to optimize the disparity: Optimize the above disparity, and perform left-right consistency check, filtering, and interpolation operations;

[0045] After obtaining the point-by-point parallax map, based on the relevant theory of RPC forward intersection in formulas (5)-(6), calculate the three-dimensional coordinates of ground points point by point.

[0046] Compared with the prior art, the present invention provides a method for making conjugate epipolar lines of large images based on conjugate epipolar lines on the image plane, which has the following beneficial effects:

[0047] This method performs block processing on large-size remote sensing images, makes conjugate epipolar lines one by one, refines them precisely, and better completes the conjugate epipolar lines of large-size images, and can obtain an approximate conjugate epipolar line image pair with sub-pixel error, thereby providing support for subsequent processing such as remote sensing image matching and 3D modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic flowchart of the present invention;

[0049] Figure 2 It is a schematic diagram of image block division of the present invention;

[0050] Figure 3 It is a schematic diagram of generating conjugate epipolar line pairs by the projection trajectory method based on the maximum and minimum elevation values of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0052] Refer to Figures 1-3 , the present invention provides a method for making conjugate epipolar lines of large images based on conjugate epipolar lines on the image plane, including the following steps:

[0053] Step 1: Define the original image as the left image, divide the left image into blocks, and make the starting points of conjugate epipolar lines;

[0054] Step 2: Make a projection plane according to the image, and make sets of conjugate epipolar line points of the left and right images through the starting points of conjugate epipolar lines and the projection plane;

[0055] Step 3: Fit a straight line to the conjugate epipolar line points of the left and right images, count the distances from each point to the straight line, and make conjugate epipolar lines;

[0056] Step 4: Evaluate the conjugate epipolar lines, and judge whether the processing results are satisfied according to the evaluation results;

[0057] Step 5: Make a DSM in blocks and obtain the three-dimensional coordinates of ground points;

[0058] Step 6: After the block image processing is completed, process each block image of the left image according to the above steps until all the block images are processed.

[0059] As a further preferred embodiment of the present invention, in the first step, after the left image is segmented, the left image obtains n×n small images, and the center point of the small image is used as the starting point of the epipolar line.

[0060] As a further preferred embodiment of the present invention, the second step includes defining the starting point as p, extracting the actual maximum and minimum values of the original image to establish a projection plane, the maximum value is Hmax, the minimum value is Hmin, and using the rational function model to calculate the coordinates of point p on the Hmax and Hmin projection planes through the inverse calculation formula of the left image RPC model; then calculating the corresponding two image points through the forward calculation formula of the right image RPC model; passing each image point through the Hmax and Hmin projection planes without repetition, and looping until the corresponding image point exceeds the image range, obtaining a set of epipolar line points on the left and right images respectively, the set of left image epipolar line points is denoted as P, and the set of left image epipolar line points is denoted as Q.

[0061] As a further preferred embodiment of the present invention, the third step includes performing linear fitting on the sets of epipolar line points P and Q of the left and right images respectively, and statistically calculating the distance error of each point from the fitted line. When the maximum line error is within 1 pixel, the current segmented image is rotated by the corresponding angle to make the epipolar line; when the maximum line error exceeds 1 pixel, it is converted to the conventional projection trajectory method for epipolar line acquisition one by one.

[0062] As a further preferred embodiment of the present invention, the fourth step specifically uses the feature point matching method to evaluate the epipolar line performance, checks the row coordinate difference. If the row coordinate difference meets the conditions, it is considered that the epipolar line image performance is excellent and meets the processing result; if it does not meet the conditions, it goes back to analyze the reasons, checks the positioning model parameters and the epipolar line linear fitting error; when the above parameters are all correct but the epipolar line row coordinate performance still does not meet the index, it goes back to use the projection trajectory method for processing.

[0063] As a further preferred embodiment of the present invention, in the fifth step, through the segmented epipolar line image, the dense matching algorithm SGM is used to obtain the disparity of corresponding points, and at the same time, the "trailing" phenomenon in the disparity map is eliminated by this algorithm; the three-dimensional coordinates of ground points are obtained.

[0064] As a further preferred embodiment of the present invention, the image points are projected onto the ground maximum elevation and minimum elevation planes using the rational function model, and the image point coordinates (r n , c n ) are described as the ratio of polynomials with the corresponding ground coordinates (X n , Y n , Z n ) as independent variables, and the formula is:

[0065]

[0066] Equation (1) is the direct calculation formula for calculating the image point coordinates from ground points; in the formula, a ijk , b ijk , c ijk , d ijk represent rational function parameters (RPCs), n represents the number of target points, i + j + k is not greater than 3, representing the degree of the rational function model, that is, i + j + k ≤ 3; taking the numerator P1(X, Y, Z) on the right side of r n as an example, that is its form is:

[0067]

[0068] The inverse calculation formula for calculating ground points from image point coordinates can be obtained by RPC forward intersection. To increase the reliability of numerical solution and avoid excessive parameter differences, translation and scaling parameters are usually introduced to standardize the original object-image points, and the standardized values are between (-1.0 ~ +1.0); during RFM stereo positioning, Equation (1) is transformed into:

[0069]

[0070] In the formula, F(X n , Y n , Z n ), G(X n , Y n , Z n ) respectively represent the right sides of the equations corresponding to r n , c n in Equation (2). The error equation is obtained by Taylor expansion of Equation (3):

[0071]

[0072] For a single target point, if the image coordinates of the corresponding points on the left and right images (r l , c l ), (r r , c r ) are known, then the following 4 error equations can be listed:

[0073]

[0074] If the above error equation is written as V = AΔ - l, then the least squares solution of the coordinate correction Δ is:

[0075] Δ = [ΔX ΔY ΔZ] T = (A T A) -1 A T l (6),

[0076] The above formulas (3)-(6) are the inverse calculation formulas for calculating the object space coordinates from image points.

[0077] As a further preferred embodiment of the present invention, the straight line form of the straight line fitting can be expressed as y = ax + b or ax + by + c = 0. The former is suitable for the case where the angle is not too large. When the straight line is close to vertical, the latter is used.

[0078] As a further preferred embodiment of the present invention, the epipolar line performance evaluation specifically includes evaluating the epipolar line performance by using the feature point matching method. The SIFT operator is used to extract feature points in the left and right block epipolar line images, and the row coordinate difference of the feature points is checked. If the average value of the coordinate differences is within 1 pixel and the maximum difference does not exceed 2 pixels, it is considered that the epipolar line image performance is excellent and meets the processing results; if the above conditions are not met, the reason is analyzed by falling back, the positioning model parameters and the epipolar line straight line fitting error are checked. If the above parameters are correct but the epipolar line row coordinate performance still does not meet the index, the projection trajectory method is used for processing by falling back.

[0079] As a further preferred embodiment of the present invention, the acquisition of the three-dimensional coordinates of the ground points includes using the dense matching algorithm SGM (Semi-Global Matching) for the block epipolar line images to obtain the disparity of corresponding points. The SGM algorithm is generally divided into three stages;

[0080] The first stage is the calculation of the pixel matching cost: The matching cost defined by SGM based on mutual information is:

[0081] C MI (p,d) = -h l (p) - h r (p,d) + h l,r (p,d) (7),

[0082]

[0083]

[0084] In the formula is the mutual information of the images given based on the information entropy, I bp is the gray value of the image point p of the reference image, I mn is the gray value of the image point q at the corresponding position in the matching image to p, q = e bm (p,d), e bm (p,d) is the epipolar line corresponding to the point p of the image to be matched, that is, e bm (p,d) = [p x +d, p y ; P I 、 are the gray probability distributions of the image to be matched, that is, the joint probability distributions; is a Gaussian convolution, and n is the total number of pixels;

[0085] The second stage is to calculate the disparity. Based on the matching cost of the above pixels, the SGM algorithm defines the energy function as follows:

[0086]

[0087] In the formula, C MI (p, d p ) is the matching cost in (7), p represents any image point, N p is the neighborhood of this image point, P1 and P2 represent the penalty coefficients for disparity changes, D is the dense disparity map of the image to be matched, and T is a Boolean function. To solve the global minimization problem, a multi-path dynamic programming algorithm is used to optimize the disparity:

[0088] At the same time, this algorithm uses the 8-path or 16-path dynamic programming idea to calculate the matching cost L r (p, d), and solve the "trailing" phenomenon that appears in the disparity map:

[0089]

[0090]

[0091] The third stage is to optimize the disparity: optimize the above disparity, and perform left-right consistency check, filtering, and interpolation operations;

[0092] After obtaining the point-by-point disparity map, based on the relevant theory of RPC forward intersection in formulas (5)-(6), calculate the three-dimensional coordinates of the ground points point by point.

[0093] As the first specific embodiment of the present invention:

[0094] First, divide the remotely sensed image with a large size into multiple small images of the same size; then, taking the center point of the small image as the starting point, project the left image points to the ground maximum elevation and minimum elevation surfaces using the rational function model, and then reverse calculate to the right image to obtain the corresponding two right image epipolar points. Loop until the corresponding image points exceed the image range. At this time, the epipolar pair of the current small image based on the left image starting point is obtained; then, perform a linear fitting on the point set on this epipolar pair to obtain the linear fitting parameters and fitting errors; finally, rotate the small image by the corresponding angle to obtain the epipolar image. Through the rotation angle parameter, the conversion relationship between the epipolar image and the original image can be established, and the large image completes all processing through block-by-block cycling. This algorithm is applicable to situations such as IKONOS, WorldView-3, and WorldView-3 with good parallel characteristics and large image sizes, and can obtain an approximate epipolar image pair close to sub-pixel error, thereby providing support for subsequent processing such as remotely sensed image matching and 3D modeling.

[0095] As the second specific embodiment of the present invention:

[0096] Step 1: Segment the large-size image. Assume that the number of rows (height) and columns (width) of the large-size image are Line_big and Sample_big respectively, and the number of rows and columns of the segmented small images are set as Line_small and Sample_small. Considering the processing power of the computer, it is impossible to directly allocate more than 4G of memory. Therefore, the recommended sizes of Line_small and Sample_small here are 1000 pixels. The step sizes for segmenting the large image are: Ystep = Line_big / Line_small in the row direction, and Xstep = Sample_big / Sample_small in the column direction. The sizes of the small images in the last row and the last column are subject to the remaining sizes. After completing the segmentation, select the center point of the left image of each block as the starting point for epipolar line generation. As shown in the following figure, the original image is I. After segmentation, n×n small images are obtained, and the center points of the small images are the starting points of the epipolar lines;

[0097] Step 2: Taking the center point of the segmented image as the starting point, generate epipolar line pairs based on the projection trajectory method. As Figure 2 shown, assume that the center point of the left segmented image is p, and the maximum and minimum elevation values provided by the digital elevation model (DEM) of the ground are Hmax and Hmin respectively. Then, the light rays passing through point p intersect the projection planes Hmax and Hmin to obtain 2 ground points. The specific coordinates of these 2 ground points are calculated by the inverse calculation formula of the left image RPC model (calculating the object space coordinate position from the image point coordinates). These 2 ground points can obtain 2 corresponding image points q1 and q2 on the right image by using the forward calculation formula of the right image RPC (calculating the image point coordinate position from the object space coordinates). At this time, the straight line passing through q1 and q2 is the approximate epipolar line of the right image. In the present invention, this straight line is not directly used, but more epipolar line point pairs on the left and right images are continuously projected to increase the reliability of epipolar line fitting. That is, we project from q1 to H_min and then calculate forward to the left image to obtain p′, and project from q2 to H_max and then calculate forward to the left image to obtain p″. Following the idea that each image point is projected onto the H_min and H_max elevation planes without repetition, a set of epipolar line point pairs can be obtained on the left and right images by looping. Subsequently, a more accurate straight line is fitted with these epipolar line point pairs as the epipolar line. Here, the sets of epipolar line points of the left and right images are denoted as p and q respectively,

[0098] Regarding the rational function model, the image point coordinates (r n , c n ) are described as the ratio of polynomials with the corresponding ground coordinates (X n , Y n , Z n ) of the point as the independent variables. The formula is:

[0099]

[0100] Equation (1) is the direct calculation formula for calculating the image point coordinates from ground points. In the formula, a ijk , b ijk , c ijk , d ijk represent rational function parameters (RPCs), n represents the number of target points, and i + j + k is usually not greater than 3, representing the degree of the rational function model, that is, i + j + k ≤ 3. Taking the numerator P1(X, Y, Z) on the right side of r n as an example, its form is:

[0101]

[0102] The inverse calculation formula for calculating ground points from image point coordinates can be obtained by RPC forward intersection. To increase the reliability of numerical solution and avoid too large parameter differences, translation and scaling parameters are usually introduced to standardize the original object-image points, and the standardized values are between (-1.0 and +1.0). When performing RFM stereo positioning, Equation (2) is transformed into:

[0103]

[0104] In the formula, F(X n , Y n , Z n ), G(X n , Y n , Z n ) respectively represent the right sides of the equations corresponding to r n , c n in Equation (2). Expanding Equation (3) by Taylor series gives the error equation:

[0105]

[0106] For a single target point, if the image coordinates of the homologous points in the left and right images (r l , c l ), (r r , c r ) are known, then the following 4 error equations can be listed:

[0107]

[0108] If the above error equation is written as V = AΔ - l, then the least squares solution of the coordinate correction Δ is:

[0109] Δ = [ΔX ΔY ΔZ] T = (A T A) -1 A T l (6)

[0110] The above is the inverse calculation formula for calculating the object space coordinates from image points.

[0111] Step 3: Line fitting to evaluate the epipolar line features and generate epipolar images. Perform line fitting on the left and right epipolar point sets p and q respectively. The line form can be expressed as y = ax + b or ax + by + c = 0. The former is suitable for cases where the angle is not too large. When the line is close to vertical, the latter is used. After calculating the unknown parameters of the line, statistically analyze the distance error of each point to the fitted line. If the maximum error of the line is within 1 pixel, rotate the current sub-block image by the corresponding angle to create the epipolar line; if the maximum error exceeds 1 pixel, convert to the conventional projection trajectory method for sampling line by line.

[0112] Step 4: Epipolar line performance evaluation. Use the feature point matching method to evaluate the epipolar line performance. Use the SIFT operator to extract feature points from the left and right sub-block epipolar images, and check the row coordinate difference of the feature points. If the average value of the coordinate difference is within 1 pixel and the maximum difference does not exceed 2 pixels, it is considered that the epipolar image performance is excellent and meets the processing results; if the above conditions are not met, analyze the reasons for fallback, check the positioning model parameters and the epipolar line fitting error. If the above parameters are correct but the epipolar line row coordinate performance still does not meet the requirements, fallback to the projection trajectory method for processing.

[0113] Step 5: Generate DSM for each sub-block. Using the above sub-block epipolar images, obtain the disparity of corresponding points by using the dense matching algorithm SGM (Semi-Global Matching). The matching cost of SGM based on mutual information is defined as:

[0114] C MI (p, d) = -h l (p) - h r (p, d) + h l,r (p, d) (7)

[0115] At the same time, this algorithm uses the 8-path or 16-path dynamic programming idea to calculate the matching cost L r (p, d) to solve the "trailing" phenomenon in the disparity map:

[0116]

[0117] After obtaining the point-by-point disparity map, based on the relevant theory of RPC forward intersection in formulas (5)-(6), calculate the three-dimensional coordinates of ground points point by point.

[0118] Step 6: After processing the sub-block images, process each sub-block image according to the above process until all sub-blocks are processed.

[0119] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for making conjugate epipolar lines of large images based on image plane conjugate epipolar pairs, comprising the following steps: Step 1: Define the original image as the left image. After dividing the left image into blocks, the left image obtains n×n small images, and the center points of the small images are used as the starting points of the conjugate epipolar lines. Step 2: Establish a projection plane through the maximum and minimum elevation values provided by the ground digital elevation model. The maximum value is Hmax, and the minimum value is Hmin. Define the starting point as p. Use the inverse calculation formula of the left image RPC model to calculate the coordinates of point p on the Hmax and Hmin projection planes through the rational function model; then use the forward calculation formula of the right image RPC model to calculate the corresponding two image points; Pass each image point through the Hmax and Hmin projection planes without repetition, and loop until the corresponding image point exceeds the image range, and a set of conjugate epipolar line points are obtained on the left and right images respectively. Step 3: Perform linear fitting on the conjugate epipolar line points of the left and right images respectively, and statistically analyze the distance from each point to the line to make conjugate epipolar lines, including performing linear fitting on the conjugate epipolar line points of the left and right images respectively, and statistically analyzing the distance error from each point to the fitted line. When the maximum line error is within 1 pixel, rotate the current divided image by the corresponding angle to make conjugate epipolar lines; when the maximum line error exceeds 1 pixel, convert to the conventional projection trajectory method for collecting conjugate epipolar lines one by one. Step 4: Evaluate the conjugate epipolar lines, and judge whether the processing result is satisfied according to the evaluation result. Use the feature point matching method to evaluate the performance of the conjugate epipolar lines, and check the row coordinate difference. If the row coordinate difference meets the conditions, it is considered that the performance of the conjugate epipolar line image is excellent and meets the processing result; if it does not meet the conditions, roll back to analyze the reasons, and check the positioning model parameters and the conjugate epipolar line linear fitting error; when the above parameters are correct but the conjugate epipolar line row coordinate performance still does not meet the index, roll back to use the projection trajectory method for processing. Step 5: Make a DSM in blocks to obtain the three-dimensional coordinates of ground points. Step 6: After the divided image processing is completed, process each divided image of the left image according to the above steps until all the divided images are processed.

2. The method for making conjugate epipolar lines of large images based on image plane conjugate epipolar pairs according to claim 1, wherein in step 5, through the divided conjugate epipolar line images, the dense matching algorithm SGM is used to obtain the disparity of corresponding points, and at the same time, the "trailing" phenomenon appearing in the disparity map is eliminated by this algorithm; the three-dimensional coordinates of ground points are obtained.

3. The method for making conjugate epipolar lines of large images based on image plane conjugate epipolar pairs according to claim 1, wherein the image points use the rational function model to describe the image point coordinates (r n , c n ) as the ground coordinates of the corresponding points (X n , Y n , Z n) is a polynomial ratio with the independent variable, and the formula is: Equation (1) is the forward calculation formula for calculating the image point coordinates from the ground points; in the formula, a ijk , b ijk , c ijk , d ijk represent rational function parameters (RPCs), n represents the number of target points, m i , m j , m k represent the polynomial orders corresponding to X, Y, and Z. i + j + k is not greater than 3, indicating the degree of the rational function model, that is, i + j + k ≤ 3; taking the numerator P1(X, Y, Z) on the right side of r n as an example, that is Its form is: The inverse calculation formula for calculating the ground points from the image point coordinates can be obtained by RPC forward intersection. To increase the reliability of numerical solution and avoid too large parameter differences, translation and scaling parameters are usually introduced to standardize the original object-image points. The standardized values are between (-1.0 and +1.0); when performing RFM stereo positioning, Equation (1) is transformed into: In the formula, F(X n , Y n , Z n ), G(X n , Y n , Z n ) respectively represent the right sides of the equations corresponding to r n , c n in Equation (1). The error equation is obtained by Taylor expanding Equation (3): In the formula, represents taking the partial derivative, and respectively represent the initial values of the row and column numbers of the image point; For a single target point, if the image coordinates (r l , c l ), (r r , c r ) of the corresponding points in the left and right images are known, then the following 4 error equations are listed: If the above error equation is written as V = AΔ - l, the meanings of each symbol correspond one by one from left to right according to Equation (5). Δ represents [ΔX ΔY ΔZ] T , T represents matrix transpose, then the least squares solution of the coordinate correction Δ is: Δ = [ΔX ΔY ΔZ] T = (A T A) -1 A T l (6), The above equations (3)-(6) are the inverse calculation formulas for calculating the object space coordinates from the image points.

4. A method for making epipolar lines of large images based on epipolar line pairs on the image plane according to claim 1, wherein the form of the straight line fitted by the straight line can be expressed as y = ax + b, where a is the slope and b is the x-axis intercept, or ax + by + c = 0, where a + b ≠ 0; the former is suitable for the case where the angle is not too large, and when the straight line is close to vertical, the latter is used to represent it.

5. A method for making epipolar lines of large images based on epipolar line pairs on the image plane according to claim 1, wherein the evaluation of the epipolar line performance specifically includes evaluating the epipolar line performance by using the feature point matching method, extracting feature points in the left and right sub-blocked epipolar line images using the SIFT operator, checking the difference in the row coordinates of the feature points. If the mean value of the coordinate difference is within 1 pixel and the maximum difference does not exceed 2 pixels, it is considered that the epipolar line image performance is excellent and meets the processing results; if the above conditions are not met, the reason is analyzed by reverting, checking the positioning model parameters and the epipolar line straight line fitting error. If the above parameters are correct but the epipolar line row coordinate performance still does not meet the index, then revert to the projection trajectory method for processing.

6. A method for making epipolar lines of large images based on epipolar line pairs on the image plane according to claim 2, wherein the obtaining of the three-dimensional coordinates of the ground points includes obtaining the disparity of corresponding points of the sub-blocked epipolar line images by using the dense matching algorithm SGM; the SGM algorithm is generally divided into three stages; The first stage is the calculation of the pixel matching cost: The matching cost of SGM based on mutual information is defined as: C MI (p, d) = -h l (p) - h r (p, d) + h l,r (p, d) (7), In the formula is the mutual information of the images given based on the information entropy, I bp is the gray value of the reference image point p, I mn is the gray value of the image point q at the corresponding position to p in the matching image, q = e bm (p, d), e bm (p, d) is the epipolar line corresponding to the p point of the image to be matched, that is, e bm (p, d) = [p x + d, p y ; P I , They are the gray - level probability distribution of the images to be matched, i.e., the joint probability distribution; is the Gaussian convolution, and n is the total number of pixels; The second stage is to calculate the disparity. Based on the matching cost of the above - mentioned pixels, the SGM algorithm defines the energy function as follows: In the formula, C MI (p, d p ) is the matching cost in (7), p represents any image point, N p is the neighborhood of this image point, P1 and P2 represent the penalty coefficients of the disparity change, D is the dense disparity map of the images to be matched, and T is a Boolean function. To solve the global minimization problem, a multi - path dynamic programming algorithm is used to optimize the disparity: The third stage is to optimize the disparity: optimize the above - mentioned disparity, and perform left - right consistency check, filtering, and interpolation operations; After obtaining the point - by - point disparity map, based on the relevant theory of RPC forward intersection in formulas (5) - (6), calculate the three - dimensional coordinates of ground points point by point.

Images