Laser altimetry data and high-resolution stereo image registration method and device

Abstract

The present invention discloses a laser altimetry data and high-resolution stereo image registration method and device, the method comprising: obtaining a laser point, a laser footprint image and a stereo image to be registered; taking the laser point elevation as a reference, correcting the reference image and the stereo image to be matched on the current layer of the pyramid to obtain the corrected image; using the sub-pixel phase correlation algorithm and the resolution of the current layer of the pyramid to calculate the geographic offset of the corrected stereo image to be matched relative to the corrected reference image; repeating the correction and offset calculation at each layer of the pyramid until the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image is calculated; completing the positioning of the laser point on the stereo image to be registered according to the geographic offset to obtain the registered image. By implementing the present invention, high-precision registration of satellite laser footprints and high-resolution stereo images is achieved, laying a foundation for using laser high-precision elevation data to assist in improving the application of stereo image mapping.

Description

Technical Field

[0001] The present invention relates to the technical field of photogrammetry and remote sensing image processing, and in particular to a method and device for registering laser altimetry data and high-resolution stereo images. Background Art

[0002] Satellite photogrammetry uses stereo measurement to obtain terrain information from remote sensing images acquired at different angles. With the continuous improvement of remote sensing image resolution, the accuracy of satellite on-orbit attitude measurement has become an important factor affecting the accuracy of high-resolution stereo image mapping. However, due to the limitations of attitude measurement hardware equipment, the improvement of attitude measurement accuracy is limited. Satellite laser altimetry can obtain high-precision surface elevation data. Therefore, improving the geometric positioning accuracy of remote sensing images with the assistance of satellite laser altimetry has become a development trend in satellite photogrammetry. The experimental laser altimeter carried by the "Resource-3" 02 satellite has increased the optical stereo image elevation measurement accuracy from 5m to about 3m. The "Gaofen-7" satellite, which is used for high-precision stereo mapping at a scale of 1:10,000, will be equipped with a two-line array stereo camera and a laser altimeter, mainly using laser altimetry to assist optical stereo images to achieve the expected elevation accuracy.

[0003] Using laser altimetry to improve the geometric positioning accuracy of high-resolution stereo images, it is a critical step to accurately position the laser point on each high-resolution stereo image. Domestic satellite laser altimetry systems used to obtain global elevation control points are usually equipped with laser footprint cameras. The laser footprint camera shares the optical path with the laser, is highly integrated, and has a very stable relationship. Therefore, the registration of laser altimetry data and high-resolution stereo images can be achieved with the help of laser footprint images.

[0004] At present, there are two main types of remote sensing image registration (or matching): spatial domain and frequency domain. In the spatial domain, there are mainly feature-based matching methods and region-based matching methods. Feature-based matching methods mainly extract specific image feature points (such as SIFT, etc.) and obtain matching feature point pairs (i.e., image points with the same name) through feature similarity. Feature matching can usually overcome the rotation and scaling between images, but it involves steps such as feature point extraction and matching, and false matching elimination. The processing is relatively complicated, and the matching accuracy is limited by the feature point extraction accuracy. Region-based matching calculates the similarity measure of two image regions (such as correlation coefficient, mutual information, etc.). The region matching accuracy is high, but the image resolution needs to be consistent. In addition, spatial domain matching usually relies on the grayscale information of the image and is more sensitive to the nonlinear radiation changes of the image. Frequency domain matching methods mainly determine the translation between images based on the principle that the translation in the spatial domain is equivalent to the phase shift in the frequency domain. Generally, the image resolution is required to be consistent or close, and there is no large angle rotation.

[0005] The footprint camera carried by the satellite laser altimeter is different from the traditional high-resolution stereo camera. The traditional high-resolution stereo camera uses the TDICCD linear array imaging mode, which not only has high resolution but also good radiation quality. The laser footprint camera uses a new CMOS area array sensor, and its resolution and imaging quality are not as good as the traditional high-resolution stereo camera. There is a large difference in resolution between the laser footprint image and the stereo image, and due to the different sensors used to acquire the images, the radiation difference between the laser footprint image and the remote sensing image may also be large. In addition, the available high-precision laser points are usually located in areas with scarce textures, and it is difficult for existing methods to robustly achieve the registration of the two. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a method and device for aligning laser altimetry data and high-resolution stereo images to solve the technical problem in the prior art that it is difficult to robustly align laser altimetry data and high-resolution stereo images with the help of laser footprint influence.

[0007] The technical solution proposed by the present invention is as follows:

[0008] A first aspect of an embodiment of the present invention provides a method for registering laser altimetry data and high-resolution stereo images, comprising: obtaining a laser point, a laser footprint image, and a stereo image to be registered, wherein the laser footprint image is a reference image of the laser point; correcting the reference image and the stereo image to be matched on a current layer of a pyramid based on the elevation of the laser point to obtain a corrected reference image and a corrected stereo image to be matched; calculating the geographic offset of the corrected stereo image to be matched relative to the corrected reference image using a sub-pixel phase correlation algorithm and the resolution of the current layer of the pyramid; repeating the correction and offset calculation on each layer of the pyramid until the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image is calculated, and the size of the image at each layer of the pyramid remains unchanged; and completing the positioning of the laser point on the stereo image to be registered according to the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image.

[0009] Optionally, the laser altimetry data and high-resolution stereo image registration method further includes: determining a new reference image based on the positioning result, repeating the correction and geographic offset calculation described in the first aspect of the embodiment of the present invention, and obtaining the geographic offsets of other stereo images to be matched at the bottom layer of the pyramid relative to the new reference image; and completing the positioning of the laser point on the other stereo images to be registered based on the geographic offsets of the other stereo images to be matched relative to the new reference image.

[0010] Optionally, based on the laser point elevation, the reference image and the stereo image to be matched are corrected on the current layer of the pyramid to obtain a corrected reference image and a corrected stereo image to be matched, including: calculating the resolution of the current layer of the pyramid according to the initial resolution; calculating the geographic coordinates corresponding to each pixel of the corrected image according to the geographic coordinates of the reference image and the resolution of the current layer of the pyramid; based on the laser point elevation, back-projecting the geographic coordinates corresponding to each pixel of the corrected image onto the reference image and the stereo image to be matched to obtain a corrected reference image and a corrected stereo image to be matched.

[0011] Optionally, based on the laser point elevation, the reference image and the stereo image to be matched are corrected on the current layer of the pyramid to obtain a corrected reference image and a corrected stereo image to be matched, and the method further includes: obtaining the corrected reference image coordinates and the corrected stereo image coordinates to be matched according to the RPC model; correcting the corrected stereo image coordinates to be matched according to the geographic offset of the previous layer of the pyramid to obtain the corrected stereo image coordinates to be matched; and resampling the corrected reference image coordinates and the corrected stereo image coordinates to be matched according to a bilinear interpolation algorithm to obtain a resampled reference image and a resampled stereo image to be matched.

[0012] Optionally, the geographic offset of the corrected stereo image to be matched relative to the corrected reference image is calculated using a sub-pixel phase correlation algorithm and the resolution of the current layer of the pyramid, including: calculating the offset of the corrected reference image and the corrected stereo image to be matched according to the sub-pixel phase correlation algorithm; and calculating the geographic offset of the corrected stereo image to be matched relative to the corrected reference image according to the offset and the resolution of the current layer of the pyramid.

[0013] Optionally, the offset of the corrected reference image and the corrected stereo image to be matched is calculated according to a sub-pixel phase correlation algorithm, including: performing Fourier transform on the corrected reference image and the corrected stereo image to be matched; calculating the mutual power spectrum between the reference image after Fourier transform and the stereo image to be matched; performing inverse Fourier transform on the mutual power spectrum to obtain an impulse function related to the translation amount; and calculating the offset of the corrected reference image and the corrected stereo image to be matched according to the coordinate values ​​of the impulse function.

[0014] Optionally, the offset of the corrected reference image and the corrected stereo image to be matched is calculated according to the coordinate values ​​of the pulse function, including: calculating a first translation according to the coordinate values ​​corresponding to the maximum value of the pulse function; calculating a second translation according to the coordinate values ​​corresponding to the neighborhood values ​​of the maximum value of the pulse function; and calculating the offset of the corrected reference image and the corrected stereo image to be matched according to the first translation and the second translation.

[0015] According to a second aspect of an embodiment of the present invention, there is provided a laser altimetry data and high-resolution stereo image registration device, comprising: an image acquisition module, used to acquire a laser point, a laser footprint image and a stereo image to be registered, wherein the laser footprint image is a reference image of the laser point; a correction module, used to correct the reference image and the stereo image to be matched on the current layer of a pyramid based on the elevation of the laser point, to obtain a corrected reference image and a corrected stereo image to be matched; an offset calculation module, used to calculate the geographic offset of the corrected stereo image to be matched relative to the corrected reference image by using a sub-pixel phase correlation algorithm and the resolution of the current layer of the pyramid, wherein the size of each layer of the pyramid remains unchanged; a pyramid repetition calculation module, used to repeat the correction and offset calculation on each layer of the pyramid until the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image is calculated; and a registration module, used to complete the positioning of the laser point on the stereo image to be registered according to the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image.

[0016] A third aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the laser altimetry data and high-resolution stereo image registration method as described in the first aspect of the embodiment of the present invention and any one of the first aspects.

[0017] A fourth aspect of an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the laser altimetry data and high-resolution stereo image registration method as described in the first aspect of the embodiment of the present invention and any one of the first aspects.

[0018] The technical solution provided by the present invention has the following effects:

[0019] The laser altimetry data and high-resolution stereo image registration method and device provided by the embodiment of the present invention use pyramid phase-correlated laser altimetry data to register with the high-resolution stereo image to be matched; during the registration, the high-precision laser footprint elevation obtained by laser altimetry is first used to correct the laser footprint image and the high-resolution optical stereo image to be matched, effectively overcoming the problem that phase correlation cannot be used for image registration with rotation and scaling transformations; at the same time, the phase registration is robust to radiation differences, and a pyramid strategy from coarse to fine is adopted to achieve rough offset detection in the upper layer of the pyramid and fine offset detection in the bottom layer of the pyramid. Unlike the conventional pyramid construction process in which the image size gradually increases, the pyramid used in this method keeps the image size unchanged, and reduces the ground range corresponding to the image by gradually reducing the resolution. Compared with the conventional pyramid image, which will increase the amount of calculation exponentially as the image gradually increases, the image size of this registration method remains unchanged, and the calculation efficiency is improved; moreover, the conventional pyramid image size increases, the resolution decreases, and the image ground range remains unchanged. This registration method keeps the image size unchanged, and reduces the image ground range by reducing the resolution, which can more accurately obtain the local offset, that is, taking into account both the overall large offset and the local small offset. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 is a flow chart of a method for registering laser altimetry data and high-resolution stereo images according to an embodiment of the present invention;

[0022] FIG2( a ) is a schematic diagram of an existing conventional pyramid image;

[0023] FIG2( b ) is a schematic diagram of a pyramid image according to an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of laser altimetry data and high-resolution stereo image registration results according to an embodiment of the present invention;

[0025] Figure 4 is a flow chart of a method for registering laser altimetry data and high-resolution stereo images according to another embodiment of the present invention;

[0026] Figure 5 is a schematic diagram of resampling and correcting a reference image of a laser point and a stereo image to be matched according to an embodiment of the present invention;

[0027] Figure 6 is a flow chart of a method for registering laser altimetry data and high-resolution stereo images according to another embodiment of the present invention;

[0028] Figure 7 is a schematic diagram of a phase-related pulse function peak according to an embodiment of the present invention;

[0029] Figure 8 is a structural block diagram of a laser altimetry data and high-resolution stereo image registration device according to an embodiment of the present invention;

[0030] Fig. 9 is a schematic diagram of the structure of a computer-readable storage medium provided according to an embodiment of the present invention;

[0031] Fig.10 is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0033] The embodiment of the present invention provides a method for registering laser altimetry data and high-resolution stereo images. Figure 1 As shown, the method comprises the following steps:

[0034] Step S101: acquiring a laser point, a laser footprint image and a stereo image to be registered, wherein the laser footprint image is a reference image of the laser point.

[0035] Specifically, when laser altimetry is used to improve the geometric positioning accuracy of high-resolution stereo images, it is necessary to use the laser points emitted by the satellite laser altimeter and the laser footprint images taken by the laser footprint camera. Among them, the laser pulse emitted by the satellite laser altimeter will form a laser ground spot after reaching the ground. After data processing, the three-dimensional coordinates of the ground spot are obtained, which is called a laser point. The laser point has precise geographic coordinates, which include latitude, longitude and elevation.

[0036] The laser footprint camera used will capture images around the laser ground spot to form a laser footprint image. The laser footprint image, like the optical stereo image, has geometric positioning information, such as the RPC model. The position of the laser point on the laser footprint image can be determined based on its coordinates through the RPC model of the footprint influence.

[0037] In this embodiment, the laser footprint image has high-precision geographic coordinates (lon, lat, alt), including plane coordinates and elevation. At the same time, the laser footprint image and the stereo image to be registered have geometric positioning information, such as an RPC model. Specifically, when matching the laser point to different stereo images, it is necessary to form a correction image around the laser point. During the first matching, the laser footprint image is used as the reference for correction, and the laser footprint image at this time is the reference image. If the registered image has been obtained through registration, the registered image can be used as the reference, and the reference image at this time selects the registered image, so that the advantage of the high resolution of the stereo image can be used to ensure accurate matching.

[0038] In addition to obtaining the laser point, the laser footprint image, and the stereo image to be registered, the pyramid level and the size of the matching window can also be predetermined. In one embodiment, the pyramid level N can be 3, that is, the pyramid level is 3; the matching window size (w×h) is (64×64).

[0039] Step S102: taking the laser point elevation as a reference, correcting the reference image and the stereo image to be matched on the current layer of the pyramid to obtain a corrected reference image and a corrected stereo image to be matched.

[0040] Specifically, during correction, the reference image, i.e., the ground coordinates of the laser footprint image, is taken as the center, the laser point elevation is taken as the benchmark, and the accurate elevation obtained by laser altimetry is used to perform orthorectification on the reference image and the stereo image to be matched, so that the corrected reference image and the stereo image to be matched have the same coordinate projection and resolution, thereby overcoming the influence of rotation and scaling on image phase correlation.

[0041] Step S103: Calculate the geographic offset of the corrected stereo image to be matched relative to the corrected reference image using a sub-pixel phase correlation algorithm and the resolution of the current pyramid layer.

[0042] Specifically, when calculating the geographic offset, the offset between the corrected reference image and the stereo image to be matched can be calculated based on the sub-pixel phase correlation algorithm, and then the geographic offset of the corrected stereo image to be matched relative to the corrected reference image is calculated by combining the offset and the resolution of the current layer of the pyramid.

[0043] When calculating the geographic offset, the calculation may be started from the top layer of the pyramid; thus, the current layer may be the top layer of the pyramid.

[0044] Step S104: Repeat the correction and offset calculation at each layer of the pyramid until the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image is calculated, and the size of each layer of the pyramid remains unchanged. After the geographic offset is calculated at the top layer of the pyramid, the correction and geographic offset calculation process can be repeated layer by layer downward; and when calculating the corrected stereo image to be matched at the next layer, the geographic offset calculated at the previous layer is used for correction. Thus, by proceeding layer by layer, the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image is finally calculated.

[0045] As shown in Figure 2(a) and Figure 2(b), compared with the conventional pyramid construction process, the image size gradually increases and the calculation amount increases exponentially. In the process of using the pyramid for calculation, this registration method keeps the image size unchanged and gradually reduces the image range by changing the resolution of each level of the pyramid. In this way, rough offset detection is achieved in the upper layer of the pyramid and fine offset detection is achieved in the lower layer without significantly increasing the calculation amount.

[0046] Step S105: completing the positioning of the laser point on the stereo image to be registered according to the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image.

[0047] In determining the offset at the bottom of the pyramid (ΔP N ',ΔL' N ), it can be used as the final offset. That is, the final offset (ΔP, ΔL) can be expressed as:

[0048]

[0049] Then the final position of the laser point on the stereo image to be matched is:

[0050] (sx,sy)=RPC(lon+ΔP,lat+ΔL,alt)

[0051] In this way, the precise positioning of the laser point on the stereo image to be matched can be completed, that is, the position of the laser point on the stereo image to be registered can be obtained, such as Figure 3 shown.

[0052] The laser altimetry data and high-resolution stereo image registration method provided by the embodiment of the present invention uses pyramid phase-correlated laser altimetry data to register with the high-resolution stereo image to be matched; during the registration, the high-precision laser footprint elevation obtained by laser altimetry is first used to correct the laser footprint image and the high-resolution optical stereo image to be matched, effectively overcoming the problem that phase correlation cannot be used for image registration with rotation and scaling transformations; at the same time, the phase registration is robust to radiation differences, and a pyramid strategy from coarse to fine is adopted to achieve rough offset detection in the upper layer of the pyramid and fine offset detection in the bottom layer of the pyramid. Unlike the conventional pyramid construction process in which the image size gradually increases, the pyramid used in this method keeps the image size unchanged, and reduces the ground range corresponding to the image by gradually reducing the resolution. Compared with the conventional pyramid image, which will increase the amount of calculation exponentially as the image gradually increases, the image size of this registration method remains unchanged, and the calculation efficiency is improved; moreover, the conventional pyramid image size increases, the resolution decreases, and the image ground range remains unchanged. This registration method keeps the image size unchanged, and reduces the image ground range by reducing the resolution, which can more accurately obtain the local offset, that is, taking into account both the overall large offset and the local small offset.

[0053] In one embodiment, the laser altimetry data and high-resolution stereo image registration method further includes: determining a new reference image based on the positioning result, repeating the correction and geographic offset calculation described in the above registration method, and obtaining the geographic offset of other stereo images to be matched at the bottom layer of the pyramid relative to the new reference image; and completing the positioning of the laser point on other stereo images to be matched based on the geographic offset of other stereo images to be matched relative to the new reference image.

[0054] Specifically, after the laser footprint image is used as a reference image to be matched for registration and positioning, when registration is needed again, the positioning result can be used to determine a new reference image, and then the same method can be used to match the laser point to other stereo images. Due to the low resolution of the laser footprint image, if the laser footprint image is still used as a reference image and matched to other high-resolution stereo images, the matching point accuracy of the laser footprint image on each high-resolution stereo image is limited and cannot meet the requirements of regional network adjustment. Therefore, subsequent registration needs to use the registered stereo image as a reference and register it to other stereo images so that the point registration accuracy of the laser point on each stereo image reaches the sub-pixel level.

[0055] In one embodiment, if Figure 4 and Figure 5 As shown, based on the laser point elevation, the reference image and the stereo image to be matched are corrected on the current layer of the pyramid to obtain the corrected reference image and the corrected stereo image to be matched, including the following steps:

[0056] Step S201: Calculate the resolution of the current layer of the pyramid according to the initial resolution.

[0057] Specifically, for the initial resolution, if the reference image is a laser footprint image, the initial resolution is the resolution of the laser footprint image; if the registered image is used as the reference image, since the resolution of the stereo image is equivalent, the initial resolution is 4 to 8 times lower than the resolution of the laser footprint image.

[0058] After determining the initial resolution gsd0, the resolution of the current layer of the pyramid is calculated using the following formula:

[0059] gsd n =gsd0×Z n-1

[0060] Wherein, Z is the pyramid zoom factor, which can generally be 1 / 2, and n represents the level of the current layer of the pyramid, that is, n=1,...,N.

[0061] Since the correction starts from the top layer of the pyramid, that is, the current layer n of the pyramid is 1, the resolution of the current layer of the pyramid is the initial resolution; if the reference image is a laser footprint image, the resolution of the current layer of the pyramid is the ground resolution of the laser footprint image.

[0062] Step S202: Calculate the geographic coordinates corresponding to each pixel of the corrected image according to the geographic coordinates of the reference image and the resolution of the current layer of the pyramid.

[0063] When calculating geographic coordinates, the laser footprint ground coordinates (lon, lat, alt) are used as the center and the following formula is used:

[0064]

[0065] Among them, L i , P j are the longitude and latitude coordinates corresponding to row i and column j of the image respectively.

[0066] Step S203: Based on the laser point elevation, the geographic coordinates corresponding to each pixel of the corrected image are back-projected onto the reference image and the stereo image to be matched, so as to obtain a corrected reference image and a corrected stereo image to be matched.

[0067] Specifically, during correction, according to the geometric positioning model of the image (generally the RPC model), the ground coordinates (P j ,L i,H) back-projected onto the reference image and the stereo image to be matched. This process actually uses the accurate elevation obtained by laser altimetry to orthorectify the reference image and the stereo image to be matched, so that the corrected reference image and the stereo image to be matched have the same coordinate projection and resolution, thereby overcoming the influence of rotation and scaling on image phase correlation.

[0068] Step S204: Obtain the coordinates of the corrected reference image and the coordinates of the corrected stereo image to be matched according to the RPC model. For the corrected image, since the matching window is set small, the corresponding ground range is valid, and the registered laser points are usually on flat terrain, the influence of the projection difference caused by the terrain undulation on the phase correlation can be ignored. Therefore, the RPC model can be used to obtain the image coordinates.

[0069] Assume that the image coordinates after back-projection of (L, P, H) are (x, y), and the calculation formula is as follows:

[0070]

[0071] Among them, LAT_OFF, LAT_SCALE, LON_OFF, LON_SCALE, HEIGHT_OFF, HEIGHT_SCALE, SAMP_OFF, SAMP_SCALE, LINE_OFF, LINE_SCALE, a t 、b t 、c t d t (t=0,1,…,19) are all known RPC model parameters, i,j,k=0,1,2, s and l are intermediate variables (normalized coordinate values) in the RPC calculation process.

[0072] For ease of expression, the above RPC model calculation is uniformly expressed as the following formula:

[0073] (x,y)=RPC(P,L,H)

[0074] Therefore, the ground coordinates (L i ,P j ,H) corresponds to the reference image coordinates (px,py), then:

[0075] (px,py)=RPC(P i ,L j ,H)

[0076] Step S205: correcting the corrected coordinates of the to-be-matched stereo image according to the geographic offset of the previous layer of the pyramid to obtain corrected coordinates of the to-be-matched stereo image.

[0077] For the corrected stereo image to be matched, it is necessary to use the geographic offset calculated by the previous layer of the pyramid for correction. If the current layer is the top layer of the pyramid, the geographic offset value is 0. Therefore, the coordinates of the corrected stereo image to be matched are expressed by the following formula:

[0078] (px,py)=RPC(P i +ΔP n ,L j +ΔL n ,H)

[0079] Among them, ΔL n and ΔP n They are respectively the latitude and longitude offsets of the stereo images to be registered relative to the laser points calculated in the previous layer.

[0080] Step S206: Resample the corrected reference image coordinates and the corrected stereo image coordinates to be matched according to the bilinear difference algorithm to obtain a resampled reference image and a resampled stereo image to be matched. Since (px, py) after back-projection is generally not an integer, bilinear interpolation is used to obtain the pixel value f(i, j) of the pixel (i, j). The bilinear interpolation calculation formula is as follows:

[0081]

[0082] in, is a rounding operation, (ix, iy) is the integer pixel coordinate corresponding to (px, py) after back-projection, (dx, dy) is the decimal part, f is the resampled image, and I is the laser point reference image or the stereo image to be matched. After resampling the reference image and the stereo image to be registered, they have the same resolution and coordinate projection, which eliminates the rotation and scaling between the images.

[0083] In one embodiment, if Figure 6 As shown, the geographic offset of the corrected stereo image to be matched relative to the corrected reference image is calculated using the sub-pixel phase correlation algorithm and the current layer resolution of the pyramid, including the following steps:

[0084] Step S301: Calculate the offset of the corrected reference image and the corrected stereo image to be matched according to the sub-pixel phase correlation algorithm. Through the above steps S201 to S206, the resampled reference image and the stereo image to be matched are obtained after the correction and resampling steps. The resampled reference image and the stereo image to be matched are divided into f(x, y) and g(x, y), and then Fourier transformed respectively to obtain the Fourier transformed reference image and the stereo image to be matched F(u, v) and G(u, v), which are expressed by the following formula:

[0085] F(u,v)=Fourier(f(x,y))

[0086] G(u,v)=Fourier(g(x,y))

[0087] Then the transformation relationship between F(u,v) and G(u,v) is expressed as:

[0088] The phase of the cross power spectrum of the Fourier transform of the image is equal to the phase difference between the original images. By inversely transforming it, we can get the impulse function of (x-x0, y-y0) in the (x, y) space, and it has an obvious peak at (x0, y0), such as Figure 7 As shown, the translation amount between the original images f(x, y) and g(x, y) can be obtained by detecting the peak value.

[0089] Specifically, the cross power spectrum between the reference image after Fourier transformation and the stereo image to be matched is calculated according to the cross power spectrum calculation formula, that is, the cross power spectrum is expressed as:

[0090]

[0091] Among them G * is the complex conjugate of G.

[0092] Perform inverse Fourier transform on the cross power spectrum to obtain the impulse function m(u,v) about the translation (x0,y0):

[0093] m(x,y)=Fourier -1 (M(u,v))

[0094] After obtaining the impulse function, find the coordinate value corresponding to the maximum value of the impulse function, which is the integer translation (x0, y0) between the original images f(x, y) and g(x, y), or the first translation in the offset.

[0095] (x0,y0)=argmax(m(x,y))

[0096] The above phase correlation method obtains the integer pixel translation between images, which usually does not meet the requirements of high-precision registration. The value of the pulse peak neighborhood is obtained, and the high-precision translation is calculated by weighted calculation to obtain the decimal part of the offset (dx0, dy0), or the second translation in the offset:

[0097]

[0098] The values ​​of i and j are in the 3×3 or 5×5 neighborhood of (x0, y0).

[0099] Thus, the precise offset (x′, y′) between the original images f(x, y) and g(x, y) is calculated by the above first translation and second translation:

[0100]

[0101] Step S302: Calculate the geographic offset of the corrected stereo image to be matched relative to the corrected reference image based on the offset and the resolution of the current layer of the pyramid. After obtaining the offset between the two, the geographic offset of the corrected stereo image to be matched relative to the corrected reference image can be calculated in combination with the resolution of the current layer of the pyramid:

[0102]

[0103] Where, ΔL n and ΔP n They are respectively the latitude and longitude offsets of the stereo images to be registered relative to the laser points calculated in the previous layer.

[0104] After the geographic offset of the corrected stereo image to be matched relative to the corrected reference image is calculated by using the above steps S201 to S206 and steps S301 to S302 at the top layer of the pyramid, it is possible to move to the next layer and calculate the geographic offset again by using the above steps S201 to S206 and steps S301 to S302; wherein the initial offset value of the stereo image to be matched relative to the reference image at the next layer is the offset after refinement of the previous layer, that is:

[0105]

[0106] Then repeat the above steps at each level until you reach the bottom level (i.e., level N) of the pyramid, and you can get the final geographic offset (ΔP, ΔL) of the stereo image relative to the laser point.

[0107] The laser altimetry data and high-resolution stereo image registration method provided by the embodiment of the present invention utilizes the pyramid phase-correlated laser footprint and high-resolution stereo image registration, which can be applied to the fields of satellite surveying and mapping and remote sensing to achieve high-precision registration of satellite laser altimetry data and stereo images, thereby laying a foundation for achieving laser altimetry data-assisted improvement of stereo image mapping accuracy.

[0108] The embodiment of the present invention also provides a laser altimetry data and high-resolution stereo image registration device, such as Figure 8 As shown, the device comprises:

[0109] An image acquisition module, used to acquire a laser point, a laser footprint image and a stereo image to be registered, wherein the laser footprint image is a reference image of the laser point;

[0110] The correction module is used to correct the reference image and the stereo image to be matched on the current layer of the pyramid based on the laser point elevation to obtain a corrected reference image and a corrected stereo image to be matched; the specific content refers to the corresponding part of the above method embodiment, which will not be repeated here.

[0111] The offset calculation module is used to calculate the geographic offset of the corrected stereo image to be matched relative to the corrected reference image by using the sub-pixel phase correlation algorithm and the resolution of the current layer of the pyramid, and the size of each layer of the pyramid remains unchanged; the specific content refers to the corresponding part of the above method embodiment, which will not be repeated here.

[0112] The pyramid recalculation module is used to repeat the correction and offset calculation at each layer of the pyramid until the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image is calculated; for specific contents, please refer to the corresponding part of the above method embodiment, which will not be repeated here.

[0113] The registration module is used to complete the positioning of the laser point on the stereo image to be registered according to the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image. For details, please refer to the corresponding part of the above method embodiment, which will not be repeated here.

[0114] The laser altimetry data and high-resolution stereo image registration device provided by the embodiment of the present invention uses pyramid phase-correlated laser altimetry data to register with the high-resolution stereo image to be matched; during the registration, the high-precision laser footprint elevation obtained by laser altimetry is first used to correct the laser footprint image and the high-resolution optical stereo image to be matched, effectively overcoming the problem that phase correlation cannot be used for image registration with rotation and scaling transformations; at the same time, the phase registration is robust to radiation differences, and a pyramid strategy from coarse to fine is adopted to achieve rougher offset detection in the upper layer of the pyramid and fine offset detection in the bottom layer of the pyramid. Unlike the conventional pyramid construction process in which the image size gradually increases, the pyramid used by the device keeps the image size unchanged, and reduces the ground range corresponding to the image by gradually reducing the resolution. Compared with the conventional pyramid image, which gradually increases the amount of calculation, the image size of the registration device remains unchanged, and the calculation efficiency is improved; moreover, the conventional pyramid image size increases, the resolution decreases, and the image ground range remains unchanged. The registration device keeps the image size unchanged, reduces the image ground range by reducing the resolution, and can more accurately obtain the local offset, that is, taking into account both the overall large offset and the local small offset.

[0115] For a detailed description of the functions of the laser altimetry data and high-resolution stereoscopic image registration device provided in an embodiment of the present invention, please refer to the description of the laser altimetry data and high-resolution stereoscopic image registration method in the above embodiment.

[0116] The embodiment of the present invention also provides a storage medium, such as Fig. 9 As shown, a computer program 601 is stored thereon, and when the instruction is executed by the processor, the steps of the laser altimetry data and high-resolution stereoscopic image registration method in the above embodiment are implemented. The storage medium also stores audio and video stream data, feature frame data, interaction request signaling, encrypted data, and preset data size, etc. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0117] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0118] The embodiment of the present invention further provides an electronic device, such as Fig.10 As shown, the electronic device may include a processor 51 and a memory 52, wherein the processor 51 and the memory 52 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0119] The processor 51 may be a central processing unit (CPU). The processor 51 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0120] The memory 52 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the corresponding program instructions / modules in the embodiments of the present invention. The processor 51 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 52, that is, to implement the laser height measurement data and high-resolution stereo image registration method in the above method embodiment.

[0121] The memory 52 may include a program storage area and a data storage area, wherein the program storage area may store an application required for operating the device and at least one function; the data storage area may store data created by the processor 51, etc. In addition, the memory 52 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 52 may optionally include a memory remotely arranged relative to the processor 51, and these remote memories may be connected to the processor 51 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0122] The one or more modules are stored in the memory 52 and when executed by the processor 51, perform the following steps: Figure 1 -7 shows a laser altimetry data and a high-resolution stereo image registration method in the embodiment.

[0123] For details of the above electronic equipment, please refer to Figures 1 to 7 The corresponding related descriptions and effects in the illustrated embodiments can be understood and will not be repeated here.

[0124] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for registering laser altimetry data and high-resolution stereo images, characterized in that: include: Acquire a laser point, a laser footprint image, and a stereo image to be registered, wherein the laser footprint image is a reference image of the laser point; Based on the laser point elevation, orthorectify the reference image and the stereo image to be matched on the current layer of the pyramid to obtain a corrected reference image and a corrected stereo image to be matched; The geographic offset of the corrected stereo image to be matched relative to the corrected reference image is calculated using the sub-pixel phase correlation algorithm and the current layer resolution of the pyramid; Repeating the correction and offset calculation at each layer of the pyramid until the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image is calculated, and the size of the image at each layer of the pyramid remains unchanged; The positioning of the laser point on the stereo image to be registered is completed according to the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image; Based on the laser point elevation, the reference image and the stereo image to be matched are orthorectified on the current layer of the pyramid to obtain a corrected reference image and a corrected stereo image to be matched, including: Calculate the resolution of the current layer of the pyramid based on the initial resolution; Calculate the geographic coordinates corresponding to each pixel of the corrected image according to the geographic coordinates of the laser point and the resolution of the current layer of the pyramid, the corrected image is an orthorectified image formed around the laser point when the laser point is matched to different stereo images, and the correction is performed based on the laser footprint image during the first matching; Based on the laser point elevation, the geographic coordinates corresponding to each pixel of the corrected image are back-projected onto the reference image and the stereo image to be matched, so as to obtain the corrected reference image and the corrected stereo image to be matched; Based on the laser point elevation, the reference image and the stereo image to be matched are corrected on the current layer of the pyramid to obtain a corrected reference image and a corrected stereo image to be matched, and further comprising: According to the RPC model, the coordinates of the corrected reference image and the coordinates of the corrected stereo image to be matched are obtained; The corrected coordinates of the to-be-matched stereo image are corrected according to the geographic offset of the upper layer of the pyramid to obtain the corrected coordinates of the to-be-matched stereo image; Resampling the coordinates of the corrected reference image and the corrected stereo image to be matched according to a bilinear difference algorithm to obtain a resampled reference image and a resampled stereo image to be matched as the corrected reference image and the corrected stereo image to be matched; The geographic offset of the corrected stereo image to be matched relative to the corrected reference image is calculated using the sub-pixel phase correlation algorithm and the current pyramid resolution, including: Calculate the offset of the corrected reference image and the corrected stereo image to be matched according to the sub-pixel phase correlation algorithm; The geographic offset of the corrected stereo image to be matched relative to the corrected reference image is calculated according to the offset and the resolution of the current layer of the pyramid.

2. The laser altimetry data and high-resolution stereo image registration method according to claim 1, characterized in that: Also includes: Determine a new reference image based on the positioning result, repeat the correction and geographic offset calculation described in claim 1, and obtain the geographic offset of other stereo images to be matched at the bottom layer of the pyramid relative to the new reference image; The positioning of the laser point on the other stereo images to be registered is completed according to the geographic offset of the other stereo images to be matched relative to the new reference image.

3. The laser altimetry data and high-resolution stereo image registration method according to claim 1, characterized in that: The offset of the corrected reference image and the corrected stereo image to be matched is calculated according to the sub-pixel phase correlation algorithm, including: Performing Fourier transform on the corrected reference image and the corrected stereo image to be matched; Calculate the cross power spectrum between the Fourier transformed reference image and the stereo image to be matched; Performing inverse Fourier transform on the cross power spectrum to obtain an impulse function related to the translation amount; The offset of the corrected reference image and the corrected stereo image to be matched is calculated according to the pulse function coordinate value.

4. The laser altimetry data and high-resolution stereo image registration method according to claim 3, characterized in that: Calculating the offset of the corrected reference image and the corrected stereo image to be matched according to the pulse function coordinate value, including: A first translation amount is calculated according to the coordinate value corresponding to the maximum value of the pulse function; A second translation amount is calculated according to the coordinate value corresponding to the neighborhood value of the maximum value of the pulse function; The offsets of the corrected reference image and the corrected stereo image to be matched are calculated according to the first translation amount and the second translation amount.

5. A laser altimetry data and high-resolution stereo image registration device, characterized in that: include: An image acquisition module, used to acquire a laser point, a laser footprint image and a stereo image to be registered, wherein the laser footprint image is a reference image of the laser point; A correction module is used to perform ortho-correction on the reference image and the stereo image to be matched on the current layer of the pyramid based on the laser point elevation to obtain a corrected reference image and a corrected stereo image to be matched; An offset calculation module is used to calculate the geographic offset of the corrected stereo image to be matched relative to the corrected reference image by using a sub-pixel phase correlation algorithm and the resolution of the current layer of the pyramid, wherein the size of each layer of the pyramid remains unchanged; A pyramid recalculation module is used to repeat the correction and offset calculation at each layer of the pyramid until the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image is calculated; A registration module, used for completing the positioning of the laser point on the stereo image to be registered according to the geographic offset of the stereo image to be matched at the bottom layer of the pyramid relative to the reference image; Based on the laser point elevation, the reference image and the stereo image to be matched are orthorectified on the current layer of the pyramid to obtain a corrected reference image and a corrected stereo image to be matched, including: Calculate the resolution of the current layer of the pyramid based on the initial resolution; Calculate the geographic coordinates corresponding to each pixel of the corrected image according to the geographic coordinates of the laser point and the resolution of the current layer of the pyramid, the corrected image is an orthorectified image formed around the laser point when the laser point is matched to different stereo images, and the correction is performed based on the laser footprint image during the first matching; Based on the laser point elevation, the geographic coordinates corresponding to each pixel of the corrected image are back-projected onto the reference image and the stereo image to be matched, so as to obtain the corrected reference image and the corrected stereo image to be matched; Based on the laser point elevation, the reference image and the stereo image to be matched are corrected on the current layer of the pyramid to obtain a corrected reference image and a corrected stereo image to be matched, and further comprising: According to the RPC model, the coordinates of the corrected reference image and the coordinates of the corrected stereo image to be matched are obtained; The corrected coordinates of the to-be-matched stereo image are corrected according to the geographic offset of the upper layer of the pyramid to obtain the corrected coordinates of the to-be-matched stereo image; Resampling the coordinates of the corrected reference image and the corrected stereo image to be matched according to a bilinear difference algorithm to obtain a resampled reference image and a resampled stereo image to be matched as the corrected reference image and the corrected stereo image to be matched; The geographic offset of the corrected stereo image to be matched relative to the corrected reference image is calculated using the sub-pixel phase correlation algorithm and the current pyramid resolution, including: Calculate the offset of the corrected reference image and the corrected stereo image to be matched according to the sub-pixel phase correlation algorithm; The geographic offset of the corrected stereo image to be matched relative to the corrected reference image is calculated according to the offset and the resolution of the current layer of the pyramid.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the laser altimetry data and high-resolution stereo image registration method as described in any one of claims 1 to 4.

7. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the laser altimetry data and high-resolution stereo image registration method as described in any one of claims 1 to 4 by executing the computer instructions.

Images