Multi-point outdoor consistent scale depth estimation method and device based on remote sensing image

Abstract

This application discloses a multi-point outdoor consistent-scale depth estimation method and apparatus based on remote sensing images. The method includes: acquiring a remote sensing image of a target area; acquiring a first panoramic ground image under a cylindrical projection of the target area based on at least one ground point in the target area, and performing cube projection to obtain a second panoramic ground image under the cube projection; extracting features from the first panoramic ground image, the second panoramic ground image, and the remote sensing image to determine the relative positioning of the first panoramic ground image and the remote sensing image; constructing a multi-point scale constraint mask for the remote sensing image; predicting the uniform scale coefficient of the first panoramic ground image; acquiring an initial depth map of the first panoramic ground image; correcting the scale of the initial depth map to obtain a multi-point outdoor consistent-scale depth estimation result. This solves the problem in related technologies where the large scale differences in the predicted depth of a uniform object at different locations reduce the accuracy of depth estimation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for estimating multi-point outdoor consistent scale depth based on remote sensing images. Background Technology

[0002] In related technologies, outdoor depth estimation for a single point is usually achieved based on deep learning methods. This can learn and estimate the depth information corresponding to the current single point by learning each pixel and its context within the image.

[0003] However, due to the significant differences in the contextual information within images obtained from different image captures, the predicted depth of the same object varies considerably from location to location, reducing the accuracy of depth estimation and failing to meet the needs of downstream applications based on depth data. This issue urgently needs to be addressed. Summary of the Invention

[0004] This application provides a multi-point outdoor consistent scale depth estimation method and apparatus based on remote sensing images to solve the problem in related technologies where the large differences in the contextual information within images obtained from different images lead to large scale differences in the predicted depth of the same object at different locations, thus reducing the accuracy of depth estimation.

[0005] The first aspect of this application provides a method for estimating multi-point outdoor consistent scale depth based on remote sensing images, comprising the following steps: acquiring a remote sensing image of a target area; acquiring a first ground panoramic image under a cylindrical projection of the target area based on at least one ground point in the target area; performing cube projection on the first ground panoramic image to obtain a second ground panoramic image under the cube projection; extracting at least one feature from the first ground panoramic image, the second ground panoramic image, and the remote sensing image to obtain at least one feature vector; determining the relative positioning of the first ground panoramic image and the remote sensing image based on the at least one feature vector; constructing a multi-point scale constraint mask of the remote sensing image based on the relative positioning; predicting the uniform scale coefficient of the first ground panoramic image; acquiring an initial depth map of the first ground panoramic image; correcting the scale of the initial depth map based on the multi-point scale constraint mask and the uniform scale coefficient to obtain a multi-point outdoor consistent scale depth map, and obtaining the multi-point outdoor consistent scale depth estimation result.

[0006] Optionally, in one embodiment of this application, acquiring the remote sensing image of the target area, based on at least one ground point in the target area, and collecting a first ground panoramic image under the projection of the target cylinder, includes: selecting the target area, collecting the first ground panoramic image from the at least one ground point in the target area; and selecting a remote sensing image covering the first ground panoramic image from the remote sensing data according to the geographical location information of the target area.

[0007] Optionally, in one embodiment of this application, the step of performing cube projection on the first ground panoramic image to obtain a second ground panoramic image under the cube projection, extracting at least one feature from the first ground panoramic image, the second ground panoramic image, and the remote sensing image to obtain at least one feature vector, and obtaining the relative positioning of the first ground panoramic image and the remote sensing image based on the at least one feature vector, includes: performing cube projection transformation on the first ground panoramic image to obtain a second ground panoramic image under the cube projection; extracting at least one feature from the first ground panoramic image, the second ground panoramic image, and the remote sensing image to obtain a first feature vector of the first ground panoramic image, a second feature vector of the second ground panoramic image, and a third feature vector of the remote sensing image, respectively; concatenating the first feature vector and the second feature vector to obtain a fourth feature vector; concatenating the fourth feature vector and the third feature vector to obtain a fifth feature vector; obtaining a feature vector group based on the fifth feature vector; and adjusting the feature dimensions of the feature vector group using a target multilayer perceptron (MLP) to obtain the position of the first ground panoramic image relative to the center of the remote sensing image.

[0008] Optionally, in one embodiment of this application, the step of constructing a multi-point scale constraint mask for the remote sensing image based on the relative positioning and predicting the uniform scale coefficient of the first ground panoramic image includes: calculating the Euclidean distance between at least one binary number in a preset binary array and a target binary number; obtaining a parameter matrix through the Euclidean distance; using the parameter matrix to obtain the cross-pixel distance between the first ground panoramic image and the target ground panoramic image from the remote sensing perspective; converting the cross-pixel distance matrix from the remote sensing perspective into a cross-ground distance matrix according to the ground sampling distance (GSD) of the remote sensing image; calculating the average value of the distance between the first ground panoramic image and the target ground panoramic image in the cross-ground distance matrix to obtain the depth prediction range coefficient of the at least one ground point.

[0009] Optionally, in one embodiment of this application, the step of correcting the scale of the initial depth map based on the multi-point scale constraint mask and the target scale coefficient to obtain a multi-point outdoor consistent scale depth map includes: obtaining a relatively proportional cross-ground distance matrix based on each preset distance in the cross-ground distance matrix and the corresponding value in the cross-ground distance vector; obtaining the multi-point scale mask based on the relatively proportional cross-ground distance matrix; performing scale correction on the initial depth map based on the multi-point scale mask to obtain a process depth image; and performing channel conversion on the process depth image based on the image generation network to obtain a finally corrected consistent scale depth image.

[0010] A second aspect of this application provides a multi-point outdoor consistent scale depth estimation device based on remote sensing images, comprising: an acquisition module for acquiring remote sensing images of a target area, and acquiring a first ground panoramic image under a target cylindrical projection based on at least one ground point in the target area; a determination module for performing cube projection on the first ground panoramic image to obtain a second ground panoramic image under the cube projection, extracting at least one feature from the first ground panoramic image, the second ground panoramic image, and the remote sensing image to obtain at least one feature vector, and determining the relative position of the first ground panoramic image and the remote sensing image based on the at least one feature vector; a construction module for constructing a multi-point scale constraint mask of the remote sensing image based on the relative position, and predicting a uniform scale coefficient of the first ground panoramic image; an acquisition module for acquiring an initial depth map of the first ground panoramic image; and a correction module for correcting the scale of the initial depth map based on the multi-point scale constraint mask and the uniform scale coefficient to obtain a multi-point outdoor consistent scale depth map, thereby obtaining the multi-point outdoor consistent scale depth estimation result.

[0011] Optionally, in one embodiment of this application, the acquisition module includes: an acquisition unit, configured to select the target area and acquire the first ground panoramic image from the at least one ground point within the target area; and a selection unit, configured to select a remote sensing image covering the first ground panoramic image from remote sensing data based on the geographical location information of the target area.

[0012] Optionally, in one embodiment of this application, the determining module includes: a first acquisition unit, configured to perform cube projection transformation on the first ground panoramic image to obtain a second ground panoramic image under the cube projection; a second acquisition unit, configured to extract at least one feature from the first ground panoramic image, the second ground panoramic image, and the remote sensing image, respectively obtaining a first feature vector of the first ground panoramic image, a second feature vector of the second ground panoramic image, and a third feature vector of the remote sensing image; a first determining unit, configured to concatenate the first feature vector and the second feature vector to obtain a fourth feature vector, concatenate the fourth feature vector and the third feature vector to obtain a fifth feature vector, and obtain a feature vector group based on the fifth feature vector; and a second determining unit, configured to adjust the feature dimensions of the feature vector group using a target multilayer perceptron (MLP) to obtain the position of the first ground panoramic image relative to the center of the remote sensing image.

[0013] Optionally, in one embodiment of this application, the construction module includes: a calculation unit, configured to calculate the Euclidean distance between at least one binary number in a preset binary array and a target binary number, obtain a parameter matrix through the Euclidean distance, and use the parameter matrix to obtain the cross-pixel distance between the first ground panoramic image and the target ground panoramic image under the remote sensing view; a conversion unit, configured to convert the cross-pixel distance matrix under the remote sensing view into a cross-ground distance matrix according to the ground sampling distance (GSD) of the remote sensing image; and a third determination unit, configured to calculate the average value of the distance between the first ground panoramic image and the target ground panoramic image in the cross-ground distance matrix, and obtain the depth prediction range coefficient of the at least one ground point.

[0014] Optionally, in one embodiment of this application, the correction module includes: a fourth determining unit, configured to obtain a relatively proportional cross-ground distance matrix based on each preset distance in the cross-ground distance matrix and the corresponding value in the cross-ground distance vector, and to obtain the multi-point scale mask based on the relatively proportional cross-ground distance matrix; a correction unit, configured to perform scale correction on the initial depth map based on the multi-point scale mask to obtain a process depth image; and a fifth determining unit, configured to perform channel conversion on the process depth image based on the image generation network to obtain a final corrected consistent scale depth image.

[0015] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-point outdoor consistent scale depth estimation method based on remote sensing images as described in the above embodiments.

[0016] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multi-point outdoor consistent scale depth estimation method based on remote sensing images.

[0017] This application embodiment can acquire remote sensing images of a target area, collect a first panoramic ground image under the projection of a target cylinder based on at least one ground point in the target area, and perform cube projection to obtain a second panoramic ground image under the cube projection. Features are extracted from the first panoramic ground image, the second panoramic ground image, and the remote sensing image. The relative positioning of the first panoramic ground image and the remote sensing image is determined based on the obtained feature vectors. A multi-point scale constraint mask for the remote sensing image is constructed, and a uniform scale coefficient of the first panoramic ground image is predicted. An initial depth map of the first panoramic ground image is obtained, and the scale of the initial depth map is corrected to obtain a multi-point outdoor consistent scale depth map, thereby effectively improving the accuracy of the scale depth estimation results. This solves the problem in related technologies where the large differences in contextual information within images acquired from different image captures lead to significant scale differences in the predicted depth of the same object at different locations, reducing the accuracy of depth estimation.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0019] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0020] Figure 1 This is a flowchart of a multi-point outdoor consistent scale depth estimation method based on remote sensing images provided in an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of the structure of a multi-point outdoor consistent scale depth estimation device based on remote sensing images according to an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0024] The following describes a method and apparatus for multi-point outdoor consistent-scale depth estimation based on remote sensing images, according to embodiments of this application, with reference to the accompanying drawings. Addressing the problem mentioned in the background art that significant differences in contextual information within images acquired from different images lead to large scale differences in the predicted depth of the same object at different locations, reducing the accuracy of depth estimation, this application provides a method for multi-point outdoor consistent-scale depth estimation based on remote sensing images. In this method, a remote sensing image of the target area is acquired; a first panoramic ground image under the projection of a target cylinder is collected based on at least one ground point in the target area; and a cube projection is performed to obtain a second panoramic ground image under the cube projection. Features are extracted from the first panoramic ground image, the second panoramic ground image, and the remote sensing image; the relative positioning of the first panoramic ground image and the remote sensing image is determined based on the obtained feature vectors; a multi-point scale constraint mask of the remote sensing image is constructed; the consistent scale coefficient of the first panoramic ground image is predicted; an initial depth map of the first panoramic ground image is obtained; and the scale of the initial depth map is corrected to obtain a multi-point outdoor consistent-scale depth map, thereby effectively improving the accuracy of the scale depth estimation results. This solves the problem in related technologies where the large differences in contextual information within images obtained from different image captures lead to significant scale differences in the predicted depth of the same object at different locations, thus reducing the accuracy of depth estimation.

[0025] Specifically, Figure 1 This is a flowchart illustrating a multi-point outdoor consistent scale depth estimation method based on remote sensing images, provided in an embodiment of this application.

[0026] like Figure 1 As shown, this multi-point outdoor consistent-scale depth estimation method based on remote sensing images includes the following steps:

[0027] In step S101, a remote sensing image of the target area is acquired, and a first panoramic ground image under the projection of the target cylinder is acquired based on at least one ground point in the target area.

[0028] It is understood that the embodiments of this application can obtain remote sensing images of the target area in the following steps, and acquire a first panoramic ground image under the projection of the target cylinder based on at least one ground point in the target area, thereby effectively improving the feasibility of scale depth estimation.

[0029] In one embodiment of this application, acquiring a remote sensing image of a target area and collecting a first ground panoramic image under the projection of a target cylinder based on at least one ground point in the target area includes: selecting a target area and collecting a first ground panoramic image at at least one ground point within the target area; and selecting a remote sensing image covering the first ground panoramic image from the remote sensing data based on the geographical location information of the target area.

[0030] In actual implementation, embodiments of this application can select an area for depth prediction, and within the selected area, use a panoramic camera to acquire first panoramic ground images under cylindrical projection at several locations, which can be represented as I1, I2, ..., I n Where n represents the number of first ground panoramic images acquired. In addition, remote sensing images that can cover all first ground panoramic images can be selected from the remote sensing data based on the geographical location information of the selected area. The remote sensing images are represented as S, and the ground sampling distance GSD parameter of the remote sensing camera is recorded, which effectively improves the feasibility of scale depth estimation.

[0031] In step S102, the first ground panoramic image is cube-projected to obtain a second ground panoramic image under cube projection. At least one feature is extracted from the first ground panoramic image, the second ground panoramic image, and the remote sensing image to obtain at least one feature vector. The relative positioning of the first ground panoramic image and the remote sensing image is determined based on the at least one feature vector.

[0032] It is understood that, in the embodiments of this application, the first ground panoramic image in the following steps can be cube-projected to obtain a second ground panoramic image under cube projection, at least one feature of the first ground panoramic image, the second ground panoramic image and the remote sensing image can be extracted to obtain at least one feature vector, and the relative positioning of the first ground panoramic image and the remote sensing image can be determined based on at least one feature vector, effectively avoiding large scale differences in the depth predicted for the same object at different locations.

[0033] In one embodiment of this application, a first ground panoramic image is cube-projected to obtain a second ground panoramic image under cube projection. At least one feature is extracted from the first ground panoramic image, the second ground panoramic image, and the remote sensing image to obtain at least one feature vector. The relative positioning of the first ground panoramic image and the remote sensing image is obtained based on the at least one feature vector. This includes: performing cube projection transformation on the first ground panoramic image to obtain a second ground panoramic image under cube projection; extracting at least one feature from the first ground panoramic image, the second ground panoramic image, and the remote sensing image to obtain a first feature vector of the first ground panoramic image, a second feature vector of the second ground panoramic image, and a third feature vector of the remote sensing image; concatenating the first feature vector and the second feature vector to obtain a fourth feature vector; concatenating the fourth feature vector and the third feature vector to obtain a fifth feature vector; obtaining a feature vector group based on the fifth feature vector; and adjusting the feature dimensions of the feature vector group using a target multilayer perceptron (MLP) to obtain the position of the first ground panoramic image relative to the center of the remote sensing image.

[0034] As one possible implementation, embodiments of this application can perform cube projection on the first ground panoramic image to obtain a second ground panoramic image under cube projection, which can be represented as C1, C2, ..., C n The second ground panoramic image can be represented by six monocular images, representing image information in six directions: front, back, left, right, top, and bottom.

[0035] Next, the second ground panoramic image C1, C2, ..., C can be used. n This process yields 6n feature vectors of size 1×C, where n represents the number of second ground panoramic images. Each second ground panoramic image yields 6 feature vectors. Then, these 6 feature vectors are output from each second ground panoramic image and averaged to finally obtain n first feature vectors of size 1×C, which can be represented as F. C1 ,F C2 ,...,F Cn , where 1×C means that the number of elements on the vertical axis of each feature vector is 1, and the number of elements on the horizontal axis is C.

[0036] Secondly, the obtained first ground panoramic images I1,I2,...,I ​​can be processed... n By performing feature extraction, we can obtain n second feature vectors of size 1×C, which can be represented as F. E1 ,F E2 ,...,F En Furthermore, the remote sensing image S can be used for feature extraction to obtain a third feature vector of size 1×C, which can be represented as F. S .

[0037] Furthermore, embodiments of this application can utilize the features F of each first ground panoramic image. E1 ,F E2 ,...,F En and the features F of the second ground panoramic image C1 ,F C2 ,...,F Cn Concatenate the first n fourth feature vectors of size 1×2C to obtain the second n fourth feature vectors of size 1×2C. Then concatenate the second n fourth feature vectors of size 1×2C with the third feature vector of the remote sensing image to obtain the third feature vector of size 1×3C. Finally, obtain the third feature vector of size 1×3C based on the third feature vector.

[0038] Next, the feature vector group n×3C is gradually reduced to n×C and n×2 through two MLPs. Each binary number represents the position of the first ground panoramic image relative to the center of the remote sensing image. The binary array n×2 composed of n binary numbers can be represented as L0.

[0039] It should be noted that feature extraction is set by those skilled in the art according to the actual situation, and no specific limitations are made here.

[0040] In step S103, a multi-point scale constraint mask for the remote sensing image is constructed based on relative positioning to predict the uniform scale coefficient of the first ground panoramic image.

[0041] It is understood that the embodiments of this application can construct a multi-point scale constraint mask for remote sensing images based on the relative positioning in the following steps, and predict the uniform scale coefficient of the first ground panoramic image, thereby effectively improving the positioning accuracy of outdoor images.

[0042] In one embodiment of this application, constructing a multi-point scale constraint mask for a remote sensing image based on relative positioning and predicting the uniform scale coefficient of a first ground panoramic image includes: calculating the Euclidean distance between at least one binary number in a preset binary array and a target binary number; obtaining a parameter matrix through the Euclidean distance; using the parameter matrix to obtain the cross-pixel distance between the first ground panoramic image and the target ground panoramic image from the remote sensing perspective; converting the cross-pixel distance matrix from the remote sensing perspective into a cross-ground distance matrix according to the ground sampling distance (GSD) of the remote sensing image; and calculating the average value of the distance between the first ground panoramic image and the target ground panoramic image in the cross-ground distance matrix to obtain the depth prediction range coefficient of at least one ground point.

[0043] In some embodiments, this application embodiment can calculate the binary array L0 of relative positions and perform cross distance calculation. Specifically, for any binary number in the binary array L, the Euclidean distance of that binary number to the other n-1 binary numbers is calculated, which is the number of pixels, and finally a parameter matrix L of size n×(n-1) is obtained. S , where each parameter represents the predicted distance between the first ground panoramic image and the target ground panoramic image, i.e., the pixel distance from the remote sensing perspective.

[0044] Next, based on the GSD of the remote sensing image S, the cross-pixel distance matrix from the remote sensing perspective is converted into the cross-ground distance matrix L. G1 The specific calculation method is as follows:

[0045] L g1 =L S ×GSD

[0046] Among them, L G1 Let L be the cross-ground distance matrix. S It is a parameter matrix.

[0047] Furthermore, embodiments of this application can be used to analyze the cross-ground distance matrix L. GThe average distance between each first ground panoramic image and the target ground panoramic image is taken, that is, the n×(n-1) parameter matrix is ​​converted into a parameter vector L of size n×1. G2 , where the parameter vector L G2 The value of each element in the value represents the depth prediction range coefficient of the ground points in the target area.

[0048] In step S104, the initial depth map of the first ground panoramic image is obtained.

[0049] It is understood that the embodiments of this application can obtain an initial depth map of the first ground panoramic image. For example, an initial depth value prediction can be performed on each first ground panoramic image based on an image generation network, that is, for each first ground panoramic image I1, I2, ..., I with a size of H×W×3... n Where H is the number of pixels along the vertical axis of the first panoramic image, and W is the number of pixels along the horizontal axis of the first panoramic image, an initial depth image N1, N2, ..., N of size H×W×1 is obtained through an image generation network. n In this context, the value of each pixel represents the initial depth value of the ground point in the selected area, thus effectively avoiding the problem of large differences in the contextual information within images obtained from different image captures.

[0050] In step S105, the scale of the initial depth map is corrected based on the multi-point scale constraint mask and the unified scale coefficient to obtain a multi-point outdoor consistent scale depth map, thus obtaining the multi-point outdoor consistent scale depth estimation result.

[0051] It is understood that, based on the multi-point scale constraint mask and uniform scale coefficient in the following steps, the scale of the initial depth map can be corrected to obtain a multi-point outdoor consistent scale depth map, thereby obtaining a multi-point outdoor consistent scale depth estimation result. This makes the depth estimation result of the first ground panoramic image within the coverage area of ​​the remote sensing image have a uniform scale range, effectively improving the accuracy of the scale depth estimation result.

[0052] In one embodiment of this application, the scale of the initial depth map is corrected based on a multi-point scale constraint mask and a target scale coefficient to obtain a multi-point outdoor consistent scale depth map. This includes: obtaining a relatively proportional cross-ground distance matrix based on each preset distance in the cross-ground distance matrix and the corresponding value in the cross-ground distance vector; obtaining a multi-point scale mask based on the relatively proportional cross-ground distance matrix; performing scale correction on the initial depth map based on the multi-point scale mask to obtain a process depth image; and performing channel conversion on the process depth image based on an image generation network to obtain the final corrected consistent scale depth image.

[0053] As one possible implementation method, embodiments of this application can be based on the cross-ground distance matrix LG1 Divide each 1×(n-1) distance by the cross distance vector L G2 The corresponding values ​​are used to obtain the relative proportional cross-ground distance matrix, also known as the multi-point scale mask L. G3 , where the size is n×(n-1), and each 1×(n-1) vector represents the depth distance scale coefficient between ground points in the target area and other points.

[0054] Next, a multi-point scale mask L can be used. G3 The initial predicted depth map is scaled, and each initial depth map of size H×W×1, N1, N2, ..., N2, is resized. n After multi-scale masking L G3 Multiply and superimpose the results to obtain process depth images M1, M2, ..., M with size H×W×(n-1). n .

[0055] Secondly, process depth images M1, M2, ..., M can be generated based on image generation networks. n Perform channel conversion to convert each process depth image M of size H×W×(n-1) i The transformation yields a final, consistent-scale depth image D1, D2, ..., D of size H × W × 1. n This yields consistent outdoor scale depth estimation results at multiple points, ensuring that the depth estimation results of the first ground panoramic image within the coverage area of ​​the remote sensing image have a uniform scale range, effectively improving the accuracy of the scale depth estimation results.

[0056] The multi-point outdoor consistent-scale depth estimation method based on remote sensing images proposed in this application can acquire remote sensing images of a target area, collect a first panoramic ground image under the projection of a target cylinder based on at least one ground point in the target area, and perform cube projection to obtain a second panoramic ground image under the cube projection. Features are extracted from the first panoramic ground image, the second panoramic ground image, and the remote sensing image. The relative positioning of the first panoramic ground image and the remote sensing image is determined based on the obtained feature vectors. A multi-point scale constraint mask for the remote sensing image is constructed, and the uniform scale coefficient of the first panoramic ground image is predicted. An initial depth map of the first panoramic ground image is obtained, and the scale of the initial depth map is corrected to obtain a multi-point outdoor consistent-scale depth map, thereby effectively improving the accuracy of the scale depth estimation results. This solves the problem in related technologies where large differences in the contextual information within images acquired from different images lead to significant scale differences in the predicted depth of the same object at different locations, reducing the accuracy of depth estimation.

[0057] Next, referring to the accompanying drawings, a multi-point outdoor consistent scale depth estimation device based on remote sensing images, according to an embodiment of this application, is described.

[0058] Figure 2 This is a block diagram of a multi-point outdoor consistent scale depth estimation device based on remote sensing images, according to an embodiment of this application.

[0059] like Figure 2 As shown, the multi-point outdoor consistent scale depth estimation device 10 based on remote sensing images includes: an acquisition module 100, a determination module 200, a construction module 300, an acquisition module 400, and a correction module 500.

[0060] Specifically, the acquisition module 100 is used to acquire remote sensing images of the target area and acquire a first panoramic ground image under the projection of the target cylinder based on at least one ground point in the target area.

[0061] The determination module 200 is used to perform cube projection on the first ground panoramic image to obtain a second ground panoramic image under cube projection, extract at least one feature from the first ground panoramic image, the second ground panoramic image and the remote sensing image, obtain at least one feature vector, and determine the relative position of the first ground panoramic image and the remote sensing image based on the at least one feature vector.

[0062] Module 300 is used to construct a multi-point scale-constrained mask for remote sensing images based on relative positioning, and to predict the uniform scale coefficient of the first ground panoramic image.

[0063] The acquisition module 400 is used to acquire the initial depth map of the first ground panoramic image.

[0064] The correction module 500 is used to correct the scale of the initial depth map based on the multi-point scale constraint mask and the unified scale coefficient, so as to obtain a multi-point outdoor consistent scale depth map and obtain the multi-point outdoor consistent scale depth estimation result.

[0065] Optionally, in one embodiment of this application, the acquisition module 100 includes an acquisition unit and a selection unit.

[0066] The acquisition unit is used to select a target area and acquire a first ground panoramic image of at least one ground point within the target area.

[0067] The selection unit is used to select a remote sensing image that covers the first panoramic ground image from the remote sensing data based on the geographical location information of the target area.

[0068] Optionally, in one embodiment of this application, the determining module 200 includes: a first acquiring unit, a second acquiring unit, a first determining unit, and a second determining unit.

[0069] The first acquisition unit is used to perform cube projection transformation on the first ground panoramic image to obtain a second ground panoramic image under cube projection.

[0070] The second acquisition unit is used to extract at least one feature from the first ground panoramic image, the second ground panoramic image, and the remote sensing image, and to acquire the first feature vector of the first ground panoramic image, the second feature vector of the second ground panoramic image, and the third feature vector of the remote sensing image, respectively.

[0071] The first determining unit is used to concatenate the first feature vector with the second feature vector to obtain the fourth feature vector, concatenate the fourth feature vector with the third feature vector to obtain the fifth feature vector, and obtain a feature vector group based on the fifth feature vector.

[0072] The second determining unit is used to adjust the feature dimensions of the feature vector group using the target multilayer perceptron (MLP) to obtain the position of the first ground panoramic image relative to the center of the remote sensing image.

[0073] Optionally, in one embodiment of this application, the construction module 300 includes: a calculation unit, a conversion unit, and a third determination unit.

[0074] The calculation unit is used to calculate the Euclidean distance between at least one binary number in the preset binary array and the target binary number, obtain the parameter matrix through the Euclidean distance, and use the parameter matrix to obtain the cross-pixel distance between the first ground panoramic image and the target ground panoramic image from the perspective of remote sensing.

[0075] The conversion unit is used to convert the cross-pixel distance matrix under the remote sensing view into a cross-ground distance matrix based on the ground sampling distance (GSD) of the remote sensing image.

[0076] The third determining unit is used to calculate the average distance between the first ground panoramic image and the target ground panoramic image in the cross ground distance matrix, and to obtain the depth prediction range coefficient of at least one ground point.

[0077] Optionally, in one embodiment of this application, the correction module 500 includes: a fourth determining unit, a correction unit, and a fifth determining unit.

[0078] The fourth determining unit is used to obtain a relatively proportional cross-ground distance matrix based on the corresponding values ​​in the cross-ground distance vector for each preset distance in the cross-ground distance matrix, and to obtain a multi-point scale mask based on the relatively proportional cross-ground distance matrix.

[0079] The correction unit is used to perform scale correction on the initial depth map based on the multi-point scale mask to obtain the process depth image.

[0080] The fifth determining unit is used to perform channel conversion on the process depth image based on the image generation network to obtain the final corrected consistent scale depth image.

[0081] It should be noted that the foregoing explanation of the embodiment of the multi-point outdoor consistent scale depth estimation method based on remote sensing images also applies to the multi-point outdoor consistent scale depth estimation device based on remote sensing images in this embodiment, and will not be repeated here.

[0082] The multi-point outdoor consistent scale depth estimation device based on remote sensing images proposed in this application can acquire remote sensing images of a target area, collect a first panoramic ground image under the projection of a target cylinder based on at least one ground point in the target area, and perform cube projection to obtain a second panoramic ground image under the cube projection. Features are extracted from the first panoramic ground image, the second panoramic ground image, and the remote sensing image. The relative positioning of the first panoramic ground image and the remote sensing image is determined based on the obtained feature vectors. A multi-point scale constraint mask for the remote sensing image is constructed, and the uniform scale coefficient of the first panoramic ground image is predicted. An initial depth map of the first panoramic ground image is obtained, and the scale of the initial depth map is corrected to obtain a multi-point outdoor consistent scale depth map, thereby effectively improving the accuracy of the scale depth estimation results. This solves the problem in related technologies where large differences in the contextual information within images acquired from different images lead to significant scale differences in the predicted depth of the same object at different locations, reducing the accuracy of depth estimation.

[0083] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0084] The memory 301, the processor 302, and the computer program stored on the memory 301 and capable of running on the processor 302.

[0085] When the processor 302 executes the program, it implements the multi-point outdoor consistent scale depth estimation method based on remote sensing images provided in the above embodiments.

[0086] Furthermore, electronic devices also include:

[0087] Communication interface 303 is used for communication between memory 301 and processor 302.

[0088] The memory 301 is used to store computer programs that can run on the processor 302.

[0089] The memory 301 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0090] If the memory 301, processor 302, and communication interface 303 are implemented independently, then the communication interface 303, memory 301, and processor 302 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0091] Optionally, in a specific implementation, if the memory 301, processor 302, and communication interface 303 are integrated on a single chip, then the memory 301, processor 302, and communication interface 303 can communicate with each other through an internal interface.

[0092] Processor 302 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0093] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multi-point outdoor consistent scale depth estimation method based on remote sensing images.

[0094] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0095] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0096] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0097] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0098] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0099] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0100] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0101] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A multi-point outdoor consistent-scale depth estimation method based on remote sensing images, characterized in that, Includes the following steps: Acquire remote sensing images of the target area, and acquire a first panoramic ground image based on at least one ground point in the target area under the projection of the target cylinder; The first ground panoramic image is cube-projected to obtain a second ground panoramic image under the cube projection. At least one feature is extracted from the first ground panoramic image, the second ground panoramic image, and the remote sensing image to obtain at least one feature vector. The relative positioning of the first ground panoramic image and the remote sensing image is determined based on the at least one feature vector. Constructing a multi-point scale constraint mask for the remote sensing image based on the relative positioning, and predicting the uniform scale coefficient of the first ground panoramic image, wherein the construction of the multi-point scale constraint mask for the remote sensing image based on the relative positioning and the prediction of the uniform scale coefficient of the first ground panoramic image includes: calculating the Euclidean distance between at least one binary number in a preset binary array and a target binary number; obtaining a parameter matrix through the Euclidean distance; using the parameter matrix to obtain the cross-pixel distance between the first ground panoramic image and the target ground panoramic image under the remote sensing viewpoint; converting the cross-pixel distance matrix under the remote sensing viewpoint into a cross-ground distance matrix according to the ground sampling distance (GSD) of the remote sensing image; calculating the average value of the distance between the first ground panoramic image and the target ground panoramic image in the cross-ground distance matrix, and obtaining the depth prediction range coefficient of the at least one ground point; Obtain the initial depth map of the first ground panoramic image; and Based on the multi-point scale constraint mask and the unified scale coefficient, the scale of the initial depth map is corrected to obtain a multi-point outdoor consistent scale depth map, and the multi-point outdoor consistent scale depth estimation result is obtained. The process of correcting the scale of the initial depth map based on the multi-point scale constraint mask and the unified scale coefficient to obtain a multi-point outdoor consistent scale depth map includes: obtaining a relatively proportional cross-ground distance matrix based on each preset distance in the cross-ground distance matrix and the corresponding value in the cross-ground distance vector; obtaining a multi-point scale mask based on the relatively proportional cross-ground distance matrix; performing scale correction on the initial depth map based on the multi-point scale mask to obtain a process depth image; and performing channel conversion on the process depth image based on the image generation network to obtain the final corrected consistent scale depth map.

2. The method according to claim 1, characterized in that, The acquisition of remote sensing images of the target area, including acquiring a first panoramic ground image based on at least one ground point in the target area and projecting it onto the target cylinder, comprises: Select the target area, and acquire the first ground panoramic image from the at least one ground point within the target area; Based on the geographical location information of the target area, a remote sensing image covering the first ground panoramic image is selected from the remote sensing data.

3. The method according to claim 1, characterized in that, The step of performing cube projection on the first ground panoramic image to obtain a second ground panoramic image under the cube projection, extracting at least one feature from the first ground panoramic image, the second ground panoramic image, and the remote sensing image to obtain at least one feature vector, and obtaining the relative positioning of the first ground panoramic image and the remote sensing image based on the at least one feature vector, includes: Perform cube projection transformation on the first ground panoramic image to obtain a second ground panoramic image under the cube projection; At least one feature is extracted from the first ground panoramic image, the second ground panoramic image, and the remote sensing image, and a first feature vector of the first ground panoramic image, a second feature vector of the second ground panoramic image, and a third feature vector of the remote sensing image are obtained respectively. The first feature vector is concatenated with the second feature vector to obtain the fourth feature vector. The fourth feature vector is concatenated with the third feature vector to obtain the fifth feature vector. A feature vector group is obtained based on the fifth feature vector. By adjusting the feature dimensions of the feature vector group using a target multilayer perceptron (MLP), the position of the first ground panoramic image relative to the center of the remote sensing image is obtained.

4. A multi-point outdoor consistent-scale depth estimation device based on remote sensing images, characterized in that, include: The acquisition module is used to acquire remote sensing images of the target area and acquire a first panoramic ground image under the projection of the target cylinder based on at least one ground point in the target area. The determination module is used to perform cube projection on the first ground panoramic image to obtain a second ground panoramic image under the cube projection, extract at least one feature from the first ground panoramic image, the second ground panoramic image and the remote sensing image to obtain at least one feature vector, and determine the relative position of the first ground panoramic image and the remote sensing image based on the at least one feature vector. A construction module is used to construct a multi-point scale constraint mask for the remote sensing image based on the relative positioning, and predict the uniform scale coefficient of the first ground panoramic image. The construction of the multi-point scale constraint mask based on the relative positioning and the prediction of the uniform scale coefficient of the first ground panoramic image includes: calculating the Euclidean distance between at least one binary tuple in a preset binary array and a target binary tuple; obtaining a parameter matrix through the Euclidean distance; using the parameter matrix to obtain the cross-pixel distance between the first ground panoramic image and the target ground panoramic image under the remote sensing viewpoint; converting the cross-pixel distance matrix under the remote sensing viewpoint into a cross-ground distance matrix according to the ground sampling distance (GSD) of the remote sensing image; and calculating the average distance between the first ground panoramic image and the target ground panoramic image in the cross-ground distance matrix to obtain the depth prediction range coefficient of the at least one ground point. The acquisition module is used to acquire the initial depth map of the first ground panoramic image; and The correction module is used to correct the scale of the initial depth map based on the multi-point scale constraint mask and the unified scale coefficient to obtain a multi-point outdoor consistent scale depth map and obtain the multi-point outdoor consistent scale depth estimation result. The step of correcting the scale of the initial depth map based on the multi-point scale constraint mask and the unified scale coefficient to obtain the multi-point outdoor consistent scale depth map includes: obtaining a relatively proportional cross-ground distance matrix based on each preset distance in the cross-ground distance matrix and the corresponding value in the cross-ground distance vector; obtaining a multi-point scale mask based on the relatively proportional cross-ground distance matrix; performing scale correction on the initial depth map based on the multi-point scale mask to obtain a process depth image; and performing channel conversion on the process depth image based on the image generation network to obtain the final corrected consistent scale depth map.

5. The apparatus according to claim 4, characterized in that, The acquisition module includes: Acquisition unit, configured to select the target area and acquire the first ground panoramic image from at least one ground point within the target area; The selection unit is used to select a remote sensing image that covers the first ground panoramic image from the remote sensing data based on the geographical location information of the target area.

6. The apparatus according to claim 4, characterized in that, The determining module includes: The first acquisition unit is used to perform cube projection conversion on the first ground panoramic image to obtain a second ground panoramic image under the cube projection. The second acquisition unit is used to extract at least one feature from the first ground panoramic image, the second ground panoramic image and the remote sensing image, and to acquire the first feature vector of the first ground panoramic image, the second feature vector of the second ground panoramic image and the third feature vector of the remote sensing image, respectively. The first determining unit is used to concatenate the first feature vector with the second feature vector to obtain a fourth feature vector, concatenate the fourth feature vector with the third feature vector to obtain a fifth feature vector, and obtain a feature vector group based on the fifth feature vector. The second determining unit is used to adjust the feature dimensions of the feature vector group using a target multilayer perceptron (MLP) to obtain the position of the first ground panoramic image relative to the center of the remote sensing image.

7. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the multi-point outdoor consistent scale depth estimation method based on remote sensing images as described in any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the multi-point outdoor consistent scale depth estimation method based on remote sensing images as described in any one of claims 1-3.

Images