Photogrammetry method and apparatus, device and storage medium

Abstract

The present disclosure relates to a photogrammetry method and apparatus, a device and a storage medium. The method comprises: acquiring a plurality of image frames, wherein the images are obtained by using a camera to perform image acquisition on a measured object with mark points on the surface; performing mark point identification on the images to obtain corresponding mark point position information, and performing image feature extraction on the information to obtain corresponding image feature information; on the basis of the mark point position information and the image feature information, determining matched mark point pairs corresponding to matched image pairs among the plurality of image frames; and, on the basis of the matched mark point pairs, determining corresponding depth information, and poses corresponding to the images.

Description

Photogrammetry method, device, equipment and storage medium

[0001] Cross-reference

[0002] This disclosure claims priority to Chinese patent application number 202311744621.6, filed with the Patent Office of China on December 19, 2023, entitled “Photogrammetry Method, Device, Equipment and Storage Medium,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a photogrammetry method, apparatus, device, and storage medium. Background Art

[0004] Photogrammetry refers to the technology of using a combination of cameras and film to measure the shape, size and spatial position of an object. It is widely used in industry, architecture, biology and other fields.

[0005] The photogrammetry method involved in the related technology is high in cost and low in accuracy. Therefore, there is an urgent need for a low-cost and high-accuracy photogrammetry method. Summary of the Invention

[0006] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a photogrammetry method, apparatus, device and storage medium.

[0007] A first aspect of the present disclosure provides a photogrammetry method, the method comprising:

[0008] Acquire multiple frames of images, wherein the images are acquired by a camera capturing images of an object having marking points set on its surface;

[0009] For the image, perform landmark recognition on it to obtain the corresponding landmark position information, and perform image feature extraction on it to obtain the corresponding image feature information;

[0010] Based on the marker point position information and the image feature information, determining the matching marker point pairs corresponding to the matching image pairs in the multiple frames of images;

[0011] Based on the matching landmark point pairs, the corresponding depth information and the corresponding pose of the image are determined.

[0012] A second aspect of the present disclosure provides a photogrammetry device, the device comprising:

[0013] A first acquisition module is configured to acquire multiple frames of images, wherein the images are acquired by a camera capturing images of an object having marking points set on its surface;

[0014] The first extraction module is configured to perform landmark recognition on the image to obtain corresponding landmark position information, and to perform image feature extraction on the image to obtain corresponding image feature information;

[0015] A first determining module is configured to determine matching image pairs and matching landmark point pairs corresponding to the matching image pairs in the multiple frames of images based on the landmark point position information and the image feature information;

[0016] The first reconstruction module is configured to determine corresponding depth information and a corresponding pose of the image based on matching landmark point pairs.

[0017] A third aspect of an embodiment of the present disclosure provides an electronic device, the server comprising: a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the method of the first aspect above.

[0018] A fourth aspect of an embodiment of the present disclosure provides a non-volatile computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method of the first aspect described above can be implemented.

[0019] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art:

[0020] The embodiment of the present disclosure can obtain multiple frames of images, wherein the images are obtained by performing image acquisition on the surface of the object to be measured with marker points provided by a camera; for the image, the marker points are identified to obtain the corresponding marker point position information, and the image feature is extracted to obtain the corresponding image feature information; based on the marker point position information and the image feature information, the matching marker point pairs corresponding to the matching image pairs in the multiple frames of images are determined; based on the matching marker point pairs, the corresponding depth information and the corresponding posture of the image are determined. With the above technical solution, it is sufficient to use a camera to acquire images of the object to be measured. The cost of the camera is generally low, which is conducive to reducing costs. In the process of marker point matching (or registration), the external points can be filtered through the image feature information, so that the determined matching marker point pairs are more accurate, which is conducive to improving the accuracy of the depth information and posture. It can be seen that the photogrammetry method of the embodiment of the present disclosure has the advantages of low cost and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] FIG1 is a flow chart of a photogrammetry method provided by some embodiments of the present disclosure;

[0024] FIG2 is a flow chart of another photogrammetry method provided by some embodiments of the present disclosure;

[0025] FIG3 is a flow chart of another photogrammetry method provided by some embodiments of the present disclosure;

[0026] FIG4 is a schematic structural diagram of a photogrammetry device provided by some embodiments of the present disclosure;

[0027] FIG5 is a schematic structural diagram of an electronic device in some embodiments of the present disclosure. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0030] FIG1 is a flow chart of a photogrammetry method provided by an embodiment of the present disclosure. The method can be performed by an electronic device. The electronic device can be exemplarily understood as a device such as a mobile phone, tablet computer, laptop computer, desktop computer, smart TV, etc. As shown in FIG1 , the method provided by this embodiment includes the following steps:

[0031] S110 , obtaining multiple frames of images, wherein the images are obtained by a camera capturing images of a measured object with marking points set on its surface.

[0032] In the disclosed embodiments, prior to capturing images of the object under test, a number of markers may be set on the surface of the object under test. A camera may then be used to capture images of the object under test at different locations on the surface with the markers set thereon, thereby obtaining multiple frames of images. It should be noted that the camera may be a fixed-focus SLR camera or a fixed-focus industrial camera, but is not limited thereto.

[0033] In some embodiments of the present disclosure, a marker point can be a coding marker point or a non-coding marker point, without limitation. It is understood that a coding marker point generally includes multiple points regularly arranged on the same circumference, the area of ​​the points is relatively small, and the position information of the entire coding marker point is provided by the position of the center of the circle of the central point. The presence or absence of the points at each coding position determines the different coded information. In addition, the coding marker point can also be a two-dimensional code, a one-dimensional code, a combination of large and small dots, etc., without limitation here. One-dimensional codes and two-dimensional codes can encode and decode information using preset algorithms, while the combination of large and small dots can be implemented using preset rules. For example, in the case where the coding marker point is a combination of large and small dots, the coding marker point can be a dot matrix, where the combination of large and small dots is used to represent different information (such as a large dot represents 1 and a small dot represents 0), and the information is encoded by the size, position, and arrangement of the dots. However, in some embodiments of the present disclosure, a non-coding marker point includes a single dot, the area of ​​the dot is relatively large, and there is no coded information between a non-coding marker point and other non-coding marker points. In other words, each non-coding marker point is independent. In this way, compared with setting coded marking points on the surface of the object to be measured, setting non-coded marking points on the surface of the object to be measured is conducive to shortening the time spent on setting marking points on the surface of the object to be measured and saving labor costs.

[0034] It should be noted that the shape of the non-coding markers can be set by those skilled in the art according to actual conditions and is not limited here. For example, they can be circular, rectangular, etc., but are not limited thereto. It should also be noted that the distribution of the non-coding markers on the surface of the object being measured can also be set by those skilled in the art according to actual conditions and is not limited here. For example, non-coding markers of a certain density can be randomly affixed to the surface of the object being measured to ensure that each frame of the image includes at least four non-coding markers.

[0035] S120 , performing landmark recognition on the image to obtain corresponding landmark position information, and performing image feature extraction on the image to obtain corresponding image feature information.

[0036] In some embodiments of the present disclosure, the marker point position information is the image coordinates of the marker point in the image to which it belongs. It should be noted that any possible marker point recognition algorithm can be used to perform marker point recognition on the image, and this is not limited here.

[0037] In some embodiments of the present disclosure, image feature information is information used to characterize image features. Image features may include, but are not limited to, color features, texture features, shape features, and / or spatial relationship features. It should be noted that any possible image feature extraction algorithm may be used to extract image features from an image, and this is not limited herein.

[0038] S130 : Determine matching landmark point pairs corresponding to matching image pairs in the multiple frames of images based on the landmark point position information and the image feature information.

[0039] In some embodiments of the present disclosure, a matching image pair includes two different frames of images, the two frames of images include overlapping image areas, and the overlapping image areas include matching landmark point pairs.

[0040] In some embodiments of the present disclosure, for two different frames of images, if one frame of the image has a marker point A (2D marker point) in the overlapping image area, and the other frame of the image has a marker point A' (2D marker point) in the overlapping image area, if the marker point A and the marker point A' meet the matching conditions based on the image feature information of the two frames of images, then it means that the two correspond to the same marker point (3D marker point) on the surface of the object being measured, and the two constitute a matching marker point pair. It should be noted that the image features of the two frames of images refer to quantitative descriptions that can be extracted from the images and used to characterize the image attributes, including grayscale features (such as the grayscale value of the image), texture features (such as roughness, smoothness), color features (such as color histogram, color moment), shape features (such as edges, corners, contours), spatial relationship features (i.e., the relative position and distance between different objects or pixels in the image), and other one or more combinations of features. In some embodiments of the present application, distance features and grayscale features can be used to jointly construct image feature information. Specifically, a depth sensor or a stereo matching algorithm can be used to extract the distance information of each pixel point (i.e., distance features) from two frames of images, and the grayscale value of each pixel point (i.e., grayscale features) can be extracted from the image; a 128-dimensional vector of each marker point A and A' is constructed, which contains distance features and grayscale features. For example, the first 64 dimensions can be used to store distance features, and the last 64 dimensions can be used to store grayscale features; the constructed 128-dimensional vector of marker point A and the 128-dimensional vector of marker point A' are used to determine whether the two constitute a matching marker point pair. In some embodiments of the present application, distance features can also be used to construct image feature information. Specifically, feature points of markers A and A' can be extracted from two frames of images using a feature detection algorithm, such as SIFT (Scale-Invariant Feature Transform) and ORB (Oriented FAST and Rotated BRIEF). The descriptor of each feature point (a vector used to characterize feature points in an image) is calculated. The Euclidean distance between the descriptors is calculated. Based on the Euclidean distance between the descriptors of the two feature points, it is determined whether the two constitute a matching marker point pair. The specific content of the matching conditions can be set by those skilled in the art according to actual conditions and is not limited here. For example, based on the image feature information of the two frames of images, the basic matrix (i.e., F) of the two frames of images is determined. If PAFPA is less than or equal to a second preset threshold (e.g., 0), where PA is the marker point position information (i.e., image coordinates) of marker point A in the image to which it belongs, and PA' is the marker point position information (i.e., image coordinates) of marker point A in the image to which it belongs, then marker point A and marker point A' constitute a matching marker point pair.

[0041] S140 : Determine corresponding depth information and the corresponding pose of the image based on the matching landmark point pairs.

[0042] In some embodiments of the present disclosure, S140 may include: S141 , selecting an initial reconstructed image pair from the matching image pairs.

[0043] In some embodiments of the present disclosure, S141 may include: selecting an initial image from multiple frames of images, wherein the sum of the matching marker point pairs corresponding to the initial image is the largest, and the sum of the matching marker point pairs corresponding to an image is: the sum of the number of matching marker point pairs corresponding to each matching image pair including the image; selecting an initial reconstructed image pair from the matching image pairs, wherein the initial reconstructed image pair includes the initial image and the matching image, wherein the matching image is an image having the largest number of matching marker point pairs with the initial image among images whose frame angle with the initial image is greater than a preset angle threshold.

[0044] Of course, in some other embodiments of the present disclosure, S142 may include: randomly selecting one of the matching image pairs as the initial reconstructed image pair.

[0045] S142 : triangulate the corresponding matching landmark points based on the initial reconstructed image to obtain corresponding depth information.

[0046] In some embodiments of the present disclosure, for the matching marker point pairs corresponding to the initial reconstructed image pair, the coordinates of the marker points set on the surface of the measured object corresponding to the two marker points in the matching marker point pair in three-dimensional space (i.e., depth information) can be calculated based on the marker point position information of the two marker points in the matching marker point pair.

[0047] Of course, for the initial reconstructed image pair, the corresponding posture of the matching image can be determined based on the coordinates of the marker points set on the surface of the measured object corresponding to the matching marker point pair in three-dimensional space, and the marker point position information (i.e., image coordinates) of the marker points in the matching marker point pair located in the matching image.

[0048] S143. When a preset reconstruction stop condition is not met, determine the next reconstructed image pair and perform reconstruction processing on the next reconstructed image pair until the preset reconstruction stop condition is met, wherein determining the next reconstructed image pair and performing reconstruction processing on the next reconstructed image pair includes: determining the adjacent image corresponding to the previous reconstructed image pair, and the next reconstructed image pair including the adjacent image, wherein the adjacent image and the previous reconstructed image pair have the most common view marker points; determining the corresponding posture of the adjacent image based on the depth information of the common view marker point and the position information of the common view marker point in the adjacent image; and performing triangulation processing based on the matching marker point pair corresponding to the next reconstructed image pair to obtain the corresponding depth information.

[0049] In some embodiments of the present disclosure, the specific content of the preset reconstruction stop condition can be set by those skilled in the art based on actual circumstances and is not limited herein. For example, each matching image pair corresponding to multiple frames of image has participated in the triangulation process, or the number of matching image pairs that have participated in the triangulation process is greater than a sixth preset threshold, etc., but the present invention is not limited thereto.

[0050] In some embodiments of the present disclosure, the image with the most common view landmark points with the previous reconstructed image pair is selected from multiple frames of images as the adjacent image; based on the coordinates of the common view landmark points in three-dimensional space (i.e., depth information) and the landmark point position information of the common view landmark points in the adjacent image (i.e., image coordinates), the corresponding postures of the adjacent images can be determined.

[0051] In some embodiments of the present disclosure, the process of selecting the next reconstructed image pair may include: selecting the one with the largest number of matching landmark point pairs from matching image pairs that have never participated in triangulation processing and include adjacent images as the next reconstructed image pair.

[0052] Of course, in other embodiments of the present disclosure, the process of selecting the next reconstructed image pair may also include: randomly selecting one of the matching image pairs that have never participated in the triangulation process and include adjacent images as the next reconstructed image pair.

[0053] In some embodiments of the present disclosure, for the matching marker point pair corresponding to the next reconstructed image pair, the coordinates of the marker points set on the surface of the measured object corresponding to the two marker points in the matching marker point pair can be calculated based on the marker point position information of the two marker points in the matching marker point pair.

[0054] Of course, for the next reconstructed image pair, based on the coordinates of the marker points set on the surface of the measured object corresponding to the matching marker point pair in three-dimensional space, and the marker point position information (i.e., image coordinates) of the marker points in the matching marker point pair located in another image different from the adjacent image, the posture corresponding to the other image different from the adjacent image can be determined.

[0055] It can be understood that determining the depth information of the marker points based on triangulation processing and determining the corresponding posture of the image based on the 3D-2D coordinate relationship of the marker points can simplify the method of determining the depth information and posture, which is conducive to reducing the difficulty of implementation.

[0056] In some embodiments of the present disclosure, the method further includes: optimizing depth information, posture, and camera intrinsic parameters.

[0057] In some embodiments of the present disclosure, the initial camera intrinsic parameters can be calculated using any possible method. For example, when the camera is a single-lens reflex camera, the initial camera intrinsic parameters can be estimated using Exchangeable Image File Format (EXIF) information. When the camera is an industrial camera, the initial camera intrinsic parameters can be calculated based on the camera lens information.

[0058] In some embodiments of the present disclosure, the specific timing of optimization may be determined by those skilled in the art based on actual circumstances and is not limited herein. For example, optimization may be performed after determining the poses corresponding to all images, or after determining the poses corresponding to a certain number of images, or the error corresponding to depth information, pose, and camera intrinsic parameters may be calculated in real time, and optimization may be triggered when the error exceeds a first preset error threshold, etc., but the present invention is not limited thereto.

[0059] It should be noted that any possible optimization method (such as BA (Bundle Adjustment) etc.) can be used for optimization, and there is no limitation to this.

[0060] It is understandable that by optimizing the depth information, pose, and camera intrinsic parameters, the above parameters can be made more accurate, which is conducive to improving the accuracy of photogrammetry.

[0061] The disclosed embodiment only requires a single camera to capture images of the object being measured, which helps reduce costs. Furthermore, during the landmark point matching (or registration) process, outliers can be filtered using image feature information, making the determined matching landmark point pairs more precise, thereby improving the accuracy of depth information and pose. Thus, the photogrammetry method of the disclosed embodiment has the advantages of low cost and high accuracy.

[0062] Figure 2 is a flow chart of another photogrammetry method provided by an embodiment of the present disclosure. The present disclosure embodiment is optimized based on the above embodiment, and the present disclosure embodiment can be combined with various optional solutions in the above one or more frame embodiments.

[0063] As shown in FIG2 , the photogrammetry method may include the following steps.

[0064] S210 , obtaining multiple frames of images, wherein the images are obtained by capturing images of the object to be measured with marking points set on the surface by a camera.

[0065] In some embodiments of the present disclosure, S210 is similar to S110 and will not be described again here.

[0066] S220 , performing landmark recognition on the image to obtain corresponding landmark position information, and performing image feature extraction on the image to obtain corresponding image feature information.

[0067] In some embodiments of the present disclosure, S220 is similar to S120 and will not be described again here.

[0068] S230: Determine at least one initial image pair based on multiple frames of images.

[0069] In some embodiments of the present disclosure, S230 may include: for multiple frames of images, forming an initial image pair with every two frames of images.

[0070] For example, if the number of the multi-frame images is N, each of the two frames forms an initial image pair, and C_N^2 initial image pairs can be obtained.

[0071] Of course, in some embodiments of the present disclosure, S230 may also include: forming an initial image pair based on multiple frame images according to the principle that an initial image pair can be formed when the frame spacing between two frame images is less than or equal to a seventh preset threshold (for example, 10, etc.).

[0072] S240 . Perform image feature matching on the initial image pair based on corresponding image feature information to obtain a corresponding first matching feature point pair, and determine a corresponding basic matrix based on the first matching feature point pair.

[0073] In some embodiments of the present disclosure, the first matching feature point pair includes two mutually matching feature points from different images.

[0074] It should be noted that any possible image feature matching algorithm can be used to perform image feature matching on the initial image pair, and this is not limited here.

[0075] Of course, a scene graph (i.e., a graph) corresponding to the initial image pairs of multiple frames can also be drawn based on the first matching feature point pairs corresponding to each initial image pair. The scene graph includes nodes and edges. Nodes are used to represent images, and edges are used to represent that the two frames connected by them constitute an initial image pair. In addition, the edges can also carry the first matching feature point pairs corresponding to the initial image pairs connected by them. In this way, it is possible to clearly indicate which two frames of multiple frames constitute the initial image pair and the first matching feature point pairs between the initial image pairs.

[0076] In some embodiments of the present disclosure, the fundamental matrix (i.e., F) constrains the relationship between 3D points in the image coordinate systems of two cameras. If the same 3D point is imaged as a 2D point B and a 2D point B' in two different images, then PBFPB' = 0, where PB is the image coordinate of 2D point B in its own image, and PB' is the image coordinate of 2D point B' in its own image. The first matching feature point pair is the imaging result of the same 3D point in two different images, obtained through image feature matching. Therefore, the fundamental matrix can be determined based on the first matching feature point pair.

[0077] It should be noted that any possible algorithm (such as the eight-point method, etc.) can be used to calculate the basic matrix corresponding to the initial image pair, and there is no limitation to this.

[0078] S250 : For the initial image pair, determine a corresponding matching landmark point pair based on the corresponding basic matrix, the first matching feature point pair, and the landmark point position information.

[0079] In some embodiments of the present disclosure, S250 may include: for the initial image pair, determining the corresponding initial marker point pair based on the first matching feature point pair and the marker point position information; if the product of the position information of the initial marker point pair and the basic matrix is ​​less than or equal to a second preset threshold, determining the initial marker point pair as a matching marker point pair.

[0080] In some embodiments of the present disclosure, for an initial image pair, the corresponding first matching feature point pair includes position information (i.e., image coordinates) of two feature points that match each other. If the position information (i.e., image coordinates) of two marker points from the initial image pair and located in different images coincides with the position information of a first matching feature point pair or the deviation is less than a first preset deviation threshold, then the two marker points (2D marker points) corresponding to the two marker point position information constitute an initial marker point pair.

[0081] In some embodiments of the present disclosure, the initial landmark point pairs are verified based on the basic matrix, and the initial landmark points that pass the verification are the matching landmark point pairs.

[0082] It should be noted that the specific value of the second preset threshold can be set by those skilled in the art according to actual circumstances and is not limited here. For example, if the second preset threshold is 0, and the initial marker point pair includes marker point A and marker point A', if PAFPA = 0, where PA is the marker point position information (i.e., image coordinates) of marker point A in the image to which it belongs, and PA' is the marker point position information (i.e., image coordinates) of marker point A in the image to which it belongs, then the initial marker point pair passes the verification and can be determined to be a matching marker point pair.

[0083] It can be understood that by verifying the initial landmark point pairs through the basic matrix, the final matching landmark point pairs can be made more accurate, which is beneficial to improving the photogrammetry accuracy.

[0084] Of course, in other embodiments of the present disclosure, the basic matrix may not be used to verify the initial landmark point pairs, but each initial landmark point pair may be determined as a matching landmark point pair.

[0085] S260: Determine an initial image pair whose number of matching landmark point pairs meets a first preset threshold as a matching image pair.

[0086] It should be noted that the specific value of the first preset threshold can be set by those skilled in the art according to actual conditions and is not limited here.

[0087] Of course, the edges corresponding to the initial image pairs that do not belong to the matching image pairs in the scene graph may also be deleted, and only the edges corresponding to the matching image pairs are retained.

[0088] S270: Determine corresponding depth information and the corresponding pose of the image based on the matching landmark point pairs.

[0089] In some embodiments of the present disclosure, S270 is similar to S140 and will not be described again here.

[0090] The disclosed embodiment can perform image feature matching based on the image feature information corresponding to the initial image pair to obtain the corresponding first matching feature point pair, and determine the corresponding basic matrix and initial marker point pair based on the first matching feature point pair, and then verify the initial marker point pair based on the basic matrix, filter out unreliable initial marker point pairs, and retain matching marker point pairs with higher credibility. In this way, the accuracy of the final matching marker point pair can be made higher, which is conducive to improving the accuracy of depth information and posture, and thus improving the accuracy of photogrammetry.

[0091] Figure 3 is a flow chart of another photogrammetry method provided by an embodiment of the present disclosure. The present disclosure embodiment is optimized based on the above embodiment, and the present disclosure embodiment can be combined with various optional solutions in the above one or more frame embodiments.

[0092] As shown in FIG3 , the photogrammetry method may include the following steps.

[0093] S310 , obtaining multiple frames of images, wherein the images are obtained by a camera collecting images of a measured object with marking points set on its surface.

[0094] In some embodiments of the present disclosure, S310 is similar to S110 and will not be described again here.

[0095] S320: Obtain IMU information between two adjacent image frames.

[0096] In the disclosed embodiments, during image acquisition of the measured object, an IMU measurement unit can be used to detect the camera's IMU information in real time to obtain IMU information between adjacent images. It should be noted that the IMU measurement unit's measurement frame rate can be greater than the camera's image acquisition frame rate, and this is not specifically limited here.

[0097] In some embodiments of the present disclosure, IMU information may include velocity, acceleration, angular velocity, angular acceleration, etc., but is not limited thereto.

[0098] S330 , performing landmark recognition on the image to obtain corresponding landmark position information, and performing image feature extraction on the image to obtain corresponding image feature information.

[0099] In some embodiments of the present disclosure, S330 is similar to S120 and will not be described again here.

[0100] S340 , determining matching landmark point pairs corresponding to matching image pairs in the multiple frames of images based on the landmark point position information, the image feature information, and the IMU information.

[0101] In some embodiments of the present disclosure, S340 may include: S341. For multiple frames of images, group two frames of images whose relative positions between frames meet a preset condition into a matching image pair.

[0102] In some embodiments of the present disclosure, the specific content of the preset conditions can be set by those skilled in the art according to actual circumstances and is not limited here. For example, according to the principle that an image can form a matching image pair with a preset number of images acquired after and / or before the acquisition time, a matching image pair is formed based on multiple frames of images. Alternatively, according to the principle that a matching image pair can be formed when the frame spacing between two frames of images is less than or equal to a seventh preset threshold (e.g., 10, etc.), a matching image pair is formed based on multiple frames of images. However, this is not limited to this.

[0103] S342 : For the matching image pair, perform image feature matching based on the corresponding image feature information to obtain a corresponding first matching feature point pair.

[0104] Of course, a scene graph (i.e., a graph) corresponding to the matching image pairs of multiple frames can also be drawn based on the first matching feature point pairs corresponding to each matching image pair. The scene graph includes nodes and edges. Nodes are used to represent images, and edges are used to represent that the two frames connected by them constitute a matching image pair. In addition, the edges can also carry the first matching feature point pairs corresponding to the matching image pairs connected by them. In this way, it is possible to clearly indicate which two frames of multiple images constitute a matching image pair and the first matching feature point pairs between the matching image pairs.

[0105] Of course, for two matching image pairs, let one matching image pair include image M and image N, and the other matching image pair include image N and image P. If the number of first matching feature point pairs corresponding to image M and image N is greater than the eighth preset threshold, and the number of first matching feature point pairs corresponding to image N and image P is greater than the eighth preset threshold, then image M and image P can be combined into a new matching image pair, and their corresponding matching landmark point pairs can be determined.

[0106] S343. For the matching image pairs, if the number of corresponding first matching feature point pairs is greater than the third preset threshold, the motion direction between the two frames of the matching image pair is determined based on the corresponding IMU information, and the corresponding second matching feature point pairs are determined based on the motion direction; if the number of second matching feature point pairs is greater than the fourth preset threshold, the corresponding basic matrix is ​​determined based on the second matching feature point pairs, and the corresponding matching marker point pairs are determined based on the corresponding basic matrix, the second matching feature point pairs and the marker point position information.

[0107] It should be noted that the specific value of the third preset threshold can be set by those skilled in the art according to actual conditions and is not limited here.

[0108] In some embodiments of the present disclosure, a matching image pair includes two frames of images, and the IMU information measured by the IMU measurement unit between the acquisition moments of the two frames of images is the IMU information corresponding to the matching image pair.

[0109] In some embodiments of the present disclosure, the direction of motion can be represented by a rotation matrix R and a translation matrix T. For a matching image pair, the pose corresponding to one frame of the image is recorded as the first pose, and the pose corresponding to the other frame of the image is recorded as the second pose. The first pose can be obtained by transforming the second pose based on the rotation matrix R and the translation matrix T.

[0110] In some embodiments of the present disclosure, for a matching image pair, the two frames of images in the matching image pair can be unified into the same image coordinate system based on the direction of motion. After being unified into the same image coordinate system, two feature points in the two frames of images whose position information (i.e., image coordinates) is the same or whose deviation is less than a second preset deviation threshold constitute a second matching feature point pair.

[0111] It should be noted that the specific value of the fourth preset threshold can be set by those skilled in the art according to actual conditions and is not limited here.

[0112] It should also be noted that any possible algorithm (such as the eight-point method, etc.) can be used to calculate the basic matrix corresponding to the matching image pair, and there is no limitation to this.

[0113] In some embodiments of the present disclosure, determining the corresponding matching marker point pairs based on the corresponding basic matrix, the second matching feature point pairs and the marker point position information may include: for the matching image pair, determining the corresponding initial marker point pairs based on the second matching feature point pairs and the marker point position information, if the product of the position information of the initial marker point pair and the basic matrix is ​​less than or equal to a second preset threshold, determining the initial marker point pair as the matching marker point pair.

[0114] In some embodiments of the present disclosure, for a matching image pair, the corresponding second matching feature point pair includes the position information (i.e., image coordinates) of two feature points that match each other. If the position information (i.e., image coordinates) of two marker points from the matching image pair and located in different images coincides with the position information of the second matching feature point pair or the deviation is less than the first preset deviation threshold, then the two marker points (2D marker points) corresponding to the two marker point position information constitute an initial marker point pair.

[0115] In some embodiments of the present disclosure, the initial landmark point pairs are verified based on the basic matrix, and the initial landmark points that pass the verification are the matching landmark point pairs.

[0116] It should be noted that the specific value of the second preset threshold can be set by those skilled in the art according to actual circumstances and is not limited here. For example, if the second preset threshold is 0, and the initial marker point pair includes marker point A and marker point A', if PAFPA = 0, where PA is the marker point position information (i.e., image coordinates) of marker point A in the image to which it belongs, and PA' is the marker point position information (i.e., image coordinates) of marker point A in the image to which it belongs, then the initial marker point pair passes the verification and can be determined to be a matching marker point pair.

[0117] It can be understood that by verifying the initial landmark point pairs through the basic matrix, the final matching landmark point pairs can be made more accurate, which is beneficial to improving the photogrammetry accuracy.

[0118] In some embodiments of the present disclosure, S340 may also include: S344. For the matching image pairs, if the number of second matching feature point pairs is less than or equal to a fourth preset threshold, the corresponding initial marker point pairs are determined based on the second matching feature point pairs and the marker point position information, and the initial marker point pairs are used as the matching marker point pairs.

[0119] In some embodiments of the present disclosure, S340 may also include: S345. For a matching image pair, if the number of first matching feature point pairs is less than or equal to a third preset threshold, determining the motion direction between the two frames of images in the matching image pair based on the corresponding IMU information, and determining the corresponding matching landmark point pairs based on the motion direction and landmark point position information.

[0120] In some embodiments of the present disclosure, for a matching image pair, the two frames of images in the matching image pair can be unified into the same image coordinate system based on the direction of motion. After being unified into the same image coordinate system, two marker points (2D marker points) in the two frames of images whose marker point position information (i.e., image coordinates) is the same or whose deviation is less than a third preset deviation threshold constitute a matching marker point pair.

[0121] It can be understood that by determining the corresponding motion direction based on the IMU information corresponding to the matching image pair, and participating in the determination of the matching landmark points based on the motion direction, when the surface features of the object being measured are unclear or difficult to identify, the matching landmark point pairs corresponding to the matching image pair can still be determined. This is conducive to improving the stability and robustness of photogrammetry.

[0122] Of course, in other embodiments of the present disclosure, S340 may also include: inputting the marker point position information, image feature information and IMU information into the trained network model, and obtaining the matching marker point pairs corresponding to the matching image pairs output by the network model.

[0123] S350: Determine corresponding depth information and the corresponding pose of the image based on the matching landmark point pairs.

[0124] In some embodiments of the present disclosure, S350 is similar to S140 and will not be described again here.

[0125] The disclosed embodiments can determine matching landmark point pairs corresponding to matching image pairs in multiple frames of images based on landmark point position information, image feature information, and IMU information. This allows for more comprehensive and more comprehensive factors to be considered in determining matching landmark point pairs, thereby improving the accuracy of matching landmark point pairs and, in turn, the accuracy of photogrammetry.

[0126] FIG4 is a schematic diagram of the structure of a photogrammetric device provided by an embodiment of the present disclosure. The photogrammetric device can be understood as the above-mentioned electronic device or a portion of the functional modules in the above-mentioned electronic device. As shown in FIG4 , the photogrammetric device 400 includes:

[0127] A first acquisition module 410 is configured to acquire multiple frames of images, wherein the images are acquired by a camera capturing images of an object having markers set on its surface;

[0128] The first extraction module 420 is configured to perform landmark recognition on the image to obtain corresponding landmark position information, and perform image feature extraction on the image to obtain corresponding image feature information;

[0129] A first determining module 430 is configured to determine matching landmark point pairs corresponding to matching image pairs in the multiple frames of images based on landmark point position information and image feature information;

[0130] The first reconstruction module 440 is configured to determine corresponding depth information and a corresponding pose of the image based on matching landmark point pairs.

[0131] In another embodiment of the present disclosure, the first determining module 430 may include:

[0132] A first determination submodule is configured to determine at least one initial image pair based on multiple frames of images;

[0133] A second determination submodule is configured to perform image feature matching on the initial image pair based on corresponding image feature information to obtain a corresponding first matching feature point pair, and determine a corresponding basic matrix based on the first matching feature point pair;

[0134] A third determination submodule is configured to determine, for the initial image pair, a corresponding matching landmark point pair based on the corresponding basic matrix, the first matching feature point pair and the landmark point position information;

[0135] The fourth determination submodule is configured to determine an initial image pair whose number of matching landmark point pairs meets a first preset threshold as a matching image pair.

[0136] In yet another embodiment of the present disclosure, the first determining submodule is specifically configured to target multiple frames of images, with every two frames of images forming an initial image pair.

[0137] In another embodiment of the present disclosure, the third determination submodule is specifically configured to determine the corresponding initial marker point pair for the initial image pair based on the first matching feature point pair and the marker point position information; if the product of the position information of the initial marker point pair and the basic matrix is ​​less than or equal to the second preset threshold, the initial marker point pair is determined to be a matching marker point pair.

[0138] In another embodiment of the present disclosure, the device further comprises:

[0139] The second acquisition module is configured to acquire IMU information between two adjacent frames of images;

[0140] The first determination module 430 may include: a fifth determination submodule, configured to determine matching marker point pairs corresponding to matching image pairs in multiple frames of images based on marker point position information, image feature information and IMU information.

[0141] In yet another embodiment of the present disclosure, the fifth determining submodule may include:

[0142] The first composition unit is configured to form a matching image pair from two frames of images whose relative positions between the frames meet a preset condition;

[0143] a first matching unit configured to perform image feature matching on the matching image pair based on corresponding image feature information to obtain a corresponding first matching feature point pair;

[0144] The first determination unit is configured to determine, for a matching image pair, a motion direction between two frames of images in the matching image pair based on the corresponding IMU information if the number of corresponding first matching feature point pairs is greater than a third preset threshold, and determine the corresponding second matching feature point pairs based on the motion direction; if the number of second matching feature point pairs is greater than a fourth preset threshold, determine the corresponding basic matrix based on the second matching feature point pairs, and determine the corresponding matching marker point pairs based on the corresponding basic matrix, the second matching feature point pairs, and the marker point position information.

[0145] In another embodiment of the present disclosure, when the first determination unit determines the corresponding matching marker point pair based on the corresponding basic matrix, the second matching feature point pair and the marker point position information, it is specifically configured to determine the corresponding initial marker point pair based on the second matching feature point pair and the marker point position information. If the product of the position information of the initial marker point pair and the basic matrix is ​​less than or equal to the second preset threshold, the initial marker point pair is determined to be a matching marker point pair.

[0146] In yet another embodiment of the present disclosure, the fifth determining submodule may further include:

[0147] The second determination unit is configured to determine, for a matching image pair, a motion direction between two frames of images in the matching image pair based on corresponding IMU information if the number of first matching feature point pairs is less than or equal to a third preset threshold, and determine a corresponding matching marker point pair based on the motion direction and marker point position information.

[0148] In yet another embodiment of the present disclosure, the first reconstruction module 440 is specifically configured to select an initial reconstructed image pair from the matching image pair;

[0149] Based on the initial reconstructed image, the corresponding matching landmark points are triangulated to obtain the corresponding depth information;

[0150] When the preset reconstruction stop condition is not met, the next reconstructed image pair is determined and the next reconstructed image pair is reconstructed until the preset reconstruction stop condition is met, wherein determining the next reconstructed image pair and reconstructing the next reconstructed image pair includes: determining the adjacent image corresponding to the previous reconstructed image pair, and the next reconstructed image pair including the adjacent image, wherein the adjacent image and the previous reconstructed image pair have the most common view landmarks; determining the corresponding posture of the adjacent image based on the depth information of the common view landmark and the position information of the common view landmark in the adjacent image; and performing triangulation based on the matching landmark pair corresponding to the next reconstructed image pair to obtain the corresponding depth information.

[0151] In another embodiment of the present disclosure, the device further comprises:

[0152] The optimization module is set to optimize depth information, pose, and camera intrinsic parameters.

[0153] The device provided in this embodiment can execute the method of any of the above embodiments, and its execution method and beneficial effects are similar, which will not be repeated here.

[0154] An embodiment of the present disclosure further provides an electronic device, which includes: a memory storing a computer program; and a processor configured to execute the computer program. When the computer program is executed by the processor, the method of any of the above embodiments can be implemented.

[0155] FIG5 is a schematic diagram of the structure of an electronic device in an embodiment of the present disclosure. Specific reference is now made to FIG5 , which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing an embodiment of the present disclosure. The electronic device 500 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and the like, as well as fixed terminals such as digital TVs, desktop computers, and the like. The electronic device shown in FIG5 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0156] As shown in Figure 5, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0157] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although FIG5 shows the electronic device 500 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0158] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0159] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a non-volatile computer-readable storage medium or any combination of the two. Non-volatile computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples of non-volatile computer-readable storage media may include, but are not limited to: an electrical connection with one or more frame conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a non-volatile computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a non-volatile computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), or the like, or any suitable combination thereof.

[0160] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as the HyperText Transfer Protocol (HTTP), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0161] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0162] The above-mentioned computer-readable medium carries one or more frame programs. When the above-mentioned one or more frame programs are executed by the electronic device, the electronic device: obtains multiple frames of images, wherein the images are obtained by a camera performing image acquisition on an object under test with marker points set on the surface; performs marker point recognition on the image to obtain corresponding marker point position information, and performs image feature extraction on the image to obtain corresponding image feature information; determines matching marker point pairs corresponding to matching image pairs in the multiple frames of images based on the marker point position information and the image feature information; and determines corresponding depth information and the corresponding posture of the image based on the matching marker point pairs.

[0163] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more frames of executable instructions for realizing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0165] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0166] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0167] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more frame lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0168] The embodiments of the present disclosure also provide a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method of any of the above embodiments can be implemented. The execution method and beneficial effects are similar and will not be repeated here.

[0169] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device that includes the element.

[0170] The foregoing are merely specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not to be limited to the embodiments described herein, but is to be construed in the broadest manner consistent with the principles and novel features disclosed herein. Industrial Applicability

[0171] The solution provided by the embodiment of the present disclosure can be applied to the field of computer technology. In the embodiment of the present disclosure, multiple frames of images can be obtained, wherein the images are obtained by a camera performing image acquisition on an object to be measured with marker points set on the surface; for the image, the marker points are identified to obtain the corresponding marker point position information, and the image feature is extracted to obtain the corresponding image feature information; based on the marker point position information and the image feature information, the matching marker point pairs corresponding to the matching image pairs in the multiple frames of images are determined; based on the matching marker point pairs, the corresponding depth information and the corresponding posture of the image are determined. With the above technical solution, the camera can be used to capture the image of the object to be measured. The cost of the camera is usually low, which is conducive to reducing costs. In the process of marker point matching (or registration), the external points can be filtered through the image feature information, so that the determined matching marker point pairs are more accurate, which is conducive to improving the accuracy of the depth information and posture. It can be seen that the photogrammetry method of the embodiment of the present disclosure has the advantages of low cost and high accuracy.

Claims

1. A photogrammetry method, comprising: Acquire multiple frames of images, wherein the images are acquired by a camera collecting images of an object with marking points set on its surface; For the image, performing marker point recognition on the image to obtain corresponding marker point position information, and performing image feature extraction on the image to obtain corresponding image feature information; Based on the marker point position information and the image feature information, determining matching marker point pairs corresponding to matching image pairs in the multiple frames of images; Based on the matching landmark point pairs, corresponding depth information and a corresponding position and posture of the image are determined.

2. The method according to claim 1, wherein: The determining, based on the marker point position information and the image feature information, matching marker point pairs corresponding to matching image pairs in the multiple frames of images comprises: Determine at least one initial image pair based on the multiple frames of images; For the initial image pair, image feature matching is performed based on the corresponding image feature information to obtain a corresponding first matching feature point pair, and a corresponding basic matrix is ​​determined based on the first matching feature point pair; For the initial image pair, determining a corresponding matching marker point pair based on the corresponding basic matrix, the first matching feature point pair and the marker point position information; An initial image pair whose number of matching marker point pairs meets a first preset threshold is determined as the matching image pair.

3. The method according to claim 2, wherein: The determining at least one initial image pair based on the multiple frames of images comprises: For the multiple frames of images, every two frames of images constitute an initial image pair.

4. The method according to claim 2, wherein: The determining the corresponding matching landmark point pair based on the corresponding basic matrix, the first matching feature point pair and the landmark point position information comprises: Based on the first matching feature point pair and the marker point position information, a corresponding initial marker point pair is determined. If the product of the position information of the initial marker point pair and the basic matrix is ​​less than or equal to a second preset threshold, the initial marker point pair is determined to be the matching marker point pair.

5. The method according to claim 2, wherein: Also includes: Get the IMU information between two adjacent frames of images; Wherein, determining the matching marker point pairs corresponding to the matching image pairs in the multiple frames of images based on the marker point position information and the image feature information includes: Based on the marker point position information, the image feature information and the IMU information, matching marker point pairs corresponding to matching image pairs in the multiple frames of images are determined.

6. The method according to claim 5, wherein: The determining, based on the marker point position information, the image feature information and the IMU information, matching marker point pairs corresponding to matching image pairs in the multiple frames of images comprises: For the multiple frames of images, two frames of images whose relative positions between frames meet a preset condition are combined into a matching image pair; For the matching image pair, performing image feature matching based on the corresponding image feature information to obtain a corresponding first matching feature point pair; For the matching image pair, if the number of the corresponding first matching feature point pairs is greater than the third preset threshold, the movement direction between the two frames of the image in the matching image pair is determined based on the corresponding IMU information, and the corresponding second matching feature point pairs are determined based on the movement direction; if the number of the second matching feature point pairs is greater than the fourth preset threshold, the corresponding basic matrix is ​​determined based on the second matching feature point pairs, and the corresponding matching marker point pairs are determined based on the corresponding basic matrix, the second matching feature point pairs and the marker point position information.

7. The method according to claim 6, wherein: The determining the corresponding matching landmark point pair based on the corresponding basic matrix, the second matching feature point pair and the landmark point position information comprises: Based on the second matching feature point pair and the marker point position information, the corresponding initial marker point pair is determined. If the product of the position information of the initial marker point pair and the basic matrix is ​​less than or equal to a second preset threshold, the initial marker point pair is determined to be the matching marker point pair.

8. The method according to claim 6, wherein: Also includes: For the matching image pair, if the number of the first matching feature point pairs is less than or equal to the third preset threshold, the movement direction between the two frames of images in the matching image pair is determined based on the corresponding IMU information, and the corresponding matching marker point pairs are determined based on the movement direction and the marker point position information.

9. The method according to claim 1, wherein: The determining corresponding depth information and a posture corresponding to the image based on the matching landmark point pair includes: Selecting an initial reconstructed image pair from the matching image pair; Performing triangulation processing on the matching landmark point pairs corresponding to the initial reconstructed image to obtain corresponding depth information; In the case where a preset reconstruction stop condition is not met, a next reconstructed image pair is determined and the next reconstructed image pair is reconstructed until the preset reconstruction stop condition is met, wherein the determining the next reconstructed image pair and the reconstructing the next reconstructed image pair include: determining an adjacent image corresponding to a previous reconstructed image pair, and a next reconstructed image pair including the adjacent image, wherein the adjacent image has the most common view marker points with the previous reconstructed image pair; determining the corresponding pose of the adjacent image based on the depth information of the common view marker points and the position information of the common view marker points in the adjacent image; and triangulating the matching marker point pair corresponding to the next reconstructed image pair to obtain corresponding depth information.

10. The method according to claim 9, wherein: Also includes: The depth information, the pose, and the camera intrinsic parameters are optimized.

11. A photogrammetric device, wherein: include: A first acquisition module is configured to acquire multiple frames of images, wherein the images are acquired by a camera performing image acquisition on an object with marking points set on its surface; A first extraction module is configured to perform marker point recognition on the image to obtain corresponding marker point position information, and perform image feature extraction on the image to obtain corresponding image feature information; A first determination module is configured to determine matching image pairs in the multiple frames of images and matching landmark point pairs corresponding to the matching image pairs based on the landmark point position information and the image feature information; The first reconstruction module is configured to determine corresponding depth information and a corresponding position and posture of the image based on the matching landmark point pairs.

12. An electronic device, wherein: include: A processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the method according to any one of claims 1 to 10.

13. A non-volatile computer-readable storage medium, wherein: The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images