Spatial scene positioning method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure disclose a spatial scene positioning method and device, electronic equipment and storage medium, wherein the method comprises: acquiring a first video stream by using an image acquisition device to acquire images in a target space; determining a plurality of second images matched with a first image in the first video stream from a model image library; wherein the model image library stores a plurality of three-dimensional models corresponding to the target space and a plurality of spatial two-dimensional images with color information, each three-dimensional model corresponds to at least one spatial two-dimensional image; determining at least one three-dimensional model corresponding to the first image based on the plurality of second images, and determining at least one set of first pose information corresponding to the image acquisition device based on the at least one three-dimensional model and the first image; verifying and screening the at least one set of first pose information to determine target pose information of the image acquisition device.

Description

Technical Field

[0001] This disclosure relates to computer vision technology, and in particular to a spatial scene localization method, apparatus, electronic device, and storage medium. Background Technology

[0002] When users arrive in an unfamiliar place, they are likely to get lost without navigation. The same applies in large indoor spaces such as factories, shopping malls, and parking lots. Users often struggle to determine their location and destination within these complex environments. Finding one's location and easily locating businesses is a significant challenge. Asking for directions manually can sometimes be inaccurate, and indoor signage suffers from various directional issues. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a spatial scene positioning method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of the present disclosure, a spatial scene positioning method is provided, comprising:

[0005] Image acquisition is performed in the target space using an image acquisition device to obtain a first video stream; wherein the first video stream includes at least one frame of a first image;

[0006] Based on a first image frame in the first video stream, multiple second images matching the first image are determined from a model image library; wherein, the model image library stores a three-dimensional model corresponding to the target space and multiple two-dimensional spatial images with color information;

[0007] Multiple pairs of first matching points are determined based on the plurality of second images and the first image; wherein each pair of first matching points includes a feature point in the first image and a feature point in the second image;

[0008] Based on the multiple pairs of first matching points and the three-dimensional model, at least one set of first pose information corresponding to the image acquisition device is determined;

[0009] The target pose information of the image acquisition device is determined by verifying and filtering the at least one set of first pose information.

[0010] Optionally, before determining a plurality of second images matching the first image from the model image library based on a frame of the first image in the first video stream, the method further includes:

[0011] The three-dimensional model corresponding to the target space and multiple two-dimensional images of the space are preprocessed to determine the descriptive feature information and three-dimensional coordinate information corresponding to multiple feature points.

[0012] Optionally, the preprocessing of the three-dimensional model corresponding to the target space and the multiple two-dimensional images of the space to determine the descriptive feature information and three-dimensional coordinate information corresponding to multiple feature points includes:

[0013] Feature extraction is performed on multiple spatial two-dimensional images using at least one feature extraction network to obtain descriptive feature information corresponding to each of the multiple feature points;

[0014] Based on the three-dimensional model, the point cloud data corresponding to the target space is determined, and the three-dimensional spatial information corresponding to each of the plurality of feature points is determined based on the point cloud data.

[0015] Optionally, determining a plurality of second images matching the first image from a model image library based on a frame of the first image in the first video stream includes:

[0016] Feature extraction is performed on the first image to obtain the first image features;

[0017] The first image features are matched with the spatial image features corresponding to multiple pre-stored two-dimensional spatial images in the model image library, and the multiple second images are determined based on the matching results.

[0018] Optionally, determining multiple pairs of matching points based on the plurality of second images and the first image includes:

[0019] Feature point matching is performed on the first image and each of the plurality of second images to obtain multiple pairs of second matching points;

[0020] The multiple pairs of second matching points are filtered to obtain multiple pairs of first matching points.

[0021] Optionally, the step of filtering the plurality of second matching point pairs to obtain plurality of first matching point pairs includes:

[0022] For each of the multiple pairs of second matching point pairs, based on the two-dimensional planar information of the feature points corresponding to the first image and the three-dimensional spatial information of the feature points corresponding to the second image, the second pose information corresponding to each second image is determined;

[0023] Based on the gravity direction information corresponding to the image acquisition device, multiple second pose information are filtered to obtain at least one filtered second pose information;

[0024] The plurality of first matching point pairs are determined based on at least one second matching point pair corresponding to the second pose information.

[0025] Optionally, the step of filtering the plurality of second matching point pairs to obtain plurality of first matching point pairs includes:

[0026] For each of the multiple pairs of matching points, based on the two-dimensional planar information of the feature points corresponding to the first image and the three-dimensional spatial information of the feature points corresponding to the second image, the second pose information corresponding to each second image is determined.

[0027] By using a stochastic consensus algorithm based on at least one prior information, multiple second pose information are filtered to obtain at least one filtered second pose information;

[0028] The plurality of first matching point pairs are determined based on at least one second matching point pair corresponding to the second pose information.

[0029] Optionally, determining at least one set of first pose information corresponding to the image acquisition device based on the plurality of first matching point pairs and the three-dimensional model includes:

[0030] Based on each pair of first matching point pairs in the plurality of first matching point pairs, determine the plurality of third matching point pairs corresponding to each pair of first matching point pairs;

[0031] Based on the bundle adjustment method, the correct matching pairs of the first and third matching point pairs are determined, and the correct matching pairs of the fourth matching point pairs are determined according to the judgment results.

[0032] Based on the multiple pairs of fourth matching points, at least one set of first pose information corresponding to the image acquisition device is determined.

[0033] Optionally, verifying and filtering the at least one set of first pose information to determine the target pose information of the image acquisition device includes:

[0034] The target pose information of the image acquisition device is determined by verifying and filtering the at least one set of first pose information based on at least one prior information; wherein the prior information includes at least one of the following: the gravity direction of the image acquisition device, the position information of the preset positioning device, and the third pose information of the image acquisition device determined by visual odometry based on the first image.

[0035] Optionally, it also includes:

[0036] Based on the target pose information of the image acquisition device and the destination input by the user, a navigation route is determined in the target space.

[0037] According to another aspect of the embodiments of this disclosure, a spatial scene positioning device is provided, comprising:

[0038] An image acquisition module is used to acquire images in a target space using an image acquisition device to obtain a first video stream; wherein the first video stream includes at least one frame of a first image;

[0039] The image matching module is used to determine multiple second images that match the first image from a model image library based on a first image frame in the first video stream; wherein, the model image library stores a three-dimensional model corresponding to the target space and multiple spatial two-dimensional images with color information;

[0040] The point-to-point matching module is used to determine multiple pairs of first matching point pairs based on the plurality of second images and the first image; wherein each second image corresponds to at least one pair of first matching point pairs, and each pair of first matching point pairs includes a feature point in the first image and a feature point in the second image;

[0041] The pose estimation module is used to determine multiple sets of first pose information corresponding to the image acquisition device based on the multiple pairs of first matching points and the three-dimensional model;

[0042] The verification and filtering module is used to verify and filter the multiple sets of first pose information to determine the target pose information of the image acquisition device.

[0043] Optionally, the device further includes:

[0044] The preprocessing module is used to preprocess the three-dimensional model corresponding to the target space and multiple two-dimensional images of the space to determine the descriptive feature information and three-dimensional coordinate information corresponding to multiple feature points.

[0045] Optionally, the preprocessing module is specifically used to perform feature extraction on the multiple spatial two-dimensional images using at least one feature extraction network to obtain descriptive feature information corresponding to each of the multiple feature points; based on the three-dimensional model, determine the point cloud data corresponding to the target space, and based on the point cloud data, determine the three-dimensional spatial information corresponding to each of the multiple feature points.

[0046] Optionally, the image matching module is specifically used to extract features from the first image to obtain first image features; match the first image features with spatial image features corresponding to multiple spatial two-dimensional images pre-stored in the model image library, and determine the multiple second images based on the matching results.

[0047] Optionally, the point-to-point matching module includes:

[0048] An initial matching unit is used to perform feature point matching on the first image and each of the plurality of second images to obtain multiple pairs of second matching point pairs;

[0049] The filtering processing unit is used to filter the multiple pairs of second matching point pairs to obtain multiple pairs of first matching point pairs.

[0050] Optionally, the filtering processing unit is specifically configured to, for each of the plurality of pairs of second matching point pairs, determine the second pose information corresponding to each second image based on the two-dimensional planar information of the feature points corresponding to the first image and the three-dimensional spatial information of the feature points corresponding to the second image; filter the plurality of second pose information based on the gravity direction information corresponding to the image acquisition device to obtain at least one filtered second pose information; and determine the plurality of pairs of first matching point pairs based on the second matching point pairs corresponding to at least one second pose information.

[0051] Optionally, the filtering processing unit is specifically used to, for each of the multiple pairs of matching point pairs, determine the second pose information corresponding to each of the second images based on the two-dimensional planar information of the feature points corresponding to the first image and the three-dimensional spatial information of the feature points corresponding to the second image; filter the multiple pairs of second pose information using a random consistency algorithm based on at least one prior information to obtain at least one filtered pair of second pose information; and determine the multiple pairs of first matching point pairs based on the second matching point pairs corresponding to at least one pair of second pose information.

[0052] Optionally, the pose estimation module is specifically used to determine multiple pairs of third matching point pairs corresponding to each pair of first matching point pairs based on each pair of first matching point pairs; to determine multiple pairs of correctly matched fourth matching point pairs based on the judgment result of the multiple pairs of first matching point pairs and the multiple pairs of third matching point pairs using the bundle adjustment method; and to determine at least one set of first pose information corresponding to the image acquisition device based on the multiple pairs of fourth matching point pairs.

[0053] Optionally, the verification and filtering module is specifically used to verify and filter the at least one set of first pose information based on at least one prior information to determine the target pose information of the image acquisition device; wherein, the prior information includes at least one of the following: the gravity direction of the image acquisition device, the position information of the preset positioning device, and the third pose information of the image acquisition device determined by visual odometry based on the first image.

[0054] Optionally, the device further includes:

[0055] The navigation module is used to determine a navigation route in the target space based on the target pose information of the image acquisition device and the destination input by the user.

[0056] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0057] Memory, used to store computer program products;

[0058] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the method described in any of the above embodiments.

[0059] According to another aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the method described in any of the above embodiments.

[0060] According to another aspect of the present disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the method described in any of the above embodiments.

[0061] Based on the above embodiments of this disclosure, a spatial scene localization method, apparatus, electronic device, and storage medium are provided. The method includes: acquiring an image in a target space using an image acquisition device to obtain a first video stream; determining a plurality of second images matching the first image from a model image library based on a first image frame in the first video stream; wherein the model image library stores a plurality of three-dimensional models corresponding to the target space and a plurality of spatial two-dimensional images with color information, each three-dimensional model corresponding to at least one spatial two-dimensional image; determining at least one three-dimensional model corresponding to the first image based on the plurality of second images; determining at least one set of first pose information corresponding to the image acquisition device based on at least one three-dimensional model and the first image; verifying and filtering the at least one set of first pose information to determine the target pose information of the image acquisition device. This embodiment, by obtaining a first image in the target space, combining image matching and feature point matching, and using known three-dimensional information in the three-dimensional model, can determine the three-dimensional information corresponding to the feature points in the two-dimensional first image, thereby determining at least one set of first pose information, and determining the target pose information through verification and filtering, improves the accuracy of the determined target pose information.

[0062] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0063] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0064] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0065] Figure 1 This is a flowchart illustrating a spatial scene positioning method provided in an exemplary embodiment of this disclosure;

[0066] Figure 2 This is a public announcement Figure 1 A flowchart illustrating step 104 in the illustrated embodiment;

[0067] Figure 3 This is a public announcement Figure 1 A flowchart illustrating step 106 in the illustrated embodiment;

[0068] Figure 4 This is a public announcement Figure 1 A flowchart illustrating step 108 in the illustrated embodiment;

[0069] Figure 5 This is a schematic diagram of the structure of a spatial scene positioning device provided in an exemplary embodiment of this disclosure;

[0070] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0071] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0072] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0073] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0074] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0075] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0076] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship. The data referred to in this disclosure can include unstructured data such as text, images, and videos, as well as structured data.

[0077] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0078] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0079] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0080] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0081] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0082] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0083] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0084] Exemplary methods

[0085] Figure 1 This is a schematic flowchart of a spatial scene positioning method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 1 As shown, it includes the following steps:

[0086] Step 102: Use an image acquisition device to acquire images in the target space to obtain the first video stream.

[0087] The first video stream includes at least one frame of a first image.

[0088] In this embodiment, the target space can be any scene space, such as a parking lot, supermarket, factory, hospital, etc.; it only needs to have a corresponding three-dimensional space model. This embodiment does not limit the application scenario. The image acquisition device can be any device with image acquisition function, such as a mobile phone, camera, camera, etc. Optionally, the user holds the image acquisition device and enters the target space to acquire images of the current scene. Based on the user's movement in the target space, image acquisition can be continuously performed to obtain a first video stream. Since the first video stream consists of at least one first image, the first image can be any frame in the video stream. Obtaining the first image enables the subsequent determination of the pose of the image acquisition device when it obtains the first image, that is, the positioning of the image acquisition device in the target space is realized. This embodiment uses the first image as an example for explanation. It can be understood that the processing of other frames is similar to that of the first image.

[0089] Step 104: Based on a first image frame in the first video stream, determine multiple second images from the model image library that match the first image.

[0090] The model image library stores the three-dimensional model corresponding to the target space and multiple two-dimensional spatial images with color information.

[0091] In one embodiment, the target space is a known space. Before executing this embodiment, three-dimensional information and two-dimensional color information of the target space are collected. Furthermore, by collecting the overlapping points of the three-dimensional and two-dimensional information, the three-dimensional information of each point in the target space can be correlated with the two-dimensional information. That is, the three-dimensional information and two-dimensional information of each point in the three-dimensional model are known. In addition, in order to facilitate image retrieval, the two-dimensional images of the space are stored independently in the model image library, and the correspondence between the two-dimensional images of the space and the three-dimensional model is saved. That is, the three-dimensional information of each point in the two-dimensional image of the space can be determined in the corresponding three-dimensional model.

[0092] Step 106: Determine multiple pairs of first matching points based on multiple second images and the first image.

[0093] Each pair of first matching points includes a feature point in the first image and a feature point in the second image.

[0094] Optionally, after implementing two-dimensional image retrieval, since there will inevitably be some incorrect matches among the multiple second images obtained, this embodiment obtains multiple pairs of first matching point pairs by matching the corresponding feature points in the second image and the first image. At this time, the second image without a matching first matching point pair indicates that the second image is an incorrect match and will be filtered out. Therefore, the filtering of multiple second images is achieved by feature point matching, which improves the accuracy of the filtered second images.

[0095] Step 108: Based on multiple pairs of first matching points and the 3D model, determine multiple sets of first pose information corresponding to the image acquisition device.

[0096] Optionally, based on the 3D information of the 3D model, the 2D and 3D information corresponding to each point in the first image can be determined. Given the known 2D and 3D information of each point, at least one set of first pose information can be determined using existing pose estimation methods. Each set of first pose information includes 6 degrees of freedom (3 translational degrees of freedom corresponding to the x, y, and z axes, and 3 rotational degrees of freedom, respectively). The position information of the image acquisition device can then be determined based on the translational degrees of freedom, achieving initial positioning. Furthermore, determining the pose information by using the corresponding 2D and 3D spatial information of the first matched point pair after matching processing can effectively improve the accuracy of the first pose information.

[0097] Step 110: Verify and filter at least one set of first pose information to determine the target pose information of the image acquisition device.

[0098] In this embodiment, prior information with verification function can be obtained by at least one method. Based on the prior information, at least one first pose information can be verified and filtered, which can improve the accuracy of the target pose information and make the final positioning result achieve a better positioning effect.

[0099] The spatial scene localization method provided in the above embodiments of this disclosure includes: acquiring images in a target space using an image acquisition device to obtain a first video stream; determining multiple second images matching the first image from a model image library based on a first image frame in the first video stream; wherein the model image library stores multiple three-dimensional models corresponding to the target space and multiple spatial two-dimensional images with color information, each three-dimensional model corresponding to at least one spatial two-dimensional image; determining at least one three-dimensional model corresponding to the first image based on the multiple second images; determining at least one set of first pose information corresponding to the image acquisition device based on at least one three-dimensional model and the first image; verifying and filtering the at least one set of first pose information to determine the target pose information of the image acquisition device; this embodiment, by obtaining a first image in the target space, combining image matching and feature point matching, and the three-dimensional information in the known three-dimensional model, can determine the three-dimensional information corresponding to the feature points in the two-dimensional first image, thereby determining at least one set of first pose information, and determining the target pose information through verification and filtering, thus improving the accuracy of the determined target pose information.

[0100] In some optional embodiments, prior to performing step 104, the following may also be included:

[0101] The three-dimensional model corresponding to the target space and multiple two-dimensional spatial images are preprocessed to determine the descriptive feature information and three-dimensional coordinate information corresponding to multiple feature points.

[0102] The 3D model provided in this embodiment mainly includes color information and geometric information. Color information helps us match the video stream used for positioning, while geometric information provides spatial location. Data preprocessing primarily aims to extract color features (descriptive feature information obtained from 2D planar images) and corresponding spatial locations (coordinates of 3D points, i.e., the x, y, and z coordinates of feature points in the 3D model) from redundant data for efficient matching. This embodiment performs sparsity processing on the points in the 3D model, extracting only a portion of representative points as feature points (e.g., corner points or feature points of different categories after classification by a neural network).

[0103] Optionally, the three-dimensional model corresponding to the target space and multiple two-dimensional spatial images are preprocessed to determine the descriptive feature information and three-dimensional coordinate information corresponding to multiple feature points, including:

[0104] Feature extraction is performed on multiple spatial two-dimensional images using at least one feature extraction network to obtain descriptive feature information corresponding to each feature point among multiple feature points; wherein, the descriptive feature information may include, but is not limited to, high-dimensional uninterpretable features (e.g., feature vectors output by a certain layer in a neural network, features that describe the image, similar to feature fingerprints, etc.), low-dimensional manually defined features, and category features (information such as table, sofa, stairs, KFC logo, etc.), semantic features, etc.

[0105] Based on the 3D model, the point cloud data corresponding to the target space is determined, and the 3D spatial information corresponding to each feature point among multiple feature points is determined based on the point cloud data.

[0106] In this embodiment, when restoring point cloud data using a 3D model, interpolation and culling processes ensure the accuracy of the point cloud data and improve the accuracy of the 3D spatial information corresponding to the feature points, avoiding the problem of being unable to obtain 3D spatial information. Optionally, the position of the feature points in space can be obtained from the surface of the 3D model, but for special objects (determined by semantic segmentation, such as mirrors, leaves, etc., where mirrors can lead to semantic segmentation errors and leaves can cause objects to be occluded), pre-labeling can improve the accuracy of the 3D spatial information.

[0107] like Figure 2 As shown above, in the above Figure 1 Based on the illustrated embodiment, step 104 may include the following steps:

[0108] Step 1041: Extract features from the first image to obtain the features of the first image.

[0109] Optionally, features can be extracted from the first image using at least one neural network, which may correspond to the neural network used in preprocessing to extract features from a two-dimensional image in the target space. For example, it may include a classification network, a segmentation network, etc. The obtained first image features can enable fast image retrieval.

[0110] Step 1042: Match the first image features with the spatial image features corresponding to multiple pre-stored two-dimensional spatial images in the model image library, and determine multiple second images based on the matching results.

[0111] This embodiment can also describe the image as a whole as a feature vector (corresponding to the first image feature), and determine the similarity between the spatial image features in the form of feature vectors pre-stored in the model image library and the first image features (the similarity can be determined by the distance between the feature vectors). Multiple second images can be determined based on the similarity. Optionally, the spatial two-dimensional images with the highest similarity can be selected as the second images based on the similarity size. Alternatively, a similarity threshold can be set (the specific value is determined according to the actual application scenario), and the spatial two-dimensional images with similarity greater than the similarity threshold can be selected as the second images, and so on.

[0112] like Figure 3 As shown above, in the above Figure 1 Based on the illustrated embodiment, step 106 may include the following steps:

[0113] Step 1061: Perform feature point matching on the first image and each of the multiple second images to obtain multiple pairs of second matching points.

[0114] Optionally, since the preprocessing determines the descriptive feature information and three-dimensional coordinate information of multiple feature points for each spatial two-dimensional image, in order to improve the matching accuracy, this embodiment proposes to perform feature point matching between the second image and the first image, so that the matching feature point pairs can more accurately determine the pose information of the image acquisition device.

[0115] Step 1062: Filter the multiple pairs of second matching points to obtain multiple pairs of first matching points.

[0116] In this embodiment, feature points are matched based on their positions in the image and their descriptors (corresponding to multiple image points within a defined range around the feature point). This improves the accuracy of feature point matching. Through feature point matching, this embodiment obtains a set of second matching point pairs between the first and second images. To further improve the accuracy of the matched feature point pairs, this embodiment also performs filtering on the second matching point pairs, for example, by eliminating some incorrect matches using prior information.

[0117] Optionally, in some alternative embodiments, step 1062 may include:

[0118] For each pair of second matching point pairs in the multiple pairs of second matching point pairs, the second pose information corresponding to each second image is determined based on the two-dimensional planar information of the feature points corresponding to the first image and the three-dimensional spatial information of the feature points corresponding to the second image.

[0119] Based on the gravity direction information corresponding to the image acquisition device, multiple second pose information are filtered to obtain at least one filtered second pose information.

[0120] Based on at least one second matching point pair corresponding to second pose information, multiple pairs of first matching point pairs are determined.

[0121] In this embodiment, the second matching point pair includes feature points in the first image and feature points in the second image. The feature points in the first image can determine two-dimensional information, and the feature points in the second image can determine the corresponding three-dimensional information through the three-dimensional model corresponding to the second image. Based on the two-dimensional and three-dimensional information corresponding to the matched feature points, the second pose information corresponding to each second image can be determined by the extrinsic parameter calibration method in the prior art (e.g., PnP algorithm, etc.). The image acquisition device usually has a gravimeter, which can determine the gravity direction of the image acquisition device. In this embodiment, the gravity direction is used as prior information to filter the multiple calculated second pose information. The filtering process can be to match the degrees of freedom representing the gravity direction in the second pose with the gravity direction information, and only retain the second pose information that matches the gravity direction. Relatively accurate second pose information is obtained through the prior information of the image acquisition device.

[0122] Alternatively, in some other alternative embodiments, step 1062 may include:

[0123] For each pair of matching points in the multiple pairs of matching points, the second pose information corresponding to each second image is determined based on the two-dimensional planar information of the feature points corresponding to the first image and the three-dimensional spatial information of the feature points corresponding to the second image.

[0124] By using a stochastic consensus algorithm based on at least one prior information, multiple second pose information are filtered to obtain at least one filtered second pose information;

[0125] Based on at least one second matching point pair corresponding to second pose information, multiple pairs of first matching point pairs are determined.

[0126] The method for determining multiple second pose information in this embodiment is the same as in the above embodiments, and will not be repeated here. For the filtering of second pose information, this embodiment adopts the Random Consensus (RANSAC) algorithm. The Random Consensus algorithm randomly selects several sets of matches, solves the solution of the mathematical model (in this embodiment, it solves the spatial position), and verifies the interior and exterior points of the entire set. The above process is repeated multiple times to obtain the optimal solution result. The RANSAC algorithm is a multiplier algorithm that requires a lot of computation. In order to speed up the filtering speed and improve the accuracy, this embodiment proposes a RANSAC scheme based on at least one prior information. The prior information may include, but is not limited to: 1. Gravity direction: After the subset is extracted by RANSAC, since the gravity direction is known, the dimension of the mathematical model solver is reduced from six degrees of freedom (three-dimensional spatial position and three rotational components) to four degrees of freedom (three-dimensional spatial position and one rotation about the gravity direction). 1. Since fewer degrees of freedom mean fewer variables to solve, the solution is more stable and faster. 2. Prior information for auxiliary positioning tools such as Bluetooth beacons and / or WiFi beacons (the auxiliary positioning tools are pre-set at preset positions in the target space, and the position information of each auxiliary positioning tool is known): After RANSAC extracts a subset, the mathematical model is still six degrees of freedom. However, since Bluetooth, WiFi, and other beacons can provide a roughly accurate spatial position, the results obtained by RANSAC in each round need to be verified in two ways: a) there are enough interior points; b) the spatial position of the solution is consistent with the prior information to a certain extent (the error of Bluetooth and WiFi beacons is relatively large, usually considered to be around 0.5 meters), and finally a more stable solution is obtained. 3. Visual odometry (visual odometry (using visual odometry to estimate the pose information of the image acquisition device based on the first captured image) serves as a priori. Typically, visual odometry pose estimation is relatively accurate over a short period (if visual information is lost, IMU integration over a short period will not have a significant problem). Therefore, if the pose of the previous frame is solved, the pose + visual odometry = the pose of the current frame. If a certain degree of deviation is acceptable, the current matching result can be filtered out within a certain threshold. If there is a covariance matrix of multiple sensors such as vision and IMU, the filtering threshold is defined based on the covariance information; otherwise, the threshold is defined by the time stamp distance to filter out outliers (i.e., the closer in time, the smaller the odometry drift error).

[0127] In this embodiment, the prior information can be acquired by setting up devices such as lasers, gravimeters, and inertial sensors in the image acquisition device, thereby enabling preliminary positioning in the unknown space.

[0128] like Figure 4 As shown above, in the above Figure 1Based on the illustrated embodiment, step 108 may include the following steps:

[0129] Step 1081: Based on each pair of first matching points in the multiple pairs of first matching point pairs, determine the multiple pairs of third matching point pairs corresponding to each pair of first matching point pairs.

[0130] Step 1082: Based on the bundle adjustment method, determine whether multiple pairs of first matching point pairs and multiple pairs of third matching point pairs are correct, and determine the multiple pairs of fourth matching point pairs that are correctly matched based on the judgment results.

[0131] Step 1083: Based on multiple pairs of fourth matching points, determine at least one set of first pose information corresponding to the image acquisition device.

[0132] In this embodiment, to improve the accuracy of the first pose information, a preset number of point pairs are acquired around each first matching point pair as third matching point pairs. The aggregation of these third matching point pairs enables stable and accurate determination of the first pose information. Bundle adjustment, using the camera's pose and the 3D coordinates of the measurement points as unknown parameters, and the coordinates of feature points detected on the image for forward intersection as observation data, performs adjustment to obtain the optimal camera parameters and world point coordinates. This embodiment aims to determine whether the first and third matching point pairs among multiple images are correct, and the position of the images in the spatial model.

[0133] In some alternative embodiments, step 110 may include:

[0134] Based on at least one prior information, at least one set of first pose information is verified and filtered to determine the target pose information of the image acquisition device.

[0135] The prior information includes, but is not limited to, at least one of the following: the gravity direction of the image acquisition device, the position information of the preset positioning device, and the third pose information of the image acquisition device determined based on the first image using a visual odometry.

[0136] In this embodiment, after obtaining at least one set of first pose information, to obtain more accurate target pose information, this embodiment provides a method for verifying and filtering each set of first pose information based on prior information. Prior information can be obtained from known hardware devices, such as obtaining the gravity direction of the image acquisition device through a gravimeter built into the image acquisition device, assisting spatial positioning by setting multiple Bluetooth and / or WiFi beacons at multiple known locations, and verifying the accuracy of the first pose information using third pose information estimated by visual odometry. The first pose information verified by at least one prior information is used as the target pose information to complete the positioning of the image acquisition device in the target space. This embodiment achieves efficient positioning, including positioning correction and repositioning functions. Repositioning is performed during the initialization process of the entire system and when positioning is lost. Positioning correction corrects the drift of the front-end system during long-distance operation during navigation.

[0137] In some optional embodiments, the method provided in this embodiment further includes:

[0138] Based on the target pose information from the image acquisition device and the destination input by the user, a navigation route is determined in the target space.

[0139] After locating the image acquisition device, its position in the 3D model can be determined. This position can then be applied to AR positioning and navigation to guide the user. In addition to navigating to the user's input destination, it can also provide routes for obtaining items based on the user's input. This overcomes the problem that users cannot reach their destination correctly due to a lack of understanding of the target space. This embodiment can be applied in various scenarios (e.g., supermarkets, factories, hospitals, etc.).

[0140] Any spatial scene localization method provided in the embodiments of this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any spatial scene localization method provided in the embodiments of this disclosure can be executed by a processor, such as by a processor executing any spatial scene localization method mentioned in the embodiments of this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0141] Exemplary device

[0142] Figure 5 This is a schematic diagram of the structure of a spatial scene positioning device provided in an exemplary embodiment of this disclosure. Figure 5 As shown, the apparatus provided in this embodiment includes:

[0143] The image acquisition module 51 is used to acquire images in the target space using an image acquisition device to obtain a first video stream. The first video stream includes at least one frame of a first image.

[0144] The image matching module 52 is used to determine multiple second images that match the first image from the model image library based on a frame of the first image in the first video stream.

[0145] The model image library stores the three-dimensional model corresponding to the target space and multiple two-dimensional spatial images with color information.

[0146] The point-to-point matching module 53 is used to determine multiple pairs of first matching points based on multiple second images and the first image.

[0147] Each second image corresponds to at least one pair of first matching points, and each pair of first matching points includes a feature point in the first image and a feature point in the second image.

[0148] The pose estimation module 54 is used to determine multiple sets of first pose information corresponding to the image acquisition device based on multiple pairs of first matching point pairs and the three-dimensional model.

[0149] The verification and filtering module 55 is used to verify and filter multiple sets of first pose information to determine the target pose information of the image acquisition device.

[0150] The spatial scene positioning device provided in the above embodiments of this disclosure includes: acquiring images in a target space using an image acquisition device to obtain a first video stream; determining multiple second images matching the first image from a model image library based on a first image frame in the first video stream; wherein the model image library stores multiple three-dimensional models corresponding to the target space and multiple spatial two-dimensional images with color information, each three-dimensional model corresponding to at least one spatial two-dimensional image; determining at least one three-dimensional model corresponding to the first image based on the multiple second images, and determining at least one set of first pose information corresponding to the image acquisition device based on at least one three-dimensional model and the first image; verifying and filtering the at least one set of first pose information to determine the target pose information of the image acquisition device; this embodiment, by obtaining a first image in the target space, combining image matching and feature point matching, and the three-dimensional information in the known three-dimensional model, can determine the three-dimensional information corresponding to the feature points in the two-dimensional first image, thereby determining at least one set of first pose information, and determining the target pose information through verification and filtering, thus improving the accuracy of the determined target pose information.

[0151] In some optional embodiments, the apparatus provided in this embodiment further includes:

[0152] The preprocessing module is used to preprocess the 3D model corresponding to the target space and multiple 2D spatial images to determine the descriptive feature information and 3D coordinate information corresponding to multiple feature points.

[0153] Optionally, the preprocessing module is specifically used to perform feature extraction on multiple spatial two-dimensional images using at least one feature extraction network to obtain descriptive feature information corresponding to each feature point among multiple feature points; based on the three-dimensional model, determine the point cloud data corresponding to the target space, and determine the three-dimensional spatial information corresponding to each of the multiple feature points based on the point cloud data.

[0154] In some optional embodiments, the image matching module 52 is specifically used to extract features from the first image to obtain first image features; match the first image features with spatial image features corresponding to multiple spatial two-dimensional images pre-stored in the model image library, and determine multiple second images based on the matching results.

[0155] In some optional embodiments, the point-to-point matching module 53 includes:

[0156] An initial matching unit is used to perform feature point matching on the first image and each of the multiple second images to obtain multiple pairs of second matching point pairs;

[0157] The filtering processing unit is used to filter multiple pairs of second matching point pairs to obtain multiple pairs of first matching point pairs.

[0158] In some optional embodiments, the filtering processing unit is specifically used to determine the second pose information corresponding to each second image for each of the multiple pairs of second matching point pairs, based on the two-dimensional planar information of the feature points corresponding to the first image and the three-dimensional spatial information of the feature points corresponding to the second image; to filter the multiple second pose information based on the gravity direction information corresponding to the image acquisition device, to obtain at least one filtered second pose information; and to determine multiple pairs of first matching point pairs based on the second matching point pairs corresponding to at least one second pose information.

[0159] In some alternative embodiments, the filtering processing unit is specifically used to determine the second pose information corresponding to each second image for each of the multiple pairs of matching point pairs, based on the two-dimensional planar information of the feature points corresponding to the first image and the three-dimensional spatial information of the feature points corresponding to the second image; to filter the multiple second pose information using a random consistency algorithm based on at least one prior information to obtain at least one filtered second pose information; and to determine multiple pairs of first matching point pairs based on the second matching point pairs corresponding to at least one second pose information.

[0160] In some optional embodiments, the pose estimation module 54 is specifically used to determine multiple pairs of third matching point pairs corresponding to each pair of first matching point pairs based on each pair of first matching point pairs; to determine multiple pairs of correctly matched fourth matching point pairs based on the judgment result of the multiple pairs of first matching point pairs and multiple pairs of third matching point pairs using the bundle adjustment method; and to determine at least one set of first pose information corresponding to the image acquisition device based on the multiple pairs of fourth matching point pairs.

[0161] In some optional embodiments, the verification and filtering module 55 is specifically used to verify and filter the at least one set of first pose information based on at least one prior information to determine the target pose information of the image acquisition device; wherein, the prior information includes at least one of the following: the gravity direction of the image acquisition device, the position information of the preset positioning device, and the third pose information of the image acquisition device determined by the visual odometry based on the first image.

[0162] In some optional embodiments, the apparatus provided in this embodiment further includes:

[0163] The navigation module is used to determine the navigation route in the target space based on the target pose information of the image acquisition device and the destination input by the user.

[0164] Exemplary electronic devices

[0165] Below, for reference Figure 6 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0166] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0167] like Figure 6 As shown, the electronic device includes one or more processors and memory.

[0168] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0169] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the spatial scene positioning methods of the various embodiments of this disclosure described above and / or other desired functions.

[0170] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0171] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0172] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0173] Of course, for the sake of simplicity, Figure 6 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0174] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the spatial scene positioning methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0175] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0176] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the spatial scene positioning methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0177] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0178] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0179] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0180] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0181] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0182] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0183] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0184] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A spatial scene positioning method, characterized in that, include: Image acquisition is performed in the target space using an image acquisition device to obtain a first video stream; wherein the first video stream includes at least one frame of a first image; Based on a first image frame in the first video stream, multiple second images matching the first image are determined from a model image library; wherein, the model image library stores a three-dimensional model corresponding to the target space and multiple two-dimensional spatial images with color information; Multiple pairs of first matching points are determined based on the plurality of second images and the first image; wherein each pair of first matching points includes a feature point in the first image and a feature point in the second image; Based on the multiple pairs of first matching points and the three-dimensional model, at least one set of first pose information corresponding to the image acquisition device is determined; The target pose information of the image acquisition device is determined by verifying and filtering the at least one set of first pose information.

2. The method according to claim 1, characterized in that, Before determining multiple second images matching the first image from the model image library based on a first image frame in the first video stream, the method further includes: The three-dimensional model corresponding to the target space and multiple two-dimensional images of the space are preprocessed to determine the descriptive feature information and three-dimensional coordinate information corresponding to multiple feature points.

3. The method according to claim 2, characterized in that, The preprocessing of the three-dimensional model corresponding to the target space and the multiple two-dimensional images of the space to determine the descriptive feature information and three-dimensional coordinate information corresponding to multiple feature points includes: Feature extraction is performed on multiple spatial two-dimensional images using at least one feature extraction network to obtain descriptive feature information corresponding to each of the multiple feature points; Based on the three-dimensional model, the point cloud data corresponding to the target space is determined, and the three-dimensional spatial information corresponding to each of the plurality of feature points is determined based on the point cloud data.

4. The method according to any one of claims 1-3, characterized in that, The step of determining multiple second images matching the first image from a model image library based on a first image frame in the first video stream includes: Feature extraction is performed on the first image to obtain the first image features; The first image features are matched with the spatial image features corresponding to multiple pre-stored two-dimensional spatial images in the model image library, and the multiple second images are determined based on the matching results.

5. The method according to any one of claims 1-3, characterized in that, The step of determining multiple pairs of matching points based on the plurality of second images and the first image includes: Feature point matching is performed on the first image and each of the plurality of second images to obtain multiple pairs of second matching points; The multiple pairs of second matching points are filtered to obtain multiple pairs of first matching points.

6. The method according to claim 5, characterized in that, The filtering process for the plurality of second matching point pairs to obtain plurality of first matching point pairs includes: For each of the multiple pairs of second matching point pairs, based on the two-dimensional planar information of the feature points corresponding to the first image and the three-dimensional spatial information of the feature points corresponding to the second image, the second pose information corresponding to each second image is determined; Based on the gravity direction information corresponding to the image acquisition device, multiple second pose information are filtered to obtain at least one filtered second pose information; The plurality of first matching point pairs are determined based on at least one second matching point pair corresponding to the second pose information.

7. The method according to claim 5, characterized in that, The filtering process for the plurality of second matching point pairs to obtain plurality of first matching point pairs includes: For each of the multiple pairs of matching points, based on the two-dimensional planar information of the feature points corresponding to the first image and the three-dimensional spatial information of the feature points corresponding to the second image, the second pose information corresponding to each second image is determined. By using a stochastic consensus algorithm based on at least one prior information, multiple second pose information are filtered to obtain at least one filtered second pose information; The plurality of first matching point pairs are determined based on at least one second matching point pair corresponding to the second pose information.

8. The method according to any one of claims 1-3, characterized in that, The step of determining at least one set of first pose information corresponding to the image acquisition device based on the multiple pairs of first matching points and the three-dimensional model includes: Based on each pair of first matching point pairs in the plurality of first matching point pairs, determine the plurality of third matching point pairs corresponding to each pair of first matching point pairs; Based on the bundle adjustment method, the correct matching pairs of the first and third matching point pairs are determined, and the correct matching pairs of the fourth matching point pairs are determined according to the judgment results. Based on the multiple pairs of fourth matching points, at least one set of first pose information corresponding to the image acquisition device is determined.

9. The method according to any one of claims 1-3, characterized in that, The step of verifying and filtering the at least one set of first pose information to determine the target pose information of the image acquisition device includes: The target pose information of the image acquisition device is determined by verifying and filtering the at least one set of first pose information based on at least one prior information; wherein the prior information includes at least one of the following: the gravity direction of the image acquisition device, the position information of the preset positioning device, and the third pose information of the image acquisition device determined by visual odometry based on the first image.

10. The method according to any one of claims 1-3, characterized in that, Also includes: Based on the target pose information of the image acquisition device and the destination input by the user, a navigation route is determined in the target space.

11. A spatial scene positioning device, characterized in that, include: The image acquisition module is used to acquire images in the target space using an image acquisition device to obtain a first image; An image matching module is used to determine multiple second images that match the first image from a model image library; wherein, the model image library stores a three-dimensional model corresponding to the target space and multiple two-dimensional spatial images with color information; The point-to-point matching module is used to determine multiple pairs of first matching point pairs based on the plurality of second images and the first image; wherein each second image corresponds to at least one pair of first matching point pairs, and each pair of first matching point pairs includes a feature point in the first image and a feature point in the second image; The pose estimation module is used to determine multiple sets of first pose information corresponding to the image acquisition device based on the multiple pairs of first matching points and the three-dimensional model; The verification and filtering module is used to verify and filter the multiple sets of first pose information to determine the target pose information of the image acquisition device.

12. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-10.

13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-10.

Images