Cloud relocation method and apparatus, and storage medium

Abstract

The present disclosure relates to a cloud relocation method and device and storage medium, which can receive a relocation request data sent by a terminal through a server, the relocation request data comprising real-time images collected by the terminal and first camera parameters corresponding to the real-time images; perform feature extraction on the real-time images through a first pre-trained deep learning model to obtain global image features and local image features; perform feature matching on the real-time images and a plurality of preset images respectively according to the global image features to obtain matching images, the preset images being images used for creating an augmented reality (AR) point cloud map; and relocate a first pose of the terminal in the AR point cloud map according to the local image features, the matching images and the first camera parameters.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of cloud visual relocalization of AR, and particularly relates to a cloud relocalization method and device and a storage medium. BACKGROUND

[0002] Currently, AR (Augmented Reality) cloud is the hottest technical direction in the AR field, and is also one of the decisive technologies for determining whether AR devices (such as AR phones or AR glasses) can mature. For example, technology companies such as Google, Niantic, Apple, etc. have released AR cloud SDK (Software Development Kit) or products, and some domestic scientific research teams have also released AR cloud related products. The cloud visual relocalization technology based on AR cloud can be applied to many AR application scenarios, such as persistent display of AR virtual content and wide-range tracking of terminals based on the visual relocalization technology of AR cloud. SUMMARY

[0003] To overcome the problems in the related art, the present disclosure provides a cloud relocalization method and device and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a cloud relocalization method is provided, applied to a server, and the method comprises:

[0005] receiving relocalization request data sent by a terminal, wherein the relocalization request data comprises a real-time image collected by the terminal and a first camera parameter corresponding to the real-time image;

[0006] extracting features of the real-time image by using a first pre-trained deep learning model to obtain global image features and local image features;

[0007] performing feature matching on the real-time image and a plurality of preset images according to the global image features to obtain matching images, wherein the preset images are images used for creating an AR (Augmented Reality) point cloud map;

[0008] relocating a first pose of the terminal in the AR point cloud map according to the local image features, the matching images, and the first camera parameter.

[0009] Optionally, the first pre-trained deep learning model comprises a first model and a second model, and the extracting features of the real-time image by using the first pre-trained deep learning model to obtain the global image features and the local image features comprises:

[0010] The global image feature is obtained by feature extraction on the real-time image through the first model, and the local image feature is obtained by feature extraction on the real-time image through the second model.

[0011] Optionally, the matching image is obtained by respectively matching the real-time image with a plurality of preset images according to the global image feature, comprising:

[0012] The matching image consistent with the global image feature of the real-time image is output from a plurality of preset images according to the global image feature through a K-neighbor algorithm.

[0013] Optionally, the first pose of the terminal in the AR point cloud map is repositioned according to the local image feature, the matching image and the first camera parameter, comprising:

[0014] The real-time image is respectively registered with each matching image according to the local image feature, and a first feature point pair with successful registration is obtained;

[0015] The first pose of the terminal in the AR point cloud map is repositioned according to the first feature point pair and the first camera parameter.

[0016] Optionally, the local image feature comprises a plurality of feature points, and the real-time image is respectively registered with each matching image according to the local image feature, and a first feature point pair with successful registration is obtained, comprising:

[0017] For each matching image, a plurality of feature points corresponding to the real-time image and a plurality of feature points corresponding to the matching image are input into a second pre-trained deep learning model, and the first feature point pair with successful registration is obtained.

[0018] Optionally, the first feature point pair comprises a first feature point of the real-time image and a second feature point of the matching image; and the first pose of the terminal in the AR point cloud map is repositioned according to the first feature point pair and the first camera parameter, comprising:

[0019] For each first feature point pair, the first 3D point of the first feature point in the AR point cloud map is determined according to the second feature point in the first feature point pair;

[0020] The first pose of the terminal in the AR point cloud map is calculated through a preset pose algorithm according to a plurality of first feature point pairs, the first 3D point corresponding to each first feature point, and the first camera parameter.

[0021] Optionally, after the first pose of the terminal in the AR point cloud map is repositioned according to the local image features, the matching image, and the first camera parameters, the method further comprises:

[0022] sending the first pose to the terminal, so that the terminal determines a mapping relationship between the AR point cloud map and a local world coordinate system of the terminal according to the first pose and a second pose, the second pose being a pose of the terminal relative to the local world coordinate system calculated by a simultaneous localization and mapping (SLAM) tracking algorithm.

[0023] Optionally, the AR point cloud map is pre-created by the following method:

[0024] receiving environment data sent by the terminal, the environment data including multiple frames of environment images of an environment in which the terminal is currently located and image parameters corresponding to each frame of environment image;

[0025] for each frame of environment image, extracting global features and local features corresponding to the frame of environment image by using the first pre-trained deep learning model, and determining a preset number of similar images corresponding to the frame of environment image from other images according to the global features corresponding to the frame of environment image;

[0026] for each frame of similar image, performing image registration on the frame of environment image and the frame of similar image according to the local features of the frame of environment image and the local features of the frame of similar image to obtain a registration result, the registration result including a plurality of second feature point pairs that are successfully registered;

[0027] determining a second 3D point corresponding to each second feature point pair by a preset depth estimation algorithm (such as a triangulation algorithm, a depth filtering algorithm, etc.) according to the plurality of second feature point pairs, the image parameters of the frame of environment image, and the image parameters of the frame of similar image;

[0028] generating the AR point cloud map according to the second 3D points.

[0029] Optionally, the image parameters include camera shooting parameters when the frame of environment image is collected and a terminal pose of the terminal relative to the local world coordinate system; and the determination of the second 3D point corresponding to each second feature point pair according to the plurality of second feature point pairs, the image parameters of the frame of environment image, and the image parameters of the frame of similar image by the preset depth estimation model comprises:

[0030] calculating a first pose conversion matrix between cameras corresponding to the frame of environment image and the frame of similar image according to the terminal pose corresponding to the frame of environment image and the terminal pose corresponding to the frame of similar image;

[0031] inputting the plurality of second feature point pairs, the camera shooting parameters of the frame environment image, the camera shooting parameters of the frame similar image, and the first pose conversion matrix into the preset depth estimation model to obtain the second 3D point corresponding to each second feature point pair respectively.

[0032] Optionally, after the second 3D point corresponding to each second feature point pair is determined by the preset depth estimation model according to the plurality of second feature point pairs, the image parameters of the frame environment image, and the image parameters of the frame similar image, the method further comprises:

[0033] For each second 3D point, the second 3D point is re-projected to a designated image to obtain a projection point, the designated image comprising the frame environment image or the frame similar image;

[0034] According to the projection point and the initial feature point corresponding to the second 3D point in the designated image, it is determined whether the second 3D point satisfies a preset error threshold condition;

[0035] In a case where it is determined that the second 3D point does not satisfy the preset error threshold condition, the second 3D point is deleted, so as to filter the second 3D point to obtain a target 3D point;

[0036] The generating the AR point cloud map according to the second 3D point comprises:

[0037] The AR point cloud map is generated according to the target 3D point.

[0038] Optionally, the method further comprises:

[0039] For each second 3D point, a parameter optimization algorithm is used to perform parameter optimization on the second 3D point to obtain an optimized 3D point;

[0040] The AR point cloud map is positionally adjusted according to the optimized 3D point.

[0041] Optionally, after the AR point cloud map is generated according to the second 3D point, the method further comprises:

[0042] For each two point cloud maps in the plurality of AR point cloud maps, it is determined whether the two point cloud maps correspond to a common picture; the common picture is a picture representing a common area existing between the two point cloud maps;

[0043] In a case where the two point cloud maps correspond to the common picture, the two point cloud maps are merged to obtain a target point cloud map.

[0044] Optionally, the two point cloud maps include a first point cloud map and a second point cloud map, the common pictures include pictures used to create the first point cloud map, and the merging of the two point cloud maps to obtain a target point cloud map includes:

[0045] For each common picture, the camera that collects the common picture is positioned at a third pose in the second point cloud map;

[0046] A second pose transformation matrix between the first point cloud map and the second point cloud map is determined according to the third pose and a fourth pose of the camera that collects the common picture relative to the first point cloud map;

[0047] The two point cloud maps are merged according to the second pose transformation matrix to obtain a target point cloud map.

[0048] Optionally, after the AR point cloud map is generated according to the second 3D point, the method further includes:

[0049] The AR point cloud map is subjected to dense processing to obtain a dense point cloud map;

[0050] An object parameter corresponding to a preset AR object is obtained, and the preset AR object is added in the dense point cloud map according to the object parameter.

[0051] Optionally, after the AR point cloud map is generated according to the second 3D point, the method further includes:

[0052] A target image and a second camera parameter when the terminal collects the target image are received, the target image being an image to which a target AR object needs to be added;

[0053] A repositioning pose of the terminal relative to the AR point cloud map is repositioned according to the target image and the second camera parameter to obtain a repositioning pose;

[0054] The repositioning pose is sent to the terminal, so that, when a seventh pose of the target AR object relative to a local world coordinate system of the terminal is obtained, the eighth pose of the target AR object relative to the AR point cloud map is determined according to the repositioning pose and the seventh pose, the target AR object being an AR object added by a user on the target image;

[0055] The target AR object and the eighth pose sent by the terminal are received, and the target AR object is added in the AR point cloud map according to the eighth pose.

[0056] According to a second aspect of the embodiments of the present disclosure, a cloud repositioning method applied to a terminal is provided, and the method includes:

[0057] collecting a real-time image and obtaining a first camera parameter at the time of collecting the real-time image;

[0058] sending a relocalization request data to a server, the relocalization request data comprising the real-time image and the first camera parameter, so that the server extracts features of the real-time image by a first pre-trained deep learning model to obtain global image features and local image features, respectively matches the real-time image with a plurality of preset images according to the global image features to obtain a matching image, and relocalizes a first pose of the terminal in the AR point cloud map according to the local image features, the matching image, and the first camera parameter;

[0059] receiving a relocalization result sent by the server, the relocalization result comprising the first pose.

[0060] Optionally, the method further comprises:

[0061] determining a mapping relationship between the AR point cloud map and a local world coordinate system of the terminal according to the first pose and a second pose, the second pose being a pose of the terminal relative to the local world coordinate system calculated by a SLAM tracking algorithm of the terminal;

[0062] receiving a fifth pose of a preset AR object relative to the AR point cloud map sent by the server;

[0063] determining a sixth pose of the preset AR object relative to the local world coordinate system according to the mapping relationship and the fifth pose;

[0064] rendering the preset AR object on the real-time image according to the sixth pose, so as to add the preset AR object on the real-time image.

[0065] Optionally, the relocalization result further comprises a re-projection constraint relationship, the re-projection constraint relationship being a corresponding relationship between a target feature point of the real-time image and a 3D point corresponding to the target feature point on the AR point cloud map; the re-projection constraint relationship comprising a plurality of target 3D points and the target feature point corresponding to each target 3D point; the method further comprises:

[0066] for each target 3D point, re-projecting the target 3D point to the real-time image by a preset re-projection algorithm to obtain a re-projection point corresponding to the target 3D point on the real-time image;

[0067] determining a residual equation according to the target feature point corresponding to the target 3D point and the re-projection point;

[0068] The residual equation is used as a residual constraint term of the SLAM tracking algorithm to perform parameter optimization on a camera pose calculated by the SLAM tracking algorithm, to obtain an optimized camera pose.

[0069] Optionally, the method further includes:

[0070] sending, to a server, a target image and a second camera parameter of the terminal when the target image is collected by the terminal, the target image being an image to which a target AR object needs to be added;

[0071] receiving, from the server, a relocalization pose of the terminal in the AR point cloud map, the relocalization pose being determined by the server according to the target image and the second camera parameter;

[0072] obtaining a seventh pose of the target AR object relative to a local world coordinate system of the terminal, the seventh pose being added by a user on the target image;

[0073] determining an eighth pose of the target AR object relative to the AR point cloud map according to the relocalization pose and the seventh pose;

[0074] sending, to the server, the target AR object and the eighth pose, so that the server adds the target AR object in the AR point cloud map according to the eighth pose.

[0075] According to a third aspect of the embodiments of the present disclosure, a cloud relocalization apparatus is provided, applied to a server, and the apparatus includes:

[0076] a first receiving module configured to receive relocalization request data sent by a terminal, the relocalization request data including a real-time image collected by the terminal and a first camera parameter corresponding to the real-time image;

[0077] a feature extraction module configured to perform feature extraction on the real-time image by using a first pre-trained deep learning model, to obtain global image features and local image features;

[0078] an image matching module configured to perform feature matching on the real-time image and a plurality of preset images respectively according to the global image features, to obtain a matching image, the preset images being images used to create an augmented reality (AR) point cloud map;

[0079] a relocalization module configured to perform relocalization on a first pose of the terminal in the AR point cloud map according to the local image features, the matching image, and the first camera parameter.

[0080] According to a fourth aspect of the embodiments of the present disclosure, a cloud relocalization apparatus is provided, applied to a terminal, and the apparatus includes:

[0081] a data collection module configured to collect a real-time image and obtain a first camera parameter at a time of collecting the real-time image;

[0082] a second sending module configured to send, to a server, a repositioning request data including the real-time image and the first camera parameter, so that the server performs feature extraction on the real-time image by using a first pre-trained deep learning model to obtain global image features and local image features, performs feature matching on the real-time image and a plurality of preset images respectively according to the global image features to obtain matching images, and repositions a first pose of the terminal in an AR point cloud map according to the local image features, the matching images, and the first camera parameter;

[0083] a second receiving module configured to receive a repositioning result sent by the server, the repositioning result including the first pose.

[0084] According to a fifth aspect of the embodiments of the present disclosure, a cloud repositioning apparatus is provided, applied to a server, and including:

[0085] a processor;

[0086] a memory for storing processor-executable instructions;

[0087] The processor is configured to:

[0088] receive repositioning request data sent by a terminal, the repositioning request data including a real-time image collected by the terminal and a first camera parameter corresponding to the real-time image;

[0089] perform feature extraction on the real-time image by using a first pre-trained deep learning model to obtain global image features and local image features;

[0090] perform feature matching on the real-time image and a plurality of preset images respectively according to the global image features to obtain matching images, the preset images being images used for creating an augmented reality (AR) point cloud map;

[0091] reposition a first pose of the terminal in the AR point cloud map according to the local image features, the matching images, and the first camera parameter.

[0092] According to a sixth aspect of the embodiments of the present disclosure, a cloud repositioning apparatus is provided, applied to a terminal, and including:

[0093] a processor;

[0094] a memory for storing processor-executable instructions;

[0095] The processor is configured to:

[0096] acquire a real-time image and obtain a first camera parameter at the time of acquiring the real-time image;

[0097] send a repositioning request data to a server, the repositioning request data comprising the real-time image and the first camera parameter, so that the server extracts features of the real-time image by a first pre-trained deep learning model to obtain global image features and local image features, respectively matches the real-time image with a plurality of preset images according to the global image features to obtain a matching image, and repositions a first pose of the terminal in the AR point cloud map according to the local image features, the matching image, and the first camera parameter;

[0098] receive a repositioning result sent by the server, the repositioning result comprising the first pose.

[0099] According to a seventh aspect of embodiments of the present disclosure, a computer readable storage medium is provided, and the computer readable storage medium stores computer program instructions. The computer program instructions are executed by a processor to implement the steps of the cloud repositioning method provided in the first aspect of the present disclosure.

[0100] According to an eighth aspect of embodiments of the present disclosure, a computer readable storage medium is provided, and the computer readable storage medium stores computer program instructions. The computer program instructions are executed by a processor to implement the steps of the cloud repositioning method provided in the second aspect of the present disclosure.

[0101] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: A server receives relocation request data sent by a terminal, the relocation request data including a real-time image acquired by the terminal and first camera parameters corresponding to the real-time image; a first pre-trained deep learning model is used to extract features from the real-time image to obtain global image features and local image features; based on the global image features, the real-time image is matched with multiple preset images to obtain a matched image, the preset images being images used to create augmented reality (AR) point cloud maps; the server then performs feature matching on the terminal based on the local image features, the matched image, and the first camera parameters. Relocalization is performed on the first pose in the AR point cloud map. In this way, compared with image registration directly in multiple preset images based on local manual feature points, this disclosure first filters images from multiple preset images based on the global image features of the real-time image to obtain a matching image, and then performs image registration from the matching image based on local image features. The relocalization result is more accurate and the relocalization efficiency is also relatively high. Furthermore, the global image features and local image features in this disclosure are extracted by pre-trained deep learning models. Compared with the image registration scheme using manual feature points, the relocalization result has better stability and accuracy.

[0102] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0103] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0104] Figure 1 This is a flowchart illustrating a cloud-based relocation method according to an exemplary embodiment.

[0105] Figure 2 It is based on Figure 1 The illustrated embodiment shows a flowchart of a cloud-based relocation method.

[0106] Figure 3 This is a flowchart illustrating a method for creating an AR point cloud map according to an exemplary embodiment.

[0107] Figure 4 This is a schematic diagram illustrating a process for creating an AR point cloud map according to an exemplary embodiment.

[0108] Figure 5 It is based on Figure 3 The illustrated embodiment shows a flowchart of a method for creating an AR point cloud map.

[0109] Figure 6 is a flowchart of a method of creating an AR point cloud map according to an example embodiment. Figure 3

[0110] Figure 7 is a flowchart of a cloud-based relocalization method according to an example embodiment.

[0111] Figure 8 is a flowchart of a method of AR persistent display according to an example embodiment. Figure 7

[0112] Figure 9 is a process diagram of AR persistent display according to an example embodiment.

[0113] Figure 10 is a flowchart of a method of implementing large-scale tracking based on AR point cloud map relocalization according to an example embodiment. Figure 7

[0114] is a process diagram of implementing large-scale tracking based on AR point cloud map relocalization according to an example embodiment. Figure 11

[0115] is a block diagram of a cloud-based relocalization device according to an example embodiment. Figure 12

[0116] is a block diagram of a cloud-based relocalization device according to an example embodiment. Figure 13 Figure 12 is a block diagram of a cloud-based relocalization device according to an example embodiment.

[0117] Figure 14 is a block diagram of a cloud-based relocalization device according to an example embodiment.

[0118] Figure 15 Figure 14 is a block diagram of a cloud-based relocalization device according to an example embodiment.

[0119] Figure 16 is a block diagram of a cloud-based relocalization device according to an example embodiment. Figure 14

[0120] is a block diagram of a cloud-based relocalization device according to an example embodiment. Figure 17 Figure 14 is a block diagram of a device for cloud-based relocalization according to an example embodiment.

[0121] Figure 18 is a block diagram of a device for cloud-based relocalization according to an example embodiment.

[0122] Figure 19 ​​​​​is a block diagram of an apparatus for cloud relocation according to an exemplary embodiment. DETAILED DESCRIPTION

[0123] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same or similar elements, unless otherwise represented. The embodiments described in the following exemplary embodiments do not represent all the implementations in accordance with this disclosure. Instead, they simply represent exemplary devices and methods in accordance with some aspects of this disclosure, as detailed in the appended claims.

[0124] It should be noted that all the actions of acquiring signals, information or data in this application are carried out in accordance with the corresponding data protection regulations and policies of the country where the device is located, and with the authorization given by the owner of the corresponding device.

[0125] The present disclosure mainly relates to a cloud visual relocation technology of augmented reality (AR), and the cloud visual relocation technology based on an AR cloud can be applied to many AR application scenarios, such as persistent display of AR virtual content and wide-range tracking of terminals based on the visual relocation technology of the AR cloud. The cloud visual relocation technology can also be applied in the field of robots. When a robot performs autonomous navigation, the robot needs to know the initial position in the initialization stage, which can be provided by the cloud visual relocation. When the tracking range is particularly large, the cloud visual relocation can effectively reduce the global tracking error and improve the autonomous navigation accuracy of the robot. In addition, the relocation technology based on the cloud high-precision map can also provide real-time high-precision real-time pose for an autonomous vehicle.

[0126] In the related art, in the process of visual relocation based on a pre-created AR cloud map, local artificial feature points (such as SIFT feature points and ORB feature points) of a query image are directly matched with corresponding images of an AR point cloud map, but for images with weak light or weak texture, the relocation accuracy is not high when directly matching based on artificial feature points, and the stability of relocation is poor.

[0127] In addition, in the process of creating an AR cloud map in the related art, special equipment (such as radar and panoramic camera) is needed to collect image data for creating a cloud map, which is not convenient and has high cost.

[0128] To solve the above problems, the present disclosure provides a cloud-based relocation method, device and storage medium. After receiving the relocation request data sent by the terminal, the server can extract the features of the real-time image in the relocation request data through a deep learning model to obtain global image features and local image features, perform image screening from multiple preset images according to the global image features of the real-time image to obtain a matching image, and then perform image registration from the matching image according to the local image features, thereby realizing the relocation of the first pose of the terminal in the AR point cloud map. Compared with directly performing image registration in multiple preset images based on local artificial feature points, the present disclosure first performs image screening from multiple preset images according to the global image features of the real-time image to obtain a matching image, and then performs image registration from the matching image according to the local image features. The accuracy of the relocation result is higher, and the efficiency of the relocation is relatively higher. Further, the global image features and the local image features in the present disclosure are both extracted by a pre-trained deep learning model. Compared with the image registration scheme using artificial feature points, the stability and accuracy of the relocation result are better.

[0129] In addition, when creating an AR point cloud map, the present disclosure can directly use a local terminal (such as an AR phone, AR glasses, etc.) to collect the environment image of the terminal currently located, and send the environment image to the server, so that the server creates an AR point cloud map according to the environment image. No additional image acquisition device is required. Compared with using panoramic cameras or radar devices to collect images to create an AR point cloud map, the cost of image acquisition is lower and more convenient and universal.

[0130] The specific embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0131] Figure 1 is a flowchart of a cloud-based relocation method according to an exemplary embodiment, applied to a server, as shown in Figure 1 The method comprises the following steps:

[0132] In step S101, the relocation request data sent by the terminal is received, and the relocation request data includes a real-time image collected by the terminal and a first camera parameter corresponding to the real-time image.

[0133] The terminal refers to a terminal with AR function, such as AR phone, AR glasses, etc.

[0134] The local AR terminal tracks the local environment in real time by running a SLAM (simultaneous localization and mapping) tracking system, and establishes a local world coordinate system at initialization. The terminal captures a real-time image of the surrounding environment through the camera of the terminal, and then sends the real-time image to the server. The terminal also records the camera intrinsic parameters when each real-time image is captured by the camera, and the camera pose of the terminal (or the camera) relative to the local world coordinate system when each real-time image is captured. The camera intrinsic parameters include focal length and the position of the optical center. The camera pose (or terminal pose) refers to the 6-DOF pose of the terminal in the local world coordinate system calculated by the SLAM tracking system. The 6-DOF pose includes the spatial position of the terminal in the local world coordinate system and the rotation amount, such as roll, pitch, and yaw. The terminal sends the camera intrinsic parameters and the camera pose to the server when the real-time image is captured by the camera, at the same time as sending the real-time image to the server. Therefore, the first camera parameters include the camera intrinsic parameters and the camera pose.

[0135] It should be noted that the terminal can send the relocalization request data to the server every preset time.

[0136] In step S102, the real-time image is feature-extracted by a first pre-trained deep learning model to obtain global image features and local image features.

[0137] The first pre-trained deep learning model includes a first model and a second model. The first model is used to extract global image features of the real-time image. The global image features (or global feature descriptors) are global feature descriptor vectors of the entire image, which are used to represent the features of the entire image. The first model can be any one of the two deep learning models of NetVLad and SOLAR, for example. The second model is used to extract local image features of the real-time image. The local image features include a plurality of feature points. Each feature point includes a key point and a local feature descriptor, such as a SuperPoint feature point. The second model can include a Repeatability model, for example.

[0138] That is, in this step, the real-time image can be feature-extracted by the first model to obtain the global image features, and the real-time image can be feature-extracted by the second model to obtain the local image features.

[0139] In step S103, the real-time image is feature-matched with a plurality of preset images according to the global image features to obtain matching images. The preset images are images used to create an augmented reality (AR) point cloud map.

[0140] The preset image refers to an environment image collected by an AR terminal for pre-creating the AR point cloud map.

[0141] In this step, the K-neighbor algorithm can be used to output a plurality of matching images from the plurality of preset images that are consistent with the global image features of the real-time image according to the global image features.

[0142] In step S104, the first pose of the terminal in the AR point cloud map is repositioned according to the local image features, the matching images, and the first camera parameters.

[0143] In this step, for each matching image, the real-time image and the matching image can be registered according to the local image features to obtain a first feature point pair that is successfully registered; and the first pose of the terminal in the AR point cloud map is repositioned according to the first feature point pair and the first camera parameters.

[0144] The local image features include a plurality of feature points, and the first pose is similar to a camera pose and includes a 6-degree-of-freedom pose of the terminal in the AR point cloud map.

[0145] Here, the real-time image and each matching image can be registered according to the local image features to obtain a first feature point pair that is successfully registered in the following manner:

[0146] For each matching image, a plurality of feature points corresponding to the real-time image and a plurality of feature points corresponding to the matching image are input into a second pre-trained deep learning model to obtain the first feature point pair that is successfully registered.

[0147] The second pre-trained deep learning model can be, for example, a Superglue model, the first feature point pair can include a plurality of first feature points of the real-time image and a plurality of second feature points of the matching image, each first feature point pair includes a first feature point of the real-time image and a second feature point of the matching image, and the first feature point and the second feature point are in a one-to-one correspondence.

[0148] In the actual registration process, the key points and local feature descriptors corresponding to each feature point of the real-time image and the matching image can be input into the second pre-trained deep learning model to obtain the first feature point pair that is successfully registered.

[0149] The first feature point pair includes a first feature point of the real-time image and a second feature point of the matching image, so that in the process of repositioning the first pose of the terminal in the AR point cloud map according to the first feature point pair and the first camera parameter, for each first feature point pair, the first 3D point of the first feature point in the first feature point pair in the AR point cloud map can be determined according to the second feature point in the first feature point pair; and the first pose of the terminal in the AR point cloud map is calculated by a preset pose algorithm according to the first feature point pair, the first 3D point corresponding to each first feature point pair, and the first camera parameter.

[0150] The preset pose algorithm may, for example, include a PNP (Perspective-n-Point) algorithm, and the first pose can be represented by a pose conversion matrix of a camera coordinate system corresponding to the real-time image and a point cloud map coordinate system (i.e., an anchor coordinate system).

[0151] The pre-constructed AR point cloud map corresponds to a plurality of data files, and the correspondence between each 3D point in the AR point cloud map and the 2D feature point on the preset image (i.e., the feature point of the local image feature in the preset image) is stored in one data file. Therefore, the server can directly read the 3D point corresponding to each second feature point in the AR point cloud map from the data file.

[0152] By way of example, for ease of description, the real-time image is represented as A1, it is assumed that after step S103 is performed, three matching images B1, B2 and B3 are obtained after the real-time image A1 is respectively matched with the plurality of preset images according to the global image features of the real-time image A1, and taking the local image feature registration process of the two images A1 and B1 as an example, it is assumed that the real-time image A1 includes 100 feature points, and the matching image B1 includes 120 feature points, the 100 feature points and the 120 feature points are input into the Superglue model to obtain 50 feature point pairs, each feature point pair includes two corresponding feature points (i.e., a first feature point and a second feature point), for each feature point pair, the second feature point in the feature point pair can be read to correspond to the first 3D point in the AR point cloud map by querying the data file corresponding to the AR point cloud map, and then it can be determined that the first feature point in the first feature point pair also corresponds to the first 3D point in the AR point cloud map, so that a group of 2D-3D point pairs can be obtained, wherein the 2D point in the 2D-3D point pair is the first feature point of the real-time image in the feature point pair, and the 3D point is the first 3D point in the AR point cloud map corresponding to the first feature point; then the plurality of first feature point pairs, the first 3D point corresponding to each first feature point, and the first camera parameter can be taken as inputs of the PNP algorithm, and the first pose of the terminal in the AR point cloud map is calculated based on the PNP algorithm. The above example is only illustrative, and the present disclosure is not limited in this regard.

[0153] By using the above method, after receiving the repositioning request data sent by the terminal, the server can extract the global image features and the local image features from the real-time image in the repositioning request data by using the deep learning model, perform image screening from the plurality of preset images according to the global image features of the real-time image to obtain matching images, and then perform image registration from the matching images according to the local image features, thereby realizing repositioning of the first pose of the terminal in the AR point cloud map. Compared with directly performing image registration from the plurality of preset images based on the local artificial feature points, the accuracy of the repositioning result is higher, and the efficiency of the repositioning is relatively higher. Further, the global image features and the local image features in the present disclosure are both extracted by using the pre-trained deep learning model, and compared with the image registration scheme using artificial feature points, the stability and accuracy of the repositioning result are better.

[0154] It has been mentioned above that in actual AR application scenarios, the AR virtual content can be persistently displayed based on the repositioning result of the terminal in the AR point cloud map. Here, persistent display refers to the virtual content placed by the user at a certain time, which can be visually repositioned at any time thereafter and accurately restored to the position of the virtual content, and the AR virtual content including preset multimedia files such as text, pictures, videos, 3D models, 2D / 3D special effects, etc. In order to realize the persistent display of the AR virtual content, the server needs to send the first pose to the terminal, so that the terminal can perform persistent display of the AR virtual content according to the first pose.

[0155] Therefore, Figure 2 is a flowchart of a cloud repositioning method according to the embodiment shown in Figure 1 As shown in Figure 2 The method further includes the following steps:

[0156] In step S105, the first pose is sent to the terminal, so that the terminal determines the mapping relationship between the AR point cloud map and the local world coordinate system of the terminal according to the first pose and the second pose, which is the pose of the terminal relative to the local world coordinate system calculated by the terminal through a simultaneous localization and mapping (SLAM) tracking algorithm.

[0157] After the terminal determines the mapping relationship, it can further calculate the sixth pose of the preset AR object relative to the local world coordinate system according to the mapping relationship and the fifth pose of the preset AR object relative to the AR point cloud map sent by the server, and then render the preset AR object on the real-time image according to the sixth pose, so as to add the preset AR object on the real-time image, thereby realizing the persistent display of the preset AR object.

[0158] The specific implementation of the terminal to persistently display the preset AR object on the terminal side according to the first pose, the second pose, and the fifth pose of the preset AR object relative to the AR point cloud map sent by the server will be described in detail in the embodiment of the terminal, which is not limited herein.

[0159] In addition, it is mentioned above that in actual AR application scenarios, a large-range tracking of the terminal can also be implemented based on the visual repositioning technology of the AR cloud. The large-range tracking refers to that when a user moves in a large-range (usually an area of more than 1,000 square meters) scene, the centimeter-level precision positioning can still be implemented, and the accurate AR virtual-real superimposition effect can be achieved throughout the whole process. In the related art, the SLAM tracking system running locally on the AR terminal is mainly relied on to track and position the terminal in real time. However, the pose estimation of the current frame of the SLAM system is calculated on the basis of the last frame. For a large-range moving scene, this will cause the error of the system to be accumulated continuously, and the accumulated global error is relatively large, which cannot meet the tracking precision requirement of AR. Therefore, in the present disclosure, the accumulated error can be eliminated based on the visual repositioning technology of the AR cloud. The reason is that the AR point cloud map can be considered to be of high precision, and each 3D point in the AR point cloud map is one-to-one mapped with the spatial position in the real world. This means that after the pose of the terminal in the AR point cloud map coordinate system is obtained through AR point cloud repositioning, the local terminal can determine the accurate position of itself in the real world space. If there is accumulated error in the SLAM system of the local terminal, the accumulated error can be eliminated based on the accurate pose of the terminal in the AR point cloud map sent by the server.

[0160] As described above, in the process of repositioning the first pose of the terminal on the AR point cloud map by the server, a set of 2D-3D point pairs can be obtained, wherein the 2D point in the 2D-3D point pair is the first feature point of the real-time image, and the 3D point is the first 3D point in the AR point cloud map corresponding to the first feature point. The set of 2D-3D point pairs can be used as a re-projection constraint relationship. The server sends the re-projection constraint relationship to the terminal, so that the terminal can determine a residual equation according to the re-projection constraint relationship, and then performs parameter optimization on the camera pose (i.e., the terminal pose) calculated by the SLAM tracking algorithm by taking the residual equation as a residual constraint term of the SLAM tracking algorithm, to obtain an optimized camera pose.

[0161] The specific implementation manner of implementing the large-range high-precision tracking of the terminal based on the visual repositioning technology of the AR cloud will be described in detail in the embodiments corresponding to the terminal, which is not limited here.

[0162] The pre-creation process of the AR point cloud map will be described below.

[0163] Figure 3 is a flowchart of a method for pre-creating an AR point cloud map according to an example embodiment, Figure 3 The process of creating the AR point cloud map shown is usually an offline creation process, as shown in Figure 3 The method includes the following steps:

[0164] In step S301, environment data sent by a terminal is received, the environment data including multiple frames of environment images of a current environment in which the terminal is located and image parameters corresponding to each frame of environment image.

[0165] The frame of environment image parameters includes camera shooting parameters when the frame of environment image is shot and a terminal pose of the terminal relative to a local world coordinate system when the frame of environment image is shot. The camera shooting parameters may include, for example, focal length and position of an optical center. The terminal pose refers to a 6-DOF pose of the terminal in the local world coordinate system calculated by a SLAM tracking system. The 6-DOF pose includes spatial position and rotation amount of the terminal in the local world coordinate system. The rotation amount includes, for example, roll angle, pitch angle and yaw angle. In addition, the environment data may also include one or more of depth environment images collected by the terminal through a depth camera, three-axis pose angle (or angular rate) and acceleration of the terminal measured by an IMU (Inertial Measurement Unit), and GPS positioning information of the terminal.

[0166] In step S302, for each frame of environment image, a global feature and a local feature corresponding to the frame of environment image are extracted by using the first pre-trained deep learning model, and a preset number of similar images corresponding to the frame of environment image are determined from other environment images according to the global feature corresponding to the frame of environment image.

[0167] The global feature is also a feature representing the entire image. The local feature may include multiple feature points. Similar to steps S102 and S103, in this step, the first model in the first pre-trained deep learning model may also be used to extract features of the frame of environment image to obtain the global feature, and the second model may be used to extract features of the frame of environment image to obtain the local feature. Then, K similar images corresponding to the frame of environment image may be determined according to the global feature by using a K-neighbor algorithm.

[0168] In step S303, for each frame of similar image, image registration is performed on the frame of environment image and the frame of similar image according to the local feature of the frame of environment image and the local feature of the frame of similar image to obtain a registration result. The registration result includes multiple second feature point pairs that are successfully registered.

[0169] For example, after multiple feature points of a frame of image and multiple feature points corresponding to the frame of similar image are input into a Superglue model, the second feature point pairs that are successfully registered are obtained. Each second feature point pair includes a third feature point of the frame of environment image and a fourth feature point of the frame of similar image, and the third feature point and the fourth feature point also belong to a one-to-one correspondence.

[0170] In step S304, a second 3D point corresponding to each second feature point pair is determined according to the plurality of second feature point pairs, the image parameters of the frame environment image, and the image parameters of the frame similar image by a preset depth estimation algorithm.

[0171] The preset depth estimation algorithm may include, for example, a triangulation algorithm, a depth filtering algorithm, etc.

[0172] In this step, a first pose transformation matrix between the cameras corresponding to the frame environment image and the frame similar image can be calculated according to the terminal pose corresponding to the frame environment image and the terminal pose corresponding to the frame similar image; the plurality of second feature point pairs, the camera shooting parameters of the frame environment image, the camera shooting parameters of the frame similar image, and the first pose transformation matrix are input into the preset depth estimation model to obtain the second 3D point corresponding to each second feature point pair.

[0173] In step S305, the AR point cloud map is generated according to the second 3D point.

[0174] With continuous registration between pictures, the second 3D point corresponding to each second feature point pair is obtained, and then the plurality of second 3D points are continuously aggregated to form the AR point cloud map.

[0175] For example, Figure 4 is a process diagram for creating an AR point cloud map according to an example embodiment, as Figure 4 As shown, an AR terminal (such as an AR phone, AR glasses, etc. in the figure) 101 is used to collect pictures and other sensor data 102, generate an AR point cloud map 103, and finally register a preset AR virtual content 103-B into the AR point cloud map. Here, only an example is illustrated, and the disclosure is not limited in this regard.

[0176] Considering that the second feature point pair obtained by registration between two images based on local features may have errors, in order to improve the accuracy of the created AR point cloud map, after determining the second 3D point corresponding to each second feature point pair according to the plurality of second feature point pairs, the image parameters of the frame environment image, and the image parameters of the frame similar image by the preset depth estimation model, each second 3D point can be checked, and the AR point cloud map can be adjusted according to the checking result. The specific implementation process is shown in Figure 5 .

[0177] Figure 5 is a flowchart of a method for creating an AR point cloud map according to the embodiment shown in Figure 3 , as Figure 5 shown, the method further includes the following steps:

[0178] In step S306, for each of the second 3D points, the second 3D point is reprojected to a designated image to obtain a projection point, the designated image including the frame environment image or the frame similar image.

[0179] This step can reproject the second 3D point to the designated image to obtain the projection point based on a preset re-projection algorithm, and the specific re-projection process can refer to the description in the related literature, which is not limited herein.

[0180] In step S307, according to the projection point and the initial feature point corresponding to the second 3D point in the designated image, it is determined whether the second 3D point satisfies a preset error threshold condition.

[0181] For example, the preset error threshold condition can be that the difference between the pixel values of the projection point and the initial feature point is less than or equal to a preset pixel value.

[0182] In step S308, in a case where it is determined that the second 3D point does not satisfy the preset error threshold condition, the second 3D point is deleted, so as to select the target 3D point from the second 3D points.

[0183] For example, taking the re-projection of the second 3D point X to the frame environment image as an example, the initial feature point corresponding to the second 3D point X in the frame environment image is denoted as x1, the pixel coordinates of the initial feature point x1 are (100, 100), and the projection point of the second 3D point X after being re-projected to the frame environment image is denoted as x2, the pixel coordinates of the projection point x2 are (100, 103). At this time, it can be determined that the projection point x2 and the initial feature point x1 differ by 3 pixels, and if the preset error threshold condition is that the difference between the pixel values of the projection point and the initial feature point is less than or equal to 2 pixel values, therefore, according to the pixel difference (i.e., 3 pixels) between the projection point x2 and the initial feature point x1, it can be determined that the second 3D point X does not satisfy the preset error threshold condition, and at this time, the second 3D point X needs to be deleted. After checking each second 3D point based on the above method, the remaining second 3D points are the target 3D points. The above is only an example for illustration, and the present disclosure is not limited thereto.

[0184] In this way, when step S305 is performed, the AR point cloud map can be generated according to the target 3D points, so that the accuracy of the created AR point cloud map can be improved.

[0185] In addition, in order to further improve the accuracy of the created AR point cloud map, the parameters of each 3D point corresponding to the created AR point cloud map can also be optimized. In a possible implementation manner of the present disclosure, for each of the second 3D points, the second 3D point can be parameter-optimized to obtain an optimized 3D point through a preset parameter optimization algorithm; and the AR point cloud map can be position-adjusted according to the optimized 3D point.

[0186] The preset parameter optimization algorithm here can be, for example, the BA (Bundle Adjustment) algorithm. The specific implementation of the BA algorithm to optimize the parameters of the second 3D point to obtain the optimized 3D point can be found in the description in the relevant literature, and is not limited here.

[0187] In addition, while optimizing the parameters of the second 3D point, the camera pose can also be optimized. Specifically, each camera pose and 3D point can be used as optimization parameters, and a reprojection error function can be established for BA optimization.

[0188] After creating the AR point cloud map, the server can store the global feature vector of each image in a global feature description file. This allows for feature matching between the global image features and each global feature vector stored in the global feature description file during subsequent relocalization based on the created AR point cloud map. In addition, after creating the AR point cloud map, the server can also record the correspondence between each 3D point in the AR point cloud map and each 2D feature point in the image, which can be used for subsequent relocalization.

[0189] In the actual process of creating AR point cloud maps, the AR point cloud maps created based on the above steps cannot be too large, as this will lead to a decrease in accuracy. Therefore, in one possible implementation method disclosed herein, the above process can be repeated to form small point cloud maps, and then the small point cloud maps can be merged into a large map.

[0190] Understandably, two point cloud maps need to have a common image in order to be merged. The common image is an image that represents the overlapping common area between the two point cloud maps.

[0191] Figure 6 It is based on Figure 3 The illustrated embodiment presents a flowchart of a method for creating an AR point cloud map, as shown in the flowchart. Figure 6 As shown, the method includes the following steps:

[0192] In step S309, for every two point cloud maps in the multiple AR point cloud maps, it is determined whether the two point cloud maps correspond to a common image. The common image is an image that represents a common area that is repeated between the two point cloud maps.

[0193] In one possible implementation of this step, for each of the two point cloud maps, the GPS positioning data corresponding to the image that created the point cloud map is determined, and then the two point cloud maps are determined to correspond to the common image based on the GPS positioning data corresponding to the image.

[0194] In step S310, in the case that the two point cloud maps correspond to a common image, the two point cloud maps are merged to obtain a target point cloud map.

[0195] The two point cloud maps include a first point cloud map and a second point cloud map, and the common image includes multiple images used to create the first point cloud map. In this step, for each common image, the third pose of the camera collecting the common image in the second point cloud map can be located; the specific location process is referred to steps S102-S104, which will not be described here. Then, the second pose transformation matrix between the first point cloud map and the second point cloud map can be determined according to the third pose and the fourth pose of the camera collecting the common image relative to the first point cloud map (the fourth pose can be directly obtained from the data file corresponding to the first point cloud map). In this way, the two point cloud maps can be merged according to the second pose transformation matrix to obtain a target point cloud map.

[0196] It should be noted that for multiple common images, multiple second pose transformation matrices can be calculated. In order to improve the accuracy of the merged map, the RANSAC algorithm can be used to determine the optimal pose transformation matrix from the multiple second pose transformation matrices. In this way, the two point cloud maps can be merged based on the optimal pose transformation matrix to obtain a target point cloud map.

[0197] In addition, as mentioned above, in order to provide users with more convenient AR persistent display capabilities, the preset AR object needs to be added to the created AR point cloud map. In one possible implementation, the server can be used to set the preset AR object offline. In the present disclosure, the AR point cloud map can be processed to obtain a dense point cloud map. Then, the object parameters corresponding to the preset AR object are obtained, which can include a preset model corresponding to the preset AR object and corresponding model parameters (for example, taking a pet model as an example, the model parameters can include the size, color, expression, and action of the pet model). After that, the preset AR object is added to the dense point cloud map after rendering according to the object parameters.

[0198] In another possible implementation of the present disclosure, the AR virtual object can also be added online in real time according to the user's operation. Specifically, the target image and the second camera parameter when the terminal collects the target image are received, and the target image is an image to which a target AR object needs to be added.

[0199] For example, in an actual application scenario, if a user wants to add a virtual pet on a table in front of him, the user can capture a target image corresponding to the table through a camera of a terminal (such as an AR mobile phone), and then send the target image and a second camera parameter to a server, the second camera parameter also including a camera shooting parameter when the terminal captures the target image and a camera pose calculated by a SLAM tracking system.

[0200] Then, a repositioning pose is obtained by repositioning a pose of the terminal relative to the AR point cloud map according to the target image and the second camera parameter.

[0201] The specific implementation manner of this step is similar to the repositioning process based on the AR point cloud map in steps S102-S104, and will not be described here again.

[0202] Then, the repositioning pose can be sent to the terminal, so that the terminal determines an eighth pose of the target AR object relative to the AR point cloud map according to the repositioning pose and a seventh pose of the target AR object relative to a local world coordinate system of the terminal, the target AR object being an AR object added by the user on the target image.

[0203] The user can manually add the target AR object on the target image through the terminal, and then the terminal can obtain the seventh pose. After obtaining the repositioning pose, the terminal can calculate a conversion matrix between the local world coordinate system of the terminal and the AR point cloud map coordinate system according to the repositioning pose and the seventh pose, and then determine the eighth pose of the target AR object relative to the AR point cloud map based on the conversion matrix, record the target AR object and the eighth pose, and then send the target AR object and the eighth pose to the server.

[0204] Then, the target AR object and the eighth pose sent by the terminal are received, and the target AR object is added in the AR point cloud map according to the eighth pose.

[0205] In this step, the server can render the target AR object in the AR point cloud map according to the eighth pose, thereby realizing online addition of the target AR object in the AR point cloud map.

[0206] After adding the AR virtual object (including online addition of the target AR object and / or offline addition of a preset AR object) in the AR point cloud map, each AR virtual object and a 6-DOF pose thereof in the AR point cloud map can be saved into an AR content pose file, so that the AR virtual objects in the AR content pose file can be persistently displayed subsequently.

[0207] By using the above method for creating an AR point cloud map, an environment image in which a local terminal (such as an AR mobile phone, AR glasses, etc.) currently exists can be directly collected by the local terminal, and the environment image is sent to a server, so that the server creates an AR point cloud map according to the environment image, without the need to be equipped with an additional image collection device. Compared with using a panoramic camera or a radar device to collect images to create an AR point cloud map, the image collection cost is lower and more convenient and universal.

[0208] Figure 7 FIG. 1 is a flowchart of a cloud-based relocation method according to an exemplary embodiment, applied to a terminal, which is an AR-capable terminal (such as an AR mobile phone, AR glasses, etc.), as shown in FIG. 2, the method includes the following steps: Figure 7

[0209] In step S701, a real-time image is collected and a first camera parameter when the real-time image is collected is obtained.

[0210] The local AR terminal tracks the local environment in real time by running a SLAM tracking system, and establishes a local world coordinate system at initialization. A real-time image of the surrounding environment is captured by a camera of the AR terminal, and then the real-time image is sent to a server. The terminal also needs to record the camera intrinsic parameter when each real-time image is captured by the camera and the camera pose of the terminal (or the camera) relative to the local world coordinate system when each real-time image is captured. The camera intrinsic parameter includes the focal length and the position of the optical center. The camera pose (or terminal pose) refers to the 6-DOF pose of the terminal in the local world coordinate system calculated by the SLAM tracking system, which includes the spatial position of the terminal in the local world coordinate system and the rotation amount, such as the roll angle roll, the pitch angle pitch, and the yaw angle yaw. The terminal sends the camera intrinsic parameter and the camera pose when the real-time image is captured by the camera to the server at the same time as the real-time image is sent to the server. Therefore, the first camera parameter includes the camera intrinsic parameter and the camera pose.

[0211] In step S702, a relocation request data is sent to the server, the relocation request data includes the real-time image and the first camera parameter, so that the server extracts features from the real-time image by using a first pre-trained deep learning model to obtain global image features and local image features, respectively matches the real-time image with a plurality of preset images according to the global image features to obtain a matching image, and relocates a first pose of the terminal in the AR point cloud map according to the local image features, the matching image, and the first camera parameter.

[0212] The first pose can be represented by a pose conversion matrix of a camera coordinate system corresponding to the real-time image and a point cloud map coordinate system (i.e., an anchor coordinate system).​

[0213] In step S703, the relocation result sent by the server is received, and the relocation result includes the first pose.

[0214] Using the above method, the terminal can send relocation request data to the server. The relocation request data includes a real-time image captured by the terminal and first camera parameters corresponding to the real-time image. The server can then extract features from the real-time image using a first pre-trained deep learning model to obtain global image features and local image features. Based on the global image features, the server performs feature matching between the real-time image and multiple preset images to obtain a matching image. The preset images are images used to create augmented reality (AR) point cloud maps. Based on the local image features, the matching image, and the first camera parameters, the server relocates the terminal to the AR point cloud map. The first pose in the cloud map is used for relocalization. In this way, compared with the method of directly performing image registration in multiple preset images based on local manual feature points, this disclosure first selects images from multiple preset images based on the global image features of the real-time image to obtain matching images, and then performs image registration from the matching images based on local image features. The relocalization result is more accurate and the relocalization efficiency is also relatively high. Furthermore, the global image features and local image features in this disclosure are extracted by pre-trained deep learning models. Compared with the image registration scheme using manual feature points, the relocalization result has better stability and accuracy.

[0215] Figure 8 It is based on Figure 7 The illustrated embodiment shows a flowchart of a method for persistent AR display, as shown in the figure. Figure 8 As shown, the method includes the following steps:

[0216] In step S704, the mapping relationship between the AR point cloud map and the local world coordinate system of the terminal is determined based on the first pose and the second pose. The second pose is the pose of the terminal relative to the local world coordinate system calculated by the SLAM tracking algorithm.

[0217] For example, suppose the first pose is represented by pose matrix P and the second pose is represented by pose matrix P1. The mapping relationship is the pose transformation matrix P*P1.

[0218] In step S705, the fifth pose of the preset AR object relative to the AR point cloud map is received from the server.

[0219] The preset AR object can include text, pictures, videos, 3D models, 2D / 3D special effects, and other preset multimedia files. In a possible implementation manner, the server can obtain the fifth pose from an AR virtual content pose data file corresponding to the AR point cloud map.

[0220] In step S706, a sixth pose of the preset AR object relative to the local world coordinate system is determined according to the mapping relationship and the fifth pose.

[0221] Suppose the fifth pose is a pose matrix P2, and the sixth pose of the preset AR object relative to the local world coordinate system determined according to the mapping relationship and the fifth pose is P2*P*P1.

[0222] In step S707, the preset AR object is rendered on the real-time image according to the sixth pose, so as to add the preset AR object on the real-time image.

[0223] By using the AR content persistent display method, the server can reposition the first pose of the terminal in the AR point cloud map based on the AR point cloud map. Then, the terminal can determine the sixth pose of the preset AR object relative to the local world coordinate system based on the repositioning result (i.e., the first pose), the second pose of the terminal relative to the local world coordinate system calculated by the SLAM tracking algorithm of the terminal, and the fifth pose of the preset AR object relative to the AR point cloud map. In this way, the preset AR content can be rendered at the sixth pose in the local world coordinate system, and the persistent display of the AR content is realized.

[0224] For example, Figure 9 FIG. 1 is a process schematic diagram of AR persistent display according to an example embodiment. Figure 9 As shown in FIG. 1, after the AR terminal is started, a SLAM system 105 is locally run to track the local environment in real time and establish a local world coordinate system. After the AR terminal sends a query picture 104 and is registered with a point cloud map 103, visual repositioning 106 is realized. After visual repositioning, the point cloud map coordinate system can be mapped to the local world coordinate system, that is, a mapping relationship between the AR point cloud map and the local world coordinate system of the terminal is obtained. A fifth pose of a preset AR object 103-B relative to the point cloud map is known. When the coordinate mapping of visual repositioning is completed, a sixth pose of the preset AR object 103-B in the local world coordinate system can be calculated. According to the sixth pose, the preset AR object 103-B is rendered at the corresponding position in the local world coordinate system, and the persistent display of the AR content is realized. This is only an example, and the present disclosure is not limited in this regard.

[0225] In addition, the repositioning result further includes a re-projection constraint relationship, which is a correspondence relationship between the target feature points of the real-time image and 3D points corresponding to the target feature points on the AR point cloud map; the re-projection constraint relationship includes a plurality of target 3D points and the target feature points corresponding to each target 3D point respectively. As described above, the re-projection constraint relationship can be a set of 2D-3D point pairs, the 2D point being the target feature point of the real-time image, and the 3D point being the 3D point in the AR point cloud map corresponding to the target feature point. The terminal can realize large-range high-precision tracking of the terminal according to the re-projection constraint relationship, and the specific implementation is as shown in Figure 10 .

[0226] Figure 10 According to the embodiment shown in Figure 7 , a flowchart of a method for realizing large-range tracking based on AR point cloud map repositioning is shown in Figure 10 , and the method further includes the following steps:

[0227] In step S708, for each target 3D point, the target 3D point is re-projected to the real-time image by a preset re-projection algorithm to obtain a re-projection point corresponding to the target 3D point on the real-time image.

[0228] The specific re-projection process of re-projecting the target 3D point to the real-time image to obtain the re-projection point based on the preset re-projection algorithm in this step can refer to the description in the related literature, which is not limited here.

[0229] In step S709, a residual equation is determined according to the target feature point corresponding to the target 3D point and the re-projection point.

[0230] For example, the residual equation can be determined by the BA algorithm according to the target feature point and the re-projection point to determine the residual equation.

[0231] In step S710, the residual equation is taken as a residual constraint term of the SLAM tracking algorithm to perform parameter optimization on the camera pose calculated by the SLAM tracking algorithm, and an optimized camera pose is obtained.

[0232] Through the above optimization, the global error of the SLAM system can be eliminated, so that the AR terminal can also ensure high precision when tracking in a large scene. It should be noted that in the case of cumulative error generated when only using the local SLAM system for tracking, the pose of the AR terminal in the real space calculated according to the camera pose obtained from the local SLAM system will be inconsistent with the repositioning result. If the repositioning result is directly used to replace the positioning result of the SLAM system, it will cause obvious jumps in the system positioning. Based on the above pose optimization method, the repositioning result is regarded as a residual constraint term of the local SLAM tracking algorithm to optimize the camera pose calculated by the SLAM tracking algorithm, which can gradually eliminate the cumulative error and ensure the smoothness of the positioning result.

[0233] For example, Figure 11 is a process diagram for realizing large-scale tracking based on AR point cloud map repositioning according to an example embodiment, as Figure 11 shown, the AR terminal starts to run the SLAM system 105 locally, performs real-time tracking and positioning, and establishes a local world coordinate system during initialization; the AR terminal sends a query image 104 (i.e. a real-time image) to the 3D point cloud map for registration, obtains a visual re-projection constraint relationship 108, adds the visual re-projection constraint relationship to the local SLAM backend optimization framework, realizes a tightly coupled system 109 of SLAM and visual repositioning, determines a residual equation based on the tightly coupled system according to the re-projection constraint relationship; the residual equation is used as a residual constraint term of the SLAM tracking algorithm to optimize the camera pose calculated by the SLAM tracking algorithm, and the final system output pose can eliminate global cumulative error, realize real-time tracking and positioning 110 in a large-scale scene. The above example is only illustrative, and the present disclosure is not limited in this regard.

[0234] In addition, in order to realize real-time online addition of AR virtual objects according to user operations, the terminal can also send a target image and a second camera parameter when the terminal collects the target image to the server, the target image being an image to which a target AR object needs to be added, for example, if a user wants to add a virtual pet to a table in front of him, he can take a picture of the target image corresponding to the table through the camera of the terminal (such as an AR phone); then receive the repositioning pose of the terminal in the AR point cloud map determined by the server according to the target image and the second camera parameter; obtain a seventh pose of the target AR object relative to the local world coordinate system of the terminal added by the user on the target image; determine an eighth pose of the target AR object relative to the AR point cloud map according to the repositioning pose and the seventh pose; send the target AR object and the eighth pose to the server, so that the server adds the target AR object in the AR point cloud map according to the eighth pose.

[0235] Figure 12 is a block diagram of a cloud relocation device according to an exemplary embodiment, applied to a server, such as Figure 12 As shown, the device comprises:

[0236] A first receiving module 1201 configured to receive relocation request data sent by a terminal, the relocation request data comprising a real-time image collected by the terminal and a first camera parameter corresponding to the real-time image;

[0237] A feature extraction module 1202 configured to perform feature extraction on the real-time image by a first pre-trained deep learning model to obtain global image features and local image features;

[0238] An image matching module 1203 configured to perform feature matching on the real-time image and a plurality of preset images respectively according to the global image features to obtain matching images, the preset images being images used to create an augmented reality (AR) point cloud map;

[0239] A relocation module 1204 configured to relocate a first pose of the terminal in the AR point cloud map according to the local image features, the matching images, and the first camera parameter.

[0240] Optionally, the first pre-trained deep learning model comprises a first model and a second model; the feature extraction module 1202 is configured to perform feature extraction on the real-time image by the first model to obtain the global image features, and perform feature extraction on the real-time image by the second model to obtain the local image features.

[0241] Optionally, the image matching module 1203 is configured to output, according to the global image features, the matching images consistent with the global image features of the real-time image in the plurality of preset images by a K- nearest neighbor algorithm.

[0242] Optionally, the relocation module 1204 is configured to perform registration on the real-time image and each of the matching images according to the local image features to obtain a first feature point pair successfully registered; and relocate the first pose of the terminal in the AR point cloud map according to the first feature point pair and the first camera parameter.

[0243] Optionally, the local image features comprise a plurality of feature points, and the relocation module 1204 is configured to, for each of the matching images, input the plurality of feature points corresponding to the real-time image and the plurality of feature points corresponding to the matching image into a second pre-trained deep learning model to obtain the first feature point pair successfully registered.

[0244] Optionally, the first feature point pair comprises a first feature point of the real-time image and a second feature point of the matching image; the repositioning module 1204 is configured to, for each first feature point pair, determine a first 3D point of the first feature point in the AR point cloud map according to the second feature point in the first feature point pair; and calculate a first pose of the terminal in the AR point cloud map according to the first pose algorithm, the first 3D points corresponding to the first feature points in the plurality of first feature point pairs respectively, and the first camera parameters.

[0245] Optionally, Figure 13 is according to Figure 12 a block diagram of a cloud repositioning device shown in an embodiment, as Figure 13 shown, the device further comprises:

[0246] The first sending module 1205 is configured to send the first pose to the terminal, so that the terminal determines a mapping relationship between the AR point cloud map and a local world coordinate system of the terminal according to the first pose and a second pose, the second pose being a pose of the terminal relative to the local world coordinate system calculated by a simultaneous localization and mapping (SLAM) tracking algorithm.

[0247] Optionally, as Figure 13 shown, the device further comprises an AR cloud creating module 1206 configured to pre-create in the following manner:

[0248] receiving environment data sent by the terminal, the environment data comprising a plurality of frames of environment images of an environment currently located by the terminal and image parameters corresponding to each frame of environment image respectively;

[0249] For each frame of environment image, extracting global features and local features corresponding to the frame of environment image by using the first pre-trained deep learning model, and determining a preset number of similar images corresponding to the frame of environment image from other images according to the global features corresponding to the frame of environment image;

[0250] For each frame of similar image, performing image registration on the frame of environment image and the frame of similar image according to the local features of the frame of environment image and the local features of the frame of similar image to obtain a registration result, the registration result comprising a plurality of second feature point pairs successfully registered;

[0251] determining a second 3D point corresponding to each second feature point pair respectively by a preset depth estimation algorithm according to the second feature point pairs, the image parameters of the frame of environment image, and the image parameters of the frame of similar image;

[0252] generating the AR point cloud map according to the second 3D points.

[0253] Optionally, the image parameters comprise camera shooting parameters when the frame of environment image is captured and a terminal pose of the terminal relative to a local world coordinate system; the AR cloud creating module 1206 is configured to calculate a first pose conversion matrix between cameras corresponding to the frame of environment image and the frame of similar image according to the terminal pose corresponding to the frame of environment image and the terminal pose corresponding to the frame of similar image; input the plurality of second feature point pairs, the camera shooting parameters of the frame of environment image, the camera shooting parameters of the frame of similar image and the first pose conversion matrix into the preset depth estimation model to obtain the second 3D point corresponding to each second feature point pair respectively.

[0254] Optionally, the AR cloud creating module 1206 is further configured to, for each second 3D point, re-project the second 3D point to a projection point in a specified image, the specified image comprising the frame of environment image or the frame of similar image; determine whether the second 3D point satisfies a preset error threshold condition according to the projection point and the initial feature point corresponding to the second 3D point in the specified image; in a case where it is determined that the second 3D point does not satisfy the preset error threshold condition, delete the second 3D point, so as to filter the second 3D point to obtain a target 3D point; generate the AR point cloud map according to the target 3D point.

[0255] Optionally, the AR cloud creating module 1206 is further configured to, for each second 3D point, perform parameter optimization on the second 3D point by a preset parameter optimization algorithm to obtain an optimized 3D point; and perform position adjustment on the AR point cloud map according to the optimized 3D point.

[0256] Optionally, the AR cloud creating module 1206 is further configured to, for each two point cloud maps in the plurality of AR point cloud maps, determine whether the two point cloud maps correspond to a common picture; the common picture is a picture representing a common area existing between the two point cloud maps; in a case where the two point cloud maps correspond to the common picture, merge the two point cloud maps to obtain a target point cloud map.

[0257] Optionally, the two point cloud maps comprise a first point cloud map and a second point cloud map, and the common picture comprises a plurality of pictures used to create the first point cloud map; the AR cloud creating module 1206 is configured to, for each common picture, locate a third pose of a camera capturing the common picture in the second point cloud map; determine a second pose conversion matrix between the first point cloud map and the second point cloud map according to the third pose and a fourth pose of the camera capturing the common picture relative to the first point cloud map; and merge the two point cloud maps according to the second pose conversion matrix to obtain a target point cloud map.

[0258] Optionally, asFigure 13 As shown, the device further includes: an AR object adding module 1207, configured to perform densification processing on the AR point cloud map to obtain a dense point cloud map; obtain object parameters corresponding to a preset AR object; and add the preset AR object to the dense point cloud map according to the object parameters.

[0259] Optionally, the AR object adding module 1207 is further configured to receive a target image sent by the terminal and second camera parameters when the terminal acquires the target image, wherein the target image is an image to which a target AR object needs to be added; reposition the terminal relative to the AR point cloud map according to the target image and the second camera parameters to obtain a repositioned pose; send the repositioned pose to the terminal so that, when the terminal obtains the seventh pose of the target AR object relative to the terminal's local world coordinate system, it determines the eighth pose of the target AR object relative to the AR point cloud map according to the repositioned pose and the seventh pose, wherein the target AR object is an AR object added by the user on the target image; receive the target AR object and the eighth pose sent by the terminal, and add the target AR object to the AR point cloud map according to the eighth pose.

[0260] Figure 14 This is a block diagram illustrating a cloud-based relocation device according to an exemplary embodiment, applied to a terminal, such as... Figure 14 As shown, the device includes:

[0261] The data acquisition module 1401 is configured to acquire real-time images and obtain first camera parameters when acquiring the real-time images;

[0262] The second sending module 1402 is configured to send relocation request data to the server. The relocation request data includes the real-time image and the first camera parameters, so that the server can extract features from the real-time image using a first pre-trained deep learning model to obtain global image features and local image features. Based on the global image features, the server can perform feature matching between the real-time image and multiple preset images to obtain a matching image. Based on the local image features, the matching image, and the first camera parameters, the server can relocate the first pose of the terminal in the AR point cloud map.

[0263] The second receiving module 1403 is configured to receive the relocation result sent by the server, the relocation result including the first pose.

[0264] Optionally, Figure 15 It is based on Figure 14 The illustrated embodiment shows a block diagram of a cloud-based relocation device, as shown in the figure. Figure 15As shown, the device further comprises:

[0265] The first determining module 1404 is configured to determine a mapping relationship between the AR point cloud map and a local world coordinate system of the terminal according to the first pose and a second pose, the second pose being a pose of the terminal relative to the local world coordinate system calculated by the terminal through a SLAM tracking algorithm;

[0266] The third receiving module 1405 is configured to receive a fifth pose of a preset AR object relative to the AR point cloud map sent by the server;

[0267] The second determining module 1406 is configured to determine a sixth pose of the preset AR object relative to the local world coordinate system according to the mapping relationship and the fifth pose;

[0268] The AR object rendering module 1407 is configured to render the preset AR object on the real-time image according to the sixth pose, so as to add the preset AR object on the real-time image.

[0269] Optionally, Figure 16 is according to Figure 14 A block diagram of a cloud-side repositioning device according to the embodiment shown in the figure, the repositioning result further comprises a re-projection constraint relationship, the re-projection constraint relationship is a corresponding relationship between a target feature point of the real-time image and a 3D point corresponding to the target feature point on the AR point cloud map; the re-projection constraint relationship comprises a plurality of target 3D points and the target feature point corresponding to each target 3D point respectively; as Figure 16 As shown, the device further comprises:

[0270] The re-projection module 1408 is configured to, for each target 3D point, re-project the target 3D point to the real-time image through a preset re-projection algorithm to obtain a re-projection point corresponding to the target 3D point on the real-time image;

[0271] The third determining module 1409 is configured to determine a residual error equation according to the target feature point corresponding to the target 3D point and the re-projection point;

[0272] The parameter optimization module 1410 is configured to perform parameter optimization on a camera pose calculated by the SLAM tracking algorithm by taking the residual error equation as a residual error constraint term of the SLAM tracking algorithm, to obtain an optimized camera pose.

[0273] Optionally, Figure 17 is according to Figure 14 A block diagram of a cloud-side repositioning device according to the embodiment shown in the figure, as Figure 17 As shown, the device further comprises:

[0274] The third sending module 1411 is configured to send a target image and a second camera parameter of the terminal when collecting the target image to a server, the target image being an image requiring a target AR object to be added; receive a relocalization pose of the terminal in the AR point cloud map determined by the server according to the target image and the second camera parameter and sent by the server; obtain a seventh pose of the target AR object added by a user on the target image relative to a local world coordinate system of the terminal; determine an eighth pose of the target AR object relative to the AR point cloud map according to the relocalization pose and the seventh pose; and send the target AR object and the eighth pose to the server, so that the server adds the target AR object in the AR point cloud map according to the eighth pose.

[0275] As to the apparatus in the above-mentioned embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described here in detail.

[0276] With the above apparatus, the terminal can send relocalization request data to the server, the relocalization request data including a real-time image collected by the terminal and a first camera parameter corresponding to the real-time image; in this way, the server can perform feature extraction on the real-time image through a first pre-trained deep learning model to obtain global image features and local image features; perform feature matching on the real-time image and a plurality of preset images respectively according to the global image features to obtain matching images, the preset images being images used to create an augmented reality (AR) point cloud map; and relocalize a first pose of the terminal in the AR point cloud map according to the local image features, the matching images, and the first camera parameter. Compared with directly performing image registration on a plurality of preset images based on local artificial feature points, the present disclosure first performs image screening from a plurality of preset images according to global image features of a real-time image to obtain matching images, and then performs image registration from the matching images according to local image features, so that the accuracy of the relocalization result is higher, and the efficiency of relocalization is relatively higher. Furthermore, the global image features and the local image features in the present disclosure are both extracted through a pre-trained deep learning model, so that the stability and accuracy of the relocalization result are better compared with an image registration scheme using artificial feature points.

[0277] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, the program instructions being executed by a processor to implement the steps of the cloud relocalization method provided by the present disclosure.

[0278] Figure 18FIG. 18 is a block diagram illustrating an apparatus 1800 for cloud relocation according to an exemplary embodiment. The apparatus can be a electronic device, for example, the apparatus 1800 can be a mobile phone, a computer, a digital broadcast terminal, a message communicator, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0279] Referring to Figure 18 The apparatus 1800 can include one or more of the following components: a processing component 1802, a memory 1804, a power component 1806, a multimedia component 1808, an audio component 1810, an input / output (I / O) interface 1812, a sensor component 1814, and a communication component 1816.

[0280] The processing component 1802 usually controls overall operations of the apparatus 1800, such as operations associated with displaying, making phone calls, data communications, camera operations, and recording operations. The processing component 1802 can include one or more processors 1820 to execute instructions to complete all or part of steps of the above cloud relocation method. Further, the processing component 1802 can include one or more modules to facilitate interaction between the processing component 1802 and other components. For example, the processing component 1802 can include a multimedia module to facilitate the interaction between the multimedia component 1808 and the processing component 1802.

[0281] The memory 1804 is configured to store various types of data to support operations of the apparatus 1800. Examples of these data include instructions for any application or method operating on the apparatus 1800, contact data, phonebook data, messages, pictures, videos, etc. The memory 1804 can be realized by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0282] The power component 1806 provides power to various components of the apparatus 1800. The power component 1806 can include a power management system, one or more power sources, and other components associated with generating, managing and distributing power for the apparatus 1800.

[0283] The multimedia component 1808 includes a screen providing an output interface between the device 1800 and a user. In some embodiments, the screen includes a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping, and gestures on the touch panel. The touch sensors can not only sense a boundary of a touching or swiping action, but also detect duration and pressure related to the touching or swiping action. In some embodiments, the multimedia component 1808 includes a front camera and / or a rear camera. When the device 1800 is in an operation mode, such as a camera mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zooming capability.

[0284] The audio component 1810 is configured to output and / or input audio signals. For example, the audio component 1810 includes a microphone (MIC) configured to receive external audio signals when the device 1800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 1804 or transmitted via the communication component 1816. In some embodiments, the audio component 1810 also includes a speaker for outputting audio signals.

[0285] The I / O interface 1812 provides an interface between the processing component 1802 and peripheral interface modules, such as a keypad, a click wheel, buttons, and so on. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0286] The sensor component 1814 includes one or more sensors configured to provide various state assessments for the device 1800. For example, the sensor component 1814 can detect an open / closed state of the device 1800, relative positioning of components, such as a display and a keypad of the device 1800, a change in position of the device 1800 or a component of the device 1800, presence or absence of user contact with the device 1800, an orientation or acceleration / deceleration of the device 1800, and a temperature change of the device 1800. The sensor component 1814 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 1814 can further include a light sensor, such as a CMOS or CCD image sensor, for use in an imaging application. In some embodiments, the sensor component 1814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0287] The communication component 1816 is configured to facilitate wired or wireless communication between the device 1800 and other devices. The device 1800 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1816 further includes a Near Field Communication (NFC) module to facilitate close proximity communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0288] In an exemplary embodiment, the device 1800 can be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components, for performing the cloud relocation method described above.

[0289] In an exemplary embodiment, a non-transitory computer readable storage medium, such as the memory 1804 including instructions, is also provided, which can be executed by the processor 1820 of the device 1800 to complete the cloud relocation method described above. For example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0290] In another exemplary embodiment, a computer program product is also provided, which contains a computer program capable of being executed by a programmable device, and the computer program has code portions for executing the cloud relocation method described above when executed by the programmable device.

[0291] Figure 19 is a block diagram of a device 1900 for cloud relocation according to an exemplary embodiment. For example, the device 1900 can be provided as a server. Referring to Figure 19 , the device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions, such as an application program, executable by the processing component 1922. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the cloud relocation method described above.

[0292] The apparatus 1900 can also include a power supply component 1926 configured to perform power management for the apparatus 1900, a wired or wireless network interface 1950 configured to connect the apparatus 1900 to a network, and an input output (I / O) interface 1958. The apparatus 1900 can operate under the control of an operating system stored in the memory 1932, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM , or the like.

[0293] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

[0294] It will be understood that the disclosure is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from its scope. The scope of the disclosure is limited only by the claims that follow.

Claims

1. A cloud-based relocation method, characterized in that, Applied to a server, the method includes: The receiving terminal sends relocation request data, which includes real-time images acquired by the terminal and first camera parameters corresponding to the real-time images. The real-time image is used to extract features through a first pre-trained deep learning model to obtain global image features and local image features. Based on the global image features, the real-time image is matched with multiple preset images to obtain a matched image. The preset images are images used to create augmented reality (AR) point cloud maps. The first pose of the terminal in the AR point cloud map is relocalized based on the local image features, the matching image, and the first camera parameters. The AR point cloud map is pre-created in the following way: The terminal receives environmental data, which includes multiple frames of environmental images of the current environment collected by the terminal and image parameters corresponding to each frame of environmental image. For each frame of environment image, the first pre-trained deep learning model is used to extract the global and local features corresponding to the frame of environment image, and a preset number of similar images corresponding to the frame of environment image are determined from other images based on the global features corresponding to the frame of environment image. For each similar image frame, image registration is performed on the environment image and the similar image frame based on the local features of the environment image and the local features of the similar image frame to obtain a registration result. The registration result includes multiple successfully registered second feature point pairs. Based on multiple pairs of second feature points, the image parameters of the environmental image of the frame, and the image parameters of similar images of the frame, a preset depth estimation algorithm is used to determine the second 3D point corresponding to each pair of second feature points. The AR point cloud map is generated based on the second 3D point.

2. The method according to claim 1, characterized in that, The first pre-trained deep learning model includes a first model and a second model; the step of extracting features from the real-time image using the first pre-trained deep learning model to obtain global image features and local image features includes: The first model is used to extract features from the real-time image to obtain the global image features, and the second model is used to extract features from the real-time image to obtain the local image features.

3. The method according to claim 1, characterized in that, The step of performing feature matching between the real-time image and multiple preset images based on the global image features to obtain a matched image includes: Based on the global image features, the K-nearest neighbor algorithm is used to output a matching image from among the preset images that has global image features consistent with those of the real-time image.

4. The method according to claim 1, characterized in that, The step of relocalizing the terminal's first pose in the AR point cloud map based on the local image features, the matching image, and the first camera parameters includes: Based on the local image features, the real-time image is registered with each of the matching images to obtain a first successfully registered feature point pair. The terminal repositions its first pose in the AR point cloud map based on the first feature point pair and the first camera parameters.

5. The method according to claim 4, characterized in that, The local image features include multiple feature points, and the step of registering the real-time image with each of the matching images based on the local image features to obtain a first successfully registered feature point pair includes: For each of the matching images, multiple feature points corresponding to the real-time image and multiple feature points corresponding to the matching image are input into the second pre-trained deep learning model to obtain the first feature point pair that has been successfully registered.

6. The method according to claim 4, characterized in that, The first feature point pair includes a first feature point of the real-time image and a second feature point of the matched image; the step of relocalizing the first pose of the terminal in the AR point cloud map based on the first feature point pair and the first camera parameters includes: For each of the first feature point pairs, the first feature point in the AR point cloud map is determined based on the second feature point in the first feature point pair. Based on multiple pairs of first feature points, the first 3D point corresponding to each first feature point, and the first camera parameters, the first pose of the terminal in the AR point cloud map is calculated using a preset pose algorithm.

7. The method according to claim 1, characterized in that, After relocalizing the terminal's first pose in the AR point cloud map based on the local image features, the matching image, and the first camera parameters, the method further includes: The first pose is sent to the terminal so that the terminal can determine the mapping relationship between the AR point cloud map and the terminal's local world coordinate system based on the first pose and the second pose. The second pose is the pose of the terminal relative to the local world coordinate system calculated by the terminal through the Simultaneous Localization and Mapping (SLAM) tracking algorithm.

8. The method according to claim 1, characterized in that, The image parameters include the camera shooting parameters when the environmental image of the frame was acquired and the terminal pose relative to the local world coordinate system; the step of determining the second 3D point corresponding to each second feature point pair based on multiple second feature point pairs, the image parameters of the environmental image of the frame, and the image parameters of similar images of the frame through a preset depth estimation model includes: The first pose transformation matrix between the cameras corresponding to the two images is calculated based on the terminal pose corresponding to the environmental image of the frame and the terminal pose corresponding to the similar image of the frame. The preset depth estimation model is input with multiple pairs of second feature points, camera shooting parameters of the environmental image of the frame, camera shooting parameters of similar images of the frame, and the first pose transformation matrix to obtain the second 3D point corresponding to each pair of second feature points.

9. The method according to claim 1, characterized in that, After determining the second 3D point corresponding to each second feature point pair based on a preset depth estimation model using multiple second feature point pairs, image parameters of the environmental image of the frame, and image parameters of similar images of the frame, the method further includes: For each of the second 3D points, the second 3D point is reprojected onto a specified image to obtain a projection point. The specified image includes the environmental image of the frame or a similar image of the frame. Based on the projection point and the initial feature point corresponding to the second 3D point in the specified image, determine whether the second 3D point meets the preset error threshold condition; If it is determined that the second 3D point does not meet the preset error threshold condition, the second 3D point is deleted so that the second 3D point can be filtered to obtain the target 3D point; The step of generating the AR point cloud map based on the second 3D point includes: The AR point cloud map is generated based on the target 3D points.

10. The method according to claim 1, characterized in that, The method further includes: For each of the second 3D points, the parameters of the second 3D point are optimized using a preset parameter optimization algorithm to obtain an optimized 3D point; The AR point cloud map is repositioned based on the optimized 3D points.

11. The method according to claim 1, characterized in that, After generating the AR point cloud map based on the second 3D points, the method further includes: For each pair of point cloud maps in the multiple AR point cloud maps, determine whether the two point cloud maps correspond to a common image; the common image is an image that represents a common area that is repeated between the two point cloud maps. If the two point cloud maps correspond to a common image, the two point cloud maps are merged to obtain the target point cloud map.

12. The method according to claim 11, characterized in that, The two point cloud maps include a first point cloud map and a second point cloud map. The common image includes multiple images used to create the first point cloud map. Merging the two point cloud maps to obtain the target point cloud map includes: For each public image, the camera that captured the public image is located in the third pose of the second point cloud map; The second pose transformation matrix between the first point cloud map and the second point cloud map is determined based on the third pose and the fourth pose of the camera that acquired the public image relative to the first point cloud map. The two point cloud maps are merged according to the second pose transformation matrix to obtain the target point cloud map.

13. The method according to claim 1, characterized in that, After generating the AR point cloud map based on the second 3D points, the method further includes: The AR point cloud map is densified to obtain a dense point cloud map; Obtain the object parameters corresponding to the preset AR object, and add the preset AR object to the dense point cloud map according to the object parameters.

14. The method according to any one of claims 8-13, characterized in that, After generating the AR point cloud map based on the second 3D points, the method further includes: The terminal receives a target image sent by the terminal and second camera parameters when the terminal acquires the target image, wherein the target image is an image to which a target AR object needs to be added; The repositioning pose of the terminal relative to the AR point cloud map is obtained by repositioning the terminal based on the target image and the second camera parameters. The repositioning pose is sent to the terminal so that, after the terminal obtains the seventh pose of the target AR object relative to the terminal's local world coordinate system, the terminal determines the eighth pose of the target AR object relative to the AR point cloud map based on the repositioning pose and the seventh pose. The target AR object is an AR object added by the user on the target image. The system receives the target AR object and the eighth pose sent by the terminal, and adds the target AR object to the AR point cloud map according to the eighth pose.

15. A cloud-based relocation method, characterized in that, Applied to a terminal, the method includes: Acquire real-time images and obtain the first camera parameters at the time of acquiring the real-time images; The server sends relocation request data, which includes the real-time image and the first camera parameters, so that the server can extract features from the real-time image using a first pre-trained deep learning model to obtain global image features and local image features. Based on the global image features, the server performs feature matching between the real-time image and multiple preset images to obtain a matching image. Based on the local image features, the matching image, and the first camera parameters, the server relocates the first pose of the terminal in the AR point cloud map. Receive the relocation result sent by the server, the relocation result including the first pose; The method further includes: Send environmental data to the server. The environmental data includes multiple frames of environmental images of the current environment collected by the terminal and image parameters corresponding to each frame of environmental image. The server pre-creates the AR point cloud map in the following manner: For each frame of environment image, the first pre-trained deep learning model is used to extract the global and local features corresponding to the frame of environment image, and a preset number of similar images corresponding to the frame of environment image are determined from other images based on the global features corresponding to the frame of environment image; For each frame of similar image, image registration is performed on the frame of environment image and the frame of similar image based on the local features of the frame of environment image and the frame of similar image, and a registration result is obtained, the registration result including multiple successfully registered second feature point pairs; Based on the multiple second feature point pairs, the image parameters of the frame of environment image and the image parameters of the frame of similar image, a preset depth estimation algorithm is used to determine the second 3D point corresponding to each second feature point pair; The AR point cloud map is generated based on the second 3D points.

16. The method according to claim 15, characterized in that, The method further includes: The mapping relationship between the AR point cloud map and the local world coordinate system of the terminal is determined based on the first pose and the second pose. The second pose is the pose of the terminal relative to the local world coordinate system calculated by the terminal through the SLAM tracking algorithm. Receive the fifth pose of the preset AR object relative to the AR point cloud map sent by the server; The sixth pose of the preset AR object relative to the local world coordinate system is determined based on the mapping relationship and the fifth pose. The preset AR object is rendered on the real-time image according to the sixth pose, so as to add the preset AR object on the real-time image.

17. The method according to claim 16, characterized in that, The relocation result also includes a reprojection constraint relationship, which is the correspondence between the target feature points of the real-time image and the corresponding 3D points of the target feature points on the AR point cloud map. The reprojection constraint relationship includes multiple target 3D points and the target feature points corresponding to each target 3D point; the method further includes: For each target 3D point, the target 3D point is reprojected onto the real-time image using a preset reprojection algorithm to obtain the reprojection point of the target 3D point on the real-time image. The residual equation is determined based on the target feature points and the reprojection points corresponding to the target 3D point; The residual equation is used as a residual constraint term in the SLAM tracking algorithm to optimize the camera pose calculated by the SLAM tracking algorithm, thereby obtaining the optimized camera pose.

18. The method according to any one of claims 15-17, characterized in that, The method further includes: Send the target image and the second camera parameters when the terminal acquires the target image to the server; the target image is an image to which a target AR object needs to be added. The terminal receives the repositioning pose in the AR point cloud map, determined by the server based on the target image and the parameters of the second camera, sent by the server. Obtain the seventh pose of the target AR object added by the user on the target image relative to the local world coordinate system of the terminal; The eighth pose of the target AR object relative to the AR point cloud map is determined based on the repositioning pose and the seventh pose. The target AR object and the eighth pose are sent to the server so that the server can add the target AR object to the AR point cloud map based on the eighth pose.

19. A cloud-based relocation device, characterized in that, Applied to a server, the device includes: The first receiving module is configured to receive relocation request data sent by the terminal. The relocation request data includes real-time images acquired by the terminal and first camera parameters corresponding to the real-time images. The feature extraction module is configured to extract features from the real-time image using a first pre-trained deep learning model to obtain global image features and local image features; The image matching module is configured to perform feature matching between the real-time image and multiple preset images based on the global image features to obtain a matching image, wherein the preset images are images used to create augmented reality (AR) point cloud maps. The relocalization module is configured to relocalize the first pose of the terminal in the AR point cloud map based on the local image features, the matching image, and the first camera parameters. The device further includes: The AR cloud creation module is configured to receive environmental data sent by the terminal. The environmental data includes multiple frames of environmental images of the current environment collected by the terminal and image parameters corresponding to each frame. For each frame of environmental image, the first pre-trained deep learning model is used to extract the global and local features corresponding to that frame of environmental image, and a preset number of similar images corresponding to that frame of environmental image are determined from other images based on the global features of that frame of environmental image. For each similar image, image registration is performed on the frame of environmental image and the similar image based on the local features of that frame of environmental image and the similar image, to obtain a registration result. The registration result includes multiple successfully registered pairs of second feature points. Based on the multiple pairs of second feature points, the image parameters of the frame of environmental image, and the image parameters of the similar image, a preset depth estimation algorithm is used to determine the second 3D point corresponding to each pair of second feature points. The AR point cloud map is generated based on the second 3D points.

20. A cloud-based relocation device, characterized in that, Applied to a terminal, the device includes: The data acquisition module is configured to acquire real-time images and obtain first camera parameters when acquiring the real-time images; The second sending module is configured to send relocation request data to the server. The relocation request data includes the real-time image and the first camera parameters, so that the server can extract features from the real-time image using a first pre-trained deep learning model to obtain global image features and local image features. Based on the global image features, the server performs feature matching between the real-time image and multiple preset images to obtain a matching image. Based on the local image features, the matching image, and the first camera parameters, the server relocates the first pose of the terminal in the AR point cloud map. The second receiving module is configured to receive the relocation result sent by the server, the relocation result including the first pose; The second sending module is further configured to send environmental data to the server, the environmental data including multiple frames of environmental images of the current environment collected by the terminal and image parameters corresponding to each frame of environmental images; The server pre-creates the AR point cloud map in the following manner: For each frame of environment image, the first pre-trained deep learning model is used to extract the global and local features corresponding to the frame of environment image, and a preset number of similar images corresponding to the frame of environment image are determined from other images based on the global features corresponding to the frame of environment image; For each frame of similar image, image registration is performed on the frame of environment image and the frame of similar image based on the local features of the frame of environment image and the frame of similar image, and a registration result is obtained, the registration result including multiple successfully registered second feature point pairs; Based on the multiple second feature point pairs, the image parameters of the frame of environment image and the image parameters of the frame of similar image, a preset depth estimation algorithm is used to determine the second 3D point corresponding to each second feature point pair; The AR point cloud map is generated based on the second 3D points.

21. A cloud-based relocation device, characterized in that, Applied to servers, including: processor; Memory used to store processor-executable instructions; The processor is configured as follows: The receiving terminal sends relocation request data, which includes real-time images acquired by the terminal and first camera parameters corresponding to the real-time images. The real-time image is used to extract features through a first pre-trained deep learning model to obtain global image features and local image features. Based on the global image features, the real-time image is matched with multiple preset images to obtain a matched image. The preset images are images used to create augmented reality (AR) point cloud maps. The first pose of the terminal in the AR point cloud map is relocalized based on the local image features, the matching image, and the first camera parameters. The AR point cloud map is pre-created in the following way: The terminal receives environmental data, which includes multiple frames of environmental images of the current environment collected by the terminal and image parameters corresponding to each frame of environmental image. For each frame of environment image, the first pre-trained deep learning model is used to extract the global and local features corresponding to the frame of environment image, and a preset number of similar images corresponding to the frame of environment image are determined from other images based on the global features corresponding to the frame of environment image. For each similar image frame, image registration is performed on the environment image and the similar image frame based on the local features of the environment image and the local features of the similar image frame to obtain a registration result. The registration result includes multiple successfully registered second feature point pairs. Based on multiple pairs of second feature points, the image parameters of the environmental image of the frame, and the image parameters of similar images of the frame, a preset depth estimation algorithm is used to determine the second 3D point corresponding to each pair of second feature points. The AR point cloud map is generated based on the second 3D point.

22. A cloud-based relocation device, characterized in that, Applied to terminals, including: processor; Memory used to store processor-executable instructions; The processor is configured as follows: Acquire real-time images and obtain the first camera parameters at the time of acquiring the real-time images; The server sends relocation request data, which includes the real-time image and the first camera parameters, so that the server can extract features from the real-time image using a first pre-trained deep learning model to obtain global image features and local image features. Based on the global image features, the server performs feature matching between the real-time image and multiple preset images to obtain a matching image. Based on the local image features, the matching image, and the first camera parameters, the server relocates the first pose of the terminal in the AR point cloud map. Receive the relocation result sent by the server, the relocation result including the first pose; The processor is further configured to send environmental data to the server, the environmental data including multiple frames of environmental images of the current environment collected by the terminal and image parameters corresponding to each frame of environmental image; The server pre-creates the AR point cloud map in the following manner: For each frame of environment image, the first pre-trained deep learning model is used to extract the global and local features corresponding to the frame of environment image, and a preset number of similar images corresponding to the frame of environment image are determined from other images based on the global features corresponding to the frame of environment image; For each frame of similar image, image registration is performed on the frame of environment image and the frame of similar image based on the local features of the frame of environment image and the frame of similar image, and a registration result is obtained, the registration result including multiple successfully registered second feature point pairs; Based on the multiple second feature point pairs, the image parameters of the frame of environment image and the image parameters of the frame of similar image, a preset depth estimation algorithm is used to determine the second 3D point corresponding to each second feature point pair; The AR point cloud map is generated based on the second 3D points.

23. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the program instructions implement the steps of the method described in any one of claims 1 to 14.

24. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the program instructions implement the steps of the method described in any one of claims 15 to 18.

Images