A method, device, equipment and storage medium for online database expansion

Abstract

Embodiments of the present disclosure provide a method, device and storage medium for online database augmentation, the database being used for spatial positioning of a terminal, the method comprising: obtaining a current image frame to be positioned; performing pose solving on the current image frame based on the database; and when the pose solving is successful, augmenting the database using the current image frame, wherein the augmented database is used for pose solving on a next image frame.

Description

Technical Field

[0001] This application relates to the field of positioning technology, and in particular to a method, apparatus, device and storage medium for online database expansion. Background Technology

[0002] Visual SLAM (Simultaneous Localization and Mapping) is a crucial technology for autonomous mobile devices equipped with visual sensors such as cameras to autonomously locate themselves and build 3D maps of their surroundings without prior environmental information. It is a fundamental technology in fields such as computer vision, robotics, autonomous vehicles, and augmented reality. Based on visual SLAM technology, the pose of image frames is calculated by tracking extracted image features, and the terminal is then positioned in 3D.

[0003] Existing terminal-cloud (server-side) positioning fusion technologies perform terminal positioning and cloud positioning separately, then fuse the two types of positioning information at the terminal to obtain the final positioning information. Terminal positioning is achieved through SLAM, while cloud positioning is achieved through a series of pose calculations. Because the terminal lacks a global pose, terminal positioning suffers from accumulated errors and drift. Cloud positioning, based on its powerful computing capabilities, has better performance and can correct the terminal positioning information using cloud positioning information during fusion processing, eliminating errors.

[0004] In AR (Augmented Reality) technology, scene effects are typically applied. The terminal captures an image of the scene and sends it to the server. Upon receiving the image to be located, the server searches and matches it against a pre-built image database, then calculates the pose based on the matched image. However, due to factors such as differences in lighting and / or viewing angle, the image captured by the terminal may not match the images in the database. Existing methods of manually collecting data and offline database expansion are not only time-consuming but also extremely costly in terms of manpower. Summary of the Invention

[0005] This disclosure provides a method, apparatus, device, and storage medium for online database expansion to provide images covering more scenarios and improve the accuracy of terminal pose calculation.

[0006] In a first aspect, embodiments of this disclosure provide a method for expanding a database online, including:

[0007] Obtain the current image frame to be located;

[0008] Pose calculation is performed on the current image frame based on the database; and

[0009] When the pose calculation is successful, the database is expanded using the current image frame, and the expanded database is used to calculate the pose for the next image frame.

[0010] Optionally, when the database only includes the first database, the step of expanding the database using the current image frame when the pose calculation is successful includes:

[0011] Create a second database, wherein the second database is independent of the first database, and the expanded database includes both the first and second databases; and

[0012] Add the current image frame to the second database.

[0013] Optionally, when the database includes a first database and a second database independent of the first database, the step of expanding the database using the current image frame when the pose calculation is successful includes:

[0014] The current image frame is added to the second database, wherein the expanded database includes the first database and the second database.

[0015] Optionally, adding the current image frame to the second database includes:

[0016] Add the global descriptor of the current image frame to the global descriptor of the second database.

[0017] Optionally, adding the current image frame to the second database further includes:

[0018] The local descriptor corresponding to the local feature point of the inlier of the current image frame is added to the second database.

[0019] Optionally, the step of calculating the pose of the current image frame based on the database includes:

[0020] Based on the similarity between the global descriptor of the current image frame and the global descriptor of each image in the first database and the second database, a predetermined number of similar image frames are recalled from the first database and the second database; and

[0021] Based on the similar image frames, the pose of the current image frame is calculated.

[0022] Optionally, the step of calculating the pose of the current image frame based on the similar image frames includes:

[0023] Perform local descriptor matching between the current image frame and each recalled similar image frame; and

[0024] Pose calculation is performed based on matched local descriptors.

[0025] Optionally, the pose calculation based on the matched local descriptor includes:

[0026] Based on the matched local descriptors, the 3D point information corresponding to the local feature points in the first database and the second database is determined; and

[0027] Pose calculation is performed based on the three-dimensional point information.

[0028] Optionally, the ratio between the number of similar image frames recalled from the second database and the number of similar image frames recalled from the first database is a predetermined value.

[0029] Optionally, the capacity of the second database is a predetermined value.

[0030] Secondly, embodiments of this disclosure provide an apparatus for online database expansion, comprising:

[0031] The acquisition unit is configured to acquire the current image frame to be located;

[0032] The calculation unit is configured to perform pose calculation on the current image frame based on a database; and,

[0033] An expansion unit is configured to expand the database using the current image frame when the pose calculation is successful, wherein the expanded database is used to perform pose calculation for the next image frame.

[0034] Optionally, when the database only includes the first database, the expansion unit is configured to expand the database using the current image frame when the pose calculation is successful, including:

[0035] Create a second database, wherein the second database is independent of the first database, and the expanded database includes both the first and second databases; and

[0036] Add the current image frame to the second database.

[0037] Optionally, when the database includes a first database and a second database independent of the first database, the expansion unit is configured to expand the database using the current image frame when the pose calculation is successful, including:

[0038] The current image frame is added to the second database, wherein the expanded database includes the first database and the second database.

[0039] Optionally, the expansion unit is configured to add the current image frame to the second database, including:

[0040] Add the global descriptor of the current image frame to the global descriptor of the second database.

[0041] Optionally, the expansion unit is configured to add the current image frame to the second database, and further includes:

[0042] The local descriptor corresponding to the local feature point of the inlier of the current image frame is added to the second database.

[0043] Optionally, the solution sheet is configured to perform pose calculation on the current image frame based on a database, including:

[0044] Based on the similarity between the global descriptor of the current image frame and the global descriptor of each image in the first database and the second database, a predetermined number of similar image frames are recalled from the first database and the second database; and

[0045] Based on the similar image frames, the pose of the current image frame is calculated.

[0046] Optionally, the calculation unit is configured to perform pose calculation on the current image frame based on the similar image frames, including:

[0047] Perform local descriptor matching between the current image frame and each recalled similar image frame; and

[0048] Pose calculation is performed based on matched local descriptors.

[0049] Optionally, the solving unit is configured to perform pose calculation based on matched local descriptors, including:

[0050] Based on the matched local descriptors, the 3D point information corresponding to the local feature points in the first database and the second database is determined; and

[0051] Pose calculation is performed based on the three-dimensional point information.

[0052] Optionally, the calculation unit is configured such that the ratio between the number of similar image frames recalled from the second database and the number of similar image frames recalled from the first database is a predetermined value.

[0053] Optionally, the capacity of the second database is a predetermined value.

[0054] Thirdly, embodiments of this disclosure provide an apparatus for online database expansion, comprising: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to implement the online database expansion method as described above, including:

[0055] Obtain the current image frame to be located;

[0056] Pose calculation is performed on the current image frame based on the database; and

[0057] When the pose calculation is successful, the database is expanded using the current image frame, and the expanded database is used to calculate the pose for the next image frame.

[0058] Optionally, when the database only includes the first database, the step of expanding the database using the current image frame when the pose calculation is successful includes:

[0059] Create a second database, wherein the second database is independent of the first database, and the expanded database includes both the first and second databases; and

[0060] Add the current image frame to the second database.

[0061] Optionally, when the database includes a first database and a second database independent of the first database, the step of expanding the database using the current image frame when the pose calculation is successful includes:

[0062] The current image frame is added to the second database, wherein the expanded database includes the first database and the second database.

[0063] Optionally, adding the current image frame to the second database includes:

[0064] Add the global descriptor of the current image frame to the global descriptor of the second database.

[0065] Optionally, adding the current image frame to the second database further includes:

[0066] The local descriptor corresponding to the local feature point of the inlier of the current image frame is added to the second database.

[0067] Optionally, the step of calculating the pose of the current image frame based on the database includes:

[0068] Based on the similarity between the global descriptor of the current image frame and the global descriptor of each image in the first database and the second database, a predetermined number of similar image frames are recalled from the first database and the second database; and

[0069] Based on the similar image frames, the pose of the current image frame is calculated.

[0070] Optionally, the step of calculating the pose of the current image frame based on the similar image frames includes:

[0071] Perform local descriptor matching between the current image frame and each recalled similar image frame; and

[0072] Pose calculation is performed based on matched local descriptors.

[0073] Optionally, the pose calculation based on the matched local descriptor includes:

[0074] Based on the matched local descriptors, the 3D point information corresponding to the local feature points in the first database and the second database is determined; and

[0075] Pose calculation is performed based on the three-dimensional point information.

[0076] Optionally, the ratio between the number of similar image frames recalled from the second database and the number of similar image frames recalled from the first database is a predetermined value.

[0077] Optionally, the capacity of the second database is a predetermined value.

[0078] Fourthly, embodiments of this disclosure provide a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the online database expansion method described above, including:

[0079] Obtain the current image frame to be located;

[0080] Pose calculation is performed on the current image frame based on the database; and

[0081] When the pose calculation is successful, the database is expanded using the current image frame, and the expanded database is used to calculate the pose for the next image frame.

[0082] Optionally, when the database only includes the first database, the step of expanding the database using the current image frame when the pose calculation is successful includes:

[0083] Create a second database, wherein the second database is independent of the first database, and the expanded database includes both the first and second databases; and

[0084] Add the current image frame to the second database.

[0085] Optionally, when the database includes a first database and a second database independent of the first database, the step of expanding the database using the current image frame when the pose calculation is successful includes:

[0086] The current image frame is added to the second database, wherein the expanded database includes the first database and the second database.

[0087] Optionally, adding the current image frame to the second database includes:

[0088] Add the global descriptor of the current image frame to the global descriptor of the second database.

[0089] Optionally, adding the current image frame to the second database further includes:

[0090] The local descriptor corresponding to the local feature point of the inlier of the current image frame is added to the second database.

[0091] Optionally, the step of calculating the pose of the current image frame based on the database includes:

[0092] Based on the similarity between the global descriptor of the current image frame and the global descriptor of each image in the first database and the second database, a predetermined number of similar image frames are recalled from the first database and the second database; and

[0093] Based on the similar image frames, the pose of the current image frame is calculated.

[0094] Optionally, the step of calculating the pose of the current image frame based on the similar image frames includes:

[0095] Perform local descriptor matching between the current image frame and each recalled similar image frame; and

[0096] Pose calculation is performed based on matched local descriptors.

[0097] Optionally, the pose calculation based on the matched local descriptor includes:

[0098] Based on the matched local descriptors, the 3D point information corresponding to the local feature points in the first database and the second database is determined; and

[0099] Pose calculation is performed based on the three-dimensional point information.

[0100] Optionally, the ratio between the number of similar image frames recalled from the second database and the number of similar image frames recalled from the first database is a predetermined value.

[0101] Optionally, the capacity of the second database is a predetermined value.

[0102] In the embodiments of this application, when the cloud receives the image frame to be located sent by the terminal, it adds the successfully located image frame to the pre-built database in real time. This online addition of successfully located image frames to the database achieves online expansion of the database. The expanded database stores images more similar to the subsequent image frames to be located, providing search images with smaller lighting differences and more similar scenes, thereby improving image matching performance, recall success rate, and the number of descriptor matches, further improving the success rate and accuracy of pose calculation for subsequent image frames to be located. This application provides images covering more scenes, allowing the server to recall image frames with more similar scenes during processing. The more similar the recalled image frames are in different scenes, the more associations can be established between them, resulting in better pose calculation performance. In this implementation, by expanding the database online, the expanded database can cover more scenes and more varied images (e.g., images under different lighting conditions), enabling the search to find more matching images, thereby improving the success rate of pose calculation and terminal positioning. Furthermore, by constructing a second database associated with the object to be located, this embodiment of the disclosure will not affect the original database accessed on the server side, nor will it increase the pressure on the original database on the server side. Attached Figure Description

[0103] To more clearly illustrate the technical solutions in one or more embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0104] Figure 1 A flowchart illustrating the method for expanding an online database according to the first embodiment of this disclosure;

[0105] Figure 2 A schematic diagram of the structure of an apparatus for expanding an online database according to an embodiment of this disclosure; and

[0106] Figure 3A schematic diagram of the structure of an online extended database device provided in an embodiment of this disclosure. Detailed Implementation

[0107] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this disclosure, the technical solutions in one or more embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of the embodiments. Based on one or more embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0108] In this application, a global descriptor is a vector used to represent an image; each image has one global descriptor. A local descriptor is also a pointer used to represent a feature point. Each feature point has one local descriptor. Since computers cannot directly recognize images, in order to enable computers to "understand" images and thus possess true "vision," researchers have chosen to employ feature extraction methods. This involves extracting useful data or information from images and converting it into "non-image" representations or descriptions, such as numerical values ​​or vectors, to facilitate computer image processing.

[0109] In the current scene special effects processing, virtual models are synthesized with real point clouds to create virtual 3D scenes that are closer to the real world in public places such as scenic spots, museums, smart parks, airports, high-speed rail stations, and commercial spaces.

[0110] In one application example of this application, a user takes a picture of a scene at a scenic spot using their mobile phone and sends it to the server. Upon receiving the image to be located from the terminal, the server searches and matches it against a pre-established image database. Then, based on the matched image information, it calculates the terminal's pose. If the pose calculation is successful, scene effects are applied to the image based on the terminal's pose information. If the matched image is inaccurate, it will introduce errors in the final terminal pose calculation, causing the final scene effects to drift. Thus, the images stored in the database directly affect the result of the pose calculation for the image to be located, thereby affecting the scene effects.

[0111] Images in the server-side database are pre-stored before the system is put into use. Due to limitations in acquisition conditions, it's difficult to acquire a comprehensive set of images. For example, there are often significant differences between the server-side database and the images sent by the terminal for localization, such as variations in lighting and viewing angle. This can lead to incorrect recalled image frames or mismatched descriptors, causing localization failures. To avoid this during use, during the testing phase, if localization fails in the current scene, one approach is to offline expand the server-side database by acquiring additional images. For instance, if the issue is found to be caused by significant differences in viewing angle or lighting, additional scene images are typically needed. This requires assigning dedicated personnel for acquisition, resulting in high labor costs and a long processing time, potentially several days. Regenerating the database from the supplemented scene images and the previous dataset also takes time. Database regeneration involves calculating the pose of all images together, including not only the pose of feature points in the 2D image but also the pose of the projection of those feature points into 3D space. Therefore, offline database expansion suffers from high costs and long processing times.

[0112] Reference Figure 1 As shown, Figure 1 This is a schematic flowchart of the method for expanding an online database provided in the first embodiment of this disclosure, as shown below. Figure 1 As shown, the process includes the following steps:

[0113] Step 101: Obtain the current image frame to be located;

[0114] Step 102: Perform pose calculation on the current image frame based on the database;

[0115] Step 103: When the pose calculation is successful, the current image frame is used to expand the database. The expanded database is used to calculate the pose for the next image frame.

[0116] In actual terminal use, upon receiving a frame of an image to be located from the terminal, the frame with successfully calculated pose is added in real time to the pre-built accessed database. This online addition of successfully located frames expands the database, storing images more similar to the next frame to be located. This provides search images with smaller lighting differences and more similar scenes for subsequent frames, improving image matching performance, recall success rate, and the number of descriptor matches, further enhancing the success rate and accuracy of pose calculation for subsequent frames. This embodiment provides images covering more scenes, allowing the server to recall image frames with more similar scenes during processing. The more similar the recalled image frames are in different scenes, the more associations can be established between them, resulting in better pose calculation. Each association corresponds to an equation. When the number of equations is sufficient, the relative pose of the current frame with respect to the corresponding image frame in the database can be calculated.

[0117] In this implementation, by expanding the database online, the expanded database can cover more scenes and more varied images, such as images with significant changes in lighting. During the query, more matching images can be found, providing more similar matching images and improving the success rate of pose calculation.

[0118] In a preferred embodiment of this application, when the database only includes a first database, expanding the database using the current image frame when the pose calculation is successful includes:

[0119] Create a second database, which is independent of the first database, and the expanded database includes both the first and second databases; and

[0120] Add the current image frame to the second database.

[0121] In this embodiment, the second database corresponds to the object to be located. The object to be located refers to the terminal. Of course, it should be noted that as smart hardware gradually improves, this terminal will not be limited to mobile phones; it is highly likely to be integrated into glasses, headphones, home appliances, and even cars.

[0122] In a preferred embodiment of this application, when the database includes a first database and a second database independent of the first database, expanding the database using the current image frame when the pose calculation is successful includes:

[0123] The current image frame is added to the second database, wherein the expanded database includes the first database and the second database.

[0124] In the embodiments of this application, image frames are added to a second database in real time. These real-time added image frames can be referred to as temporal frames. The second database, together with the accessed first database, constitutes an expanded database. This application employs two databases to implement its technical solution. This is because if the information of the current image frame is directly put into the first database (which can be called the spatial frame database), when many terminals are in an application scenario, each terminal may send a sequence of images, and each terminal has its own real-time image frames. If all of these are added to the first database, the processing effect will be poor when multiple people need to be located, and it may even cause the accessed database to crash. Furthermore, the temporal frames of each terminal are not related to each other. Therefore, a second database (container) with a certain capacity is maintained for each terminal. The image frames in this second database are updated in real time without affecting the first database on the server side or other terminals.

[0125] In one embodiment of this application, the capacity of the second database is a predetermined value. As a preferred implementation, the capacity of the second database is 3 to 5 time-domain frames. The preferred capacity is 5 time-domain frames. The container storing the time-domain frames has a certain capacity limitation; it cannot be infinitely large. Not all historical image frames uploaded by the terminal are placed in it, because if a relatively long period of time has passed, the contribution of previous image frames to the current image frame is very small. Therefore, the capacity of the second database is set to 5 time-domain frames.

[0126] In a preferred embodiment of this application, adding the current image frame to the second database includes adding the global descriptor of the current image frame to the global descriptor of the second database. The global descriptor reflects global image information, and is subsequently used to find similar image frames; the higher the similarity, the better the match.

[0127] In a preferred embodiment of this application, adding the current image frame to the second database further includes adding local descriptors corresponding to local feature points of the inlier of the current image frame to the second database. After finding similar image frames using global descriptors, the local descriptors are used to find 3D (three-dimensional) point information in the database corresponding to the local feature points.

[0128] Conventional offline acquisition methods save all feature points and descriptors of an image frame. In this application, however, only the inliers and their corresponding descriptors of the current frame are saved. When there are three to four thousand feature points in total, and only a few dozen inliers in a single frame, and these inliers have already been calculated, their validity is high. Therefore, the method described in this application has lower storage costs and faster processing speed.

[0129] In a preferred embodiment of this application, the pose calculation of the current image frame based on the expanded database includes:

[0130] Based on the similarity between the global descriptor of the current image frame and the global descriptor of each image in the first database and the second database, a predetermined number of similar image frames are recalled from the first database and the second database; and

[0131] Based on the similar image frame, the pose of the current image frame is calculated.

[0132] In a preferred embodiment of this application, pose calculation of the current image frame is performed based on the similar image frames, including:

[0133] Perform local descriptor matching between the current image frame and each recalled similar image frame; and

[0134] Pose calculation is performed based on matched local descriptors.

[0135] In a preferred embodiment of this application, the pose calculation based on the matched local descriptor includes:

[0136] Based on the matched local descriptors, the 3D point information corresponding to the local feature point in the first database and the second database is determined; and

[0137] Pose calculation is performed based on this 3D point information.

[0138] Pose calculation is performed by matching the current image frame to be located with similar image frames using local descriptors. This includes finding the 3D point information corresponding to the local feature points in the database through the matched local descriptors, and calculating the pose based on the 3D point information.

[0139] The embodiments of this application only need to calculate the pose of the current 2D image feature points, without calculating the pose of the projection of these feature points in 3D space. The poses of the feature points projected in 3D space are already stored in a preset database. This application finds the 3D points in the database that are originally associated with the 2D feature points based on local descriptors. This application only needs to directly store the local and global information of the image into the database, and then find the 3D point information corresponding to the local information in the database. It is not necessary to calculate the poses of the 2D feature points projected in 3D space again. Therefore, the real-time database update method of this disclosure embodiment has the advantages of shorter time consumption and lower cost compared to the manual offline supplementary acquisition method.

[0140] In a preferred embodiment of this application, the ratio between the number of similar image frames recalled from the second database and the number of similar image frames recalled from the first database is a predetermined value.

[0141] In the embodiments of this application, preferably, the number of image frames recalled in the second database corresponding to the terminal is preset, for example, the number of image frames recalled in the second database is limited to 3, while the first database still recalls the same preset number, such as 10. Therefore, as a preferred method, the preset ratio is 3 / 10. Of course, the preset ratio can also be preset in other ways according to actual usage needs, and this application does not limit it.

[0142] In one application example of this application, when only the first database is available, 10 image frames are recalled. To provide more and better recalled frames, the method of this application recalls several more image frames from the second database. Depending on the container capacity limit, the number is between 0 and 5. If all 5 image frames are recalled, the matching cost increases because each recalled image frame must be matched with the current image frame. The more image frames recalled, the longer the matching time. Therefore, a compromise is necessary. In this application, 3 image frames is the preferred value in the embodiments, achieving the goal of recalling more similar images.

[0143] In existing applications, when the recalled image frames are insufficient—for example, the first few recalled frames are ideal, meaning the similarity meets the requirements, but some intermediate frames are not ideally recalled, meaning no valid frames can be recalled to associate with the current frame—data association is reduced. Another situation is that the recalled frames are ideal, but due to significant differences in viewing angle, the number of matching points between the two frames is too small. By adopting the method of this application embodiment, the terminal has the characteristic of continuous angle during use, and the probability of sudden large angle changes is low. The method of this application directly provides the image most similar to the scene of the current image frame, which improves the recall and matching results, avoiding the above two situations. This allows for better calculation of the current image frame pose, resulting in a higher success rate and accuracy in terminal pose calculation, better scene effects obtained by the terminal, and reduced jitter and drift in scene effects.

[0144] In the application example of this application, pose calculation is performed on all image frames to be located sent by the terminal, and the frame images with the calculated pose are added to the time domain window corresponding to the terminal, i.e., the second database.

[0145] This application embodiment provides an online database expansion method, offering image frames covering more scenes. This results in better frame matching, leading to more matching interior points and thus better acquisition of the pose corresponding to the image to be located. The successfully calculated pose sent to the terminal is more accurate, further improving the registration rate of successfully calculated frames and the accuracy of pose calculation on the server side. This ultimately leads to higher accuracy in terminal pose calculation.

[0146] Figure 2 This is a schematic diagram of the structure of an online extended database apparatus provided in an embodiment of the present disclosure, as shown below. Figure 2 As shown, the device includes:

[0147] The acquisition unit 201 is configured to acquire the current image frame to be located;

[0148] The solution unit 202 is configured to perform pose calculation on the current image frame based on the database;

[0149] The expansion unit 203 is configured to expand the database using the current image frame when the pose calculation is successful, wherein the expanded database is used to perform pose calculation for the next image frame.

[0150] In a preferred embodiment of this application, when the database only includes the first database, the expansion unit 203 is configured to expand the database using the current image frame when the pose calculation is successful, including:

[0151] Create a second database, which is independent of the first database, and the expanded database includes both the first and second databases; and

[0152] Add the current image frame to the second database.

[0153] In a preferred embodiment of this application, when the database includes a first database and a second database independent of the first database, the expansion unit 203 is configured to expand the database using the current image frame when the pose calculation is successful, including:

[0154] The current image frame is added to the second database, wherein the expanded database includes the first database and the second database.

[0155] In a preferred embodiment of this application, the expansion unit 203 is configured to add the current image frame to the second database, including:

[0156] Add the global descriptor of the current image frame to the global descriptor of the second database.

[0157] In a preferred embodiment of this application, the expansion unit 203, configured to add the current image frame to the second database, further includes:

[0158] The local descriptor corresponding to the local feature point of the inlier of the current image frame is added to the second database.

[0159] In a preferred embodiment of this application, the solving unit 202 is configured to perform pose calculation on the current image frame based on a database, including:

[0160] Based on the similarity between the global descriptor of the current image frame and the global descriptor of each image in the first database and the second database, a predetermined number of similar image frames are recalled from the first database and the second database; and

[0161] Based on the similar image frame, the pose of the current image frame is calculated.

[0162] In a preferred embodiment of this application, the solving unit 202 is configured to perform pose calculation on the current image frame based on the similar image frames, including:

[0163] Perform local descriptor matching between the current image frame and each recalled similar image frame; and

[0164] Pose calculation is performed based on matched local descriptors.

[0165] In a preferred embodiment of this application, the solving unit 202 is configured to perform pose calculation based on matched local descriptors, including:

[0166] Based on the matched local descriptors, the 3D point information corresponding to the local feature point in the first database and the second database is determined; and

[0167] Pose calculation is performed based on this 3D point information.

[0168] In a preferred embodiment of this application, the calculation unit 202 is configured such that the ratio between the number of similar image frames recalled from the second database and the number of similar image frames recalled from the first database is a predetermined value.

[0169] In a preferred embodiment of this application, the capacity of the second database is a predetermined value.

[0170] The apparatus for expanding the database online in this embodiment can implement the various processes of the aforementioned method embodiments for expanding the database online, and achieve the same effects and functions, which will not be repeated here.

[0171] One embodiment of this disclosure also provides a device for online database expansion. Figure 3A schematic diagram of the structure of an online extended database device provided in an embodiment of this disclosure is shown below. Figure 3 As shown, devices for online extended databases can vary significantly due to differences in configuration or performance. They may include one or more processors 301 and memory 302, with memory 302 storing one or more application programs or data. Memory 302 can be temporary or persistent storage. The application programs stored in memory 302 may include one or more modules (not shown), each module including a series of computer-executable instructions from the positioning device. Furthermore, processor 301 may be configured to communicate with memory 302, executing the series of computer-executable instructions in memory 302 on the online extended database device. The online extended database device may also include one or more power supplies 303, one or more wired or wireless network interfaces 304, one or more input or output interfaces 305, one or more keyboards 306, etc. It is understood that... Figure 3 This is just one possible example; in real-world applications, devices that can expand their databases online can be configured arbitrarily as needed.

[0172] In one specific embodiment, the device for online database expansion includes a processor; and a memory configured to store computer-executable instructions that, when executed, cause the processor to perform the following processes:

[0173] Obtain the current image frame to be located;

[0174] The pose of the current image frame is calculated based on the database.

[0175] When the pose calculation is successful, the current image frame is used to expand the database. The expanded database is then used to calculate the pose for the next image frame.

[0176] Optionally, when the database only includes the first database, when the pose calculation is successful, the database is expanded using the current image frame, including:

[0177] Create a second database, which is independent of the first database, and the expanded database includes both the first and second databases; and

[0178] Add the current image frame to the second database.

[0179] Optionally, when the database includes a first database and a second database independent of the first database, when the pose calculation is successful, the current image frame is used to expand the database, including:

[0180] The current image frame is added to the second database, wherein the expanded database includes the first database and the second database.

[0181] Optionally, the current image frame is added to the second database, including:

[0182] Add the global descriptor of the current image frame to the global descriptor of the second database.

[0183] Optionally, adding the current image frame to the second database also includes:

[0184] The local descriptor corresponding to the local feature point of the inlier of the current image frame is added to the second database.

[0185] Optionally, the pose calculation of the current image frame based on the database includes:

[0186] Based on the similarity between the global descriptor of the current image frame and the global descriptor of each image in the first database and the second database, a predetermined number of similar image frames are recalled from the first database and the second database; and

[0187] Based on the similar image frame, the pose of the current image frame is calculated.

[0188] Optionally, based on the similar image frame, pose calculation is performed on the current image frame, including:

[0189] Perform local descriptor matching between the current image frame and each recalled similar image frame; and

[0190] Pose calculation is performed based on matched local descriptors.

[0191] Optionally, pose calculation is performed based on the matched local descriptor, including:

[0192] Based on the matched local descriptors, the 3D point information corresponding to the local feature point in the first database and the second database is determined; and

[0193] Pose calculation is performed based on this 3D point information.

[0194] Optionally, the ratio between the number of similar image frames recalled from the second database and the number of similar image frames recalled from the first database is a predetermined value.

[0195] Optionally, the capacity of the second database is a predetermined value.

[0196] In this embodiment, when recall is unsatisfactory, or when the viewing angle difference is too large during matching, leading to unsatisfactory matching, the method described in this application embodiment can improve the matching of frames, thereby bringing more matched inliers. This allows for better calculation of the pose of the current frame, resulting in more accurate poses sent to the terminal. Consequently, it can improve the registration rate of successfully calculated frames and the accuracy of server-side recall. This leads to a higher success rate and higher accuracy in terminal pose calculation.

[0197] The device for online database expansion in this embodiment can implement the various processes of the aforementioned method embodiment for online database expansion and achieve the same effects and functions, which will not be repeated here.

[0198] One embodiment of this disclosure also provides a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the following process:

[0199] Obtain the current image frame to be located;

[0200] The pose of the current image frame is calculated based on the database.

[0201] When the pose calculation is successful, the current image frame is used to expand the database. The expanded database is then used to calculate the pose for the next image frame.

[0202] Optionally, when the database only includes the first database, when the pose calculation is successful, the database is expanded using the current image frame, including:

[0203] Create a second database, which is independent of the first database, and the expanded database includes both the first and second databases; and

[0204] Add the current image frame to the second database.

[0205] Optionally, when the database includes a first database and a second database independent of the first database, when the pose calculation is successful, the current image frame is used to expand the database, including:

[0206] The current image frame is added to the second database, wherein the expanded database includes the first database and the second database.

[0207] Optionally, the current image frame is added to the second database, including:

[0208] Add the global descriptor of the current image frame to the global descriptor of the second database.

[0209] Optionally, adding the current image frame to the second database also includes:

[0210] The local descriptor corresponding to the local feature point of the inlier of the current image frame is added to the second database.

[0211] Optionally, pose calculation is performed on the current image frame based on the database, including:

[0212] Based on the similarity between the global descriptor of the current image frame and the global descriptor of each image in the first database and the second database, a predetermined number of similar image frames are recalled from the first database and the second database; and

[0213] Based on the similar image frame, the pose of the current image frame is calculated.

[0214] Optionally, based on the similar image frame, pose calculation is performed on the current image frame, including:

[0215] Perform local descriptor matching between the current image frame and each recalled similar image frame; and

[0216] Pose calculation is performed based on matched local descriptors.

[0217] Optionally, pose calculation is performed based on the matched local descriptor, including:

[0218] Based on the matched local descriptors, the 3D point information corresponding to the local feature point in the first database and the second database is determined; and

[0219] Pose calculation is performed based on this 3D point information.

[0220] Optionally, the ratio between the number of similar image frames recalled from the second database and the number of similar image frames recalled from the first database is a predetermined value.

[0221] Optionally, the capacity of the second database is a predetermined value.

[0222] The computer-readable storage medium in this embodiment can implement the various processes of the aforementioned online database expansion method embodiment and achieve the same effects and functions, which will not be repeated here.

[0223] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0224] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0225] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0226] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0227] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

Claims

1. A method for online expansion of a database, the database being used for spatial positioning of a terminal, the method comprising: Obtain the current image frame to be located; The pose of the current image frame is calculated based on the database. as well as When the pose calculation is successful, the database is expanded using the current image frame, wherein the expanded database is used to perform pose calculation for the next image frame; When the database only includes the first database, the step of expanding the database using the current image frame when the pose calculation is successful includes: Create a second database corresponding to the terminal, wherein the second database is independent of the first database, and the expanded database includes the first database and the second database, the second database corresponds to the object to be located, and the capacity of the second database is a predetermined value; Add the current image frame to the second database.

2. The method according to claim 1, wherein, When the database includes a first database and a second database, the step of expanding the database using the current image frame when the pose calculation is successful includes: The current image frame is added to the second database, wherein the expanded database includes the first database and the second database.

3. The method according to claim 1 or 2, wherein, Adding the current image frame to the second database includes: Add the global descriptor of the current image frame to the global descriptor of the second database.

4. The method according to claim 3, wherein, Adding the current image frame to the second database further includes: The local descriptor corresponding to the local feature point of the inlier of the current image frame is added to the second database.

5. The method according to claim 2, wherein, The pose calculation of the current image frame based on the database includes: Based on the similarity between the global descriptor of the current image frame and the global descriptor of each image in the first database and the second database, a predetermined number of similar image frames are recalled from the first database and the second database; and Based on the similar image frames, the pose of the current image frame is calculated.

6. The method according to claim 5, wherein, The step of calculating the pose of the current image frame based on the similar image frames includes: Perform local descriptor matching between the current image frame and each recalled similar image frame; and Pose calculation is performed based on matched local descriptors.

7. The method according to claim 6, wherein, The pose calculation based on matched local descriptors includes: Based on the matched local descriptors, 3D point information corresponding to local feature points in the first and second databases is determined; and Pose calculation is performed based on the three-dimensional point information.

8. The method according to claim 5, wherein, The ratio between the number of similar image frames recalled from the second database and the number of similar image frames recalled from the first database is a predetermined value.

9. The method according to claim 1 or 2, wherein, The capacity of the second database is a predetermined value.

10. An apparatus for online database expansion, comprising: The acquisition unit is configured to acquire the current image frame to be located; The calculation unit is configured to perform pose calculation on the current image frame based on the database; as well as, An expansion unit is configured to expand the database using the current image frame when the pose calculation is successful, wherein the expanded database is used to perform pose calculation for the next image frame; The expansion unit is further configured to: When the database only includes the first database, a second database corresponding to the terminal is created, wherein the second database is independent of the first database, and the expanded database includes the first database and the second database, the second database corresponds to the object to be located, and the capacity of the second database is a predetermined value; Add the current image frame to the second database.

11. A device for online database expansion, comprising: processor; as well as, The memory is configured to store computer-executable instructions that, when executed, cause the processor to implement the method of online database expansion as described in any one of claims 1-9.

12. A computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the method for online expansion of a database as described in any one of claims 1-9.

Images