A method and device for updating an AR positioning map

Abstract

The present disclosure provides a method for updating an AR positioning map, including: obtaining a positioning image uploaded by a client device; calculating a first positioning result thereof in an original AR positioning map, updating the confidence level of a block to be updated, and determining whether the confidence level of the block to be updated is lower than a first preset threshold; if so, determining that the block to be updated needs to be updated, reconstructing the block to be updated to obtain a to-be-verified local scene AR positioning map; obtaining a new positioning image uploaded by the client device; calculating a second positioning result thereof in the to-be-verified local scene AR positioning map, updating the confidence level and point cloud information of the to-be-verified local scene AR positioning map, and determining whether the confidence level of the to-be-verified local scene AR positioning map is higher than a second preset threshold; if so, fusing the to-be-verified local scene AR positioning map and the original AR positioning map to obtain a new AR positioning map, and replacing the original AR positioning map with the new AR positioning map.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of augmented reality, and more particularly, to a method and apparatus for updating an AR positioning map Background Art

[0002] Since there is a problem in the AR positioning map of visual positioning that the point cloud data is inconsistent with the real-time scene due to scene changes, and this problem directly affects the positioning accuracy and even the positioning success rate of the user terminal in the scene.

[0003] Existing technologies for updating AR positioning maps generally include:

[0004] 1) Staff regularly check the changes in the scene. When the AR positioning map scene changes, the staff use special acquisition equipment to re-acquire the image data of the entire scene or a partial scene, manually complete the replacement of the image data of the updated part, and then re-build the map and update the cloud AR positioning map. It is necessary for the staff to regularly check the scene, collect, integrate and process the image data after discovering problems, which may result in a long update process, high maintenance costs, and a large amount of computing resources required for the processing and integration of the updated data.

[0005] 2) Allow users to provide real-time feedback or edit map information. Users can submit the environmental changes through the application and complete the acquisition and upload of the changed scene data under the guidance of the document. This information is used to update the map in the cloud. On the one hand, it has high requirements for user operations and the operations are complex. On the other hand, an effective review mechanism is required to filter and verify the information provided by users to prevent the entry of incorrect or false data. Summary of the Invention

[0006] Embodiments described herein provide a method and apparatus for updating an AR positioning map to solve the above technical problems.

[0007] According to a first aspect of the present disclosure, there is provided a method for updating an AR positioning map, including:

[0008] Obtaining a positioning image uploaded by a user terminal device;

[0009] Calculating a first positioning result of the positioning image in the original AR positioning map;

[0010] If the first positioning result is a positioning failure, calculating a positioning pose of the positioning failure image in the original AR positioning map according to the positioning success images in the adjacent area of the positioning failure image, and determining the position of the block to be updated according to the positioning pose;

[0011] Update the confidence level of the block to be updated according to the first positioning result, and determine whether the confidence level of the block to be updated is lower than a first preset threshold;

[0012] If the confidence level of the block to be updated is lower than the first preset threshold, determine that the block to be updated needs to be updated, and reconstruct the block to be updated according to the positioning failure image and the positioning success images in the adjacent regions of the positioning failure image to obtain a local scene AR positioning map to be verified;

[0013] Obtain a new positioning image uploaded by the client device;

[0014] Calculate a second positioning result of the new positioning image in the local scene AR positioning map to be verified;

[0015] Update the confidence level and the point cloud information of the local scene AR positioning map to be verified according to the second positioning result, and determine whether the confidence level of the local scene AR positioning map to be verified is higher than a second preset threshold;

[0016] If the confidence level of the local scene AR positioning map to be verified is higher than the second preset threshold, fuse the local scene AR positioning map to be verified and the original AR positioning map to obtain a new AR positioning map, and replace the original AR positioning map with the new AR positioning map.

[0017] In some embodiments, the step of calculating the first positioning result of the positioning image in the original AR positioning map specifically includes:

[0018] Extract features from the positioning image to extract feature points in the image;

[0019] Match the feature points with the features stored in the original AR positioning map, and determine the first positioning result according to the matching result;

[0020] If the matching fails, the first positioning result is a positioning failure;

[0021] If the matching succeeds, the first positioning result is a positioning success.

[0022] In some embodiments, the step of calculating the positioning pose of the positioning failure image in the original AR positioning map according to the positioning success images in the adjacent regions of the positioning failure image and determining the position of the block to be updated according to the positioning pose specifically includes:

[0023] Obtain the VIO pose of the positioning failure image uploaded by the client device, the VIO pose of the positioning success images in the adjacent areas of the positioning failure image, and the positioning poses of the positioning success images in the adjacent areas of the positioning failure image in the original AR positioning map;

[0024] According to the VIO pose of the positioning failure image, the VIO pose of the positioning success images in the adjacent areas of the positioning failure image, and the positioning poses of the positioning success images in the adjacent areas of the positioning failure image in the original AR positioning map, calculate the positioning pose of the positioning failure image in the original AR positioning map. The calculation method is as follows:

[0025] Set the pose of the positioning failure image I b in the VIO coordinate system of the client device as The image I b The adjacent positioning success image I a in the VIO coordinate system of the client device as And the image I a The pose in the original AR positioning map is Calculate the pose of the image I b in the original AR positioning map as Expressed as

[0026] where I represents the VIO coordinate system, m represents the original AR positioning map coordinate system, R represents a three-dimensional rotation matrix, R -1 represents the inverse matrix of the three-dimensional rotation matrix, t represents the displacement vector, 0 T represents the transpose of the matrix, and 0 represents the zero matrix;

[0027] According to Determine the position of the block to be updated.

[0028] In some embodiments, the step of updating the confidence level of the block to be updated according to the first positioning result specifically includes:

[0029] If the first positioning result is positioning failure, lower the confidence level of the block to be updated and mark the confidence level of the block to be updated as low confidence;

[0030] If the first positioning result is positioning success, raise the confidence level of the block to be updated.

[0031] In some embodiments, the step of calculating the second positioning result of the new positioning image in the AR positioning map of the local scene to be verified specifically includes:

[0032] Extract features from the new positioning image to obtain the extracted features;

[0033] Match the extracted features with the features stored in the to-be-verified local scene AR positioning map, and determine the second positioning result according to the matching result;

[0034] If the matching fails, the second positioning result is positioning failure;

[0035] If the matching succeeds, the second positioning result is positioning success.

[0036] In some embodiments, the step of updating the confidence of the to-be-verified local scene AR positioning map according to the second positioning result specifically includes:

[0037] If the second positioning result is positioning failure, lower the confidence of the to-be-verified local scene AR positioning map;

[0038] If the second positioning result is positioning success, raise the confidence of the to-be-verified local scene AR positioning map.

[0039] In some embodiments, the step of fusing the to-be-verified local scene AR positioning map and the original AR positioning map specifically includes:

[0040] For the positioning success images in the original AR positioning map, set the pose sequence of its image sequence in the to-be-verified local scene AR positioning map to be wherein, the pose sequence of this image sequence in the original AR positioning map obtained by the PNP algorithm is In an ideal case, there exists a similarity transformation matrix For the positioning success images in the original AR positioning map, set the pose sequence of its image sequence in the to-be-verified local scene AR positioning map to be wherein, the pose sequence of this image sequence in the original AR positioning map is There exists a similarity transformation matrix such that holds;

[0041] wherein, i represents the image sequence number of the image sequence, P represents the coordinate system of the to-be-verified local scene AR positioning map, m represents the coordinate system of the original AR positioning map, T represents the Euclidean transformation matrix, wherein, R represents the three-dimensional rotation matrix, 0 T represents the transpose of the matrix, 0 represents the zero matrix, and t represents the displacement vector; represents the similarity transformation matrix, and the form is as follows wherein, R represents the three-dimensional rotation matrix, t represents the displacement vector, and s represents the scaling factor

[0042] Taking as the optimization objective, the estimated result of is obtained, where r (residual) represents the residual between the calculated value and the true value, n represents the number of images used for the optimization calculation, and the point cloud information of the local scene AR positioning map to be verified passes through Inverse matrix mapping can achieve coordinate alignment with the original AR positioning map. Among them, when is true, there is R -1 represents the inverse matrix of the three-dimensional rotation matrix;

[0043] Delete the low-confidence information in the original AR positioning map.

[0044] According to a second aspect of the present disclosure, there is provided an apparatus for updating an AR positioning map, including:

[0045] An acquisition module for acquiring a positioning image uploaded by a client device;

[0046] A first calculation module for calculating a first positioning result of the positioning image in the original AR positioning map;

[0047] A first processing module, if the first positioning result is a positioning failure, calculates the positioning pose of the positioning failure image in the original AR positioning map according to the positioning success images in the adjacent area of the positioning failure image, and determines the position of the block to be updated according to the positioning pose; updates the confidence of the block to be updated according to the first positioning result, and determines whether the confidence of the block to be updated is lower than a first preset threshold; if the confidence of the block to be updated is lower than the first preset threshold, it is determined that the block to be updated needs to be updated, and the block to be updated is reconstructed according to the positioning failure image and the positioning success images in the adjacent area of the positioning failure image to obtain a local scene AR positioning map to be verified;

[0048] The acquisition module is further configured to acquire a new positioning image uploaded by the client device;

[0049] A second calculation module for calculating a second positioning result of the new positioning image in the local scene AR positioning map to be verified;

[0050] A second processing module, configured to update the confidence of the AR positioning map of the to-be-verified local scene and the point cloud information of the AR positioning map of the to-be-verified local scene according to the second positioning result, and determine whether the confidence of the AR positioning map of the to-be-verified local scene is higher than a second preset threshold; if the confidence of the AR positioning map of the to-be-verified local scene is higher than the second preset threshold, then fuse the AR positioning map of the to-be-verified local scene and the original AR positioning map to obtain a new AR positioning map, and replace the original AR positioning map with the new AR positioning map.

[0051] According to a third aspect of the present disclosure, there is provided a computer device, including a memory and a processor, where a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method for updating the AR positioning map in any one of the above embodiments are implemented.

[0052] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for updating the AR positioning map in any one of the above embodiments are implemented.

[0053] The embodiments of the present application provide a method for updating an AR positioning map, which includes obtaining a positioning image uploaded by a client device; calculating a first positioning result of the positioning image in the original AR positioning map; if the first positioning result is a positioning failure, calculating the positioning pose of the positioning failure image in the original AR positioning map according to the positioning success images in the adjacent areas of the positioning failure image, and determining the position of the block to be updated according to the positioning pose; updating the confidence level of the block to be updated according to the first positioning result, and determining whether the confidence level of the block to be updated is lower than a first preset threshold; if the confidence level of the block to be updated is lower than the first preset threshold, determining that the block to be updated needs to be updated, and reconstructing the block to be updated according to the positioning failure image and the positioning success images in the adjacent areas of the positioning failure image to obtain a to-be-verified local scene AR positioning map; obtaining a new positioning image uploaded by the client device; calculating a second positioning result of the new positioning image in the to-be-verified local scene AR positioning map; updating the confidence level and the point cloud information of the to-be-verified local scene AR positioning map according to the second positioning result, and determining whether the confidence level of the to-be-verified local scene AR positioning map is higher than a second preset threshold; if the confidence level of the to-be-verified local scene AR positioning map is higher than the second preset threshold, fusing the to-be-verified local scene AR positioning map and the original AR positioning map to obtain a new AR positioning map, and replacing the original AR positioning map with the new AR positioning map. In this way, the present disclosure automatically determines whether the map scene has changed based on the uploaded positioning images of the user's positioning in the scene, and automatically updates the local map data of the changed scene using the uploaded positioning images of the user. It does not require arranging staff to re-collect the entire scene data for the overall reconstruction of the map, nor does it require additional operation requirements for the user. Therefore, it has the advantages of high automation, low cost, and user-friendliness compared with the existing methods.

[0054] The above description is only an overview of the technical solutions of the embodiments of the present application. In order to be able to understand the technical means of the embodiments of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features, and advantages of the embodiments of the present application more obvious and understandable, the following specifically illustrates the embodiments of the present application. Brief Description of the Drawings

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly described below. It should be understood that the following described drawings only relate to some embodiments of the present disclosure and do not limit the present disclosure, where:

[0056] Figure 1It is a schematic diagram of the application environment of a method for updating an AR positioning map provided by an embodiment of the present disclosure.

[0057] Figure 2 It is a schematic diagram of the principle of a method for updating an AR positioning map provided by an embodiment of the present disclosure.

[0058] Figure 3 It is a schematic flowchart of a method for updating an AR positioning map provided by an embodiment of the present disclosure.

[0059] Figure 4 It is a schematic structural diagram of a device for updating an AR positioning map provided by an embodiment of the present disclosure.

[0060] Figure 5 It is a schematic example diagram provided by an embodiment of the present disclosure.

[0061] Figure 6 It is a schematic example diagram provided by an embodiment of the present disclosure.

[0062] Figure 7 It is a schematic example diagram provided by an embodiment of the present disclosure.

[0063] Figure 8 It is a schematic structural diagram of a computer device provided by an embodiment of the present disclosure.

[0064] It should be noted that the elements in the drawings are schematic and not drawn to scale. Detailed implementation manners

[0065] In order to make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some, but not all, of the embodiments of the present disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of the present disclosure without creative efforts shall also fall within the scope of protection of the present disclosure.

[0066] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the subject matter of the present disclosure pertains. Further, it will be understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the context of the specification and the relevant art, and shall not be interpreted in an idealized or overly formal form unless expressly so defined herein. As used herein, a statement that two or more parts are "connected" or "coupled" together shall mean that these parts are directly joined together or joined through one or more intermediate components.

[0067] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase "embodiments" in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0068] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists, A and B exist at the same time, and B exists. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. Terms such as "first" and "second" are only used to distinguish one component (or a part of a component) from another component (or another part of a component).

[0069] In the description of the present application, unless otherwise specified, "plurality" means more than two (including two), and similarly, "multiple groups" means more than two groups (including two).

[0070] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0071] In this application, it should be understood that the terms involved may be technical means for implementing a part of the present invention or other summary technical terms. For example, the terms may include:

[0072] AR (Argument Reality): Augmented reality is a technology that cleverly integrates virtual information with the real world. It simulates computer-generated virtual information such as text, images, models, videos, etc., and applies them to the real world. The two types of information complement each other to achieve "enhancement" of the real world.

[0073] Pose: position and attitude (orientation), for example: in two dimensions it is generally 3Dof (x, y, yaw), in three dimensions it is generally 6Dof (x, y, z, yaw, pitch, roll), the last three elements describe the attitude of the object, where yaw is the heading angle, rotating around the Z axis; pitch is the pitch angle, rotating around the Y axis; roll is the roll angle, rotating around the X axis.

[0074] Positioning is performed in the VIO local coordinate system, such as the VIO commonly used on AR glasses, ARKit on iOS phones, ARCore on Android phones, and AREngine on Huawei phones, with a smaller accuracy error.

[0075] VIO, namely Visual Inertial Odometry, is a positioning method that fuses visual and inertial measurement data. It combines an IMU and a camera to obtain geometric information about the environment and processes and analyzes it through computer vision technology to achieve precise positioning and navigation.

[0076] The basic principle of VIO is to fuse the IMU and the camera. The attitude information of the camera is obtained through the acceleration and angular velocity measurement values of the IMU, and the geometric information of the environment is obtained through the image information of the camera. By fusing and processing this information, the position and attitude information of the camera can be obtained, thus achieving precise positioning and navigation.

[0077] VIO has the advantages of high precision, high resolution, low cost, etc., so it has been widely used in many fields. In the fields of robot navigation, unmanned driving, and drone flight, VIO is widely used to achieve precise positioning and navigation. At the same time, VIO can also be fused with other sensors such as lidar and GPS to further improve the positioning accuracy and stability.

[0078] The accuracy and performance of VIO are affected by various factors such as lighting conditions, image quality, and motion speed. Therefore, when using VIO for visual positioning, calibration and alignment are required to eliminate errors and improve accuracy. At the same time, the data of VIO also needs to be processed and analyzed to extract useful information, and fused and optimized to achieve precise and reliable visual positioning.

[0079] IMU, namely Inertial Measurement Unit, is a measurement system based on the inertial principle, usually composed of accelerometers in three directions and gyroscopes in three directions.

[0080] The accelerometer is used to measure the acceleration of an object in three directions, while the gyroscope is used to measure the angular velocity of an object in three directions. By integrating and calculating these measurement values, information such as the attitude, speed, and position of the object can be obtained.

[0081] IMU has the advantages of good dynamic performance, strong anti-interference ability, and high accuracy, so it has been widely used in many fields. In visual positioning, the IMU can be used to assist the camera in attitude estimation and position tracking, improving the accuracy and stability of visual positioning.

[0082] The working principle of the IMU is based on physical principles such as Newton's second law and the Coriolis force. By resolving and processing the measured values, the motion state information of the object can be obtained. At the same time, the IMU can also be fused with other sensors such as GPS and lidar to further improve the measurement accuracy and reliability.

[0083] The accuracy and performance of the IMU are affected by various factors such as temperature, noise, and zero drift. Therefore, when using the IMU for visual positioning, calibration and adjustment are required to eliminate errors and improve accuracy. At the same time, it is also necessary to process and analyze the IMU data to extract useful information, and perform fusion and optimization to achieve accurate and reliable visual positioning.

[0084] SFM refers to "Structure from Motion", that is, the process of deriving the three-dimensional structure and camera motion in the scene by analyzing and processing the image sequence.

[0085] This technology generally includes the following steps:

[0086] Feature extraction: Extract features from each frame in the image sequence. These features may be corner points, edges, descriptors, etc.

[0087] Feature matching: Match similar feature points in different frames to determine their corresponding relationships in different frames.

[0088] Camera positioning: Derive the position and orientation of the camera in space, that is, the camera positioning, through known feature point matching and geometric reasoning.

[0089] Three-dimensional reconstruction: Estimate the three-dimensional structure of the objects in the scene based on the camera motion and the positions of the feature points, that is, three-dimensional reconstruction.

[0090] Bundle Adjustment: By optimizing the estimation of camera motion and three-dimensional points, making these estimations more consistent. This process is called Bundle Adjustment.

[0091] SFM is usually applied to fields such as reconstructing three-dimensional scenes, building three-dimensional models, navigation, and SLAM (Simultaneous Localization and Mapping). It has a wide range of applications in fields such as computer vision, augmented reality, virtual reality, and robotics.

[0092] PnP (Perspective-n-Point): It is a method for solving the correspondence between 3D and 2D points. This method describes how to estimate the pose of a camera when the 3D positions of n points in space are known. If the 3D positions of feature points in one of the two images are known, then at least 3 point pairs (and at least one additional verification point to verify the result) are required to calculate the camera motion.

[0093] A method for updating an AR positioning map provided in this application can be applied in an application environment such as Figure 1 shown. Figure 1 It is a schematic diagram of the application environment of a method for updating an AR positioning map provided by an embodiment of the present disclosure. Figure 2 It is a schematic diagram of the principle of a method for updating an AR positioning map provided by an embodiment of the present disclosure. As Figure 1 shown, the client device 110 moves in the target scene. During the movement, the positioning image is uploaded to the cloud server 120 to calculate the positioning result. As Figure 2 shown, the cloud server 120 automatically determines whether the scene has changed based on the positioning image uploaded by the client device 110, and automatically updates the AR positioning map based on the positioning image uploaded by the client device 110 to maintain the accuracy of the AR positioning map and provide high-precision and highly reliable positioning services for users. It should be noted that the above client device 110 can be a smart device such as a smartphone, a tablet computer, an AR glasses with VIO (Visual-IMU based odometry) function, and a head-mounted device, etc. The cloud server 120 includes, but is not limited to, a combination of one or more of the deployed Bluetooth positioning service, GPS positioning service, and WIFI positioning service.

[0094] Figure 3 It is a schematic flowchart of a method for updating an AR positioning map provided by an embodiment of the present disclosure.

[0095] As Figure 3 shown, the specific process of the method for updating the AR positioning map includes the following steps:

[0096] Step S210: Obtain the positioning image uploaded by the client device;

[0097] The camera on the client device will capture the images of the scene in real time, and these positioning images will be collected and uploaded to the cloud server.

[0098] Step S220: Calculate the first positioning result of the positioning image in the original AR positioning map.

[0099] The original AR positioning map is the AR positioning map established in the cloud server and stored in the cloud server;

[0100] Specifically, step S220 specifically includes the following steps:

[0101] The cloud server processes the received positioning image. First, feature extraction is performed to extract feature points in the image. The features in the image may include edges, colors, textures, etc. Usually, features that can reflect unique information of the object or environment are selected.

[0102] Match the extracted feature points with the features stored in the original AR positioning map in the cloud server, find the feature points belonging to the same object or scene, and determine the first positioning result according to the matching result;

[0103] If the matching fails, the first positioning result is positioning failure;

[0104] If the matching succeeds, the first positioning result is positioning success.

[0105] Step S230, if the first positioning result is positioning failure, then calculate the positioning pose of the positioning failure image in the original AR positioning map according to the positioning success images in the adjacent areas of the positioning failure image, and determine the position of the block to be updated according to the positioning pose;

[0106] The original AR positioning map can be divided into multiple blocks. When the positioning is successful, the AR positioning block can be directly determined according to the positioning result;

[0107] When the positioning fails, it can be judged that the scene has changed and this block needs to be updated. According to the calculated positioning pose, determine the position of the block to be updated.

[0108] The positioning image includes: positioning failure image and positioning success image.

[0109] After feature matching, it is also necessary to estimate the pose of the image, that is, the rotation and tilt angles of the device. This usually combines the IMU (Inertial Measurement Unit) data of the device.

[0110] In an AR (Augmented Reality) system, the pose (position and orientation) of the device is usually determined by a feature point-based visual positioning method, which requires both the positioning image of the user device and the pre-established AR positioning map.

[0111] Specifically, obtain the VIO pose of the positioning failure image uploaded by the user device, the VIO pose of the positioning success images in the adjacent areas of the positioning failure image, and the positioning pose of the positioning success images in the adjacent areas of the positioning failure image in the original AR positioning map;

[0112] Calculate the positioning pose of the positioning failure image in the original AR positioning map based on the VIO pose of the positioning failure image, the VIO poses of the positioning success images in the adjacent areas of the positioning failure image, and the positioning poses of the positioning success images in the adjacent areas of the positioning failure image in the original AR positioning map. The calculation method is as follows:

[0113] Set the successfully positioned image I in the original AR positioning map b The pose in the VIO coordinate system of the client device is The image I adjacent to b The successfully positioned image I in the original AR positioning map adjacent to a The pose in the VIO coordinate system of the client device is And the image I a The pose in the original AR positioning map is Calculate the pose of the image I b In the original AR positioning map is Expressed as formula (1):

[0114]

[0115] Where, I represents the VIO coordinate system, m represents the original AR positioning map coordinate system, R represents the three-dimensional rotation matrix, R -1 Represents the inverse matrix of the three-dimensional rotation matrix, t represents the displacement vector, 0 T Represents the transpose of the matrix, 0 represents the zero matrix;

[0116] According to Determine the position of the block to be updated.

[0117] Determine the number of successfully positioned images in the adjacent original AR positioning map as the number n of images used for the optimization calculation.

[0118] Step S240: Update the confidence of the block to be updated according to the first positioning result, and determine whether the confidence of the block to be updated is lower than the first preset threshold;

[0119] If the first positioning result is positioning failure, lower the confidence of the block to be updated;

[0120] If the first positioning result is positioning success, raise the confidence of the block to be updated.

[0121] Step S250: If the confidence level of the block to be updated is lower than the first preset threshold, it is determined that the block to be updated needs to be updated. Based on the positioning failure image and the positioning success images in the adjacent areas of the positioning failure image, the block to be updated is reconstructed to obtain a to-be-verified local scene AR positioning map.

[0122] When the user moves from a well-positioned block to the current block, the cloud server automatically detects in the background that the positioning success rate of the positioning images uploaded by the user in the current block continues to decrease significantly and is lower than the set threshold. Then, the cloud server in the background marks the decrease in the confidence level of the current block accordingly.

[0123] In some embodiments, the SFM (Structure from Motion) method can be used to complete the initial reconstruction.

[0124] The SFM (Structure from Motion) method specifically includes the following steps:

[0125] Feature extraction: Feature extraction is performed on each frame in the image sequence. These features may be corner points, edges, descriptors, etc.

[0126] Feature matching: Similar feature points in different frames are matched to determine their corresponding relationships in different frames.

[0127] Camera positioning: By using the known feature point matching and geometric reasoning, the position and pose of the camera in space are deduced, that is, the camera positioning.

[0128] Three-dimensional reconstruction: Based on the movement of the camera and the positions of the feature points, the three-dimensional structure of the objects in the scene is estimated, that is, three-dimensional reconstruction.

[0129] Bundle Adjustment: By optimizing the estimation of the camera movement and three-dimensional points, these estimations are made more consistent. This process is called Bundle Adjustment.

[0130] Step S260: Obtain new positioning images uploaded by the user device within a period of time.

[0131] Step S270: Calculate the second positioning result of the new positioning image in the to-be-verified local scene AR positioning map.

[0132] Specifically, feature extraction is performed on the new positioning image to obtain the extracted features;

[0133] The extracted features are matched with the features stored in the to-be-verified local scene AR positioning map, and the second positioning result is determined according to the matching result;

[0134] If the matching fails, the second positioning result is positioning failure;

[0135] If the matching succeeds, the second positioning result is positioning success.

[0136] Step S280: According to the second positioning result, update the confidence level of the AR positioning map of the local scene to be verified and the point cloud information of the AR positioning map of the local scene to be verified, and determine whether the confidence level of the AR positioning map of the local scene to be verified is higher than a second preset threshold.

[0137] If the second positioning result is positioning failure, lower the confidence level of the AR positioning map of the local scene to be verified;

[0138] If the second positioning result is positioning success, raise the confidence level of the AR positioning map of the local scene to be verified.

[0139] The point cloud information of the AR positioning map of the local scene to be verified is the feature information stored in the AR positioning map of the local scene to be verified. By continuously updating and supplementing the point cloud information of the AR positioning map of the local scene to be verified, the positioning accuracy can be improved.

[0140] In some embodiments, the second threshold is greater than the first threshold.

[0141] Step S290: If the confidence level of the AR positioning map of the local scene to be verified is higher than the second preset threshold, fuse the AR positioning map of the local scene to be verified and the original AR positioning map to obtain a new AR positioning map, and replace the original AR positioning map with the new AR positioning map.

[0142] Arrange the respective images in a column to obtain an image sequence;

[0143] Arrange the respective postures in a column to obtain a posture sequence.

[0144] For the images with successful positioning in the original AR positioning map, set the posture sequence of its image sequence in the AR positioning map of the local scene to be verified as wherein, the posture sequence of this image sequence obtained by the visual positioning algorithm in the original AR positioning map is Under ideal circumstances, there exists a similarity transformation matrix such that formula (2) holds;

[0145]

[0146] Among them, i represents the image sequence number of the images, P represents the coordinate system of the local scene AR positioning map to be verified, m represents the original AR positioning map coordinate system, and T represents the Euclidean transformation matrix, in the following form:

[0147]

[0148] Among them, R represents the three-dimensional rotation matrix, 0 T represents the transpose of the matrix, 0 represents the zero matrix, and t represents the displacement vector;

[0149] represents the similarity transformation matrix, in the following form:

[0150]

[0151] Among them, R represents the three-dimensional rotation matrix, t represents the displacement vector, and s represents the scaling factor, where the scaling factor is the scale parameter of the AR positioning map;

[0152] For example: when the directions of the two coordinate systems of the local scene AR positioning map to be verified and the original AR positioning map coordinate system are exactly the same, the three-dimensional rotation matrix between the two coordinate systems is R = [1, 0, 0; 0, 1, 0; 0, 0, 1].

[0153] Taking formula (3) as the optimization objective, find out the estimation result of, formula (3) is expressed as:

[0154]

[0155] Among them, r (residual) represents the residual between the pose measurement value and the estimated value, that is, the error of the result, n represents the number of images used in the optimization calculation, represents the deviation between the pose measurement value and the estimated value, and the point cloud information of the local scene AR positioning map to be verified passes through the inverse matrix mapping of can achieve coordinate alignment with the original AR positioning map, where when it is, there is

[0156] Delete the low-confidence information in the original AR positioning map.

[0157] By using the positioning results of the data collected in the local reconstruction and adjacent to the low-confidence area in the original AR positioning map, the coordinate alignment between the local scene AR positioning map to be verified and the original AR positioning map can be achieved.

[0158] After the above steps, the update of the AR positioning map is completed.

[0159] Based on the uploaded positioning image of the user's positioning in the scene, the embodiment of the present disclosure automatically determines whether the map scene has changed, and automatically updates the local map data of the changed scene by using the uploaded positioning image of the user. It is not necessary to arrange staff to re-collect the entire scene data for the overall reconstruction of the map, nor is it necessary to put forward additional operation requirements for the user. Therefore, compared with the existing methods, it has the advantages of high automation, low cost and user-friendliness.

[0160] Figure 4 The structural schematic diagram of a device for updating an AR positioning map provided by this embodiment.

[0161] As Figure 4 described, the device for updating the AR positioning map may include: an acquisition module 310, a first calculation module 320, a first processing module 330, a second calculation module 340, and a second processing module 350.

[0162] The acquisition module 310 is used to acquire the positioning image uploaded by the user terminal device;

[0163] The first calculation module 320 is used to calculate the first positioning result of the positioning image in the original AR positioning map;

[0164] The first processing module 330, if the first positioning result is a positioning failure, is used to calculate the positioning pose of the positioning failure image in the original AR positioning map according to the positioning success images in the adjacent areas of the positioning failure image, and determine the position of the block to be updated according to the positioning pose; according to the first positioning result, update the confidence of the block to be updated, and determine whether the confidence of the block to be updated is lower than a first preset threshold; if the confidence of the block to be updated is lower than the first preset threshold, it is determined that the block to be updated needs to be updated, and the block to be updated is reconstructed according to the positioning failure image and the positioning success images in the adjacent areas of the positioning failure image to obtain a to-be-verified local scene AR positioning map;

[0165] The acquisition module 310 is further used to acquire the new positioning image uploaded by the user terminal device;

[0166] The second calculation module 340 is used to calculate the second positioning result of the new positioning image in the to-be-verified local scene AR positioning map;

[0167] The second processing module 350 is configured to update the confidence of the AR positioning map of the to-be-verified local scene and the point cloud information of the AR positioning map of the to-be-verified local scene according to the second positioning result, and determine whether the confidence of the AR positioning map of the to-be-verified local scene is higher than a second preset threshold; if the confidence of the AR positioning map of the to-be-verified local scene is higher than the second preset threshold, the AR positioning map of the to-be-verified local scene and the original AR positioning map are fused to obtain a new AR positioning map, and the original AR positioning map is replaced with the new AR positioning map.

[0168] The device for updating the AR positioning map provided by the present disclosure can execute the above method embodiments. For the specific implementation principles and technical effects, reference can be made to the above method embodiments, which will not be elaborated herein.

[0169] The following uses a specific example to illustrate in detail the method for updating the AR positioning map provided by the present disclosure.

[0170] For example, a user wearing an AR device walks in a shopping mall. During the movement, the user device uploads the positioning image to the cloud server to calculate the positioning result. The cloud server automatically determines whether the scene has changed based on the positioning image uploaded by the user device. If the scene has changed, the AR positioning map is automatically updated according to the positioning image uploaded by the user device to maintain the accuracy of the AR positioning map. The specific implementation is as follows:

[0171] As Figure 5 shown, 1, 2, 3, 4, 5, 6, 7, 8, 9 are the images in the original AR positioning map.

[0172] Figure 5 In, A, B, C, D, E, F are the positioning images uploaded by the user device, and their pose sequences in the VIO coordinate system are Among them, the pose sequence of A in the VIO coordinate system is The pose sequence of B in the VIO coordinate system is The pose sequence of C in the VIO coordinate system is The pose sequence of D in the VIO coordinate system is The pose sequence of E in the VIO coordinate system is The pose sequence of F in the VIO coordinate system is The pose sequence is the poses arranged in a column. When the scene within the block (dashed box) changes, the four frames of images C, D, E, and F can no longer be successfully located in the original AR positioning map, while the two frames of images A and B outside the dashed box can still be successfully located, and the positioning result of the A frame image in the original AR positioning map is Then according to It can be deduced that the pose sequences of the four frames of images C, D, E, and F in the original AR positioning map are The cloud server can thus determine that the spatial positions of the C, D, E, and F images are within this block (dashed box) but the positioning is unsuccessful. Based on this, it can be determined that the scene of this block (dashed box) has changed, and the map of this block needs to be updated. This block is the block to be updated, and based on this, the confidence level of this block (dashed box) is marked as decreased.

[0173] Such as Figure 6 As shown, in order to update the map of this block (dashed box), within a period of time, the cloud server collects new positioning images from the user device: seven frames of images a, b, c, d, e, f, and g, and their pose sequences in the VIO coordinate system are And the creation of the AR positioning map P of the local scene to be verified is completed through the SFM method. In the AR positioning map of the local scene to be verified, the pose sequences corresponding to a, b, c, d, e, f, and g are And the three frames of images a, b, and c can be successfully positioned in the original AR positioning map, and the poses in the pose sequence in the original AR positioning map are a, b, and c are the images used for the optimization calculation, so the number n of the images used for the optimization calculation is 3.

[0174] When the AR positioning map of the local scene to be verified passes the test of the newly uploaded positioning data of the user device and the positioning success rate is greater than the threshold, the effectiveness of the AR positioning map of the local scene to be verified is recognized. Then the subsequent work of fusing the AR positioning map of the local scene to be verified with the original AR positioning map can be started. Taking as the optimization goal, it can be obtained that Through It can be calculated that

[0175] The images and point cloud information in the AR positioning map P of the local scene to be verified can be aligned with the original AR positioning map through the mapping of And the relevant map information of the invalid images 6, 7, 8, and 9 within the dashed box is deleted. A new AR positioning map is obtained, and the original AR positioning map is replaced with the new AR positioning map. As Figure 7 shown. The new AR positioning map contains twelve frames of images 1, 2, 3, 4, 5, a, b, c, d, e, f, and g. Thus, the automatic update of the AR positioning map of this local scene is completed.

[0176] The embodiment of the present application also provides a computer device. Specifically, please refer to Figure 8 , Figure 8 which is the basic structure block diagram of the computer device in this embodiment.

[0177] The computer device includes a memory 410 and a processor 420 that are communicatively connected to each other via a system bus. It should be noted that only the computer device with components 410 - 420 is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Among them, those skilled in the art of this technology can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0178] The computer device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer device can perform human-computer interaction with users through a keyboard, a mouse, a remote control, a touchpad or a voice control device and other means.

[0179] The memory 410 includes at least one type of readable storage medium, which includes non-volatile memory or volatile memory, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disc, etc. The RAM can include static RAM or dynamic RAM. In some embodiments, the memory 410 can be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. In other embodiments, the memory 410 can also be an external storage device of the computer device, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device. Of course, the memory 410 can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the memory 410 is generally used to store the operating system and various application software installed on the computer device, such as the program code of the above method. In addition, the memory 410 can also be used to temporarily store various types of data that have been output or will be output.

[0180] The processor 420 is generally used to execute the overall operations of the computer device. In this embodiment, the memory 410 is used to store program code or instructions, and the program code includes computer operation instructions. The processor 420 is used to execute the program code or instructions stored in the memory 410 or process data, such as running the program code of the above method.

[0181] In this text, the bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. This bus system can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0182] Another embodiment of the present application further provides a computer-readable medium, which can be a computer-readable signal medium or a computer-readable storage medium. The processor in the computer reads the computer-readable program code stored in the computer-readable medium, enabling the processor to perform the functional actions specified in each step or the combination of steps in the above method; and generating a device for performing the functional actions specified in each block or the combination of blocks in the block diagram.

[0183] The computer-readable medium includes but is not limited to electronic, magnetic, optical, electromagnetic, infrared memories or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. The memory is used to store program code or instructions, and the program code includes computer operation instructions. The processor is used to execute the program code or instructions of the above method stored in the memory.

[0184] For the definitions of the memory and the processor, reference can be made to the description of the foregoing computer device embodiments, which will not be elaborated herein.

[0185] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the devices or units can be in an electrical, mechanical or other form.

[0186] In each embodiment of the present application, each functional unit or module can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0187] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0188] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The "including" described in this application does not exclude the existence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the existence of a plurality of such elements. This application can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In the claims listing several units of a device, several of these units of the device can be embodied by the same item of hardware. The use of the first, second, and third, etc. does not indicate any order, and these words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

[0189] The above embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of this application.

Claims

1. A method for updating an AR positioning map, characterized in that: include: Obtaining the positioning image uploaded by the user terminal device; Calculating a first positioning result of the positioning image in the original AR positioning map; If the first positioning result is positioning failure, calculating the positioning posture of the positioning failure image in the original AR positioning map according to the positioning success image in the adjacent area to the positioning failure image, and determining the position of the block to be updated according to the positioning posture; According to the first positioning result, updating the confidence of the block to be updated, and determining whether the confidence of the block to be updated is lower than a first preset threshold; If the confidence of the block to be updated is lower than the first preset threshold, it is determined that the block to be updated needs to be updated, and the block to be updated is reconstructed according to the positioning failure image and the positioning success image of the area adjacent to the positioning failure image to obtain an AR positioning map of the local scene to be verified; Acquire a new positioning image uploaded by the user terminal device; Calculate a second positioning result of the new positioning image in the AR positioning map of the local scene to be verified; According to the second positioning result, updating the confidence of the AR positioning map of the local scene to be verified and the point cloud information of the AR positioning map of the local scene to be verified, and determining whether the confidence of the AR positioning map of the local scene to be verified is higher than a second preset threshold; If the confidence of the local scene AR positioning map to be verified is higher than the second preset threshold, the local scene AR positioning map to be verified and the original AR positioning map are fused to obtain a new AR positioning map, and the original AR positioning map is replaced by the new AR positioning map.

2. The method according to claim 1, characterized in that The step of calculating the first positioning result of the positioning image in the original AR positioning map specifically includes: Performing feature extraction on the positioning image to extract feature points in the image; Matching the feature point with the feature stored in the original AR positioning map, and determining a first positioning result according to the matching result; If the match fails, the first positioning result is positioning failure; If the match is successful, the first positioning result is positioning success.

3. The method according to claim 2, characterized in that The step of calculating the positioning posture of the positioning failure image in the original AR positioning map based on the positioning success image in the adjacent area to the positioning failure image, and determining the position of the block to be updated according to the positioning posture specifically includes: Obtain the VIO posture of the positioning failure image uploaded by the user terminal device, the VIO posture of the positioning success image in the area adjacent to the positioning failure image, and the positioning posture of the positioning success image in the area adjacent to the positioning failure image on the original AR positioning map; According to the VIO posture of the positioning failure image, the VIO posture of the positioning success image in the adjacent area to the positioning failure image, and the positioning posture of the positioning success image in the adjacent area to the positioning failure image in the original AR positioning map, the positioning posture of the positioning failure image in the original AR positioning map is calculated, and the calculation method is as follows: Set the positioning failure image I in the original AR positioning map b The posture in the VIO coordinate system of the user-side device is With image I b The adjacent image I is successfully located in the original AR positioning map a The posture in the VIO coordinate system of the user-side device is And image I a The posture in the original AR positioning map is Calculate image I b The posture in the original AR positioning map is Expressed as Among them, I represents the VIO coordinate system, m represents the original AR positioning map coordinate system, R represents the three-dimensional rotation matrix, R -1 represents the inverse matrix of the three-dimensional rotation matrix, t represents the displacement vector, 0 T represents the transpose of the matrix, and 0 represents the zero matrix; according to The position of the block to be updated is determined.

4. The method according to claim 2, characterized in that: The step of updating the confidence of the block to be updated according to the first positioning result specifically includes: If the first positioning result is positioning failure, lowering the confidence of the block to be updated, and marking the confidence of the block to be updated as low confidence; If the first positioning result is successful positioning, the confidence of the block to be updated is increased.

5. The method according to claim 1, characterized in that The step of calculating the second positioning result of the new positioning image in the AR positioning map of the local scene to be verified specifically includes: Performing feature extraction on the new positioning image to obtain extracted features; Matching the extracted features with the features stored in the AR positioning map of the local scene to be verified, and determining a second positioning result according to the matching result; If the match fails, the second positioning result is positioning failure; If the match is successful, the second positioning result is positioning success.

6. The method according to claim 5, characterized in that The step of updating the confidence of the AR positioning map of the local scene to be verified according to the second positioning result specifically includes: If the second positioning result is positioning failure, lowering the confidence of the AR positioning map of the local scene to be verified; If the second positioning result is successful positioning, the confidence of the AR positioning map of the local scene to be verified is increased.

7. The method according to claim 1, characterized in that The step of fusing the AR positioning map of the local scene to be verified with the original AR positioning map specifically includes: For the successfully positioned image in the original AR positioning map, the posture sequence of its image sequence in the AR positioning map of the local scene to be verified is set as Among them, the posture sequence of the image sequence in the original AR positioning map is There exists a similarity transformation matrix Make Established; Where i represents the image sequence number, P represents the AR positioning map coordinate system of the local scene to be verified, m represents the original AR positioning map coordinate system, T represents the Euclidean transformation matrix, Where R represents the three-dimensional rotation matrix, 0 T represents the transpose of the matrix, 0 represents the zero matrix, and t represents the displacement vector; represents the similarity transformation matrix, Among them, R represents the three-dimensional rotation matrix, t represents the displacement vector, and s represents the scaling factor; by As the optimization goal, find The estimated result is, where r (residual) represents the residual between the calculated value and the true value, n represents the number of images used in the optimization calculation, and the point cloud information of the local scene AR positioning map to be verified is Inverse Matrix The mapping can be aligned with the coordinates of the original AR positioning map, wherein when Sometimes, there are R -1 Represents the inverse matrix of a three-dimensional rotation matrix; The low-confidence information in the original AR positioning map is deleted.

8. A device for updating an AR positioning map, characterized in that: include: An acquisition module is used to acquire the positioning image uploaded by the user terminal device; A first calculation module, used to calculate a first positioning result of the positioning image in the original AR positioning map; A first processing module is configured to calculate, if the first positioning result is positioning failure, a positioning posture of the positioning failure image in the original AR positioning map according to a positioning success image in an area adjacent to the positioning failure image, and determine a position of a block to be updated according to the positioning posture; According to the first positioning result, updating the confidence of the block to be updated, and determining whether the confidence of the block to be updated is lower than a first preset threshold; If the confidence of the block to be updated is lower than the first preset threshold, it is determined that the block to be updated needs to be updated, and the block to be updated is reconstructed according to the positioning failure image and the positioning success image of the area adjacent to the positioning failure image to obtain an AR positioning map of the local scene to be verified; The acquisition module is further used to acquire a new positioning image uploaded by the user terminal device; A second calculation module, used to calculate a second positioning result of the new positioning image in the AR positioning map of the local scene to be verified; A second processing module is used to update the confidence of the AR positioning map of the local scene to be verified and the point cloud information of the AR positioning map of the local scene to be verified according to the second positioning result, and determine whether the confidence of the AR positioning map of the local scene to be verified is higher than a second preset threshold; If the confidence of the local scene AR positioning map to be verified is higher than the second preset threshold, the local scene AR positioning map to be verified and the original AR positioning map are fused to obtain a new AR positioning map, and the original AR positioning map is replaced by the new AR positioning map.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method for updating an AR positioning map as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the method for updating an AR positioning map as described in any one of claims 1 to 7 is implemented.

Images