Map generation method, positioning method, system and computer readable storage medium

Abstract

The present invention provides a map generation method, a positioning method, a system and a computer-readable storage medium. The map generation method includes: aligning a visual positioning map to a spatial geographic coordinate system and calculating the position information of each key frame in the spatial geographic coordinate system; using the position information and the main optical axis to calculate the azimuth information of each key frame relative to the spatial geographic coordinate system to obtain the orientation information of the key frame. The map positioning method includes: obtaining the second attribute information and image features of the image to be positioned; eliminating the key frames larger than the screening radius to obtain the first key frame after screening; based on the orientation information of the image to be positioned and the orientation information of the key frame, filtering out the second key frame from the visual positioning map; based on the image features, filtering out the third key frame from the key frame. The present invention can reduce the error rate caused by image retrieval during visual map positioning while using fewer map key frames for rapid positioning.

Description

Technical Field

[0001] The present invention relates to the field of computer computing, and in particular to a map generating method, a positioning method and a system thereof, and a computer-readable storage medium. Background Art

[0002] Map positioning system is the basis for realizing technologies such as robot navigation, augmented reality (AR), automatic cruise, and automatic driving. Positioning can be to find matching points in the constructed map and then confirm the current location. Map positioning system has been developed for many years, and its technical solution principle can be summarized as: visual map generation technology and visual map positioning technology.

[0003] The most common visual map generation technology is completed through the Structure-from-Motion technology, which mainly includes: taking pictures of the scene, extracting image features for matching, triangulating to obtain spatial 3D points, and finally optimizing using the Bundle Adjustment technology. The visual map generated using this technology consists of a 3D point cloud and its image features.

[0004] Visual map positioning technology is based on visual map positioning. The image currently captured by the device camera is matched with the three-dimensional point cloud of the visual map to obtain 2D-3D point pairs, and then the PNP problem is solved to obtain the current position of the device camera, thereby achieving visual positioning.

[0005] There are some obvious application problems in the existing map positioning system. Since visual positioning needs to be based on the generated visual map, the positioning efficiency of the visual positioning system is determined by the efficiency of image retrieval on the generated map. The existing visual positioning system needs to find the image that is most similar to the image to be retrieved from the database of the generated visual map through a certain method.

[0006] Common image retrieval algorithms generally include: using nearest neighbor search (K-Nearest Neighbors, KNN) and other methods for retrieval, based on the overall descriptor (global descriptor) of the image; subsequently using other algorithms to improve the recall rate of image retrieval, such as QE (Query Expansion), PQ (Product Quantization) and other methods. QE mainly improves the recall rate through two rounds of retrieval, while PQ aims to compress the size of image description words while improving their specificity. For the global descriptor search of images, there were traditional methods such as the VLAD (Vector of Locally Aggregated Descriptors) algorithm in the early days, and now there are also many descriptors and matching methods based on deep learning (NetVLAD, GEM).

[0007] However, image retrieval based only on the global descriptor of the image is simply using visual information for retrieval to achieve map positioning. The success rate and accuracy of positioning results often have bottlenecks. Using only the visual information of the image for positioning cannot exclude visually similar image features, which can easily cause matching errors and cannot avoid the error rate of positioning results. Summary of the invention

[0008] The technical problem solved by the technical solution of the present invention is how to reduce the error rate caused by image retrieval during visual map positioning.

[0009] In order to solve the above technical problems, the technical solution of the present invention provides a method for generating a visual map, comprising:

[0010] Collecting the required video sequence and first attribute information, where the first attribute information includes first GPS information;

[0011] Reconstructing a visual positioning map based on the video sequence, wherein the visual positioning map includes key frame image features and three-dimensional point cloud data corresponding to the key frame image features;

[0012] Aligning the visual positioning map to a spatial geographic coordinate system according to the first GPS information and calculating the position information of each key frame in the visual positioning map in the spatial geographic coordinate system;

[0013] Calculating the azimuth information of each key frame relative to the spatial geographic coordinate system using the position information of the key frame in the spatial geographic coordinate system and the main optical axis to obtain the orientation information of the key frame;

[0014] The visual positioning map is updated based on the key frame orientation, and the updated visual positioning map also includes: the key frame orientation information.

[0015] Optionally, reconstructing a visual positioning map based on the video sequence includes: reconstructing the visual positioning map by using a motion structure recovery algorithm or a real-time positioning and map construction algorithm.

[0016] Optionally, the key frame image features include: overall image features and local image features.

[0017] Optionally, the spatial geographic coordinate system includes: an X-axis pointing due east, a Y-axis pointing due north, and a Z-axis pointing away from the ground and upward; the calculation of the azimuth information of each key frame relative to the spatial geographic coordinate system using the position information of the key frame in the spatial geographic coordinate system and the main optical axis includes:

[0018] Select a point p(x0,y0,f) on the principal optical axis of each key frame, where (x0,y0) is the principal point of the camera and f is the focal length of the camera;

[0019] The key frame optical center o(0,0,0) and point p(x0,y0,f) in the camera coordinate system are converted into coordinates O(xo,yo,zo) and coordinates P(xp,yp,zp) in the corresponding spatial geographic coordinate system through coordinate system conversion;

[0020] Calculate the angle θ between the ray OP composed of the coordinates O(xo,yo,zo) and P(xp,yp,zp) and the X-axis, where:

[0021]

[0022] The azimuth information of ray OP is calculated according to the following conditions:

[0023] If xp>xo, the current azimuth is (π / 2-θ);

[0024] If xp<xo, the current azimuth is (3π / 2-θ);

[0025] If xp=xo and yp>yo, the current azimuth is 0;

[0026] If xp=xo and yp<yo, the current azimuth is π.

[0027] Optionally, the first attribute information further includes: first IMU information; and the visual map generation method further includes:

[0028] Based on the first GPS information and / or the first IMU information, the visual positioning map is calibrated and updated using bundle adjustment.

[0029] Optionally, the visual map generating method further includes:

[0030] The updated visual positioning map is gridded for display.

[0031] In order to solve the above technical problems, the technical solution of the present invention further provides a visual map positioning method, based on the above visual map generation method, comprising:

[0032] Acquire second attribute information and image features of the image to be located, where the second attribute information includes second GPS information and orientation information;

[0033] A screening radius is set, and key frames larger than the screening radius are removed to obtain a first key frame after screening, wherein the screening radius is the sum of a screening constant radius and a device location accuracy value set for the second GPS information;

[0034] Filtering a second key frame from the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frame;

[0035] Selecting a third key frame from the key frames based on the image features of the image to be located and the image features of the key frames;

[0036] Feature matching is performed based on the finally screened key frame image features and the features of the image to be positioned, and the three-dimensional point cloud data corresponding to the effectively matched image features in the key frame and the characteristic two-dimensional coordinate information of the image to be positioned are selected, and the device posture of the image to be positioned is obtained through the PNP algorithm.

[0037] Optionally, the selecting a second key frame from the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frame includes: selecting a second key frame from a first key frame of the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the first key frame;

[0038] The step of selecting a third key frame from key frames based on the image features of the image to be located and the image features of the key frames includes: selecting a third key frame from the second key frame based on the image features of the image to be located and the image features of the second key frame, and the key frame finally selected is the third key frame.

[0039] Optionally, the filtering out the third key frame from the key frame based on the image features of the image to be located and the image features of the key frame includes: filtering out the third key frame from the first key frame based on the image features of the image to be located and the image features of the first key frame;

[0040] The step of selecting the second key frame from the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frame includes: selecting the second key frame from the third key frame of the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the third key frame, and the key frame finally selected is the second key frame.

[0041] Optionally, removing key frames larger than the screening radius to obtain the first key frame after screening includes: removing key frames larger than the screening radius from the second key frames to obtain the first key frame after screening;

[0042] The step of selecting a third key frame from key frames based on the image features of the image to be located and the image features of the key frames includes: selecting a third key frame from the first key frame based on the image features of the image to be located and the image features of the first key frame, and the key frame finally selected is the third key frame.

[0043] Optionally, the filtering out the third key frame from the key frame based on the image features of the image to be located and the image features of the key frame includes: filtering out the third key frame from the second key frame based on the image features of the image to be located and the image features of the second key frame;

[0044] Eliminating key frames larger than the screening radius to obtain the first key frame after screening includes: eliminating key frames larger than the screening radius in the third key frame to obtain the first key frame after screening, and the key frame finally screened out is the first key frame.

[0045] Optionally, removing key frames larger than the screening radius to obtain the first key frame after screening includes: removing key frames larger than the screening radius from the third key frame to obtain the first key frame after screening;

[0046] The step of selecting a second key frame from the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frame includes: selecting a second key frame from the first key frame of the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the first key frame, and the key frame finally selected is the second key frame.

[0047] Optionally, the filtering out the second key frame from the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frame includes: filtering out the second key frame from the third key frame of the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the third key frame;

[0048] Eliminating key frames larger than the screening radius to obtain the first key frame after screening includes: eliminating key frames larger than the screening radius in the second key frame to obtain the first key frame after screening, and the key frame finally screened out is the first key frame.

[0049] Optionally, eliminating key frames larger than the filtering radius to obtain the first filtered key frame includes: taking the device location displayed by the second GPS information as the center, eliminating key frames on the visual positioning map whose radius is larger than the filtering radius.

[0050] Optionally, selecting a second key frame from the visual positioning map based on the orientation information of the image to be positioned and the key frame orientation information includes: setting an orientation threshold; comparing the key frame orientation information with the key frame orientation information, and eliminating key frames whose orientation difference exceeds the orientation threshold.

[0051] Optionally, the selecting a third key frame from the key frames based on the image features of the image to be located and the image features of the key frames includes:

[0052] Performing a KNN query on the key frame using key frame image features to obtain K key frames;

[0053] Clustering the K key frames using a K-means algorithm according to position information of the K key frames, wherein the K key frames are divided into N clusters;

[0054] The key frames whose number of three-dimensional point clouds in each cluster meets the predetermined number are taken as the third key frames, and other key frames are eliminated.

[0055] In order to solve the above technical problems, the technical solution of the present invention further provides a visual map generation system, including: a processor and a memory;

[0056] The memory stores a computer program, and the processor executes the steps of the visual map generation method as described above when executing the computer program.

[0057] In order to solve the above technical problems, the technical solution of the present invention further provides a visual map positioning system, including: a processor and a memory;

[0058] The memory stores a computer program, and the processor executes the steps of the visual map positioning method as described above when executing the computer program.

[0059] In order to solve the above technical problems, the technical solution of the present invention further provides a visual map generation system, including: a collection module, a map reconstruction module, a first calculation module and a first update module;

[0060] The acquisition module is adapted to acquire the required video sequence and first attribute information, wherein the first attribute information includes first GPS information;

[0061] The map reconstruction module is adapted to reconstruct a visual positioning map based on the video sequence, wherein the visual positioning map includes key frame image features and three-dimensional point cloud data corresponding to the key frame image features;

[0062] The first calculation module is adapted to align the visual positioning map to a spatial geographic coordinate system according to the first GPS information and calculate the position information of each key frame in the visual positioning map in the spatial geographic coordinate system;

[0063] The first calculation module is further adapted to calculate the azimuth information of each key frame relative to the spatial geographic coordinate system using the position information of the key frame in the spatial geographic coordinate system and the main optical axis to obtain the orientation information of the key frame;

[0064] The first updating module is adapted to update the visual positioning map based on the key frame orientation, and the updated visual positioning map further includes: the key frame orientation information.

[0065] Optionally, the map reconstruction module is suitable for reconstructing the visual positioning map by using a motion structure recovery algorithm or a real-time positioning and mapping algorithm.

[0066] Optionally, the spatial geographic coordinate system includes: an X-axis pointing due east, a Y-axis pointing due north, and a Z-axis pointing away from the ground and upward; the first calculation module includes: a selection unit, a conversion unit, a first calculation unit, and a second calculation unit;

[0067] The selection unit is adapted to select a point p(x0, y0, f) on the principal optical axis of each key frame, where (x0, y0) is the principal point of the camera and f is the focal length of the camera;

[0068] The conversion unit is adapted to convert the key frame optical center o(0,0,0) and point p(x0,y0,f) in the camera coordinate system into coordinates O(xo,yo,zo) and coordinates P(xp,yp,zp) in the corresponding spatial geographic coordinate system through coordinate system conversion;

[0069] The first calculation unit is adapted to calculate the angle θ of the ray OP composed of the coordinates O(xo, yo, zo) and P(xp, yp, zp) relative to the X-axis, wherein:

[0070]

[0071] The second calculation unit is adapted to calculate the azimuth information of the ray OP according to the following conditions:

[0072] If xp>xo, the current azimuth is (π / 2-θ);

[0073] If xp<xo, the current azimuth is (3π / 2-θ);

[0074] If xp=xo and yp>yo, the current azimuth is 0;

[0075] If xp=xo and yp<yo, the current azimuth is π.

[0076] Optionally, the first attribute information further includes: first IMU information; the visual map generation system further includes: a second updating module;

[0077] The second updating module is suitable for calibrating the visual positioning map and updating the visual positioning map using bundle adjustment based on the first GPS information and / or the first IMU information.

[0078] Optionally, the visual map generation system further includes: a gridding module;

[0079] The gridding module is suitable for gridding the updated visual positioning map for display.

[0080] In order to solve the above technical problems, the technical solution of the present invention further provides a visual map positioning system, based on the visual map generation system as described above, comprising: an acquisition module, a first screening module, a second screening module, a third screening module and a second calculation module;

[0081] The acquisition module is adapted to acquire second attribute information and image features of the image to be located, wherein the second attribute information includes second GPS information and orientation information;

[0082] The first screening module is adapted to set a screening radius and remove key frames larger than the screening radius to obtain a first key frame after screening, wherein the screening radius is the sum of a screening constant radius and a device location accuracy value set for the second GPS information;

[0083] The second screening module is adapted to screen out a second key frame from the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frame;

[0084] The third screening module is adapted to screen out a third key frame from the key frames based on the image features of the image to be located and the image features of the key frames;

[0085] The second calculation module is suitable for performing feature matching based on the finally screened key frame image features and the features of the image to be positioned, selecting the three-dimensional point cloud data corresponding to the effectively matched image features in the key frame and the feature two-dimensional coordinate information of the image to be positioned, and obtaining the device posture of the image to be positioned through the PNP algorithm.

[0086] In order to solve the above technical problems, the technical solution of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the visual map generation method as described above is implemented.

[0087] In order to solve the above technical problems, the technical solution of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the visual map positioning method as described above is implemented.

[0088] The beneficial effects of the technical solution of the present invention include at least:

[0089] The technical solution of the present invention can reconstruct a visual map using the attribute information that the device carries when collecting images, so as to obtain auxiliary image retrieval information of key frames in the map data, including: key frame orientation information, so that the image retrieval process in the visual image positioning process can use the employer image retrieval information to obtain a higher recall rate at a faster speed, select key frames with better positions and perspectives, reduce the error rate caused by image retrieval in the visual map positioning process, and improve the accuracy of the final visual map positioning result.

[0090] In an optional technical solution of the technical solution of the present invention, the attribute information that the device brings with it when collecting images is further utilized, such as obtaining image GPS information and / or IMU information, to calibrate the visual positioning map, so that the positioning data of the key frames of the visual positioning map are more accurate and effective.

[0091] When retrieving and locating the image to be located, the technical solution of the present invention uses the attribute information and image features of the image device to be located, and uses the GPS information and the topological relationship between the key frames of the located image to screen out the key frames within the screening radius, and then uses the device magnetometer to eliminate the key frames that are obviously oriented incorrectly with the image to be located, and further selects the key frames suitable for positioning according to the visual global descriptor from the filtered key frames, and selects the key frames that are finally filtered out after elimination for visual positioning. The technical solution of the present invention can screen out most of the key frames involved in the image retrieval process through the GPS information and magnetometer information in the image device to be located. The filtered key frames do not participate in the image retrieval of the visual global descriptor, which can greatly improve the image retrieval efficiency of visual map positioning, and also avoids the key frames that are obviously not together in spatial position but have consistent image features to produce erroneous results for map positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0093] Figure 1 A schematic diagram of a system architecture structure provided for the technical solution of the present invention;

[0094] Figure 2 A schematic diagram of a flow chart of a visual map generation method provided by the technical solution of the present invention;

[0095] Figure 3 A schematic diagram of the data storage structure of a visual positioning map file formed by the visual map generation method provided by the technical solution of the present invention;

[0096] Figure 4 A schematic diagram of another visual positioning map file data storage structure formed by the visual map generation method provided by the technical solution of the present invention;

[0097] Figure 5 A schematic flow chart of another visual map generation method provided by the technical solution of the present invention;

[0098] Figure 6 A schematic diagram of the flow chart of the first visual map positioning method provided by the technical solution of the present invention;

[0099] Figure 7 A schematic flow chart of a second visual map positioning method provided by the technical solution of the present invention;

[0100] Figure 8 A schematic diagram of a third visual map positioning method flow chart provided by the technical solution of the present invention;

[0101] Fig. 9 A schematic diagram of a fourth visual map positioning method flow chart provided by the technical solution of the present invention;

[0102] Fig.10 A schematic flow chart of the fifth visual map positioning method provided by the technical solution of the present invention. DETAILED DESCRIPTION

[0103] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0104] Unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" include plural referents. As used in this application, the terms "first" and "second" are used interchangeably to distinguish one or a type of component from another or another type, respectively, and are not intended to indicate the position or importance of an independent component.

[0105] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0106] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0107] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0108] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0109] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0110] The terminal devices 101, 102, and 103 may be various electronic devices with display screens and supporting web browsing, including but not limited to user devices, network devices, or devices formed by integrating user devices and network devices through a network. The user devices include but are not limited to any mobile electronic products that can interact with users through a touch panel, such as smart phones, tablet computers, etc. The mobile electronic products may use any operating system, such as an Android operating system, an IOS operating system, etc. Among them, the network device includes an electronic device that can automatically perform numerical calculations and information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc. The network device includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets, or a cloud composed of a plurality of servers; here, the cloud is composed of a large number of computers or network servers based on cloud computing, wherein cloud computing is a type of distributed computing, a virtual supercomputer composed of a group of loosely coupled computer sets.

[0111] The network 104 includes, but is not limited to, the Internet, a mobile communication network, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless ad hoc network (AdHoc network), etc. For example, the mobile communication network may be a 3G, 4G or 5G mobile communication system, for example, the mobile communication network may be a wideband code division multiple access (WCDMA) system, a frequency division multiple access (FDMA) system, an orthogonal frequency division multiple access (OFDMA) system, a single carrier FDMA (SC-FDMA) system, a general packet radio service (GPRS) system or a long term evolution (LTE) system, and other such communication systems. Of course, those skilled in the art should understand that the above-mentioned terminal devices are only examples, and other existing or future terminal devices that may appear, if applicable to the present application, should also be included in the scope of protection of the present application, and are hereby included by reference.

[0112] The server 105 may be a single server, a server cluster consisting of several servers, or a cloud computing service center, such as a cloud server. It may also be a server that provides various services, such as a backend server that provides support for pages displayed on the terminal devices 101, 102, and 103.

[0113] It should be noted that the visual map generation method and the visual map positioning method provided in the embodiments of the present application are generally executed by a server. Accordingly, the visual map generation system / device and the visual map positioning system / device are generally arranged in the server.

[0114] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0115] like Figure 2 FIG. 1 is a flow chart of a visual map generation method provided in an embodiment of the present application, and the visual map generation method can be executed by a cloud server. The cloud server can be Figure 1 The server 105. The visual map generation method comprises the following steps:

[0116] Step S10, collecting the required video sequence and first attribute information, wherein the first attribute information includes first GPS information.

[0117] When using a data acquisition device to collect the video sequence required for generating a visual map, the first attribute information related to the collected video sequence is obtained based on the GPS receiving chip, IMU (inertial sensor, composed of a gyroscope and an accelerometer), magnetometer, gyroscope, accelerometer sensor, etc. carried by the data acquisition device itself. When generating a positioning map, the corresponding sensor is selected for fusion according to the different acquisition devices. Generally, the acquisition device is equipped with GPS and IMU sensors, so the above sensor information will be used when generating the map. The attribute information can be: GPS information, IMU information, etc.

[0118] The video sequence required for generating the visual map collected by the data acquisition device can also be an image sequence. Depending on the data acquisition device and the acquisition method, the collected image data can be arbitrary. The required images can be collected in the required positioning area by a single lens, multiple lenses or panoramic camera. In this embodiment, while collecting the video sequence, the GPS information of the key frames of the video sequence is also obtained through the GPS receiving device of the data acquisition device.

[0119] Step S11, reconstructing a visual positioning map based on the video sequence, wherein the visual positioning map includes key frame image features and three-dimensional point cloud data corresponding to the key frame image features.

[0120] The map database includes a visual positioning map reconstructed based on the video sequence, and the reconstruction of the visual positioning map includes extracting image features of the key frames of the image, and obtaining three-dimensional point cloud data corresponding to the key frame image features based on the image features of the key frames. The method of reconstructing the visual positioning map can obtain the key frame image features of the visual positioning map and the corresponding three-dimensional point cloud data according to the method of reconstructing the visual positioning map in the prior art. The key frame image features of the visual positioning map obtained by the reconstructed visual positioning map and the corresponding three-dimensional point cloud data can be stored in a cloud server.

[0121] In this embodiment, the process of acquiring key frame image features when reconstructing the visual positioning map can refer to the following method:

[0122] A certain number of image features can be extracted from each key frame image, and the number of features can be 1000 to 2000. When the value is 3000 to 5000, better image features can be ensured. The number of image features extracted from each key frame image can be preset according to different needs, and different numbers of image features can be extracted from the local area of ​​the key frame as needed. The technical solution of the present invention does not impose any restrictions on this. The image features extracted from the local area of ​​the corresponding key frame image include at least one or more of the following image features: edges, corners, lines, curves, and regional features of other special attributes.

[0123] The algorithm for extracting image features from the local area of ​​the key frame image is widely used in computer vision. This image feature extraction algorithm can adopt the FAST corner detection algorithm (Features from Accelerated Segment Test, FAST), SIFT feature detection algorithm (Scale-Invariant Feature Transform, SIFT), Harris algorithm, ORB feature detection algorithm (Oriented FAST and Rotated BRIEF, ORB), etc., which can be used in the technical solution of the present invention. In a specific application example, the key frame image feature can be a special position point in the image, such as a corner point as described above. The FAST corner detection algorithm is to find a pixel point XP that differs from a predetermined number of pixels in the surrounding neighborhood by a threshold value or more. The pixel point XP is a corner point, and the corner point is the image feature to be extracted. The SIFT feature descriptor algorithm is used to detect and describe local features in an image, calculate the gradient of the feature point, generate a gradient histogram, and count the gradient and direction of the pixels in the selected neighborhood, so as to determine the direction of the feature point as an image feature. In another application example, the classic feature SIFT algorithm can be used to extract the above-mentioned image features, or a deep learning-based algorithm (such as LIFT, D2Net, R2D2, SuperPoint) can be used to extract the local features of the above-mentioned images.

[0124] In other embodiments, in addition to the above-mentioned extraction of image features for local areas of the image, one or more of the color features, texture features or shape features representing the overall image features of the image can also be extracted from the key frame image. In a specific application example, if the color features of the key frame image are to be extracted, a method of extracting the color histogram of the image, a method of extracting the main color histogram, a color moment method or a color set method can be used as a color feature extraction method, and the color histogram information, the main color histogram information, the color moment information or the color set information can be extracted as color features. If the texture features of the key frame image are to be extracted, a method such as the LBP method (Local binary patterns) or the grayscale co-occurrence matrix method can be used to extract the texture features of the image. If the shape features of the key frame image are to be extracted, an edge feature extraction method can be used, specifically implemented using the Canny operator edge detection or the Sobel operator edge detection algorithm.

[0125] For large-scale scene map generation, based on the above key frame image features, the Structure From Motion (SFM) algorithm or the Simultaneous Localization and Mapping (SLAM) algorithm can be used to restore the three-dimensional point cloud data corresponding to the above key frame image features to complete the visual positioning map reconstruction.

[0126] The structure from motion algorithm is a general term for the technology of recovering a three-dimensional sparse point cloud of a scene from a series of two-dimensional images. Its main steps include feature matching, triangulation of point cloud, BA optimization, etc. The technical solution of using the structure from motion algorithm to recover three-dimensional scene data point cloud from a series of two-dimensional images in the prior art can realize the process of obtaining corresponding three-dimensional point cloud data based on image features in this embodiment.

[0127] The real-time positioning and map construction algorithm is a process in which a computer starts from an unknown environment and unknown location on the map, and uses the two-dimensional image position and posture data of key frame map features (such as corners, pillars, etc.) during movement to incrementally construct a map based on its own position, thereby restoring the three-dimensional point cloud data corresponding to the key frame image features in the three-dimensional map.

[0128] The present invention does not discuss the use of algorithms to reconstruct the key frame image features visually. Figure 3 Limit the scenario.

[0129] Step S12: align the visual positioning map to the spatial geographic coordinate system according to the first GPS information and calculate the position information of each key frame in the visual positioning map in the spatial geographic coordinate system; use the position information of the key frame in the spatial geographic coordinate system and the main optical axis to calculate the azimuth information of each key frame relative to the spatial geographic coordinate system to obtain the orientation information of the key frame.

[0130] Different from the prior art, the reconstructed visual map of the technical solution of the present invention includes not only the above-mentioned key frame image features and the three-dimensional point cloud data corresponding to the key frame image features, but also key frame orientation information. Through the GPS information obtained from the above-mentioned attribute information, the visual map is aligned to the spatial geographic coordinate system (also called the world coordinate system) to obtain a calibrated visual map (Aligned Map), and the position information of each key frame after alignment with the spatial geographic coordinate system can be obtained. Based on the position information of these key frames, the azimuth of the main optical axis of each key frame in the visual map relative to the spatial geographic coordinate system can be calculated. In one example of the technical solution of the present invention, the key frame orientation information can be obtained through the following algorithm:

[0131] The spatial geographic coordinate system includes: an X-axis pointing due east, a Y-axis pointing due north, and a Z-axis pointing upward away from the ground; the azimuth information can be represented by the direction of the main optical axis of the key frame, and the azimuth information of each key frame in the visual positioning map relative to the spatial geographic coordinate system is calculated based on the position information of each key frame in the spatial geographic coordinate system and the main optical axis, including:

[0132] Select a point p(x0,y0,f) on the principal optical axis of each key frame, where (x0,y0) is the principal point of the camera and f is the focal length of the camera;

[0133] The key frame optical center o(0,0,0) and point p(x0,y0,f) in the camera coordinate system are converted to the coordinates O(xo,yo,zo) and P(xp,yp,zp) in the corresponding spatial geographic coordinate system through coordinate system conversion, where xo, yo,zo are the X-axis, Y-axis and Z-axis coordinate values ​​of the key frame optical center o(0,0,0) converted to the corresponding spatial geographic coordinate system, and xp, yp,zp are the X-axis, Y-axis and Z-axis coordinate values ​​of the point p(x0,y0,f) converted to the corresponding spatial geographic coordinate system. In this step, the key frame optical center o(0,0,0) and point p(x0,y0,f) in the camera coordinate system are converted to the coordinates O(xo,yo,zo) and P(xp,yp,zp) in the corresponding spatial geographic coordinate system through coordinate system conversion, so as to align the visual map to the spatial geographic coordinate system. The position information of the key frame in the spatial geographic coordinate system of the present invention may be the coordinates O(xo, yo, zo) converted from the key frame optical center o(0, 0, 0) in the spatial geographic coordinate system.

[0134] The angle θ between the ray OP composed of the coordinates O(xo,yo,zo) and P(xp,yp,zp) and the X-axis is calculated based on the following formula, that is:

[0135]

[0136] Since the required angle θ can only be between (-π / 2, π / 2), the size between xp and xo is also compared. The entire visual positioning map has been aligned to the North Pole, so the calculated azimuth of ray OP can be expressed as the direction of the key frame. The azimuth information of ray OP can be calculated according to the following conditions:

[0137] If xp>xo, the current azimuth is (π / 2-θ);

[0138] If xp<xo, the current azimuth is (3π / 2-θ);

[0139] If xp=xo and yp>yo, the current azimuth is 0;

[0140] If xp=xo and yp<yo, the current azimuth is π.

[0141] Step S13: updating the visual positioning map based on the key frame orientation, wherein the updated visual positioning map further includes: the key frame orientation information.

[0142] An updated visual positioning map can be obtained through the map data in the visual map database, including: key frame image features, three-dimensional point cloud data corresponding to the key frame image features, and orientation information of the key frame (that is, azimuth information).

[0143] After steps S10 to S13, a data storage format of the updated visual positioning map file can be as follows: Figure 3 shown.

[0144] J1, J2, J3…J m The key frames are P1, P2, P3...P m The image feature set, where j 11 ,j 12 …j 1N is the feature data in the image feature set J1 of key frame P1, j 21 ,j 22 …j 2N is the feature data in the image feature set J2 of key frame P2, j 31 ,j 32 …j 3N is the feature data in the image feature set J3 of key frame P3, and so on. m1 ,j m2 …j mN The key frame P m The image feature set J m Characteristic data in J1, J2, J3…J m and P1, P2, P3…P m The subscripts 1, 2, 3…m are the key frame numbers. Figure 3 Only key frames P1, P2, P3...P m The key frame image feature set J1, J2, J3…J m , J1, J2, J3…J m The key frame image features extracted from the overall key frame area are not included. Only the image features of the local area of ​​the image with the above order of magnitude N are included. 11 ,j 12 …j 1N , j 21 ,j 22 …j 2N , j 31 ,j 32 …j3N , …, j m1 ,j m2 …j mN The subscript of is composed of the numbers on the left and right, where the numbers on the left, 1, 2, 3, ... m, represent the key frame numbers; the numbers on the right represent the magnitude numbers of the local area image features, that is, 1 to N number of image features are extracted from the corresponding key frame image. N is the magnitude of the extracted image features.

[0145] The key frames P1, P2, P3, ... P can be restored by using the motion recovery structure algorithm or the real-time positioning and mapping algorithm provided in this embodiment. m The acquisition equipment parameters and the three-dimensional structure of the scene are used to obtain the three-dimensional point cloud data set S1, S2, S3…S corresponding to the image features. m . 3D point cloud data S1, S2, S3…S m They are image feature sets J1, J2, J3…J m A set of three-dimensional coordinate data placed at preset three-dimensional coordinates. More specifically, S1 is the feature point j in the image feature set J1. 11 ,j 12 …j 1N Corresponding to the three-dimensional coordinate data set, S2 is the feature point j in the image feature set J2 21 ,j 22 …j 2N Corresponding to the three-dimensional coordinate data set, S3 is the feature point j in the image feature set J3 31 ,j 32 …j 3N Corresponding to the three-dimensional coordinate data set, and so on, S m is the image feature set J m Feature point j m1 ,j m2 …j mN Corresponding to the three-dimensional coordinate data set.

[0146] Continue to refer Figure 3 , the map file also includes key frames P1, P2, P3...P m Direction information D1, D2, D3…D m and position information G1, G2, G3…G m From the data storage format of the updated visual positioning map file, the key frames P1, P2, P3…P m The image feature set J1, J2, J3…J m , 3D point cloud data set S1, S2, S3…S m , direction information D1, D2, D3…D m , location information G1, G2, G3…G mA corresponding relationship with data storage. This corresponding relationship enables the computer to obtain an image feature set corresponding to the image feature data and a three-dimensional coordinate data set corresponding to the image feature data after obtaining one of the image feature data (feature points), and obtain the three-dimensional coordinate data corresponding to the feature data based on the feature data, and also obtain the corresponding orientation information based on the image feature set or the three-dimensional coordinate data set.

[0147] For example, the computer retrieves the feature data j 11 , the feature data j can be retrieved from the three-dimensional coordinate data set S1 corresponding to the image feature set J1 11 The corresponding three-dimensional coordinate data is obtained, and the corresponding orientation information D1 and position information G1 are obtained.

[0148] For example, when the computer retrieves the orientation information D2, it can obtain the corresponding image feature set J2, the corresponding three-dimensional coordinate data set S2 and the position information G2.

[0149] In other embodiments, after steps S10 to S13, another data storage form of the updated visual positioning map file can be as follows: Figure 4 As shown. The key frames P1, P2, P3…P m The image feature set J1', J2', J3'...J m In addition to including Figure 3 The image feature set shown also includes key frame image features j extracted from the overall area of ​​the key frame 10 ,j 20 …j m0 .j 10 ,j 20 …j m0 The subscript is also composed of numbers on the left and right sides, where the numbers 1, 2, 3, ... m on the left represent the key frame numbers; the number 0 on the right represents the number of the overall regional image features with key frame numbers 1, 2, 3, ... m.

[0150] exist Figure 4 In the map file, key frames P1, P2, P3…P m The image feature set J1', J2', J3'...J m ', 3D point cloud data set S1, S2, S3...S m , direction information D1, D2, D3…D m , location information G1, G2, G3…G mWith a corresponding relationship of data storage. This corresponding relationship enables the computer to obtain one of the image feature data (which can be the overall image feature or the local image feature), and then obtain the image feature set corresponding to the image feature data, obtain the three-dimensional coordinate data set corresponding to the image feature set, and also obtain the corresponding orientation information according to the image feature set or the three-dimensional coordinate data set.

[0151] For example, the computer retrieves the feature data j 11 , we can obtain the image feature set J1, obtain the three-dimensional coordinate data set S1 corresponding to the image feature set J1, and retrieve the feature data j from the three-dimensional coordinate data set S1 corresponding to the image feature set J1 11 The corresponding three-dimensional coordinate data is obtained, and the corresponding orientation information D1 and position information G1 are obtained.

[0152] Another example is that the computer retrieves the characteristic data j 20 , the image feature set J2' can be obtained, the three-dimensional coordinate data set S2 corresponding to the image feature set J2' can be obtained, and the corresponding orientation information D2 and position information G2 can be obtained.

[0153] In a variation of the above visual map generation method, in order to improve the map accuracy, the first attribute information obtained by the device is also used to calibrate the generated visual map using the bundle adjustment method. Figure 5 The visual map generation method shown in FIG. Figure 3 In addition to steps S10 to S13, the method further includes:

[0154] Step S14, calibrating and updating the visual map using bundle adjustment.

[0155] The visual map generated by the bundle adjustment calibration can minimize the back-projection error. As Dr. Shan Jie pointed out in "A Brief History and Overview of Bundle Adjustment" (Journal of Wuhan University, Information Science Edition, 2018, 43(12): 1797-1810.), bundle adjustment is a theory and method for positioning using images that is currently common in the fields of photogrammetry, computer vision and robotics. The generated map can be optimized by establishing and solving the mathematical model of bundle adjustment. The mathematical model of bundle adjustment refers to the relationship between observation data and parameters in bundle adjustment. It first includes the imaging model, that is, the mathematical relationship between the image, the object and the camera, which usually refers to the fact that the three are on the same straight line, that is, the collinear equation. There have been several different ways to express this basic relationship over the past few decades. In addition, in the development of bundle adjustment, new observation data or conditions (constraints) have been continuously introduced into the adjustment calculation. The relationship between these observation data and unknown parameters also constitutes an extension of the traditional bundle model. On this basis, the GPS information or IMU information in the first data information is used as a constraint condition (ie, observation data) to optimize the generated visual map so that the entire optimization process proceeds in the right direction.

[0156] According to Figure 2 or Figure 5 The visual map generated by the visual map generation method shown is a sparse map. For the convenience of display, the visual map can be gridded on the basis of the above visual map generation for the convenience of display. The grid map can be generated by using multi-view stereo (MVS) and Poisson surface reconstruction methods. There are many methods for generating grid maps in the prior art, and the technical solution of the present invention does not impose any restrictions on the way of gridding the map.

[0157] Based on the embodiment of the above-mentioned visual map generation method provided by the technical solution of the present invention, based on the visual map generated by any embodiment method, such as Figure 6 As shown, the technical solution of the present invention also provides a visual map positioning method, including the following process steps:

[0158] Step S20, obtaining second attribute information and image features of the image to be located, wherein the second attribute information includes second GPS information and orientation information.

[0159] The method of extracting the second attribute information of the image to be positioned is similar to step S10, that is, using the GPS receiving chip, IMU (inertial sensor, composed of a gyroscope and an accelerometer), magnetometer, gyroscope, accelerometer sensor, etc. carried by the image acquisition device itself to obtain attribute information related to the image to be positioned, such as: GPS information, IMU information, orientation information, etc.

[0160] In step S20, the GPS information and the orientation information in the second attribute information may be used for positioning calculation of the image to be positioned in this embodiment.

[0161] Furthermore, the method for obtaining the image features of the image to be located can refer to the method for obtaining the key frame image features of the visual positioning map in step S11. Similarly, by obtaining a predetermined order of magnitude of image features from the image to be located, especially extracting different numbers of image features from the local area of ​​the image to be located, the image features of the image to be located can be extracted. The image features extracted from the local area of ​​the image to be located include at least one or more of the following image features: edges, corners, lines, curves, and regional features of other special attributes.

[0162] In other embodiments, in addition to extracting image features of a local area of ​​the image to be positioned, the overall image features of the image to be positioned may also be extracted, including one or more of color features, texture features or shape features in the overall image features.

[0163] The local area image features of the image to be positioned are extracted, or the overall image features of the image to be positioned are extracted. The specific algorithms for extracting image features can refer to the above-mentioned algorithms for extracting key frame image features, which will not be repeated here.

[0164] In the prior art, the positioning method of the visual map is based on the results of obtaining the image features of the image to be positioned. After the image features of the image to be positioned are matched with the image features of the key frame of the visual map, the first matching image features of the image to be positioned and the second matching image features of the key frame of the visual map that match each other are obtained. When the visual map is generated, the image features of the key frame of the visual map and the corresponding three-dimensional point cloud data have been obtained. Therefore, when matching and positioning the image to be positioned, the device posture of the image to be positioned is obtained through the PNP algorithm based on the three-dimensional point cloud data corresponding to the first matching image features and the second matching image features.

[0165] Different from the prior art, this embodiment screens the key frame images before matching the image features of the image to be located, thereby improving the accuracy of key frame image matching and retrieval efficiency.

[0166] Continue to refer Figure 6 In this embodiment, the visual map positioning method further includes:

[0167] Step S21, setting a screening radius, and removing key frames larger than the screening radius to obtain a first key frame after screening.

[0168] According to step S21, based on the first GPS information of the visual map, based on the second GPS information, the visual map key frames are filtered for the key frames of GPS information within the set filtering radius. If the first GPS information of the visual map key frame shows that it is within the filtering radius with the second GPS information as the center of the range, then these visual map key frames are filtered as first key frames.

[0169] The setting method of the screening radius can be predetermined according to the actual system needs. In a variation of this embodiment, the screening radius can be the sum of the screening constant radius and the device location accuracy value set for the second GPS information. Specifically: the accuracy range of the GPS at the current moment can be determined according to the system interface of the image acquisition device to be located. A , D A That is, the device accuracy value corresponding to the second GPS location; R0 is the value selected by the user; the screening radius R can be the accuracy range D A The sum of the custom value R0.

[0170] Step S22, selecting a second key frame from the first key frame of the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the first key frame;

[0171] Based on the first key frame screened out in step S21, the key frames whose GPS position information is inconsistent with the image to be located have been screened out, and the accuracy and efficiency of key frame screening and retrieval have been improved. Further, the orientation information of the image to be located obtained by the device magnetometer can be used to eliminate the key frames that are inconsistent with the orientation of the image to be located. In the comparison of the orientation information, the orientation information can be set, and the azimuth information of the main optical axis of the image formed by the device relative to the spatial geographic coordinate system can be used as the image orientation information, and compared with the orientation information of the first key frame. Compare the azimuth representing the orientation information. If the orientation information in the first key frame is compared with the orientation information of the image to be located, the difference exceeds the custom threshold, then these first key frames are eliminated, and only the first key frames whose differences do not exceed the custom threshold range are retained. This further reduces the candidate key frame set, and finally, through the comparison of the orientation information, the second key frame whose difference meets the threshold range is screened. That is to say, in the specific comparison process, the angle screening threshold can be set, and the key frames whose orientation is not within the effective viewing angle range of the orientation information of the image to be located can be eliminated to obtain the second key frame.

[0172] Step S23, filtering out a third key frame from the second key frame based on the image features of the image to be located and the image features of the second key frame, and the key frame finally filtered out is the third key frame.

[0173] Step S23 can adopt the image retrieval method of the prior art, that is, use the image features of the local area of ​​the image to be located, and search the local area image features of the second key frame within the second key frame of the visual map to obtain the key frame finally screened out. The key frame finally screened out. The retrieval method of the local area image features can be through a simple local feature matching algorithm, that is, directly calculating the Euclidean distance between the image features in the image to be located and the image features of the second key frame, and taking the key frame feature image that is closest to the image features in the image to be located as the final screened out image features.

[0174] In a variation of this embodiment, step S23 may also be implemented by the following means:

[0175] The KNN algorithm (K-Nearest Neighbor algorithm) is used to query the overall image features of the image to be located (that is, the global description sub-image features), and k key frames suitable for positioning are further selected from all the above-mentioned candidate key frames (the second key frame in this embodiment);

[0176] According to the location information of the k key frames queried, n clusters are constructed using the K-means algorithm, and the distances from the k key frames to the n clusters are calculated based on the location information, and the k key frames are assigned to the cluster with the closest distance;

[0177] In each cluster, the key frame with the most 3D points is selected as the candidate frame, and other key frames are eliminated;

[0178] Finally, the remaining key frames in the n clusters are selected as the final filtered key frames, and the n key frames in the n clusters are sorted according to the directory order (Index) returned by the KNN algorithm and used for positioning.

[0179] In this step, since the candidate key frame, i.e., the second key frame, has been screened by GPS and magnetometer orientation in steps S21 and S22, further using the KNN algorithm and the K-means algorithm to screen the key frames through image features can well select key frames of different perspectives at multiple locations, thereby improving the positioning success rate and accuracy.

[0180] Step S23 uses the image features of the visual map, that is, the key frame most similar to the image to be located is retrieved through the KNN algorithm and the K-means algorithm. The k key frames obtained by KNN are clustered using the K-means algorithm, and the k key frames are divided into n categories by clustering. When the KNN algorithm is used to query K key frames, the relative order between the k key frames, that is, the directory order returned by the KNN algorithm, will be returned to return the sorted key frames.

[0181] Step S24, perform feature matching based on the finally screened key frame image features and the features of the image to be positioned, select the three-dimensional point cloud data corresponding to the effectively matched image features in the key frame and the feature two-dimensional coordinate information of the image to be positioned, and obtain the device posture of the image to be positioned through the PNP algorithm.

[0182] The PNP (Perspective-n-Point) algorithm is a problem of estimating the pose of a calibrated camera given n 3D points on a world reference system and their corresponding 2D projections in an image. The PNP algorithm can be used to obtain the pose of the image device to be positioned based on the image features of the local area of ​​the image to be positioned (corresponding to the 2D projection) and the three-dimensional point cloud data (relative to the 3D points) corresponding to the key frame image features finally screened out. Since the PNP algorithm is widely used in existing technical solutions such as camera pose tracking, object pose tracking, AR / VR, robot operation, pose initial value solution in SLAM, etc., commonly used solutions include DLT algorithm, P3P algorithm, EPnP algorithm and UPnP algorithm, etc., which are very basic algorithm solutions in all three-dimensional vision fields, and the technical solution of the present invention will not be repeated.

[0183] Corresponds to Figure 3 The data of the visual positioning map file shown in the figure, the visual positioning map file includes: key frames P1, P2, P3...P m and its image feature set J1, J2, J3…J m , the three-dimensional point cloud data set S1, S2, S3…S corresponding to the image features m , key frames P1, P2, P3…P m Direction information D1, D2, D3…D m And position information G1, G2, G3...G m . The image feature sets J1, J2, J3…J m From key frames P1, P2, P3...P m A set of key frame image features of order N is extracted from the local area of ​​the image.

[0184] When executing step S20, the device acquires GPS information, IMU information, and orientation information of the image to be located based on the device's auxiliary device (such as GPS chip, IMU, magnetometer, etc.) when shooting the image to be located, which can be used for auxiliary positioning. After the image features are extracted from the image to be located, the GPS information, IMU information, orientation information, and image features of the image to be located are also formed to wait for positioning data.

[0185] When executing step S21, the GPS information of the image to be located is used to obtain the Figure 3 In the data of the visual positioning map file shown, position information G1, G2, G3...G m The key frames of the visual map larger than the screening radius are removed by screening, and the screened key frames are the first key frames. Assuming that after screening, the magnitude of the first key frame is f, and the first key frame sequence is x1, x2, x3, ... xf, the first key frame data includes: key frame P x1 , P x2 , P x3 …P xf and its image feature set J x1 , J x2 , J x3 …J xf , the three-dimensional point cloud data set S corresponding to the image features x1 , S x2 , S x3 …S xf , orientation information D x1 , D x2 , D x3 …D xf The order of magnitude of f is further smaller than m.

[0186] When executing step S22, the orientation information of the image to be positioned is used to obtain the orientation information D in the first key frame data. x1 , D x2 , D x3 …D xf The key frames whose orientation information is inconsistent with the orientation information of the image to be positioned are removed, and the selected key frames are the second key frames. Assume that after screening, the magnitude of the second key frame is h, and the second key frame sequence number is y1, y2, y3, ... yh. The second key frame data includes: key frame P y1 , P y2 , P y3 …P yh and its image feature set J y1 , J y2 , J y3 …J yh , the three-dimensional point cloud data set S corresponding to the image features y1 , Sy2 , S y3 …S yh The order of magnitude of h is further smaller than f.

[0187] When executing step S23, the image feature set J can be used to retrieve the image features of the image to be located. y1 , J y2 , J y3 …J yh The range is searched to obtain the key frame image features closest to the image features to be located. Assuming that the magnitude of the third key frame finally screened out is g, and the third key frame sequence is z1, z2, z3, ... zg, the third key frame data includes: image feature set J z1 , J z2 , J z3 …J zg The corresponding three-dimensional point cloud data set S z1 , S z2 , S z3 …S zg .

[0188] When executing step S23, according to the three-dimensional point cloud data set S in the third key frame data z1 , S z2 , S z3 …S zg As well as the image features of the local area of ​​the image to be positioned (corresponding to the 2D projection), the position and posture of the image device to be positioned are obtained.

[0189] Corresponds to Figure 4 The data of the visual positioning map file shown in the figure, the visual positioning map file includes: key frames P1, P2, P3...P m And image feature sets J1', J2', J3'...J m ', 3D point cloud data set S1, S2, S3...S m , direction information D1, D2, D3…D m And position information G1, G2, G3...G m . The image feature set J1', J2', J3'...J m 'From key frames P1, P2, P3...P m A set of key frame image features of order N are extracted from the local area of ​​the image and the key frame image features are extracted from the overall area of ​​the key frame.

[0190] When executing step S20, the image to be positioned has GPS information, IMU information, orientation information, overall image features, and local area image features waiting for positioning data. When executing step S21, the GPS information of the image to be positioned is used. Figure 4 In the data of the visual positioning map file shown, position information G1, G2, G3...G m The filtered key frame is the first key frame. Assuming that after filtering, the magnitude of the first key frame is still f, and the first key frame sequence number is still x1, x2, x3, ... xf, the first key frame data includes: key frame P x1 , P x2 , P x3 …P xf and its image feature set J' x1 , J' x2 , J' x3 …J' xf , the three-dimensional point cloud data set S corresponding to the image features x1 , S x2 , S x3 …S xf , orientation information D x1 , D x2 , D x3 …D xf The order of magnitude of f is further smaller than m.

[0191] Continue to correspond to Figure 4 The key frame selected by executing step S22 is the second key frame. Assuming that after the selection, the magnitude of the second key frame is still h, and the second key frame sequence number is still y1, y2, y3, ... yh, the second key frame data includes: key frame P y1 , P y2 , P y3 …P yh and its image feature set J' y1 , J' y2 , J' y3 …J' yh , the three-dimensional point cloud data set S corresponding to the image features y1 , S y2 , S y3 …S yh The order of magnitude of h is further smaller than f.

[0192] When executing step S22, the KNN algorithm and the K-means algorithm can be used to identify the overall image features of the image to be located and the image of the local area in the image feature set J'. y1 , J' y2 , J' y3 …J' yh Range, first use the KNN algorithm to find the overall area image features j of the key frame y10 ,j y20 …j yh0Extract k key frames suitable for positioning. At this time, the order of magnitude is k, and the sequence numbers are z1, z2, z3, ... zk. The filtered key frame data includes: key frame P z1 , P z2 , P z3 …P zk and its image feature set J' z1 , J' z2 , J' z3 …J' zk , 3D point cloud data set S z1 , S z2 , S z3 …S zk , location information G z1 , G z2 , G z3 …G zk The order of magnitude of k is further smaller than h.

[0193] Using k key frames P z1 , P z2 , P z3 …P zk Location information G z1 , G z2 , G z3 …G zk , construct n clusters, and calculate the distances from k key frames to n clusters, and use the key frame with the most 3D points in the n clusters as the candidate frame for each distance, so as to obtain n orders of magnitude of key frame data, the key frame sequence number is r1, r2, r3, ... rn, the key frame data includes: key frame P r1 , P r2 , P r3 …P rn and its image feature set J' r1 , J' r2 , J' r3 …J' rn , 3D point cloud data set S r1 , S r2 , S r3 …S rn . The order of magnitude is that n is further smaller than k.

[0194] Based on the corresponding Figure 4 The key frame data finally screened out, when executing step S23, the three-dimensional point cloud data set S of the key frame data sorted in the directory according to the KNN algorithm in the third key frame data is r1 , S r2 , S r3 …S rn As well as the image features of the local area of ​​the image to be positioned (corresponding to the 2D projection), the position and posture of the image device to be positioned can be obtained.

[0195] In the above embodiment, step S21 and step S22 have been filtered by GPS and magnetometer orientation, and step S23 is further filtered based on the second key frame filtered by step S21 and step S22. In other embodiments, if step S21 and step S22 are not executed, that is, GPS and magnetometer orientation filtering are not used, after step S20 is executed, step S23 is executed to filter the key frame data of the visual map directly based on the image features of the image to be located, which can also achieve the effect of improving system performance and completing positioning. Figure 7 In the variation shown, the visual map positioning method includes the following steps:

[0196] Step S30, obtaining image features of the image to be located.

[0197] In this embodiment, the acquisition of image features may include extracting overall image features of the image to be positioned and extracting image features of a local area, or may only extract image features of a local area of ​​the image to be positioned.

[0198] Step S31, filtering out key frames from the key frames of the visual map based on the image features of the image to be located and the image features of the key frames.

[0199] The screening method of step S31 can refer to step S23, except that the screening range of image feature retrieval or KNN algorithm and K-means algorithm is changed from the second key frame range of step S23 to all key frames of the visual map, which will not be repeated here.

[0200] Step S32, perform feature matching based on the finally screened key frame image features and the features of the image to be positioned, select the three-dimensional point cloud data corresponding to the effectively matched image features in the key frame and the feature two-dimensional coordinate information of the image to be positioned, and obtain the device posture of the image to be positioned through the PNP algorithm.

[0201] Step S32 is to use the PNP algorithm to obtain the position and posture of the image device to be positioned based on the image features of the local area of ​​the image to be positioned (corresponding to the 2D projection) and the three-dimensional point cloud data (relative to the 3D points) corresponding to the key frame image features finally screened out.

[0202] The screening order of step S21, step S22 and step S23 of the technical solution of the present invention can be changed arbitrarily, and the order of the steps in the positioning method flow can be arbitrarily exchanged. This arbitrary exchange of the execution order can replace the technical solution composed of steps S20 to S24. The difference between the alternative solution and the technical solution composed of steps S20 to S24 is that in different GPS accuracies and scenarios, inconsistent search results may occur, but the overall optimization core is consistent and indistinguishable.

[0203] Based on the embodiment of the above-mentioned visual map generation method provided by the technical solution of the present invention, based on the visual map generated by any embodiment method, such as Figure 8 As shown, the technical solution of the present invention also provides a visual map positioning method, which changes the execution and screening order of step S21 and step S22, and specifically includes the following process steps:

[0204] Step S40, obtaining second attribute information and image features of the image to be located, wherein the second attribute information includes second GPS information and orientation information.

[0205] The execution method of step S40 is the same as that of step S20, which will not be repeated here.

[0206] Step S41, selecting a first key frame from the key frames of the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frames of the visual map.

[0207] The difference between step S41 and step S21 is that it uses the orientation information of the image to be located obtained by the device magnetometer to remove the key frames of the original visual map that are inconsistent with the orientation of the image to be located, and obtain the first key frame data that has been orientation-screened and has the same orientation as the image to be located. The comparison method of the orientation information can refer to step S22, which will not be repeated here.

[0208] Step S42, setting a screening radius, and removing first key frames larger than the screening radius to obtain screened second key frames.

[0209] According to step S41, based on the GPS information of the key frame of the visual map, based on the second GPS information of the map to be located, the key frame of GPS information of the first key frame after the orientation information filtering in step S41 is filtered within the set filtering radius, and the second key frame data of the GPS position within the set filtering radius of the map to be located is obtained. The key frame filtering method can refer to step S21, which will not be repeated here.

[0210] Step S43, filtering out a third key frame from the second key frame based on the image features of the image to be located and the image features of the second key frame, and the key frame finally filtered out is the third key frame.

[0211] The image retrieval method of the prior art can be used to retrieve the image features of the second key frame range of the visual map to obtain the final selected key frame. It is also possible to first query the global description sub-image features of the second key frame of the visual image through the KNN algorithm to select k key frames suitable for positioning; then use the K-means algorithm to build n clusters, assign these k key frames to these n clusters, select the key frame with the most three-dimensional points in each cluster as the candidate frame, and eliminate other key frames; finally, n key frames in the directory order returned by the KNN algorithm are obtained and used for positioning. The third key frame selected is the n key frames returned according to the directory order of the KNN algorithm. The execution method of step S43 is the same as step S23, which will not be repeated here.

[0212] Step S44, perform feature matching based on the finally screened key frame image features and the features of the image to be positioned, select the three-dimensional point cloud data corresponding to the effectively matched image features in the key frame and the feature two-dimensional coordinate information of the image to be positioned, and obtain the device posture of the image to be positioned through the PNP algorithm.

[0213] The execution algorithm of step S44 is consistent with that of step S24. Both are based on the image features of the local area of ​​the image to be located (corresponding to the 2D projection) and the three-dimensional point cloud data (relative to the 3D points) corresponding to the key frame image features finally screened out to obtain the posture of the image device to be located. It will not be repeated here.

[0214] Corresponds to Figure 3 The data in the visual positioning map file shown is different from Figure 6 The visual map positioning method shown:

[0215] When executing step S41, firstly, the orientation information of the image to be positioned is used, and in the original key frame data of the visual map, the orientation information D1, D2, D3 ... D m The key frames whose orientation information is inconsistent with the orientation information of the image to be positioned are removed, and the selected key frames are the first key frames. Assume that after screening, the magnitude of the first key frame is h, and the sequence number of the first key frame is x1, x2, x3, ... xh. The first key frame data includes: key frame P x1 , P x2 , P x3 …P xh and its image feature set J x1 , J x2 , J x3 …J xh , the three-dimensional point cloud data set S corresponding to the image features x1 , S x2 , S x3 …S xh , location information G x1 , Gx2 , G x3 …G xh The order of magnitude of h is further smaller than m.

[0216] When executing step S42, the location information G is obtained by using the GPS information of the image to be located. x1 , G x2 , G x3 …G xh Screening is performed to remove the key frames of the visual map that are larger than the screening radius, and the screened key frames are the second key frames. Assuming that after screening, the magnitude of the second key frame is f, and the second key frame sequence number is y1, y2, y3, ... yf, the second key frame data includes: key frame P y1 , P y2 , P y3 …P yf and its image feature set J y1 , J y2 , J y3 …J yf , the three-dimensional point cloud data set S corresponding to the image features y1 , S y2 , S y3 …S yf The order of magnitude of f is further smaller than h.

[0217] When executing step S43, the image feature set J can be used to retrieve the image features of the image to be located. y1 , J y2 , J y3 …J yf The range is searched to obtain the key frame image features closest to the image features to be located. Assuming that the magnitude of the third key frame finally screened out is g, and the third key frame sequence is z1, z2, z3, ... zg, the third key frame data includes: image feature set J z1 , J z2 , J z3 …J zg The corresponding three-dimensional point cloud data set S z1 , S z2 , S z3 …S zg According to the three-dimensional point cloud data set S in the third key frame data z1 , S z2 , S z3 …S zg As well as the image features of the local area of ​​the image to be positioned (corresponding to the 2D projection), the position and posture of the image device to be positioned are obtained.

[0218] Corresponds to Figure 4The data of the visual positioning map file shown in the figure, because the visual positioning map file includes: image feature sets J1', J2', J3'...J m '. When executing step S41, by the direction information D1, D2, D3...D m Suppose after screening, the first key frame has an order of magnitude of h, and the first key frame sequence is x1, x2, x3, ... xh. The first key frame data includes: key frame P x1 , P x2 , P x3 …P xh and its image feature set J' x1 , J' x2 , J' x3 …J' xh , the three-dimensional point cloud data set S corresponding to the image features x1 , S x2 , S x3 …S xh , location information G x1 , G x2 , G x3 …G xh The order of magnitude is h is further smaller than m. When executing step S42, the position information G is obtained by using the GPS information of the image to be located. x1 , G x2 , G x3 …G xh Screening is performed to obtain the second key frame. Assuming that after screening, the second key frame magnitude is f, the second key frame sequence number is y1, y2, y3, ... yf, and the second key frame data includes: key frame P y1 , P y2 , P y3 …P yf and its image feature set J' y1 , J' y2 , J' y3 …J' yf , the three-dimensional point cloud data set S corresponding to the image features y1 , S y2 , S y3 …S yf The order of magnitude of f is further smaller than h.

[0219] When executing step S43, the KNN algorithm and the K-means algorithm can be used to obtain the image feature set J' based on the overall image features of the image to be located and the image of the local area. y1 , J' y2 , J' y3 …J' yf Range, first use the KNN algorithm to find the overall area image features j of the key framey10 ,j y20 …j yf0 Extract k key frames suitable for positioning. At this time, the order of magnitude is k, and the sequence numbers are z1, z2, z3, ... zk. The filtered key frame data includes: key frame P z1 , P z2 , P z3 …P zk and its image feature set J' z1 , J' z2 , J' z3 …J' zk , 3D point cloud data set S z1 , S z2 , S z3 …S zk , location information G z1 , G z2 , G z3 …G zk . The order of magnitude is further smaller than f.

[0220] Using k key frames P z1 , P z2 , P z3 …P zk Location information G z1 , G z2 , G z3 …G zk , construct n clusters, and calculate the distances from k key frames to n clusters, and use the key frame with the most 3D points in the n clusters as the candidate frame for each distance, so as to obtain n orders of magnitude of key frame data, the key frame sequence number is r1, r2, r3, ... rn, the key frame data includes: key frame P r1 , P r2 , P r3 …P rn and its image feature set J' r1 , J' r2 , J' r3 …J' rn , 3D point cloud data set S r1 , S r2 , S r3 …S rn . The order of magnitude is that n is further smaller than k.

[0221] The process flow of other steps S40 to S44 can refer to Figure 6 The execution process of the embodiment shown will not be repeated here.

[0222] Based on the embodiment of the above-mentioned visual map generation method provided by the technical solution of the present invention, based on the visual map generated by any embodiment method, such as Fig. 9As shown, the technical solution of the present invention also provides a visual map positioning method, which changes the execution and screening order of step S22 and step S23, and specifically includes the following process steps:

[0223] Step S50, obtaining second attribute information and image features of the image to be located, wherein the second attribute information includes second GPS information and orientation information.

[0224] The execution method of step S50 is the same as that of step S20, which will not be repeated here.

[0225] Step S51, setting a screening radius, and removing key frames larger than the screening radius to obtain a first key frame after screening.

[0226] The execution method of step S51 is the same as that of step S21, which will not be repeated here.

[0227] Step S52: Filter out a second key frame from the first key frame based on the image features of the image to be located and the image features of the first key frame.

[0228] The image retrieval method of the prior art can be used to retrieve the image features of the first key frame range of the visual map to obtain the selected second key frame. It is also possible to first query the global description sub-image features of the first key frame of the visual map through the KNN algorithm to select k key frames suitable for positioning; then use the K-means algorithm to build n clusters, assign these k key frames to these n clusters, select the key frame with the most three-dimensional points in each cluster as the candidate frame, and eliminate other key frames; finally, n key frames in the directory order returned by the KNN algorithm are obtained and used for positioning. The second key frame selected is the n key frames returned according to the directory order of the KNN algorithm. The other specific execution methods of step S52 are consistent with step S23 and will not be repeated here.

[0229] Step S53: Filter out a third key frame from the second key frame based on the orientation information of the image to be positioned and the orientation information of the second key frame.

[0230] The difference between step S53 and step S22 is that it uses the orientation information of the image to be located obtained by the device magnetometer. In the second key frame data range obtained by screening in the above step S52, the second key frame that is inconsistent with the orientation of the image to be located is eliminated to obtain the third key frame data that has been screened and has the same orientation as the image to be located. The third key frame data is the key frame data obtained by the final screening. The comparison method of the orientation information can refer to step S22, which will not be repeated here.

[0231] Step S54, perform feature matching based on the finally screened key frame image features and the features of the image to be positioned, select the three-dimensional point cloud data corresponding to the effectively matched image features in the key frame and the feature two-dimensional coordinate information of the image to be positioned, and obtain the device posture of the image to be positioned through the PNP algorithm.

[0232] The execution algorithm of step S54 is consistent with that of step S24. Both are based on the image features of the local area of ​​the image to be located (corresponding to the 2D projection) and the three-dimensional point cloud data (relative to the 3D points) corresponding to the key frame image features finally screened out to obtain the posture of the image device to be located. It will not be repeated here.

[0233] Corresponds to Figure 3 The data of the visual positioning map file shown in the embodiment is executed with step S50 and step S51. Figure 6 The visual map positioning method shown in FIG. 1 is consistent with that shown in FIG. 1 , and when executing step S51, the position information G1, G2, G3 ... G m The key frames of the visual map larger than the screening radius are removed. Assuming that after screening, the first key frame has an order of magnitude of f, and the first key frame sequence is x1, x2, x3, ... xf, the first key frame data includes: key frame P x1 , P x2 , P x3 …P xf and its image feature set J x1 , J x2 , J x3 …J xf , the three-dimensional point cloud data set S corresponding to the image features x1 , S x2 , S x3 …S xf , orientation information D x1 , D x2 , D x3 …D xf The differences are:

[0234] When executing step S52, the image feature set J can be obtained based on the image features of the image to be located by using the existing image retrieval method. x1 , J x2 , J x3 …J xf The range is searched to obtain the key frame image features closest to the image features to be located, so as to obtain the second key frame data obtained by screening. Assuming that the magnitude of the second key frame finally screened is g, and the second key frame sequence numbers are y1, y2, y3, ... yg, the third key frame data includes: image feature set J y1 , J y2 , J y3 …Jyg , 3D point cloud data set S y1 , S y2 , S y3 …S yg And direction information D y1 , D y2 , D y3 …D yg The order of magnitude of g is further smaller than f.

[0235] When executing step S53, the orientation information of the image to be positioned is used to obtain the orientation information D in the second key frame data of the visual map. y1 , D y2 , D y3 …D yg The key frames whose orientation information is inconsistent with the orientation information of the image to be located are removed, and the selected key frames are the third key frames. Assume that after screening, the magnitude of the third key frame is h, and the sequence number of the third key frame is z1, z2, z3, ...zh, and the third key frame data includes: key frame P z1 , P z2 , P z3 …P zh and its image feature set J z1 , J z2 , J z3 …J zh , the three-dimensional point cloud data set S corresponding to the image features z1 , S z2 , S z3 …S zh The order of magnitude of h is further smaller than g.

[0236] When executing step S54, according to the three-dimensional point cloud data set S in the third key frame data z1 , S z2 , S z3 …S zh As well as the image features of the local area of ​​the image to be positioned (corresponding to the 2D projection), the position and posture of the image device to be positioned are obtained.

[0237] Corresponds to Figure 4 The data of the visual positioning map file shown in the figure, because the visual positioning map file includes: image feature sets J1', J2', J3'...J m '. When executing step S51, position information G1, G2, G3...G m The first key frame data is obtained by filtering, which includes: key frame P x1 , P x2 , P x3 …P xf and its image feature set J' x1 , J'x2 , J' x3 …J' xf , the three-dimensional point cloud data set S corresponding to the image features x1 , S x2 , S x3 …S xf , orientation information D x1 , D x2 , D x3 …D xf The differences are:

[0238] When executing step S52, the KNN algorithm and the K-means algorithm can be used to obtain the image feature set J' based on the overall image features of the image to be located and the image of the local area. x1 , J' x2 , J' x3 …J' xf Range, first use the KNN algorithm to find the overall area image features j of the key frame x10 ,j x20 …j xf0 Extract k key frames suitable for positioning. At this time, the order of magnitude is k, and the sequence numbers are y1, y2, y3, ... yk. The filtered key frame data includes: key frame P y1 , P y2 , P y3 …P yk and its image feature set J' y1 , J' y2 , J' y3 …J' yk , 3D point cloud data set S y1 , S y2 , S y3 …S yk , location information G y1 , G y2 , G y3 …G yk . The order of magnitude is further smaller than f.

[0239] Using k key frames P y1 , P y2 , P y3 …P yk Location information G y1 , G y2 , G y3 …G yk , construct n clusters, and calculate the distances from k key frames to n clusters, and use the key frame with the most 3D points in n clusters as the candidate frame for each distance, so as to obtain n orders of magnitude of key frame data, the key frame sequence number is z1, z2, z3, ... zn, the second key frame data includes: key frame P z1, P z2 , P z3 …P zn and its image feature set J' z1 , J' z2 , J' z3 …J' zn , 3D point cloud data set S z1 , S z2 , S z3 …S zn , and orientation information D z1 , D z2 , D z3 …D zn . The order of magnitude is that n is further smaller than k.

[0240] When executing step S53, the orientation information D is obtained by using the orientation information of the image to be positioned in the second key frame data of the visual map. z1 , D z2 , D z3 …D zn The key frames whose orientation information is inconsistent with the orientation information of the image to be positioned are removed, and the selected key frames are the third key frames. Assume that after screening, the magnitude of the third key frame is h, and the sequence number of the third key frame is r1, r2, r3, ... rh. The third key frame data includes: key frame P r1 , P r2 , P r3 …P rh and its image feature set J' r1 , J' r2 , J' r3 …J' rh , the three-dimensional point cloud data set S corresponding to the image features r1 , S r2 , S r3 …S rh The order of magnitude of h is further smaller than n.

[0241] When executing step S54, according to the three-dimensional point cloud data set S in the third key frame data r1 , S r2 , S r3 …S rh As well as the image features of the local area of ​​the image to be positioned (corresponding to the 2D projection), the position and posture of the image device to be positioned are obtained.

[0242] Based on the embodiment of the above-mentioned visual map generation method provided by the technical solution of the present invention, based on the visual map generated by any embodiment method, such as Fig.10As shown, the technical solution of the present invention also provides a visual map positioning method, which changes the execution screening order of step S21 to step S23, and specifically includes the following process steps:

[0243] Step S60, obtaining second attribute information and image features of the image to be located, wherein the second attribute information includes second GPS information and orientation information.

[0244] The execution method of step S60 is the same as that of step S20, which will not be repeated here.

[0245] Step S61, selecting a first key frame from the visual map key frames based on the orientation information of the image to be located and the orientation information of the visual map key frames.

[0246] The execution method of step S61 is the same as that of step S41 and will not be repeated here.

[0247] Step S62: Filter out a second key frame from the first key frame based on the image features of the image to be located and the image features of the first key frame.

[0248] The execution method of step S62 is the same as that of step S52, which will not be repeated here.

[0249] Step S63, setting a screening radius, and removing second key frames larger than the screening radius to obtain a screened third key frame.

[0250] According to step S63, based on the GPS information of the second key frame of the visual map, the second GPS information of the map to be located performs key frame filtering of GPS information on the second key frame after the image feature filtering in step S62 within the range of the set filtering radius, and obtains the third key frame data whose GPS position is within the set filtering radius of the map to be located. The key frame filtering method can refer to step S21 or step S42, which will not be repeated here.

[0251] Step S64, perform feature matching based on the finally screened key frame image features and the features of the image to be positioned, select the three-dimensional point cloud data corresponding to the effectively matched image features in the key frame and the feature two-dimensional coordinate information of the image to be positioned, and obtain the device posture of the image to be positioned through the PNP algorithm.

[0252] The execution algorithm of step S64 is consistent with that of step S24. Both are based on the image features of the local area of ​​the image to be located (corresponding to the 2D projection) and the three-dimensional point cloud data (relative to the 3D points) corresponding to the key frame image features finally screened out to obtain the posture of the image device to be located. It will not be repeated here.

[0253] Corresponds to Figure 3 The data of the visual positioning map file shown is Figure 8The visual map positioning method shown in FIG. 1 is similar to that shown in FIG. 1 . When executing step S61, the orientation information of the image to be positioned is firstly obtained, and the orientation information D1, D2, D3 ... D m The filtered key frame is the first key frame. Assume that after filtering, the magnitude of the first key frame is h, and the first key frame sequence number is x1, x2, x3, ... xh, and the first key frame data includes: key frame P x1 , P x2 , P x3 …P xh and its image feature set J x1 , J x2 , J x3 …J xh , the three-dimensional point cloud data set S corresponding to the image features x1 , S x2 , S x3 …S xh , location information G x1 , G x2 , G x3 …G xh The order of magnitude of h is further smaller than m.

[0254] When executing step S62, the image feature set J can be used to retrieve the image features of the image to be located. x1 , J x2 , J x3 …J xh The range is searched to obtain the key frame image features closest to the image features to be located. Assuming that the magnitude of the second key frame finally screened out is g, and the second key frame sequence numbers are y1, y2, y3, ... yg, the second key frame data includes: image feature set J y1 , J y2 , J y3 …J yg The corresponding three-dimensional point cloud data set S y1 , S y2 , S y3 …S yg , location information G y1 , G y2 , G y3 …G yg The order of magnitude of g is further smaller than h.

[0255] When executing step S63, the location information G is obtained by using the GPS information of the image to be located. y1 , G y2 , G y3 …G ygScreening is performed to remove the key frames of the visual map that are larger than the screening radius, and the screened key frames are the third key frames. Assuming that after screening, the magnitude of the third key frame is f, and the sequence number of the third key frame is z1, z2, z3, ... zf, the third key frame data includes: key frame P z1 , P z2 , P z3 …P zf and its image feature set J z1 , J z2 , J z3 …J zf , the three-dimensional point cloud data set S corresponding to the image features z1 , S z2 , S z3 …S zf The order of magnitude of f is further smaller than g.

[0256] Corresponds to Figure 4 The data of the visual positioning map file shown in the figure, because the visual positioning map file includes: image feature sets J1', J2', J3'...J m '. When executing step S61, by the direction information D1, D2, D3...D m Suppose after screening, the first key frame has an order of magnitude of h, and the first key frame sequence is x1, x2, x3, ... xh. The first key frame data includes: key frame P x1 , P x2 , P x3 …P xh and its image feature set J' x1 , J' x2 , J' x3 …J' xh , the three-dimensional point cloud data set S corresponding to the image features x1 , S x2 , S x3 …S xh , location information G x1 , G x2 , G x3 …G xh The order of magnitude of h is further smaller than m.

[0257] When executing step S62, the KNN algorithm and the K-means algorithm can be used to obtain the image feature set J' based on the overall image features of the image to be located and the image of the local area. x1 , J' x2 , J' x3 …J' xh Range, first use the KNN algorithm to find the overall area image features j of the key frame x10 ,j x20…j xh0 Extract k key frames suitable for positioning. At this time, the order of magnitude is k, and the sequence numbers are y1, y2, y3, ... yk. The filtered key frame data includes: key frame P y1 , P y2 , P y3 …P yk and its image feature set J' y1 , J' y2 , J' y3 …J' yk , 3D point cloud data set S y1 , S y2 , S y3 …S yk , location information G y1 , G y2 , G y3 …G yk The order of magnitude of k is further smaller than h.

[0258] Using k key frames P y1 , P y2 , P y3 …P yk Location information G y1 , G y2 , G y3 …G yk , construct n clusters, and calculate the distances from k key frames to n clusters, and use the key frame with the most 3D points in n clusters as the candidate frame for each distance, so as to obtain n orders of magnitude of key frame data, the key frame sequence number is z1, z2, z3, ... zn, the second key frame data includes: key frame P z1 , P z2 , P z3 …P zn and its image feature set J' z1 , J' z2 , J' z3 …J' zn , 3D point cloud data set S z1 , S z2 , S z3 …S zn , and location information G z1 , G z2 , G z3 …G zn . The order of magnitude is that n is further smaller than k.

[0259] When executing step S63, the location information G is obtained by using the GPS information of the image to be located. z1 , G z2 , G z3 …G znSuppose after screening, the magnitude of the third key frame is f, the sequence number of the third key frame is r1, r2, r3, ... rf, and the third key frame data includes: key frame P r1 , P r2 , P r3 …P rf and its image feature set J' r1 , J' r2 , J' r3 …J' rf , the three-dimensional point cloud data set S corresponding to the image features r1 , S r2 , S r3 …S rf The order of magnitude of f is further smaller than n.

[0260] The step flow of other steps S60 to S64 can refer to the execution flow of other embodiments and will not be repeated here.

[0261] In addition, in the above Figure 6 In the variation of the embodiment of the visual map positioning method shown, step S23 is executed before step S21 and step S22, and after the image feature screening is completed, the process of step S21 and step S22 is executed based on the screened key frame data. Figure 8 In the variation of the embodiment of the visual map positioning method shown, step S43 is executed before step S41 and step S42, and after the image feature screening is completed, the process of step S41 and step S42 is executed based on the screened key frame data. These processes are similar and will not be repeated here.

[0262] Based on the above-mentioned visual map generation method and visual map positioning method, the technical solution of the present invention also provides a visual map generation system, including: a processor and a memory; a computer program is stored in the memory, and when the processor executes the computer program, the steps of any one of the above-mentioned visual map generation method embodiments are performed.

[0263] A visual map positioning system comprises: a processor and a memory; a computer program is stored in the memory, and when the processor executes the computer program, the steps of any one of the above-mentioned visual map positioning method embodiments are performed.

[0264] The above-mentioned visual map generation system may also include: an acquisition module, a map reconstruction module, a first calculation module and a first update module. The acquisition module is suitable for acquiring the required video sequence and the first attribute information; the map reconstruction module is suitable for reconstructing the visual positioning map based on the video sequence, and the visual positioning map includes key frames and key frame image features; the first calculation module is suitable for calculating the azimuth information of the main optical axis of each key frame in the visual positioning map relative to the spatial geographic coordinate system based on the first GPS information to obtain the key frame orientation information; the first update module is suitable for updating the visual positioning map based on the key frame orientation, and the updated visual positioning map also includes: the key frame orientation information.

[0265] The above-mentioned visual map positioning system may also include: an acquisition module, a first screening module, a second screening module, a third screening module and a second calculation module. Among them, the acquisition module is suitable for acquiring the second attribute information and image features of the image to be positioned, and the second attribute information includes the second GPS information and orientation information; the first screening module is suitable for setting a screening radius, and removing key frames larger than the screening radius to obtain filtered key frames; the second screening module is suitable for filtering key frames from the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frames; the third screening module is suitable for filtering key frames from key frames based on the image features of the image to be positioned and the image features of the key frames; the second calculation module is suitable for obtaining the device posture of the image to be positioned through the PNP algorithm according to the finally filtered key frames and the image features of the key frames.

[0266] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for generating a visual map, characterized in that: include: Collecting the required video sequence and first attribute information, where the first attribute information includes first GPS information; Reconstructing a visual positioning map based on the video sequence, wherein the visual positioning map includes key frame image features and three-dimensional point cloud data corresponding to the key frame image features; Aligning the visual positioning map to a spatial geographic coordinate system according to the first GPS information and calculating the position information of each key frame in the visual positioning map in the spatial geographic coordinate system; Calculating the azimuth information of each key frame relative to the spatial geographic coordinate system using the position information of the key frame in the spatial geographic coordinate system and the main optical axis to obtain the orientation information of the key frame; The visual positioning map is updated based on the key frame orientation, and the updated visual positioning map further includes: the key frame orientation information; The spatial geographic coordinate system includes: an X-axis pointing due east, a Y-axis pointing due north, and a Z-axis pointing away from the ground and upward; the calculation of the azimuth information of each key frame relative to the spatial geographic coordinate system using the position information of the key frame in the spatial geographic coordinate system and the main optical axis includes: Select a point p(x0,y0,f) on the principal optical axis of each key frame, where (x0,y0) is the principal point of the camera and f is the focal length of the camera; The key frame optical center o(0,0,0) and point p(x0,y0,f) in the camera coordinate system are converted into coordinates O(xo,yo,zo) and coordinates P(xp,yp,zp) in the corresponding spatial geographic coordinate system through coordinate system conversion; Calculate the angle θ between the ray OP composed of the coordinates O(xo,yo,zo) and P(xp,yp,zp) and the X-axis, where: The azimuth information of ray OP is calculated according to the following conditions: If xp>xo, the current azimuth is (π / 2-θ); If xp<xo, the current azimuth is (3π / 2-θ); If xp=xo and yp>yo, the current azimuth is 0; If xp=xo and yp<yo, the current azimuth is π.

2. The method for generating a visual map according to claim 1, wherein: The reconstructing the visual positioning map based on the video sequence includes: reconstructing the visual positioning map by using a motion structure recovery algorithm or a real-time positioning and map construction algorithm.

3. The method for generating a visual map according to claim 1, wherein: The key frame image features include: overall image features and local image features.

4. The visual map generation method according to any one of claims 1 to 3, characterized in that: The first attribute information further includes: first IMU information; and the visual map generation method further includes: Based on the first GPS information and / or the first IMU information, the visual positioning map is calibrated and updated using bundle adjustment.

5. The visual map generation method according to any one of claims 1 to 3, characterized in that: Also includes: The updated visual positioning map is gridded for display.

6. A visual map positioning method, based on the visual map generation method according to any one of claims 1 to 5, characterized in that: include: Acquire second attribute information and image features of the image to be located, where the second attribute information includes second GPS information and orientation information; A screening radius is set, and key frames larger than the screening radius are removed to obtain a first key frame after screening, wherein the screening radius is the sum of a screening constant radius and a device location accuracy value set for the second GPS information; Filtering a second key frame from the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frame; Selecting a third key frame from the key frames based on the image features of the image to be located and the image features of the key frames; Feature matching is performed based on the finally screened key frame image features and the features of the image to be positioned, and the three-dimensional point cloud data corresponding to the effectively matched image features in the key frame and the characteristic two-dimensional coordinate information of the image to be positioned are selected, and the device posture of the image to be positioned is obtained through the PNP algorithm.

7. The visual map positioning method according to claim 6, characterized in that: The selecting the second key frame from the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frame comprises: selecting the second key frame from the first key frame of the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the first key frame; The step of selecting a third key frame from key frames based on the image features of the image to be located and the image features of the key frames includes: selecting a third key frame from the second key frame based on the image features of the image to be located and the image features of the second key frame, and the key frame finally selected is the third key frame.

8. The visual map positioning method according to claim 6, characterized in that: The step of selecting the third key frame from the key frame based on the image features of the image to be located and the image features of the key frame comprises: selecting the third key frame from the first key frame based on the image features of the image to be located and the image features of the first key frame; The step of selecting the second key frame from the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frame includes: selecting the second key frame from the third key frame of the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the third key frame, and the key frame finally selected is the second key frame.

9. The visual map positioning method according to claim 6, characterized in that: The step of removing key frames larger than the screening radius to obtain the first key frame after screening includes: removing key frames larger than the screening radius from the second key frames to obtain the first key frame after screening; The step of selecting a third key frame from key frames based on the image features of the image to be located and the image features of the key frames includes: selecting a third key frame from the first key frame based on the image features of the image to be located and the image features of the first key frame, and the key frame finally selected is the third key frame.

10. The visual map positioning method according to claim 6, characterized in that: The step of selecting the third key frame from the key frame based on the image features of the image to be located and the image features of the key frame comprises: selecting the third key frame from the second key frame based on the image features of the image to be located and the image features of the second key frame; Eliminating key frames larger than the screening radius to obtain the first key frame after screening includes: eliminating key frames larger than the screening radius in the third key frame to obtain the first key frame after screening, and the key frame finally screened out is the first key frame.

11. The visual map positioning method according to claim 6, characterized in that: The step of removing key frames larger than the screening radius to obtain the first key frame after screening includes: removing key frames larger than the screening radius from the third key frame to obtain the first key frame after screening; The step of selecting a second key frame from the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frame includes: selecting a second key frame from the first key frame of the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the first key frame, and the key frame finally selected is the second key frame.

12. The visual map positioning method according to claim 6, characterized in that: The selecting the second key frame from the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frame comprises: selecting the second key frame from the third key frame of the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the third key frame; Eliminating key frames larger than the screening radius to obtain the first key frame after screening includes: eliminating key frames larger than the screening radius in the second key frame to obtain the first key frame after screening, and the key frame finally screened out is the first key frame.

13. The visual map positioning method according to any one of claims 6 to 12, characterized in that: The step of eliminating key frames larger than the screening radius to obtain the first filtered key frame includes: taking the device location displayed by the second GPS information as the center, eliminating key frames on the visual positioning map whose radius is larger than the screening radius.

14. The visual map positioning method according to any one of claims 6 to 12, characterized in that: The step of selecting a second key frame from the visual positioning map based on the orientation information of the image to be positioned and the key frame orientation information includes: setting an orientation threshold; comparing the orientation information of the key frame with the orientation information of the key frame, and eliminating key frames whose orientation difference exceeds the orientation threshold.

15. The visual map positioning method according to any one of claims 6 to 12, characterized in that: The step of selecting a third key frame from the key frames based on the image features of the image to be located and the image features of the key frames comprises: Performing a KNN query on the key frame using key frame image features to obtain K key frames; Clustering the K key frames using a K-means algorithm according to position information of the K key frames, wherein the K key frames are divided into N clusters; The key frames whose number of three-dimensional point clouds in each cluster meets the predetermined number are taken as the third key frames, and other key frames are eliminated.

16. A visual map generation system, characterized in that: include: Processor and memory; The memory stores a computer program, and the processor executes the steps of the visual map generation method according to any one of claims 1 to 5 when executing the computer program.

17. A visual map positioning system, characterized in that: include: Processor and memory; A computer program is stored in the memory, and when the processor executes the computer program, the steps of the visual map positioning method as described in any one of claims 6 to 15 are performed.

18. A visual map generation system, characterized in that: include: A collection module, a map reconstruction module, a first calculation module and a first update module; The acquisition module is adapted to acquire the required video sequence and first attribute information, wherein the first attribute information includes first GPS information; The map reconstruction module is adapted to reconstruct a visual positioning map based on the video sequence, wherein the visual positioning map includes key frame image features and three-dimensional point cloud data corresponding to the key frame image features; The first calculation module is adapted to align the visual positioning map to a spatial geographic coordinate system according to the first GPS information and calculate the position information of each key frame in the visual positioning map in the spatial geographic coordinate system; The first calculation module is further adapted to calculate the azimuth information of each key frame relative to the spatial geographic coordinate system using the position information of the key frame in the spatial geographic coordinate system and the main optical axis to obtain the orientation information of the key frame; The first updating module is adapted to update the visual positioning map based on the key frame orientation, the updated visual positioning map further comprising: the key frame orientation information; The spatial geographic coordinate system includes: an X-axis pointing due east, a Y-axis pointing due north, and a Z-axis pointing away from the ground and upward; the first calculation module includes: a selection unit, a conversion unit, a first calculation unit, and a second calculation unit; The selection unit is adapted to select a point p(x0, y0, f) on the principal optical axis of each key frame, where (x0, y0) is the principal point of the camera and f is the focal length of the camera; The conversion unit is adapted to convert the key frame optical center o(0,0,0) and point p(x0,y0,f) in the camera coordinate system into coordinates O(xo,yo,zo) and coordinates P(xp,yp,zp) in the corresponding spatial geographic coordinate system through coordinate system conversion; The first calculation unit is adapted to calculate the angle θ of the ray OP composed of the coordinates O(xo, yo, zo) and P(xp, yp, zp) relative to the X-axis, wherein: The second calculation unit is adapted to calculate the azimuth information of the ray OP according to the following conditions: If xp>xo, the current azimuth is (π / 2-θ); If xp<xo, the current azimuth is (3π / 2-θ); If xp=xo and yp>yo, the current azimuth is 0; If xp=xo and yp<yo, the current azimuth is π.

19. The visual map generation system according to claim 18, characterized in that: The map reconstruction module is suitable for reconstructing the visual positioning map through a motion structure recovery algorithm or a real-time positioning and map construction algorithm.

20. The visual map generation system according to claim 18 or 19, characterized in that: The first attribute information further includes: first IMU information; the visual map generation system further includes: a second updating module; The second updating module is suitable for calibrating the visual positioning map and updating the visual positioning map using bundle adjustment based on the first GPS information and / or the first IMU information.

21. The visual map generation system according to claim 18 or 19, characterized in that: Also includes: Gridding module; The gridding module is suitable for gridding the updated visual positioning map for display.

22. A visual map positioning system, based on the visual map generation system according to any one of claims 18 to 21, characterized in that: include: An acquisition module, a first screening module, a second screening module, a third screening module and a second calculation module; The acquisition module is adapted to acquire second attribute information and image features of the image to be located, wherein the second attribute information includes second GPS information and orientation information; The first screening module is adapted to set a screening radius and remove key frames larger than the screening radius to obtain a first key frame after screening, wherein the screening radius is the sum of a screening constant radius and a device location accuracy value set for the second GPS information; The second screening module is adapted to screen out a second key frame from the visual positioning map based on the orientation information of the image to be positioned and the orientation information of the key frame; The third screening module is adapted to screen out a third key frame from the key frames based on the image features of the image to be located and the image features of the key frames; The second calculation module is suitable for performing feature matching based on the finally screened key frame image features and the features of the image to be positioned, selecting the three-dimensional point cloud data corresponding to the effectively matched image features in the key frame and the feature two-dimensional coordinate information of the image to be positioned, and obtaining the device posture of the image to be positioned through the PNP algorithm.

23. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the visual map generation method according to any one of claims 1 to 5 is implemented.

24. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the visual map positioning method as described in any one of claims 6 to 15 is implemented.

Images