AR navigation method and device based on three-dimensional semantic map

Abstract

The present invention discloses an AR navigation method and device based on a three-dimensional semantic map. The navigation method comprises the following steps: obtaining a two-dimensional map of a collection scene; planning scene collection points based on the two-dimensional map and constructing a road network; obtaining at least one scene picture, constructing a scene picture library, and constructing a three-dimensional semantic map based on the two-dimensional map, the scene picture library and the location information of the collection points; generating a navigation path according to the user's starting point, end point and road network; and generating navigation guidance by referring to the navigation path through the three-dimensional semantic map. The AR navigation method based on a three-dimensional semantic map provided by the present invention has many advantages such as low cost, high collection efficiency, convenient implementation and easy expansion.

Description

Technical Field

[0001] The present invention relates to an AR navigation method and device based on a three-dimensional semantic map, and belongs to the technical field of visual augmented reality navigation. Background Art

[0002] AR navigation, also known as "augmented reality navigation", is an innovative way of map navigation interaction. Its principle is to capture the real picture in real time through the camera, combine it with the map navigation guidance information and the vehicles, roads, pedestrians and other scene targets recognized by visual AI, and generate a virtual 3D navigation guidance model, which is integrated and superimposed on the display screen, thereby creating a more intuitive navigation experience.

[0003] The existing AR navigation technology solutions are:

[0004] GPS positioning + 2D map navigation. Since the civilian accuracy of GPS is about 10 meters, its positioning accuracy is low. In addition, due to GPS attenuation and no signal, and the inability to distinguish floor information, this solution cannot be applied to indoor scenarios: such as shopping malls, underground parking lots, etc.

[0005] Bluetooth beacon positioning or WiFi and other professional equipment assisted positioning + 2D map navigation. Since many Bluetooth or WiFi devices need to be embedded in the scene, the deployment and maintenance costs are high. In addition, this solution is obviously not suitable for outdoor scenes and is not suitable for large-scale promotion.

[0006] High-precision three-dimensional digital map + navigation. The construction of large-scale high-precision three-dimensional digital maps requires units with navigation electronic map production (Grade A) surveying and mapping qualifications. The construction of high-precision three-dimensional digital maps requires the use of expensive lidar devices and panoramic cameras to scan the entire scene, which is very expensive and difficult for ordinary businesses to afford. The subsequent maintenance and updating of three-dimensional digital maps is also a considerable expense, which makes their production and promotion more difficult.

[0007] Therefore, it is necessary to conduct more in-depth research on AR navigation in order to provide a low-cost, easy-to-implement and easy-to-expand indoor / outdoor AR navigation solution. Summary of the invention

[0008] In order to overcome the above problems, the inventors conducted in-depth research and a large number of experiments and designed an AR navigation method based on a three-dimensional semantic map, which includes the following steps:

[0009] S1, obtaining a two-dimensional map of the acquisition scene;

[0010] S2, plan scene collection points based on the two-dimensional map and build a road network;

[0011] S3, obtaining at least one scene picture, constructing a scene picture library, and constructing a three-dimensional semantic map based on the flat two-dimensional map, the scene picture library and the location information of the collection points;

[0012] S4, generating a navigation path according to the user's starting point, end point and road network;

[0013] S5. Generate navigation guidance by referring to the navigation path through the three-dimensional semantic map.

[0014] In a preferred embodiment, in S2, the planned scene collection points include key nodes of the scene, and the key nodes refer to at least one of the starting point of the path, the bifurcation point of the path, and the user's interested node;

[0015] In S2, the planned scene collection points also include the locations of fixed markers of the scene, where the fixed markers refer to immovable reference objects on the path.

[0016] In a preferred embodiment, S3 further includes:

[0017] The scene image library is divided into training set and test set according to pre-set rules, and the classification model is trained based on the training set and test set.

[0018] In a preferred embodiment, S3 further includes the following sub-steps:

[0019] Collect several scene pictures with different orientations at each collection point to build a scene picture library;

[0020] The scene image library is divided into a training set and a test set according to preset rules, and the classification model is trained for classification. The classification model input includes at least the scene images in the scene image library, and the classification model output includes the acquisition position and orientation information corresponding to the scene images.

[0021] In a preferred embodiment, when collecting scene images at key nodes, the positions of the collection points and the orientation information of the scene images are obtained.

[0022] In a preferred embodiment, in S5, a real-time scene image of the user's location is collected through the user terminal, the first location information of the user's location is obtained through the trained classification model, and navigation guidance is generated based on the three-dimensional semantic map reference navigation path.

[0023] In a preferred embodiment, when the scene changes, the scene image is recaptured at the changed position, and S3 is repeated to achieve the expansion of the three-dimensional semantic map.

[0024] In a preferred embodiment, S5 includes the following sub-steps:

[0025] S51, collecting a real-time scene image of the user's location and obtaining the user terminal's orientation information through the user terminal;

[0026] S52, classifying the real-time scene image by using the trained classification model, and outputting first location information of the user's location;

[0027] S53, using the first location information of the user's location and the three-dimensional semantic map to determine the second location information of the user's location, that is, the precise location information of the user's location;

[0028] S54: Determine the position of the user on the navigation path according to the second position information of the user and the orientation information of the user terminal, and then generate navigation guidance.

[0029] In a preferred embodiment, in S51, the user terminal is a device having a camera and a gyroscope, wherein the camera is used to collect a scene image of the user's location, and the gyroscope is used to obtain orientation information of the user terminal.

[0030] In a preferred embodiment, in S5, during the navigation process, a scene image of the user's location is continuously obtained, the user's location coordinates and the user's orientation are updated, and the AR guide icon position is updated.

[0031] The present invention also provides an electronic device, comprising:

[0032] at least one processor; and

[0033] A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the methods described above.

[0034] The present invention also provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute any of the above-mentioned methods.

[0035] The beneficial effects of the present invention include:

[0036] (1) Low cost, which not only avoids the high cost and difficulty of burying a large number of Bluetooth or WiFi devices, but also does not require the large amount of economic investment in building high-precision three-dimensional digital maps;

[0037] (2) Easy to implement: By collecting scene images and combining them with existing maps, the user’s actual physical location information can be quickly located to achieve navigation;

[0038] (3) Easy to expand. Changes in scenes, including those caused by lighting, shadows, or seasonal changes, can be achieved by adding / updating scene images through supplementary collection. The recognition of new scenes can be achieved through machine learning and artificial intelligence, which is easy to expand. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic flow chart of an AR navigation method based on a three-dimensional semantic map according to a preferred embodiment of the present invention is shown;

[0040] Figure 2 A schematic diagram showing the selection of the orientation of a single scene point in an AR navigation method based on a three-dimensional semantic map according to a preferred embodiment of the present invention is shown.

[0041] Figure 3 A schematic diagram of acquisition positions in an AR navigation method based on a three-dimensional semantic map according to a preferred embodiment of the present invention is shown. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below through the accompanying drawings and embodiments. Through these descriptions, the characteristics and advantages of the present invention will become more clear and distinct.

[0043] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0044] According to an AR navigation method based on a three-dimensional semantic map provided by the present invention, Figure 1 As shown, the following steps are included:

[0045] S1, obtaining a two-dimensional map of the acquisition scene;

[0046] S2, plan scene collection points based on the two-dimensional map and build a road network;

[0047] S3, obtaining at least one scene picture, constructing a scene picture library, and constructing a three-dimensional semantic map based on the flat two-dimensional map, the scene picture library and the location information of the collection points;

[0048] S4, generating a navigation path based on a two-dimensional map according to the user's starting point, end point and road network;

[0049] S5. Generate navigation guidance by referring to the navigation path through the three-dimensional semantic map.

[0050] In S1, the two-dimensional map of the acquisition scene can be obtained through a third-party map platform (such as Baidu Maps and Amap), or it can be obtained through surveying and mapping, but the north direction needs to be indicated.

[0051] Preferably, in S1, the plane map is uploaded by the user. By the user uploading a customized two-dimensional plane map, the dependence on traditional maps is reduced, so that the user can flexibly create a three-dimensional semantic map of the region.

[0052] According to the present invention, in S2, the planned scene collection points include key nodes of the scene, which are the starting points of the path and the bifurcation points of the path, such as entrances and exits, stairs, elevators, three-way intersections, crossroads, etc. Figure 3 As shown in the figure, the locations of key nodes on the first floor of a shopping mall are collected.

[0053] Further preferably, the planned scene collection points also include fixed marker positions of the scene, and the fixed markers refer to immovable reference objects on the path.

[0054] Preferably, S3 further includes:

[0055] The scene image library is divided into training set and test set according to pre-set rules, and the classification model is trained based on the training set and test set.

[0056] Preferably, S3 further includes the following sub-steps:

[0057] Collect several scene pictures with different orientations at each collection point to build a scene picture library;

[0058] The scene image library is divided into a training set and a test set according to preset rules, and the classification model is trained for classification. The classification model input includes at least the scene images in the scene image library, and the classification model output includes the acquisition position and orientation information corresponding to the scene images.

[0059] In S3, pictures of the key nodes in multiple directions are collected. Preferably, pictures of the four directions of front, back, left, and right are collected, such as Figure 2 shown.

[0060] More preferably, when capturing images of fixed markers, only the front side of the fixed markers needs to be captured.

[0061] In the present invention, by collecting pictures of key nodes in multiple directions, the scene pictures are given directional attributes.

[0062] In a more preferred embodiment, in S3, the position of the acquisition point and the orientation information of the scene image are acquired while the scene image is being acquired.

[0063] Preferably, in S3, the orientation of the scene image is determined by the connection relationship between the acquisition points in the two-dimensional plane map, that is, the orientation of the scene image is all possible travel directions marked in the two-dimensional plane map, and the orientation information of the scene image is provided by a gyroscope.

[0064] In a preferred embodiment, a collection QR code is set at the collection point, and the collection QR code is provided with the collection point location and collection direction information. The collection personnel scan the collection QR code and then collect the scene picture, so that the collected scene picture can directly generate picture labels, avoiding the subsequent manual work of adding labels.

[0065] In a more preferred embodiment, a collection authorization code is also provided in the collection QR code, and the collection authorization code is used to identify the persons who are allowed to perform scene collection.

[0066] In S3, the input of the classification model is pictures of key nodes of the scene in different travel directions, and the output of the classification model is the collection position and orientation of the scene pictures.

[0067] In the present invention, the specific type of the classification model is not particularly limited, and those skilled in the art may adopt any known image recognition model, such as a CNN neural network model, as needed.

[0068] According to the present invention, the classification is performed in the server. Specifically, after obtaining the scene picture facing the user, the scene picture is transmitted to the server, and the server performs classification.

[0069] In S3, based on the flat two-dimensional map and the scene image, the location of the scene image acquisition point is mapped to the map, and the scene image library is used to perform three-dimensional reconstruction to generate a three-dimensional semantic map. In the present invention, there is no limitation on the specific method of three-dimensional reconstruction, and those skilled in the art can adopt any known method.

[0070] In a preferred embodiment, in the process of constructing a three-dimensional semantic map, three-dimensional reconstruction is not performed on all areas, but only on the key node positions of the scene to obtain multiple three-dimensional node areas. After the three-dimensional node areas are matched with the navigation paths, they are mapped on the plane map to construct a three-dimensional semantic map.

[0071] In the present invention, only the information of the locations of key nodes of the scene is used to create a three-dimensional semantic map, which simplifies the data for map creation, thereby achieving lightweight map creation, saving the cost of map creation, and improving creation efficiency.

[0072] Preferably, when the scene changes, including but not limited to the addition of scenes, changes in the appearance of local scenes, changes in seasons, etc., the scene images are recaptured for the changed positions, and S4 is repeated to achieve the expansion of the three-dimensional semantic map.

[0073] In a preferred embodiment, in S4, the collected scene images are used for both classification training of the classification model and 3D reconstruction of the scene;

[0074] Furthermore, in S3, the generated three-dimensional semantic map includes but is not limited to the two-dimensional position of the acquisition point and the orientation information of the scene image, and the point cloud mapping information corresponding to the feature points of the scene image.

[0075] The point cloud mapping information corresponding to the feature points of the scene image refers to the corresponding relationship between the scene image and the point cloud.

[0076] In S4, the starting point coordinates of the navigation path are the starting point of the user, the end point is the end point coordinates input by the user, and the navigation path passes through at least one scene collection point.

[0077] Preferably, S5 includes: collecting a real-time scene image of the user's location through the user terminal, obtaining the first location information of the user's location through the trained classification model, and generating a navigation guide based on the three-dimensional semantic map reference navigation path.

[0078] Preferably, S5 includes the following sub-steps:

[0079] S51, collecting a scene image of the user's location and obtaining the user terminal's orientation information through the user terminal;

[0080] S52, classifying the scene image by using the trained classification model, and outputting first location information of the user's location;

[0081] S53, using the first location information of the user's location and the three-dimensional semantic map to determine the second location information of the user's location, that is, the precise location information of the user's location;

[0082] S54, according to the second position information of the user's location and the orientation information of the user terminal, determine the user's location on the navigation path, and then generate navigation guidance. The inventor found that although the user's location can be accurately determined directly through the three-dimensional semantic map, since the three-dimensional semantic map contains a large amount of point cloud data, it needs to be matched and searched one by one, which will consume a lot of computing power and cause navigation jams. In the present invention, the rough location information of the user, that is, the first location information, is obtained by setting a classification model, and then, according to the first location information, the scene picture of the user's location is compared with the mapping information corresponding to the scene picture feature points in the three-dimensional semantic map, and the precise location of the user, that is, the second location information, is searched, thereby narrowing the range of the three-dimensional semantic map that needs to be searched and matched, thereby improving the speed of navigation positioning and reducing the computing power requirements during the navigation process.

[0083] Specifically,

[0084] In S51, the user terminal is a device having a camera and a gyroscope, wherein the camera is used to collect a scene picture of the user's location, and the gyroscope is used to obtain orientation information of the user terminal.

[0085] In S52, the first position information obtained through the classification model may have the problem of insufficient position accuracy. For example, the obtained position deviates from the actual position by 1-2 meters, which will have a certain impact on the navigation accuracy, especially in the navigation of narrow internal roads. Therefore, in the present invention, the user's position can be accurately determined by comparing the scene picture of the user's location with the mapping information corresponding to the feature points of the scene picture in the three-dimensional semantic map.

[0086] In S52, the trained classification model is used to classify the scene image acquired in S61 to obtain the acquisition position and orientation of the scene image as the actual position of the user and the actual orientation of the user terminal.

[0087] Furthermore, the classification is performed in a server, and the acquired scene pictures facing the user's current location are uploaded to the server, and the server classifies the scene pictures.

[0088] Furthermore, when classification fails, steps S51 to S52 are repeated, and the scene image of the user's current location is re-acquired and then classified. Preferably, when classification fails five times in a row, the user is prompted to change location, and then the scene image of the user's current location is re-acquired.

[0089] In a preferred embodiment, in S5, a scene image of the user's location is continuously obtained during the navigation process, and the user's location coordinates and the orientation of the user terminal are updated to verify the user's location and direction, thereby updating and adjusting the navigation path, and updating the AR guide icon position.

[0090] In a preferred embodiment, during the navigation process, the user position is updated in real time through SLAM.

[0091] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the present disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document is not limited here.

[0092] Various embodiments of the method described above in the present invention can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0093] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0094] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0095] To provide interaction with a user, the methods and apparatus described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0096] The methods and apparatus described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0097] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0098] The present invention has been described above in conjunction with preferred embodiments, but these embodiments are only exemplary and serve only as an illustration. On this basis, the present invention may be subjected to a variety of substitutions and improvements, all of which fall within the scope of protection of the present invention.

Claims

1. An AR navigation method based on a three-dimensional semantic map, characterized in that: The following steps are involved: S1, obtaining a two-dimensional map of the acquisition scene; S2, plan scene collection points based on the two-dimensional map and build a road network; S3, obtaining at least one scene picture, constructing a scene picture library, and constructing a three-dimensional semantic map based on the flat two-dimensional map, the scene picture library and the location information of the collection points; S4, generating a navigation path according to the user's starting point, end point and road network; S5, generating navigation guidance by referring to the navigation path through the three-dimensional semantic map; S3 also includes the following sub-steps: Collect several scene pictures with different orientations at each collection point to build a scene picture library; The scene image library is divided into a training set and a test set according to preset rules, and the classification model is trained for classification. The classification model input includes at least the scene images in the scene image library, and the classification model output includes the acquisition position and orientation information corresponding to the scene images.

2. The AR navigation method based on three-dimensional semantic map according to claim 1, characterized in that: In S2, the planned scene collection points include key nodes of the scene, and the key nodes refer to at least one of the starting point of the path, the bifurcation point of the path, and the user's interested node; In S2, the planned scene collection points also include the locations of fixed markers of the scene, where the fixed markers refer to immovable reference objects on the path.

3. The AR navigation method based on three-dimensional semantic map according to claim 1, characterized in that: Generate navigation instructions through 3D semantic map and reference navigation path, including: The real-time scene picture of the user's location is collected through the user terminal, the first location information of the user's location is obtained through the trained classification model, and the navigation guidance is generated based on the three-dimensional semantic map reference navigation path.

4. The AR navigation method based on three-dimensional semantic map according to claim 3, characterized in that: S5 includes the following sub-steps: S51, collecting a real-time scene image of the user's location and obtaining the user terminal's orientation information through the user terminal; S52, classifying the real-time scene image by using the trained classification model, and outputting first location information of the user's location; S53, using the first location information of the user's location and the three-dimensional semantic map to determine the second location information of the user's location, that is, the precise location information of the user's location; S54: Determine the position of the user on the navigation path according to the second position information of the user and the orientation information of the user terminal, and then generate navigation guidance.

5. The AR navigation method based on three-dimensional semantic map according to claim 4, characterized in that: In S51, the user terminal is a device having a camera and a gyroscope, wherein the camera is used to collect a real-time scene picture of the user's location, and the gyroscope is used to obtain the user terminal orientation information.

6. The AR navigation method based on three-dimensional semantic map according to claim 4, characterized in that: In S6, during the navigation process, a real-time scene image of the user's location is continuously obtained, the user's location coordinates and the orientation of the user terminal are updated, and the AR guide icon position is updated.

7. An electronic device comprising: at least one processor; as well as A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the method of any one of claims 1-6.

8. A computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

Images