AR-based scene tour method, ar glasses, electronic device, and storage medium

Abstract

The application relates to an AR-based scene tour method, AR glasses, an electronic device and a storage medium. Real-time images of a space region within a user's field of view are acquired; an exhibit image in the real-time images is extracted; global feature descriptors and feature point descriptors in the exhibit image are extracted; at least one similar image matched with the exhibit image is preliminarily matched from to-be-retrieved images according to the global feature descriptors; a first similar image most similar to the exhibit image is accurately matched according to the feature point descriptors; the spatial pose of the user relative to the exhibit image is determined according to the position information of the feature points in the matching feature point pairs on the exhibit image; virtual information corresponding to the first similar image is acquired according to the spatial pose, and the virtual information is superimposed and displayed on the space region within the user's field of view, thereby solving the problems of poor AR-based scene tour effect and large computing resource consumption, improving the AR-based scene tour effect and reducing the computing resource consumption.

Description

Technical Field

[0001] This application relates to the field of AR technology, and in particular to AR-based scene navigation methods, AR glasses, electronic devices, and storage media. Background Technology

[0002] The relevant AR (Augmented Reality)-based scene navigation method includes the following key steps:

[0003] Step 1: Use AR glasses to capture scene images and extract feature points from the images. Hand-designed features are used to extract these feature points.

[0004] Step 2: Establish an offline bag-of-words model. Using the feature points extracted in Step 1 on a given set of image datasets, a bag-of-words model is obtained through feature encoding algorithms based on feature descriptors, such as BOW or VLAD.

[0005] Step 3, Feature Point Matching. A random sampling consensus algorithm is used to obtain the optimal matching point pairs.

[0006] Step 4: Calculate pose. Based on the matching point pairs obtained in Step 3, calculate the corresponding transformation matrix through geometric constraints to obtain the position of the scene image.

[0007] Step 5: Display the location of the scene image for guided navigation.

[0008] The above solution has the following drawbacks:

[0009] (1) Even if the images are taken in the same scene, the scene images will be different due to factors such as complex ambient lighting and different angles of the AR glasses worn by users. Directly locating the scene position by matching the feature points of the scene image will lead to inaccurate positioning and will take a long time.

[0010] (2) Manually designed feature points are based on people's understanding of geometry and mathematics to model certain local special regions in the image to obtain the description function. The disadvantage of this type of feature point is that it only focuses on the local features of the image, is sensitive to changes in lighting and viewing angle, has weak generalization ability, and is unstable in speed. The process of building the offline bag-of-words model is based on the codebook obtained by clustering the manually designed local feature points. Therefore, it cannot describe the global features of the image, and its feature expression ability is also greatly limited.

[0011] (3) AR glasses are a portable device with low computing power and limited battery capacity, so the battery life is short. The related AR-based scene navigation method cannot balance computing power and navigation effect.

[0012] There is currently no effective solution to the problems of poor scene navigation effects and high computational resource consumption in AR-based technologies. Summary of the Invention

[0013] This embodiment provides an AR-based scene navigation method, AR glasses, electronic device, and storage medium to solve the problems of poor scene navigation effect and high computational resource consumption in related technologies.

[0014] Firstly, this embodiment provides an AR-based scene navigation method, including:

[0015] Acquire real-time images of the spatial area within the user's field of view, perform target recognition processing on the real-time images, and obtain exhibit images in the real-time images;

[0016] The exhibit image is subjected to feature extraction processing to obtain the global feature descriptor and feature point descriptor of the exhibit image;

[0017] Obtain the global feature descriptor of the image to be retrieved; based on the global feature descriptor of the exhibit image and the image to be retrieved, determine at least one similar image that matches the exhibit image from the images to be retrieved; obtain the feature point descriptor of the similar image; based on the feature point descriptor of the exhibit image and the image to be retrieved, obtain matching feature point pairs; and determine the first similar image with the most matching feature point pairs from the similar images.

[0018] The position information of the feature points in the matching feature point pair on the exhibit image is obtained. Based on the position information, the spatial pose of the user relative to the exhibit image is determined. Based on the spatial pose, virtual information corresponding to the first similar image is obtained, and the virtual information is superimposed and displayed in the spatial area within the user's field of vision.

[0019] In some embodiments, the virtual information includes the coordinates of the current exhibit within the exhibition hall, as seen from the user's field of vision.

[0020] In some of these embodiments, the virtual information includes preset navigation information pointing from the current exhibit to the next exhibit.

[0021] In some embodiments, feature extraction processing is performed on the exhibit image to obtain a global feature descriptor and feature point descriptor for the exhibit image, including:

[0022] The exhibit images are processed by feature extraction using a pre-trained convolutional neural network, wherein the pre-trained convolutional neural network includes a first network and a second network, and the hidden layer of the first network is connected to the input of the second network.

[0023] Feature extraction processing of the exhibit images using a pre-trained convolutional neural network includes:

[0024] The exhibit image is input into the first network for feature extraction processing, and shallow features and the global feature descriptor are output.

[0025] The shallow features are input into the second network for feature extraction processing, and the feature point descriptors are output.

[0026] In some embodiments, the shallow features are input into the second network for feature extraction processing, and the output feature point descriptors include:

[0027] The shallow features are input into the second network for feature extraction processing to obtain the feature map output by the last convolutional layer in the second network.

[0028] The feature map is normalized to obtain the feature point response score map;

[0029] Identify feature points in the feature point response score map that are greater than a preset threshold, and output the feature point descriptor of the feature points.

[0030] In some embodiments, the shallow features include line information and / or edge information of the exhibit image; the global feature descriptor includes the overall structural information of the exhibit image; and the feature point descriptor includes the local structural information of the exhibit image.

[0031] In some embodiments, the method for training the convolutional neural network includes:

[0032] Train the first network to obtain its network weights;

[0033] The second network is trained based on the network weights of the first network.

[0034] Secondly, this embodiment provides AR glasses, including: a camera, a display screen, and a processing unit, wherein the processing unit is connected to the camera and the display screen; wherein,

[0035] The camera is used to capture real-time images;

[0036] The processing unit is used to execute the AR-based scene navigation method described in the first aspect above;

[0037] The display screen is used to play virtual information generated by the processing unit.

[0038] Thirdly, this embodiment provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the AR-based scene navigation method described in the first aspect above.

[0039] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the AR-based scene navigation method described in the first aspect above.

[0040] Compared with related technologies, the AR-based scene navigation method, AR glasses, electronic device and storage medium provided in this embodiment solve the problems of poor scene navigation effect and high computing resource consumption in related technologies, improve the scene navigation effect of AR and reduce computing resource consumption.

[0041] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0042] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0043] Figure 1 This is a hardware structure block diagram of the terminal for the AR-based scene navigation method in this embodiment;

[0044] Figure 2 This is a flowchart of the AR-based scene navigation method in this embodiment;

[0045] Figure 3 This is a schematic diagram of the structure of the pre-trained convolutional neural network in this embodiment;

[0046] Figure 4 This is a schematic diagram of the structure of the pre-trained convolutional neural network in this preferred embodiment;

[0047] Figure 5 This is a schematic diagram of the AR glasses in this embodiment;

[0048] Figure 6 This is a flowchart illustrating the operation of the AR glasses in this embodiment. Detailed Implementation

[0049] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0050] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning as understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.

[0051] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the AR-based scene navigation method in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0052] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the AR-based scene navigation method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0053] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0054] This embodiment provides an AR-based scene navigation method. Figure 2 This is a flowchart of the AR-based scene navigation method in this embodiment, as shown below. Figure 2 As shown, the process includes the following steps:

[0055] Step S201: Obtain real-time images of the spatial area within the user's field of vision, perform target recognition processing on the real-time images, and obtain exhibit images in the real-time images.

[0056] The method in this embodiment can be applied to AR glasses. After wearing AR glasses, users can directly capture real-time images of the surrounding environment through AR glasses, preprocess the real-time images, select some candidate regions on the real-time images, extract features from these candidate regions, and then use a trained classifier to classify the extracted features. Based on the classification results, the target in the real-time image is determined.

[0057] The spatial area includes public spaces such as museums, exhibition halls, or libraries. Taking a museum as an example, when a user uses AR glasses to photograph a scene, the resulting real-time image may be a mixture of exhibits, background objects, and visitors, or more of these elements. This embodiment uses this step to distinguish the exhibits from the many other things.

[0058] Step S202: Perform feature extraction processing on the exhibit image to obtain the global feature descriptor and feature point descriptor of the exhibit image.

[0059] Directly using real-time images for feature point matching introduces numerous interfering factors, resulting in low matching accuracy and long processing times. Compared to other objects, exhibit features are relatively stable, thus the speed of feature extraction is also relatively stable. Using exhibit images for feature point matching removes many interfering factors, thereby saving computational resources for subsequent feature extraction and matching of the exhibit images.

[0060] Step S203: Obtain the global feature descriptor of the image to be retrieved; based on the global feature descriptors of the exhibit image and the image to be retrieved, determine at least one similar image that matches the exhibit image from the images to be retrieved; obtain the feature point descriptors of the similar images; based on the feature point descriptors of the exhibit image and the image to be retrieved, obtain matching feature point pairs; and determine the first similar image with the most matching feature point pairs from the similar images.

[0061] The image to be retrieved is a pre-stored exhibit image. The global feature descriptor is used to characterize the overall structure of the exhibit image. The matching time between global feature descriptors is lower than the matching time between multiple feature points. Therefore, by matching the image to be retrieved and the exhibit image through the global feature descriptor, similar images are initially screened out, which can narrow down the matching range with less matching time.

[0062] After initially identifying similar images, the matching range is narrowed down. Therefore, when performing feature point descriptor matching between similar images and exhibit images, the most matching similar images can be accurately selected without requiring a long matching time.

[0063] Step S204: Obtain the position information of the feature points in the matching feature point pair on the exhibit image; determine the user's spatial pose relative to the exhibit image based on the position information; obtain virtual information corresponding to the first similar image based on the spatial pose; and overlay the virtual information on the spatial area within the user's field of vision.

[0064] In the feature extraction process of step S202 above, in addition to extracting feature point descriptors, the position of each feature point in the image can also be extracted. In this embodiment, the essential matrix corresponding to multiple feature points can be calculated according to epipolar geometry theory, and the essential matrix can be decomposed by the singular value decomposition (SVD) algorithm to obtain the spatial pose relationship between the AR glasses' camera and the target exhibit. This allows the AR glasses to present 3D content based on different spatial poses as the viewing angle changes. For example, when the exhibit is a pottery jar, different textures of the jar can be observed from the left, front, and right sides. As long as the user's viewing angle does not move beyond a certain range, most of the feature points of the jar remain unchanged, but the essential matrix of the jar's feature points will change with the viewing angle. This allows the calculation of the user's spatial pose relative to the exhibit image and provides virtual information about the jar from different angles (e.g., magnified images of the jar at different positions), thereby enhancing the scene navigation effect and the user experience.

[0065] In some embodiments, virtual information refers to content that can be displayed on an electronic display screen, including but not limited to: text, images, videos, and audio.

[0066] Steps S201 to S204 above, in the real-time image preprocessing stage, extract exhibit images from the real-time image and perform feature matching based on the exhibit images to remove interference factors, saving computational resources for subsequent feature extraction and feature matching of exhibit images. Furthermore, by successively matching global feature descriptors and feature point descriptors, the representational ability of features on the image is enhanced, avoiding the time complexity of directly matching a unique similar image from all images to be retrieved, and avoiding the space complexity of loading the entire image into memory. When displaying virtual information of exhibit images, 3D content based on different spatial poses is presented following changes in the viewing angle, enhancing the scene navigation effect. Through these steps, the problems of poor scene navigation effect and high computational resource consumption based on AR are solved, improving the scene navigation effect of AR and reducing computational resource consumption.

[0067] In some embodiments, the virtual information may indicate the coordinates of the current exhibit in the exhibition hall as seen by the user; it may also indicate preset navigation information pointing from the current exhibit to the next exhibit; and it may simultaneously indicate the coordinates of the current exhibit in the exhibition hall as seen by the user, as well as the preset navigation information pointing from the current exhibit to the next exhibit.

[0068] For example, the virtual information of the first similar image in a certain spatial pose carries the location of an exhibition hall exit, and the preset navigation information can be pointing from the exit location of the exhibition hall to the entrance location of the next exhibition hall.

[0069] In some embodiments, the AR glasses maintain a table in the background that represents the virtual information of different exhibits. The virtual information can be updated as the exhibits move.

[0070] The feature points extracted using related technologies are manually designed, which are sensitive to changes in lighting and viewing angle, have weak generalization ability, unstable speed, and significant limitations in feature representation. To address these issues, some embodiments employ pre-trained convolutional neural networks to extract features from exhibit images. This results in features that naturally possess strong generalization and feature representation capabilities, overcoming the drawbacks of manually designed feature extraction and improving the accuracy of feature matching.

[0071] Considering the low computing power and limited battery capacity of AR glasses, in order to balance computational load and navigation effect, this problem needs to be addressed. Figure 3 A schematic diagram of the pre-trained convolutional neural network structure in this embodiment is given, as follows: Figure 3 As shown, the pre-trained convolutional neural network includes a first network and a second network, with the hidden layers of the first network connected to the input of the second network.

[0072] The exhibit image is input to the input end of the first network, and the first network performs feature extraction processing. Shallow features are output from the hidden layer of the first network, and global feature descriptors are output from the output layer of the first network.

[0073] Shallow features are input to the input of the second network, where they are processed for feature extraction. Feature point descriptors are then output from the output layer of the second network.

[0074] This embodiment combines a convolutional neural network for extracting global feature descriptors with a convolutional neural network for extracting feature point descriptors into a single network. This simplifies the complexity of deploying convolutional neural networks in AR glasses, reduces the computational load of the convolutional neural network, and significantly improves the speed of feature matching.

[0075] In this embodiment, shallow features include line information and / or edge information of the exhibit image; global feature descriptors include overall structural information of the exhibit image; and feature point descriptors include local structural information of the exhibit image.

[0076] Furthermore, in some embodiments, the shallow features are input into a second network for feature extraction, and the output feature point descriptors are achieved through the following steps:

[0077] Shallow features are input into the second network for feature extraction, resulting in the feature map output by the last convolutional layer in the second network. The feature map is then normalized to obtain the feature point response score map. Feature points in the feature point response score map that are greater than a preset threshold are identified, and feature point descriptors are output.

[0078] In this embodiment, the feature point response map contains multiple candidate feature points. The pixel value of each candidate feature point represents the probability that the pixel location is a true feature point, and its value range is 0 to 1. The feature point response map can be generated in the following way:

[0079] When supervising the training of the second network, a preset feature point response map with the same resolution as the exhibit image is set as a label. The response value of the feature point center of the preset feature point response map is 1, and the response value of the pixels near the feature point center is determined by a Gaussian distribution with a preset variance.

[0080] In this embodiment, feature point labels are generated using a heatmap method. Compared to the general method of directly regressing feature points, generating label information using a heatmap method can lead to better convergence in the training process and more stable feature point positions; it also helps to filter out feature point descriptors that do not meet preset conditions, thereby reducing the feature point descriptor matching time.

[0081] In some embodiments, the method for training a convolutional neural network includes: training a first network to obtain the network weights of the first network; and training a second network based on the network weights of the first network.

[0082] For example, when training the first network, the initial learning rate is set to 0.1, and the learning rate decays to 0.1 times the previous rate every 10 training cycles, ending training when the learning rate reaches 0.0001.

[0083] After the first network is trained, its weights are frozen, and the second network is trained jointly. When training the second network, the initial learning rate is set to 0.01, and every 5 training epochs the learning rate decays to 0.1 times the previous rate, ending training when it reaches 0.0001. Specifically, the weight ratio of the hidden layer processing feature point descriptors can be set to 0.8, and the weight ratio of the hidden layer processing feature point response maps can be set to 0.2.

[0084] Figure 4 This is a schematic diagram of the structure of the pre-trained convolutional neural network in this preferred embodiment, as shown below. Figure 4 As shown, the pre-trained convolutional neural network includes:

[0085] The first network and the second network are connected, with the hidden layer of the first network and the input of the second network connected.

[0086] The labels shown in the diagram represent the attributes of the convolution kernel. Conv and MBConv both represent convolution kernels, with Conv representing a depthwise separable convolution kernel; Channels represent the number of channels; KernelSize represents the kernel size; and Stride represents the kernel stride. For example, a convolution kernel might be defined as "16, <3×3>, 1", where 16 represents the number of channels, <3×3> represents the kernel size, and 1 represents the kernel stride.

[0087] In this preferred embodiment, the global feature descriptor is a one-dimensional vector of length 512. The first network can be trained using a large-scale landmark dataset, such as Google Landmarks Dataset v2. The classification loss function L1 used by the first network is as follows:

[0088]

[0089] Where N represents the number of samples in a batch during the training process (BatchSize).

[0090] i represents the batch number of the current sample batch.

[0091] s represents the feature scale factor.

[0092] m1 represents the multiplicative angular distance factor, m2 represents the additive angular distance factor, and m3 represents the additive cosine distance factor.

[0093] y i This represents the category to which the sample with ID i belongs.

[0094] θ represents the angle between the feature vector of the current sample i and the boundary weight of that class.

[0095] j represents the number of all training samples.

[0096] The classification loss function performs feature normalization and weight normalization, and constrains the feature boundaries through m1, m2, and m3, so that the learned global feature descriptor has stronger expressive power.

[0097] In the second network, the classification loss function L2 used to output the feature point response score map is as follows:

[0098]

[0099]

[0100] Where G represents a Gaussian distribution.

[0101] i represents the x-coordinate on the feature map.

[0102] j represents the y-coordinate on the feature map.

[0103] k represents the feature point number.

[0104] σ represents variance.

[0105] y represents the information of each feature point in the image.

[0106] This represents the true value of the training sample.

[0107] ‖‖ represents taking the L2 norm of the feature distance.

[0108] In the second network, the classification loss function L3 used to output the feature point descriptor is as follows:

[0109]

[0110] Where N represents the number of samples in a batch during the training process (BatchSize).

[0111] x represents the coordinates of the sampling point in the first image, and x′ represents the coordinates of the sampling point in the second image.

[0112] i represents the sampling number.

[0113] S represents a training sample pair, where S=1 represents a positive training sample pair and S=0 represents a negative training sample pair.

[0114] F represents the local feature at x.

[0115] ‖‖ represents taking the L2 norm of the feature distance.

[0116] This embodiment also provides AR glasses. Figure 5 This is a schematic diagram of the AR glasses in this embodiment, as shown below. Figure 5 As shown, the AR glasses include: a camera 51, a display screen 52, and a processing unit 53. The processing unit 53 is connected to the camera 51 and the display screen 52. The camera 51 is used to capture real-time images. The processing unit 53 is used to execute the AR-based scene navigation method of the above embodiment. The display screen 52 is used to play virtual information generated by the processing unit 53.

[0117] The present embodiment will now be described and illustrated through preferred embodiments.

[0118] For example, for an exhibit A in a museum, 10 photos of exhibit A are pre-stored, and the features of each photo of exhibit A are extracted using the pre-trained convolutional neural network in this application. This yields a global feature descriptor, feature point descriptor, and feature point response score map for each photo of exhibit A, which are then packaged and added to the image library to be retrieved.

[0119] When a user wears AR glasses to visit exhibit A on-site, the AR glasses will use methods such as... Figure 6 The process shown implements AR-based scene navigation, such as... Figure 6 As shown, the process includes the following steps:

[0120] Step S61: Take a picture of exhibit A to obtain a real-time image of exhibit A.

[0121] Step S62: Extract exhibit A from the real-time image to obtain the exhibit image.

[0122] Step S63: Use a pre-trained convolutional neural network to perform feature extraction processing on the exhibit image to obtain the global feature descriptor, feature point descriptor and feature point response score map of the exhibit image.

[0123] Step S64: Perform preliminary matching using global feature descriptors to retrieve the 6 most similar images that best match the exhibit images from the image database to be searched.

[0124] Step S65: Set a threshold T, filter out points in the feature point response score map that reach the threshold T as feature points, and obtain the feature point descriptors of these feature points.

[0125] Step S66: Use the feature point descriptors of the selected feature points to perform precise matching, and select the first similar image with the most matching feature point pairs from the 6 similar images.

[0126] Step S67: Obtain the position information of the feature points in the matching feature point pair on the exhibit image; determine the user's spatial pose relative to the exhibit image based on the position information; obtain virtual information corresponding to the first similar image based on the spatial pose; and overlay the virtual information onto the AR glasses.

[0127] This embodiment also provides an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0128] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0129] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0130] S1: Acquire real-time images of the spatial area within the user's field of view, perform target recognition processing on the real-time images, and obtain the exhibit images in the real-time images.

[0131] S2 performs feature extraction processing on the exhibit image to obtain the global feature descriptor and feature point descriptor of the exhibit image.

[0132] S3, obtain the global feature descriptor of the image to be retrieved, and determine at least one similar image that matches the image to be retrieved based on the global feature descriptors of the exhibit image and the image to be retrieved; obtain the feature point descriptors of the similar images, obtain matching feature point pairs based on the feature point descriptors of the exhibit image and the image to be retrieved, and determine the first similar image with the most matching feature point pairs from the similar images.

[0133] S4, obtain the position information of the feature points in the matching feature point pair on the exhibit image, determine the user's spatial pose relative to the exhibit image based on the position information, obtain virtual information corresponding to the first similar image based on the spatial pose, and overlay the virtual information on the spatial area within the user's field of vision.

[0134] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0135] Furthermore, in conjunction with the AR-based scene navigation method provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the AR-based scene navigation methods described in the above embodiments.

[0136] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0137] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0138] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0139] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. An AR-based scene navigation method, characterized in that, include: Real-time images of the spatial area within the user's field of vision are obtained through AR glasses, and target recognition processing is performed on the real-time images to obtain exhibit images in the real-time images; A pre-trained convolutional neural network is used to perform feature extraction processing on the exhibit image to obtain the global feature descriptor and feature point descriptor of the exhibit image. The pre-trained convolutional neural network includes a first network and a second network, and the hidden layer of the first network is connected to the input end of the second network. Obtain the global feature descriptor of the image to be retrieved; based on the global feature descriptor of the exhibit image and the image to be retrieved, determine at least one similar image that matches the exhibit image from the images to be retrieved; obtain the feature point descriptor of the similar image; based on the feature point descriptor of the exhibit image and the image to be retrieved, obtain matching feature point pairs; and determine the first similar image with the most matching feature point pairs from the similar images. The position information of the feature points in the matching feature point pair on the exhibit image is obtained, and the spatial pose of the user relative to the exhibit image is determined based on the position information; virtual information corresponding to the first similar image is obtained based on the spatial pose, and the virtual information is superimposed and displayed in the spatial area within the user's field of vision; The step of obtaining the position information of the feature points in the matching feature point pair on the exhibit image, determining the user's spatial pose relative to the exhibit image based on the position information, and obtaining virtual information corresponding to the first similar image based on the spatial pose, and superimposing the virtual information on the spatial area within the user's field of vision, further includes: Based on epipolar geometry theory, the essential matrix corresponding to multiple feature points is calculated, and the essential matrix is ​​decomposed by singular value decomposition algorithm to obtain the spatial pose relationship between the camera of AR glasses and the target exhibit. Thus, AR glasses can present virtual information based on different spatial poses as the viewing angle changes.

2. The AR-based scene navigation method according to claim 1, characterized in that, The virtual information includes the coordinates of the current exhibit within the exhibition hall, as seen from the user's field of vision.

3. The AR-based scene navigation method according to claim 1 or 2, characterized in that, The virtual information includes preset navigation information that guides you from the current exhibit to the next exhibit.

4. The AR-based scene navigation method according to claim 1, characterized in that, The feature extraction process for the exhibit images using a pre-trained convolutional neural network includes: The exhibit image is input into the first network for feature extraction processing, and shallow features and the global feature descriptor are output. The shallow features are input into the second network for feature extraction processing, and the feature point descriptors are output.

5. The AR-based scene navigation method according to claim 4, characterized in that, The shallow features are input into the second network for feature extraction, and the output feature point descriptors include: The shallow features are input into the second network for feature extraction processing to obtain the feature map output by the last convolutional layer in the second network. The feature map is normalized to obtain the feature point response score map; Identify feature points in the feature point response score map that are greater than a preset threshold, and output the feature point descriptor of the feature points.

6. The AR-based scene navigation method according to claim 4, characterized in that, The shallow features include line information and / or edge information of the exhibit image; the global feature descriptor includes the overall structural information of the exhibit image; and the feature point descriptor includes the local structural information of the exhibit image.

7. The AR-based scene navigation method according to claim 4, characterized in that, The methods for training the convolutional neural network include: Train the first network to obtain its network weights; The second network is trained based on the network weights of the first network.

8. An AR glasses, characterized in that, include: The system includes a camera, a display screen, and a processing unit, wherein the processing unit is connected to the camera and the display screen; wherein... The camera is used to capture real-time images; The processing unit is used to execute the AR-based scene navigation method according to any one of claims 1 to 7; The display screen is used to play virtual information generated by the processing unit.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the AR-based scene navigation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the AR-based scene navigation method according to any one of claims 1 to 7.

Images