Virtual positioning method and device based on AR / VR / MR glasses and electronic equipment

Abstract

The invention provides a virtual positioning method and device based on AR / VR / MR glasses and electronic equipment, and the method comprises the steps: obtaining a first image, first positioning information, first feature points and first reference feature points, carrying out the feature point matching of all first feature points, and constructing a three-dimensional point cloud model; then obtaining a second image, second positioning information, a second feature point and a second reference feature point, and determining a target image from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information and the second reference feature point; and finally, obtaining a matched target feature point in the first feature point and the second feature point corresponding to the target image, and determining virtual positioning information of the user in the three-dimensional point cloud model based on the target feature point. Therefore, when the positioning accuracy is met, the target image can be quickly obtained from the first image, and then the position of the user can be quickly positioned.

Description

Technical Field

[0001] The present invention relates to the field of visual positioning technology, and in particular to a virtual positioning method, device and electronic equipment based on AR / VR / MR glasses. Background Art

[0002] With the rapid development of science and technology, virtual reality (VR), augmented reality (AR) and mixed reality (MR) technologies have received widespread attention and application. These technologies generate virtual information through computers and integrate it with the real world to create an immersive experience for users, showing great application prospects in education, medical care, industry, entertainment and other fields.

[0003] In order to provide users with a good immersive experience, high-precision and fast positioning is very important. However, most existing positioning technologies can only take into account one aspect at a time. Therefore, how to ensure positioning accuracy while quickly positioning is a major problem in the fields of virtual reality (VR), augmented reality (AR) and mixed reality (MR). Summary of the invention

[0004] In view of the shortcomings of the prior art, the present application provides a virtual positioning method, device and electronic device based on AR / VR / MR glasses, aiming to solve the technical problem that the prior art cannot take into account both positioning accuracy and positioning speed. Specifically: In a first aspect, an embodiment of the present application provides a virtual positioning method based on AR / VR / MR glasses, the method comprising: obtaining a first image corresponding to a target scene and first positioning information corresponding to the first image, and performing feature extraction on the first image to determine a first feature point and a first reference feature point; performing feature point matching on all of the first feature points, and constructing a three-dimensional point cloud model based on the matching results; obtaining a second image corresponding to a real-time scene and second positioning information corresponding to the second image, and performing feature extraction on the second image to determine a second feature point and a second reference feature point; determining a target image from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information, and the second reference feature point; obtaining the first feature point corresponding to the target image and the target feature point matched in the second feature point, and determining the virtual positioning information of the user in the three-dimensional point cloud model based on the target feature point.

[0005] Furthermore, determining the target image from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information and the second reference feature point includes: determining whether there is a first reference feature point whose feature matching degree with the second reference feature point exceeds a preset threshold; if so, determining the target image from the first image based on the first reference feature point and the second reference feature point; if not, determining the target image from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information and the second reference feature point.

[0006] Furthermore, determining the target image from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information and the second reference feature point includes: obtaining the positioning matching results of the first positioning information and the second positioning information; obtaining the feature matching results of the first reference feature point and the second reference feature point; determining the matching priority based on the positioning matching results, the positioning weight, the feature matching results and the feature weight; wherein the positioning weight is greater than the feature weight; matching the second reference feature point and the first feature point in sequence based on the matching priority, stopping the matching when a first preset number of matches with a degree of match lower than a preset value appear continuously, and obtaining the first image corresponding to the first feature point with the highest degree of match from the matched results as the target image.

[0007] Furthermore, determining the target image from the first image based on the first reference feature point and the second reference feature point includes: taking the first image corresponding to the first reference feature point whose matching degree with the second reference feature point exceeds the preset threshold as the third image; and taking the third image corresponding to the first reference feature point whose matching degree with the second reference feature point is the highest as the target image.

[0008] Furthermore, performing feature point matching on all the first feature points and constructing a three-dimensional point cloud model according to the matching results includes: performing feature point matching on all the first feature points in pairs; Initial point cloud data is constructed based on feature points matched between different first feature points; the initial point cloud data having a feature point number less than a preset number is removed, and a three-dimensional point cloud model is constructed based on the remaining initial point cloud data.

[0009] Furthermore, determining the virtual positioning information of the user in the three-dimensional point cloud model based on the target feature points includes: performing coordinate matching on the target feature points in the first feature points and the second feature points corresponding to the target image based on a preset algorithm; and determining the virtual positioning information of the user in the three-dimensional point cloud model based on the coordinate matching results.

[0010] Furthermore, before obtaining the first image corresponding to the target scene and the first positioning information corresponding to the first image, the method further includes: obtaining an image set corresponding to the marked image, and obtaining an image corner point set of the image set; and determining camera parameters based on the image corner point set.

[0011] Furthermore, the first positioning information and the second positioning information are any one of GPS positioning data, Bluetooth positioning data, and WIFI positioning data.

[0012] In a second aspect, an embodiment of the present application provides a virtual positioning device based on AR / VR / MR glasses, characterized in that the device includes: a first acquisition module, used to acquire a first image corresponding to a target scene and a first positioning information corresponding to the first image, and perform feature extraction on the first image to determine a first feature point and a first reference feature point; a construction module, used to perform feature point matching on all the first feature points, and construct a three-dimensional point cloud model based on the matching results; a second acquisition module, used to acquire a second image corresponding to a real-time scene and a second positioning information corresponding to the second image, and perform feature extraction on the second image to determine a second feature point and a second reference feature point; a target acquisition module, used to determine a target image from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information and the second reference feature point; a positioning module, used to acquire the first feature point corresponding to the target image and the target feature point matched in the second feature point, and determine the virtual positioning information of the user in the three-dimensional point cloud model based on the target feature point.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: one or more processors, a memory, and one or more applications. The one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method of the first aspect.

[0014] In the technical solution provided in the present application, the first image corresponding to the target scene and the first positioning information corresponding to the first image are obtained, and the first feature points and the first reference feature points are determined by feature extraction of the first image; all the first feature points are feature point matched, and a three-dimensional point cloud model is constructed according to the matching results; the second image corresponding to the real-time scene and the second positioning information corresponding to the second image are obtained, and the second feature points and the second reference feature points are determined by feature extraction of the second image; based on the first positioning information, the first feature points, the first reference feature points, the second positioning information and the second reference feature points, the target image is determined from the first image; the target feature points matched in the first feature points and the second feature points corresponding to the target image are obtained, and the virtual positioning information of the user in the three-dimensional point cloud model is determined based on the target feature points. Thus, while satisfying the positioning accuracy, the target image can be quickly obtained from the first image and the user's position can be quickly located. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments and drawings obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] Figure 1 A schematic flow chart of a virtual positioning method based on AR / VR / MR glasses proposed in an embodiment of the present application is shown.

[0017] Figure 2 A structural block diagram of a virtual positioning device based on AR / VR / MR glasses proposed in an embodiment of the present application is shown.

[0018] Figure 3 A structural block diagram of an electronic device proposed in an embodiment of the present application is shown.

[0019] Figure 4 A structural block diagram of a computer-readable storage medium proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0022] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0023] It should be further understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0024] With the rapid development of science and technology, virtual reality (VR), augmented reality (AR) and mixed reality (MR) technologies have received widespread attention and application.

[0025] Virtual reality (VR) is the combination of virtuality and reality (using specific technology to generate a virtual situation, but people feel it is real). Theoretically speaking, virtual reality technology (VR) is a computer simulation system that can create and experience a virtual world. It uses computers to generate a simulated environment and immerse users in it. Virtual reality technology uses real-life data and electronic signals generated by computer technology, and combines them with various output devices to transform them into phenomena that people can feel. These phenomena can be real objects in reality or substances that are invisible to our naked eyes, expressed through three-dimensional models. Because these phenomena are not what we can see directly, but the real world simulated by computer technology, it is called virtual reality.

[0026] Augmented reality (AR) is a technology that superimposes and integrates virtual scenes or information with the real physical environment, and interactively presents them to users, thus creating a space where virtual and real things share the same space. In essence, augmented reality is a new interface technology that integrates software and hardware technologies such as positioning, presentation, and interaction. Its purpose is to allow users to sense the spatial and temporal connection and integration of virtual and real spaces, thereby enhancing their perception and cognition of the real environment.

[0027] Mixed reality (MR) is a further development of virtual reality technology. This technology introduces real-world scene information into the virtual environment, builds an interactive feedback information loop between the virtual world, the real world and the user, and enhances the realism of the user experience.

[0028] These technologies create an immersive experience for users by integrating computer-generated virtual information with the real world, showing great application prospects in education, medical care, industry, entertainment and other fields. In order to enable users to have a good immersive experience, high-precision and fast positioning is very important. However, most of the existing positioning technologies can only take one aspect into account. Therefore, how to quickly locate while ensuring positioning accuracy is a major problem in the fields of virtual reality (VR), augmented reality (AR) and mixed reality (MR).

[0029] Therefore, in order to solve the problems existing in the prior art, the present application proposes a virtual positioning method, device and electronic device based on AR / VR / MR glasses, which are described below in conjunction with specific embodiments.

[0030] See also Figure 1 , Figure 1 A virtual positioning method based on AR / VR / MR glasses provided in an embodiment of the present application is shown, and the method is applied to AR / VR / MR glasses. Specifically: In step S110, a first image corresponding to the target scene and first positioning information corresponding to the first image are acquired, and feature extraction is performed on the first image to determine a first feature point and a first reference feature point.

[0031] In the embodiment of the present application, the target scene is an indoor scene, and the first image is an image obtained by photographing the indoor scene. It is understandable that since a three-dimensional point cloud model corresponding to the target scene needs to be constructed based on the first image, and the imaging range of AR / VR / MR glasses has not yet reached 360°, a complete three-dimensional point cloud model cannot be constructed based on one image, that is, the number of first images is at least greater than 1. Furthermore, in order to make the subsequently constructed three-dimensional point cloud model more complete, there is an overlapping area between two adjacent images when the first image is collected.

[0032] In some implementations, images may be captured using a camera in AR / VR / MR glasses.

[0033] In the embodiment of the present application, the first positioning information is used to characterize the location where the first image is taken, the first feature point is all the feature points in the first image, the first reference feature point is the feature point of the feature point dense area in the first image, and the first reference feature point belongs to the first feature point. It can be understood that the feature point dense area is generally a relatively special area in the image, such as a house on an open space, a bird in the sky, etc.

[0034] In some implementations, the first positioning information may be determined based on any one of GPS positioning data, Bluetooth positioning data, and WIFI positioning data.

[0035] In some implementations, before step S110, the camera parameters need to be calibrated. Specifically, first obtain the image set corresponding to the marked image, then obtain the image corner point set of the image set, and finally determine the camera parameters using the image corner point set. The marked image is a chessboard drawing, the image set includes images obtained by the camera taking the chessboard drawing from different angles and distances, and the corner point set includes the chessboard corner point information on each image in the image set.

[0036] In some implementations, when capturing the first image, the user may also obtain the aperture, white balance, ISO, shooting date, etc., of the camera when the first image was captured.

[0037] In step S120, feature point matching is performed on all first feature points, and a three-dimensional point cloud model is constructed according to the matching results.

[0038] In an embodiment of the present application, after obtaining sufficient first images corresponding to the target scene and first feature points corresponding to each first image, all first feature points are matched against each other in pairs, initial point cloud data is constructed based on the feature points matched between different first feature points, initial point cloud data with a number of feature points less than a second preset number is removed, and a three-dimensional point cloud model is constructed based on the remaining initial point cloud data.

[0039] Specifically, the matched feature points among the different first feature points are used as an initial point cloud data, and then the initial point cloud data with a feature point number less than a second preset number is filtered out, and a three-dimensional point cloud model is constructed based on the remaining initial point cloud data. It can be understood that the number of feature points in the initial point cloud data is less than the second preset number, indicating that the feature points in the initial point cloud data rarely appear in the first image, so these feature points can be considered as non-valid feature points and can be directly deleted.

[0040] Exemplarily, if feature points A in images d, f, g, h, j, k, l, w, e, r, t match each other, feature points A and images d, f, g, h, j, k, l, w, e, r, t are saved in an initial point cloud data.

[0041] In step S130, a second image corresponding to the real-time scene and second positioning information corresponding to the second image are obtained, and feature extraction is performed on the second image to determine second feature points and second reference feature points.

[0042] In the embodiment of the present application, the real-time scene is the scene seen in the user's field of view when experiencing AR / VR / MR, the second image is the scene image corresponding to the scene seen in the user's field of view, the second positioning information is used to characterize the location where the second image is taken, the first feature point is all the feature points in the first image, the first reference feature point is the feature point of the feature point-dense area in the first image, and the first reference feature point belongs to the first feature point. It can be understood that the feature point-dense area is generally a relatively special area in the image, such as a house on an open space, a bird in the sky, etc.

[0043] In some implementations, the first positioning information may be determined based on any one of GPS positioning data, Bluetooth positioning data, and WIFI positioning data.

[0044] In some implementations, when capturing the second image, the user may also obtain the aperture, white balance, ISO, shooting date, etc., of the camera when the second image was shot.

[0045] In step S140, a target image is determined from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information, and the second reference feature point.

[0046] In the embodiment of the present application, it can be determined first whether there is a first reference feature point whose feature matching degree with the second reference feature point exceeds a preset threshold; if so, the target image is determined from the first image based on the first reference feature point and the second reference feature point; if not, the target image is determined from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information, and the second reference feature point. Thus, matching the first reference feature point and the second reference feature point can reduce the amount of calculation required for matching, and it can also be selected according to whether there is a first reference feature point whose feature matching degree with the second reference feature point exceeds a preset threshold to determine the target image from the first image in the future.

[0047] Furthermore, if there is no first reference feature point whose feature matching degree with the second reference feature point exceeds the preset threshold, it means that the feature point dense area in the first image for constructing the three-dimensional point cloud model does not match the feature point dense area in the second image, so it is necessary to consider the second reference feature point and all the feature points corresponding to each first image, that is, the first feature point, for feature matching. However, if the matching is performed directly and randomly, it is difficult to quickly match the required image, so it is necessary to establish a reasonable matching priority rule so that the amount of calculation can be reduced and the positioning speed can be improved.

[0048] As an implementation method, a target image can be determined from a first image based on first positioning information, a first feature point, a first reference feature point, a second positioning information and a second reference feature point, including: obtaining positioning matching results of the first positioning information and the second positioning information; obtaining feature matching results of the first reference feature point and the second reference feature point; determining a matching priority based on the positioning matching results, positioning weights, feature matching results and feature weights; wherein the positioning weight is greater than the feature weight; matching the second reference feature point and the first feature point in sequence based on the matching priority, stopping matching when a first preset number of matches with a degree of match lower than a preset value appear consecutively, and obtaining the first image corresponding to the first feature point with the highest degree of match from the matched results as the target image.

[0049] It can be understood that since both the first image and the second image have positioning information, it can be considered that the images taken at similar positions will have a higher degree of matching, so the positioning information can be used as an important indicator for priority sorting; when the first preset number of consecutive matches with a degree lower than the preset value appear, it can be considered that the results of subsequent matches will become lower and lower, so the subsequent first feature points may no longer be matched.

[0050] In some embodiments, when there is a first reference feature point whose feature matching degree with the second reference feature point exceeds a preset threshold, the first image corresponding to the first reference feature point with the highest matching degree can be directly obtained as the target image. Specifically, the first image corresponding to the first reference feature point with a matching degree with the second reference feature point exceeding the preset threshold is used as the third image, and the third image corresponding to the first reference feature point with the highest matching degree with the second reference feature point is used as the target image.

[0051] In step S150, the target feature points that match the first feature points and the second feature points corresponding to the target image are obtained, and the virtual positioning information of the user in the three-dimensional point cloud model is determined based on the target feature points.

[0052] In an embodiment of the present application, when there is no need to render the user virtual character model into the three-dimensional point cloud model, the position of the glasses can be used as the user's position, so the virtual positioning model can be the three-dimensional position and orientation of the glasses in the three-dimensional point cloud model.

[0053] In some embodiments, after obtaining the target feature points that match each other in the target image and the second image, coordinate matching can be performed on the target feature points in the first feature points and the second feature points corresponding to the target image based on a preset algorithm; and the virtual positioning information of the user in the three-dimensional point cloud model can be determined based on the coordinate matching result. The preset algorithm can be, for example, an ICP algorithm.

[0054] In some embodiments, when it is necessary to render the user's virtual character model into a three-dimensional point cloud model, the virtual positioning model can be considered as the three-dimensional position and orientation of the user itself in the three-dimensional point cloud model. To obtain the three-dimensional position and orientation of the user itself in the three-dimensional point cloud model, it is necessary to obtain the user's height, the relative position and orientation relationship between the camera and the user, and then determine the user's three-dimensional position and orientation based on the height, relative position and orientation relationship, and the three-dimensional position and orientation of the AR / VR / MR glasses obtained above.

[0055] In the technical solution provided in the present application, the first image corresponding to the target scene and the first positioning information corresponding to the first image are obtained, and the first feature points and the first reference feature points are determined by feature extraction of the first image; all the first feature points are feature point matched, and a three-dimensional point cloud model is constructed according to the matching results; the second image corresponding to the real-time scene and the second positioning information corresponding to the second image are obtained, and the second feature points and the second reference feature points are determined by feature extraction of the second image; based on the first positioning information, the first feature points, the first reference feature points, the second positioning information and the second reference feature points, the target image is determined from the first image; the target feature points matched in the first feature points and the second feature points corresponding to the target image are obtained, and the virtual positioning information of the user in the three-dimensional point cloud model is determined based on the target feature points. Thus, while satisfying the positioning accuracy, the target image can be quickly obtained from the first image and the user's position can be quickly located.

[0056] See also Figure 2 , Figure 2 A virtual positioning device 100 based on AR / VR / MR glasses proposed in an embodiment of the present application is shown, and the device includes: a first acquisition module 110, a construction module 120, a second acquisition module 130, a target acquisition module 140 and a positioning module 150. Specifically: A first acquisition module 110 is used to acquire a first image corresponding to a target scene and first positioning information corresponding to the first image, and perform feature extraction on the first image to determine a first feature point and a first reference feature point; A construction module 120, configured to perform feature point matching on all first feature points and construct a three-dimensional point cloud model according to the matching results; The second acquisition module 130 is used to acquire a second image corresponding to the real-time scene and second positioning information corresponding to the second image, and perform feature extraction on the second image to determine a second feature point and a second reference feature point; A target acquisition module 140, configured to determine a target image from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information, and the second reference feature point; The positioning module 150 is used to obtain the matching target feature points from the first feature points and the second feature points corresponding to the target image, and determine the virtual positioning information of the user in the three-dimensional point cloud model based on the target feature points.

[0057] In some embodiments, the target acquisition module 140 may include: a judgment submodule, used to determine whether there is a first reference feature point whose feature matching degree with the second reference feature point exceeds a preset threshold; a first determination submodule, used to determine the target image from the first image based on the first reference feature point and the second reference feature point if it exists; and a second determination submodule, used to determine the target image from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information and the second reference feature point if it does not exist.

[0058] Furthermore, the second determination subunit can also be used to: obtain feature matching results of the first reference feature point and the second reference feature point; determine the matching priority based on the positioning matching results, the positioning weight, the feature matching results and the feature weight; wherein the positioning weight is greater than the feature weight; match the second reference feature point and the first feature point in sequence based on the matching priority, stop matching when a first preset number of matches with a degree lower than a preset value appear consecutively, and obtain the first image corresponding to the first feature point with the highest degree of matching from the matched results as the target image.

[0059] Furthermore, the first determination subunit can also be used to: use the first image corresponding to the first reference feature point whose matching degree with the second reference feature point exceeds a preset threshold as the third image; and use the third image corresponding to the first reference feature point whose matching degree with the second reference feature point is highest as the target image.

[0060] Furthermore, the construction module 120 may also include: a matching submodule, used to match all the first feature points with each other in pairs; a first construction submodule, used to construct initial point cloud data based on the feature points matched between different first feature points; a second construction submodule, used to remove the initial point cloud data with a number of feature points less than a preset number, and construct a three-dimensional point cloud model based on the remaining initial point cloud data.

[0061] Furthermore, the positioning module 150 can also be used to: perform coordinate matching on the target feature points in the first feature points and the second feature points corresponding to the target image based on a preset algorithm; and determine the virtual positioning information of the user in the three-dimensional point cloud model based on the coordinate matching result.

[0062] Furthermore, the virtual positioning device 100 based on AR / VR / MR glasses can also include: an acquisition module, used to acquire an image set corresponding to the marked image, and acquire an image corner point set of the image set; a parameter determination module, used to determine camera parameters based on the image corner point set.

[0063] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Any processing method described in the method embodiment can be implemented by the corresponding processing module in the device embodiment, and will not be repeated one by one in the device embodiment.

[0064] See also Figure 3 , Figure 3 An electronic device 200 proposed in an embodiment of the present application is shown, and the electronic device 200 may be AR / VR / MR glasses, a head display device, etc. The electronic device 200 in the present application may include one or more of the following components: a processor 210, a memory 220, and one or more applications, wherein the one or more applications may be stored in the memory 220 and configured to be executed by one or more processors 210, and the one or more applications are configured to execute the method described in the foregoing method embodiment.

[0065] The processor 210 may include one or more processing cores. The processor 210 uses various interfaces and lines to connect various parts of the entire electronic device 200, and executes various functions and processes data of the electronic device 200 by running or executing instructions, programs, code sets or instruction sets stored in the memory 220, and calling data stored in the memory 220. Optionally, the processor 210 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 210 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 210, but may be implemented separately through a communication chip.

[0066] The memory 220 may include a random access memory (RAM) or a read-only memory (ROM). The memory 220 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 220 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a positioning function, a model building function, a matching function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a first image, a first positioning information, a second image, a second positioning information, etc.) created by the electronic device 200 during use.

[0067] See also Figure 4 , Figure 4 The structure block diagram of a computer-readable storage medium proposed in the embodiment of the present application is shown. The computer-readable storage medium 300 stores program codes, and the program codes can be called by a processor to execute the method described in the above method embodiment.

[0068] The computer-readable storage medium 300 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. Optionally, the computer-readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 300 has storage space for program code 310 for executing any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 310 can be compressed, for example, in a suitable form.

[0069] The present application provides a virtual positioning method, device and electronic device based on AR / VR / MR glasses, which obtains a first image corresponding to a target scene and first positioning information corresponding to the first image, and extracts features from the first image to determine first feature points and first reference feature points; performs feature point matching on all first feature points, and builds a three-dimensional point cloud model based on the matching results; obtains a second image corresponding to a real-time scene and second positioning information corresponding to the second image, and extracts features from the second image to determine second feature points and second reference feature points; determines a target image from the first image based on the first positioning information, first feature points, first reference feature points, second positioning information and second reference feature points; obtains the target feature points matched in the first feature points and second feature points corresponding to the target image, and determines the virtual positioning information of the user in the three-dimensional point cloud model based on the target feature points. Thus, while satisfying the positioning accuracy, the target image can be quickly obtained from the first image and the user's position can be quickly located.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A virtual positioning method based on AR / VR / MR glasses, characterized in that: The method comprises: Acquire a first image corresponding to a target scene and first positioning information corresponding to the first image, and perform feature extraction on the first image to determine a first feature point and a first reference feature point; Performing feature point matching on all the first feature points, and constructing a three-dimensional point cloud model according to the matching results; Acquire a second image corresponding to the real-time scene and second positioning information corresponding to the second image, and perform feature extraction on the second image to determine second feature points and second reference feature points; Determine a target image from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information, and the second reference feature point; Acquire the first feature points corresponding to the target image and the target feature points matched in the second feature points, and determine the virtual positioning information of the user in the three-dimensional point cloud model based on the target feature points.

2. The method according to claim 1, characterized in that The determining a target image from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information, and the second reference feature point includes: Determine whether there is a first reference feature point with a feature matching degree exceeding a preset threshold with the second reference feature point; If so, determining a target image from the first image based on the first reference feature points and the second reference feature points; If not, a target image is determined from the first image based on the first positioning information, the first feature points, the first reference feature points, the second positioning information and the second reference feature points.

3. The method according to claim 2, characterized in that The determining a target image from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information, and the second reference feature point includes: Obtaining a positioning matching result of the first positioning information and the second positioning information; Obtaining a feature matching result of the first reference feature point and the second reference feature point; Determining a matching priority based on the positioning matching result, the positioning weight, the feature matching result and the feature weight; wherein the positioning weight is greater than the feature weight; The second reference feature point and the first feature point are matched in sequence based on the matching priority, matching is stopped when a first preset number of matches with a degree lower than a preset value appear consecutively, and the first image corresponding to the first feature point with the highest degree of match is obtained from the matched results as the target image.

4. The method according to claim 2, characterized in that: The determining a target image from the first image based on the first reference feature points and the second reference feature points comprises: taking the first image corresponding to the first reference feature point whose matching degree with the second reference feature point exceeds the preset threshold as the third image; The third image corresponding to the first reference feature point with the highest matching degree with the second reference feature point is used as the target image.

5. The method according to claim 1, characterized in that The step of matching all the first feature points and constructing a three-dimensional point cloud model according to the matching results includes: Matching all the first feature points with each other in pairs; constructing initial point cloud data based on feature points matched between different first feature points; The initial point cloud data having a number of feature points less than a second preset number is removed, and a three-dimensional point cloud model is constructed based on the remaining initial point cloud data.

6. The method according to claim 1, characterized in that The determining the virtual positioning information of the user in the three-dimensional point cloud model based on the target feature point includes: Performing coordinate matching of the target feature points in the first feature points and the second feature points corresponding to the target image based on a preset algorithm; The virtual positioning information of the user in the three-dimensional point cloud model is determined based on the coordinate matching result.

7. The method according to claim 6, characterized in that Before acquiring the first image corresponding to the target scene and the first positioning information corresponding to the first image, the method further includes: Obtain an image set corresponding to the marked image, and obtain an image corner point set of the image set; Camera parameters are determined based on the set of image corner points.

8. The method according to any one of claims 1 to 7, characterized in that: The first positioning information and the second positioning information are determined according to any one of GPS positioning data, Bluetooth positioning data, and WIFI positioning data.

9. A virtual positioning device based on AR / VR / MR glasses, characterized in that: The device comprises: A first acquisition module, used to acquire a first image corresponding to a target scene and first positioning information corresponding to the first image, and perform feature extraction on the first image to determine a first feature point and a first reference feature point; A construction module, used for performing feature point matching on all the first feature points, and constructing a three-dimensional point cloud model according to the matching results; A second acquisition module is used to acquire a second image corresponding to the real-time scene and second positioning information corresponding to the second image, and perform feature extraction on the second image to determine a second feature point and a second reference feature point; a target acquisition module, configured to determine a target image from the first image based on the first positioning information, the first feature point, the first reference feature point, the second positioning information, and the second reference feature point; A positioning module is used to obtain the first feature points corresponding to the target image and the target feature points matched in the second feature points, and determine the virtual positioning information of the user in the three-dimensional point cloud model based on the target feature points.

10. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1-8.

Images