A method and device for guiding augmented reality, electronic equipment, and storage medium

Abstract

This application discloses an augmented reality guidance method applied to an augmented reality device. The method includes: acquiring a real-time environmental image and the focus of a user's gaze in the environmental image; determining a sequence of guidance markers based on the real-time environmental image; and displaying at least one guidance animation based on the guidance marker sequence. Each guidance animation is used to guide the user's gaze focus from the current guidance marker to the next guidance marker. This application also provides an augmented reality guidance device, an electronic device, and a storage medium.

Description

Technical Field

[0001] The present application relates to the technical field of electronic devices, and is related to but not limited to an augmented reality guidance method and device, electronic device, and storage medium. Background Art

[0002] With the advancement of technology, augmented reality (AR) technology has entered the lives of users more and brought new experiences to users. AR technology can apply virtual objects to real environments, so that real environments and virtual objects are superimposed on the same screen or space in real time. The application of AR technology in combination with computer vision algorithms in AR devices can perform object recognition and environment recognition on scenes displayed in AR devices, increasing the intelligence and technological sense of AR devices. Related technologies guide users to observe the environment by displaying two-dimensional scene pictures or three-dimensional scene models that need to be recognized in AR devices, such as AR glasses. However, this process requires the production of two-dimensional scene pictures or three-dimensional scene models for different scenes, which is a large workload; at the same time, due to the small field of view (FOV) of the AR device, displaying two-dimensional scene pictures or three-dimensional scene models will occupy most of the area in the field of view, resulting in a poor user experience. Summary of the invention

[0003] Embodiments of the present application provide an augmented reality guidance method and device, an electronic device, and a storage medium.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides an augmented reality guidance method, which is applied to an augmented reality device, and the method includes:

[0006] Acquire a real-time environment image and a focus of the user's sight in the environment image;

[0007] Determining a guide identification sequence based on the real-time environment image;

[0008] Based on the guide mark sequence, at least one guide animation is displayed; each of the guide animations is used to guide the focus of the user's line of sight to move from a current guide mark to a next guide mark.

[0009] In a second aspect, an embodiment of the present application provides an augmented reality guidance device, the device comprising:

[0010] An acquisition module, used to acquire a real-time environment image and a focus of the user's sight in the environment image;

[0011] A determination module, used to determine a guide identification sequence based on the real-time environment image;

[0012] The display module is used to display at least one guiding animation based on the guiding mark sequence; each guiding animation is used to guide the focus of the user's line of sight to move from the current guiding mark to the next guiding mark.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, one or more processors; a memory; a module with multiple applications installed; and one or more programs, wherein the one or more programs are stored in the memory, and when the one or more programs are executed by the processor, the electronic device implements the steps in the above method when executing the programs.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are configured to execute the method for intelligent service sharing provided above.

[0015] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0016] In an embodiment of the present application, a guide identification sequence is determined based on the real-time environmental image; at least one guide animation is displayed based on the guide identification sequence; each of the guide animations is used to guide the focus of the user's line of sight to move from the current guide identification to the next guide identification. In this way, on the one hand, the displayed guide animation can be used to guide the focus of the user's line of sight to observe the area rich in environmental feature points in the real-time environmental image, thereby improving the user's visual experience. On the other hand, by guiding the user through the displayed guide animation, there is no need to add indicative two-dimensional scene images or three-dimensional scene models, reduce the workload of image editing and production and the crowded feeling of the screen, improve the efficiency of scene recognition, and thus improve the user's AR scene recognition experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0018] Figure 1A A flowchart of an augmented reality guidance method provided in an embodiment of the present application;

[0019] Figure 1B A schematic diagram of an application scenario of an augmented reality guidance method provided in an embodiment of the present application;

[0020] Figure 1C A schematic diagram of an application scenario of an augmented reality guidance method provided in an embodiment of the present application;

[0021] Figure 1D A schematic diagram of an application scenario of an augmented reality guidance method provided in an embodiment of the present application;

[0022] Figure 1E A schematic diagram of an application scenario of an augmented reality guidance method provided in an embodiment of the present application;

[0023] Figure 1F A schematic diagram of an application scenario of an augmented reality guidance method provided in an embodiment of the present application;

[0024] Figure 1G A schematic diagram of an application scenario of an augmented reality guidance method provided in an embodiment of the present application;

[0025] Figure 2 A flowchart of an augmented reality guidance method provided in an embodiment of the present application;

[0026] Figure 3 A flowchart of an augmented reality guidance method provided in an embodiment of the present application;

[0027] Figure 4A A logic flow chart of an augmented reality guidance method provided in an embodiment of the present application;

[0028] Figure 4B A schematic diagram of an application scenario of an augmented reality guidance method provided in an embodiment of the present application;

[0029] Figure 5 A flowchart of an augmented reality guidance method provided in an embodiment of the present application;

[0030] Figure 6 A schematic diagram of an application scenario of an augmented reality guidance method provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of an augmented reality guidance device provided in an embodiment of the present application;

[0032] Figure 8 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. The following embodiments are used to illustrate the present application, but are not intended to limit the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0034] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0036] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art in the field to which the embodiments of the present application belong. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0037] Figure 1A A flowchart of an augmented reality guidance method provided in an embodiment of the present application is shown as follows: Figure 1A As shown, the method comprises at least the following steps:

[0038] Step S101, obtaining a real-time environment image and a focus of a user's sight in the environment image;

[0039] Here, the real-time environment image may be an environment image within the field of view (FOV) of the device captured in real time by the camera of the AR device. Exemplarily, when scene recognition starts, the AR device starts the camera and captures the environment image in real time. The environment image includes the environment around the AR device captured by the camera, such as machines on the product production line, products being produced, safety signs and production slogans on the product production line, etc.

[0040] Here, the focus of the user's line of sight may be any point in the FOV. For example, Figure 1B As shown, scene recognition begins, and there is a visual recognition focus point in the center of FOV to guide the user's line of sight.

[0041] Step S102, determining a guide identification sequence based on the real-time environment image;

[0042] Here, the guide mark in the guide mark sequence is used to guide the user's sight focus to stay in the area with high density of environmental feature points in the environmental image, so as to enhance the user's AR experience. Here, the environmental feature points may include: machines on the product production line, products being produced, safety marks on the product production line, and edges, corners, and blobs in production slogans.

[0043] Here, determining the guide mark sequence based on the real-time environment image includes: dividing the real-time environment image into at least one image area, determining the density of environment feature points of the at least one image area, and determining the guide mark sequence based on the density.

[0044] For example, Figure 1D As shown, the environment image includes four image areas: area 11, area 12, area 13 and area 14; wherein the density of the environment feature points in the four image areas is ranked from high to low as follows: area 11, area 12, area 13 and area 14; as shown Figure 1E As shown, the order of the guide identifiers is: area 11 is the 1st identifier in the sequence, area 12 is the 2nd identifier in the sequence, area 13 is the 3rd identifier in the sequence, and area 14 is the 4th identifier in the sequence.

[0045] Step S103: display at least one guiding animation based on the guiding mark sequence; each guiding animation is used to guide the focus of the user's line of sight to move from the current guiding mark to the next guiding mark.

[0046] Here, the guiding animation is an animation effect based on a user interface (UI). Figure 1F As shown, the animation effect may include a circle 15 displayed on the image area; Figure 1G As shown, the animation effect may also include an arc 16 indicating that the user indicates that the focus of the user's sight moves from the mark 1 to the mark 2.

[0047] In the above embodiment, a guide mark sequence is determined based on the real-time environmental image; at least one guide animation is displayed based on the guide mark sequence; each of the guide animations is used to guide the focus of the user's line of sight to move from the current guide mark to the next guide mark. In this way, on the one hand, the displayed guide animation can be used to guide the focus of the user's line of sight to observe the area rich in environmental feature points in the real-time environmental image, thereby improving the user's visual experience. On the other hand, by guiding the user through the displayed guide animation, there is no need to add indicative two-dimensional scene images or three-dimensional scene models, reduce the workload of image editing and production and the congestion of the screen, improve the efficiency of scene recognition, and thus improve the user's AR scene recognition experience.

[0048] Figure 2 A flowchart of an augmented reality guidance method provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the method comprises at least the following steps:

[0049] Step S201, obtaining a real-time environment image and a focus of the user's sight in the environment image;

[0050] Step S202, performing image recognition on the real-time environment image to obtain a first environment feature point set;

[0051] Here, image recognition can be performed by deep learning methods to obtain the first environmental feature points. For example, convolutional neural networks and residual networks are learned on real-time environmental images to identify feature points such as edges and corners of production logos in the scene.

[0052] Step S203, determining a guide identification sequence based on the first environmental feature point set;

[0053] In one practicable method, a two-dimensional image plane can be reconstructed using a first set of environmental feature points, the reconstructed two-dimensional image plane can be divided into at least one image area, and a guide marker is generated on each of the image areas shown; the density of the feature points corresponding to each area is determined, and at least one guide marker is sorted based on the density to obtain a guide marker sequence.

[0054] In another achievable manner, the feature points in the first environment feature point set may be screened to obtain a screened second environment feature point set; and a guide identification sequence may be determined based on the second environment feature point set.

[0055] Step S204: display at least one guiding animation based on the guiding mark sequence; each guiding animation is used to guide the focus of the user's line of sight to move from the current guiding mark to the next guiding mark.

[0056] In the above embodiment, the first set of environmental feature points is obtained by performing image recognition on the real-time environmental image, and the guide identification sequence is determined based on the first set of environmental feature points. In this way, the user can be guided to view the image area rich in environmental feature points, thereby improving the efficiency of scene recognition.

[0057] Figure 3 A flowchart of an augmented reality guidance method provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the method comprises at least the following steps:

[0058] Step S301, obtaining a real-time environment image and a focus of the user's sight in the environment image;

[0059] Step S302, performing image recognition on the real-time environment image to obtain a first environment feature point set;

[0060] Step S303, obtaining a labeled sample image corresponding to the real-time environment image;

[0061] Here, the marked sample image may be a sample environment image with marked environment feature points, for example, an image with marked edges and corners in a production slogan.

[0062] Step S304, based on the marked sample image, screening the environmental feature points in the first environmental feature point set to obtain a second environmental feature point set;

[0063] In one practicable manner, step S304 includes:

[0064] S3041, determining a feature descriptor for each environmental feature point in the marked sample image to obtain a first feature descriptor set;

[0065] Here, the feature descriptor is a vector used to describe a feature point. For example, when describing an angular feature point, the feature descriptor may include a vector of parameters such as an angle.

[0066] Here, each feature descriptor in the first feature descriptor set is used to correspondingly describe each environmental feature point in the labeled sample image.

[0067] S3042, determining a feature descriptor for each of the environmental feature points in the first environmental feature point set to obtain a second feature descriptor set;

[0068] S3043, determining the similarity between each feature descriptor in the first feature descriptor set and each feature descriptor in the second feature descriptor set;

[0069] In one feasible method, the labeled sample image and the real-time environment image in the same scene can be matched by similarity calculation; each feature descriptor in the labeled sample image is compared with each feature descriptor in the real-time environment image, for example, the Euclidean distance between the feature descriptors is calculated, and the pair of feature descriptors with the highest score are the two environmental feature points closest to each other, that is, the most matching features.

[0070] S3044: Determine each environmental feature point whose similarity satisfies a similarity threshold as an element in the second environmental feature point set.

[0071] In one feasible method, feature points with similarity greater than a threshold can be retained, and feature points with similarity not meeting the threshold can be filtered out, thereby filtering out environmental feature points with similarity not meeting the similarity threshold, and only retaining environmental feature points with higher similarity to obtain a second set of environmental feature points.

[0072] Step S305, determining a first environment image based on the second environment feature point set; the first environment image includes at least one image area; the density of the environment feature points in each of the image areas is greater than a density threshold;

[0073] In one practicable manner, step S305 includes:

[0074] Step S3051, reconstructing a second environment image based on the second environment feature point set;

[0075] Exemplarily, a two-dimensional image plane is established using the environmental feature point data in the second environmental feature point set, and environmental feature points with low similarity are filtered out in the two-dimensional image plane, thereby filtering out non-marked environmental feature points.

[0076] Step S3052, dividing the second environment image into at least one image area;

[0077] In one practicable manner, after the two-dimensional image plane is divided into regions, an index may be added to each of the image regions, so that different image regions may be determined by the index.

[0078] Step S3053, determining the density of environmental feature points in the at least one image area;

[0079] Step S3054: determine an area where the density of each of the environmental feature points is greater than a density threshold as an image area of ​​the first environmental image.

[0080] In one feasible method, non-core image areas can be filtered through sparsity calculation; then, based on the density clustering algorithm, the core image areas whose acquired environmental feature points meet the density threshold are further aggregated and boundary optimized, and the edge samples and image noise are filtered to obtain image areas where the environmental feature points are dense on the two-dimensional image.

[0081] Step S306, determining a guide marker sequence based on the at least one image region;

[0082] In an practicable manner, each of the image regions corresponds to a guide mark; the step S306 includes:

[0083] Step S3061, sorting the at least one image region based on the density of the environmental feature points in each of the image regions to obtain an observation order of the at least one image region;

[0084] Step S3062: Based on the observation order, determine the order of the guide identifiers to obtain a guide identifier sequence.

[0085] Step S307: display at least one guiding animation based on the guiding mark sequence; each guiding animation is used to guide the focus of the user's line of sight to move from the current guiding mark to the next guiding mark.

[0086] Exemplarily, each image area is sorted from large to small according to the density of environmental feature points in the image area, so as to determine the order of guide signs and obtain a dense to sparse guide sign sequence. The user is then guided by the real-time transformable UI animation effect to observe the corresponding image area according to the sign sequence, and the user is guided to look at the direction with rich environmental feature points, which can effectively improve the accuracy of scene recognition.

[0087] In the above embodiment, the environment feature points in the first environment feature point set are screened based on the labeled sample image to obtain the second environment feature point set; the first environment image is determined based on the second environment feature point set; the first environment image includes at least one image area; the density of the environment feature points in each of the image areas is greater than the density threshold; and the guide identification sequence is determined based on the at least one image area. In this way, the image area with few feature points can be filtered by screening the environment feature points and the image area, the image density can be reduced, and the area with dense feature points can be obtained, so that the user is guided to observe the direction with rich environment feature points, the effectiveness of the guidance can be enhanced, and the user's experience of scene recognition can be optimized.

[0088] Figure 4A A flowchart of an augmented reality guidance method provided in an embodiment of the present application is shown as follows: Figure 4AAs shown, the method comprises at least the following steps:

[0089] Step S401, obtaining a target position of the focus of the user's line of sight;

[0090] Step S402, acquiring an environment image at the target location;

[0091] Step S403: when it is detected that the environment image at the target position does not include the environment feature point, output prompt information of the moving position.

[0092] For example, Figure 4B As shown, when the density of environmental feature points in the real-time environmental image is low, a prompt message 41 is displayed to indicate that the loading has failed, and to guide the user to move.

[0093] In one achievable manner, the method further includes: step S404, outputting prompt information of successful positioning when it is detected that the focus of the user's line of sight coincides with the target position.

[0094] Step S405, obtaining a real-time environment image and a focus of the user's sight in the environment image;

[0095] Step S406, determining a guide identification sequence based on the real-time environment image;

[0096] In an achievable manner, the method further includes: displaying at least one guiding animation based on the guiding mark sequence; each guiding animation is used to guide the focus of the user's line of sight to move from a current guiding mark to a next guiding mark;

[0097] In one achievable manner, the method further includes: when the focus of the user's line of sight moves from the current guide mark to the next guide mark, the environment recognition is successful, and the display of the guide animation is stopped.

[0098] Step S407: When the focus of the user's line of sight does not move from the current guide mark to the next guide mark, the at least one guide animation is displayed in a loop.

[0099] In the above embodiment, by cyclically displaying the at least one guidance animation when the focus of the user's line of sight has not moved from the current guidance mark to the next guidance mark, the user can be repeatedly prompted with the direction of line of sight movement, thereby enhancing the user's experience of scene recognition.

[0100] With the advancement of technology, augmented reality technology has entered more into the lives of users, bringing new experiences to users. AR technology can apply virtual objects to real environments, so that the real environment and virtual objects are superimposed on the same screen or space in real time. Combining AR technology with computer vision algorithms and applying them to AR devices can perform object recognition and environment recognition on scenes displayed in AR devices, increasing the intelligence and technological sense of AR devices. Related technologies guide users to observe the environment by displaying two-dimensional scene images or three-dimensional scene models that need to be recognized in AR devices, such as AR glasses.

[0101] However, the above process requires the production of two-dimensional scene pictures or three-dimensional scene models for different scenes, which is a lot of work. At the same time, due to the small field of view (FOV) of the AR device, displaying two-dimensional scene pictures or three-dimensional scene models will occupy most of the field of view, resulting in a poor user experience.

[0102] To solve the above problems, the present application provides an augmented reality guidance method, which is applied to an augmented reality device, such as Figure 5 As shown, the method includes:

[0103] Step S501, start the augmented reality device and start recognizing the scene;

[0104] Step S502, obtaining a real-time environment image;

[0105] Step S503, environmental feature point analysis;

[0106] Here, a first set of environmental feature points is obtained by performing environmental feature point analysis on the real-time environmental image;

[0107] In one achievable manner, the step S503 includes: performing image recognition on the real-time environment image to obtain a first environment feature point set.

[0108] Step S504, screening of environmental feature points;

[0109] Here, the second environmental feature point set is obtained by screening the environmental feature points in the first environmental feature point set;

[0110] Step S505, sparsity filtering and density clustering;

[0111] Here, the second set of environmental feature points is determined by clustering based on sparsity and / or density;

[0112] Step S506, determining a guide identification sequence;

[0113] Step S507, displaying UI guide animation;

[0114] Step S508, completing scene recognition.

[0115] Here, the steps S501 to S505 are implemented through the algorithm layer; and the steps S505 to S508 are implemented through the application layer.

[0116] Here, the algorithm layer: when starting scene recognition, the AR device starts the camera and captures the environment image in real time. The environment image includes the environment around the AR device captured by the camera, such as the machines on the product production line, the products being produced, the safety signs and production language on the product production line, etc. The environmental feature points are obtained through deep learning, which can be edges, corners or spots, and then the corresponding feature descriptors are extracted for each environmental feature point. Finally, the environmental feature points are matched, and the environmental feature points whose Euclidean distance comparison results do not meet the distance threshold are filtered out through Euclidean distance comparison, and the environmental feature points whose Euclidean distance comparison results meet the distance threshold are retained to obtain the second set of environmental feature points.

[0117] The application layer: acquires the second set of environmental feature points from the algorithm layer in real time, establishes a two-dimensional image plane through the environmental feature point data in the second set of environmental feature points, divides the 2D image plane into regions, adds indexes to each image region, and filters non-core image regions through sparsity calculation; then, based on the density clustering algorithm, further aggregates and optimizes the boundaries of the core image regions where the acquired environmental feature points meet the density threshold, filters edge samples and image noise, and obtains image regions where the environmental feature points are dense on the two-dimensional image. According to the density of the environmental feature points in the image region, each image region is sorted from large to small according to the density, so as to determine the order of the guide marks, and obtain a dense to sparse guide mark sequence, so as to guide the user to observe the corresponding image region according to the mark sequence through the animation effect of the UI that can be changed in real time, and guide the user to view the direction with rich environmental feature points, which can effectively improve the accuracy of scene recognition and optimize the user's experience of scene recognition.

[0118] For example, Figure 1B As shown in the figure, scene recognition begins. There is a visual recognition focus point in the center of FOV101 to guide the user's focus. At the same time, the algorithm layer begins to analyze the real-time environment image, obtain the environment feature points, and pass them to the application layer. The guide animation is as follows Figure 1C As shown in the middle circle 17.

[0119] Exemplarily, through glance guidance, the quality of the current environment is judged, and the user is allowed to naturally move his head left and right to view. Gaze guidance: the information display type in the environmental image is Class C: following the field of vision, the application layer sorts the obtained environmental feature points according to the density, and the position with the highest density is the starting point of the guidance mark. The guidance animation is displayed on the interface. After the user focuses his eyes on the guidance animation, the guidance animation guides to the diagonal position.

[0120] For example, the area where the user's line of sight can be expanded during the surround view process. After the surround view is completed, the subsequent guidance position is determined according to the density of the newly expanded environmental feature points, and the user is guided to follow the UI guidance animation to view the feature-rich area. The information display type in the environmental image is Class A: When following the space, within the range of 120° of the front viewing angle, such as Figure 6 As shown, a circle 61 appears in the space according to the feedback of the environmental feature points to guide the focus of the user's sight to move. If the user does not follow the view, the ray-guided animation loops after 2 seconds.

[0121] In the above embodiment, environmental feature points are obtained by performing image recognition or environmental recognition on real-time environmental images; the environmental feature points are filtered; and the filtered feature points are displayed on the screen through an interface (UI) design to guide users to view locations rich in feature points, thereby improving user experience.

[0122] Based on the foregoing embodiments, the embodiments of the present application further provide an augmented reality guidance device, wherein the control device includes the modules included, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0123] Figure 7 A schematic diagram of the structure of an augmented reality guidance device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device 700 includes an acquisition module 701, a determination module 702 and a display module 703, wherein:

[0124] The acquisition module 701 is used to acquire a real-time environment image and a focus of the user's sight in the environment image;

[0125] The determination module 702 is used to determine a guide identification sequence based on the real-time environment image;

[0126] The display module 703 is used to display at least one guiding animation based on the guiding mark sequence; each guiding animation is used to guide the focus of the user's line of sight to move from the current guiding mark to the next guiding mark.

[0127] In some possible embodiments, the determination module 702 is further used to: perform image recognition on the real-time environment image to obtain a first environment feature point set; and determine a guide identification sequence based on the first environment feature point set.

[0128] In some possible embodiments, the determination module 702 includes: an acquisition submodule, used to acquire a marked sample image corresponding to the real-time environment image; a screening submodule, used to screen the environment feature points in the first environment feature point set based on the marked sample image to obtain a second environment feature point set; a first determination submodule, used to determine the first environment image based on the second environment feature point set; the first environment image includes at least one image area; the density of the environment feature points in each of the image areas is greater than a density threshold; and a second determination submodule, used to determine the guide identification sequence based on the at least one image area.

[0129] In some possible embodiments, the screening submodule is further used to: determine the feature descriptor of each environmental feature point in the marked sample image to obtain a first feature descriptor set; determine the feature descriptor of each environmental feature point in the first environmental feature point set to obtain a second feature descriptor set; determine the similarity between each feature descriptor in the first feature descriptor set and each feature descriptor in the second feature descriptor set; and determine each environmental feature point whose similarity meets a similarity threshold as an element in the second environmental feature point set.

[0130] In some possible embodiments, the first determination submodule is further used to: reconstruct a second environment image based on the second environment feature point set; divide the second environment image into at least one image area; determine the density of the environment feature points in the at least one image area; and determine an area in which the density of each of the environment feature points is greater than a density threshold as an image area of ​​the first environment image.

[0131] In some possible embodiments, each of the image areas corresponds to a guide marker; the second determination submodule is further used to: sort the at least one image area based on the density of environmental feature points in each of the image areas to obtain an observation order of the at least one image area; determine the order of the guide markers based on the observation order to obtain a guide marker sequence.

[0132] In some possible embodiments, the display module 703 is further configured to: when the focus of the user's line of sight does not move from the current guide mark to the next guide mark, cyclically display the at least one guide animation.

[0133] In some possible embodiments, the acquisition module 701 is further used to: acquire the target position of the focus of the user's line of sight; acquire the environmental image at the target position; and the display module 703 is further used to: output prompt information of the moving position when it is detected that the environmental image at the target position does not include environmental feature points.

[0134] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0135] It should be noted that in the embodiment of the present application, if the above-mentioned augmented reality guidance method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a smart phone with a camera, a tablet computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0136] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the augmented reality guiding method described in any of the above embodiments are implemented.

[0137] Correspondingly, in an embodiment of the present application, a chip is also provided, which includes a programmable logic circuit and / or program instructions. When the chip is running, it is used to implement the steps of the augmented reality guidance method described in any of the above embodiments.

[0138] Correspondingly, in an embodiment of the present application, a computer program product is also provided. When the computer program product is executed by a processor of an electronic device, it is used to implement the steps in the augmented reality guidance method described in any of the above embodiments.

[0139] Based on the same technical concept, an embodiment of the present application provides an electronic device for implementing the augmented reality guidance method recorded in the above method embodiment. Figure 8 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device 800 includes a memory 810 and a processor 820, wherein the memory 810 stores a computer program that can be run on the processor 820, and when the processor 820 executes the program, the steps in the augmented reality guiding method described in any embodiment of the present application are implemented.

[0140] The memory 810 is configured to store instructions and applications executable by the processor 820, and can also cache data to be processed or processed by the processor 820 and various modules in the electronic device (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0141] When the processor 820 executes the program, the steps of any of the above-mentioned augmented reality guiding methods are implemented. The processor 820 generally controls the overall operation of the electronic device 800.

[0142] The processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0143] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various electronic devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0144] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0145] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0146] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0148] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0149] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0150] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for the automatic test line of the device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0151] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0152] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0153] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An augmented reality guidance method, applied to an augmented reality device, the method comprising: Acquire a real-time environment image and a focus of the user's sight in the environment image; Performing image recognition on the real-time environment image to obtain a first environment feature point set; Screening the environmental feature points in the first environmental feature point set to obtain a second environmental feature point set; Based on the second set of environmental feature points, a first environmental image is determined; the first environmental image includes at least one image area; the density of environmental feature points in each of the image areas is greater than a density threshold; determining a guide identification sequence based on the at least one image region; Based on the guide mark sequence, at least one guide animation is displayed; each of the guide animations is used to guide the focus of the user's line of sight to move from a current guide mark to a next guide mark.

2. The method according to claim 1, before screening the environmental feature points in the first environmental feature point set, the method further comprises: Acquire a labeled sample image corresponding to the real-time environment image; Based on the marked sample image, the environmental feature points in the first environmental feature point set are screened to obtain a second environmental feature point set.

3. The method according to claim 2, wherein the step of screening the environmental feature points in the first environmental feature point set based on the marked sample image to obtain the second environmental feature point set comprises: Determine a feature descriptor for each environmental feature point in the marked sample image to obtain a first feature descriptor set; Determine a feature descriptor for each of the environmental feature points in the first environmental feature point set to obtain a second feature descriptor set; Determining a similarity between each feature descriptor in the first set of feature descriptors and each feature descriptor in the second set of feature descriptors; Each environmental feature point whose similarity satisfies a similarity threshold is determined as an element in the second environmental feature point set.

4. The method according to claim 1, wherein determining the first environment image based on the second environment feature point set comprises: Reconstructing a second environment image based on the second environment feature point set; dividing the second environment image into at least one image region; determining a density of environmental feature points in the at least one image region; An area where the density of each of the environmental feature points is greater than a density threshold is determined as an image area of ​​the first environmental image.

5. The method according to claim 1, wherein each of the image regions corresponds to a guide mark; and determining a guide mark sequence based on the at least one image region comprises: Sorting the at least one image region based on the density of the environmental feature points in each of the image regions to obtain an observation order of the at least one image region; Based on the observation sequence, the sequence of the guide identifiers is determined to obtain a guide identifier sequence.

6. The method according to any one of claims 1 to 5, further comprising: When the focus of the user's line of sight does not move from the current guide mark to the next guide mark, the at least one guide animation is displayed in a loop.

7. The method according to any one of claims 1 to 5, further comprising: Obtaining a target position of a focus of the user's line of sight; Acquire an environment image at the target location; When it is detected that the environment image at the target position does not include the environment feature point, prompt information of the moving position is output.

8. An augmented reality guiding device, the device comprising: An acquisition module, used to acquire a real-time environment image and a focus of the user's sight in the environment image; A determination module, configured to perform image recognition on the real-time environment image to obtain a first environment feature point set; A screening module, used for screening the environmental feature points in the first environmental feature point set to obtain a second environmental feature point set; A determination module, configured to determine a first environment image based on the second environment feature point set; the first environment image includes at least one image area; the density of environment feature points in each of the image areas is greater than a density threshold; and further configured to determine a guide identification sequence based on the at least one image area, and determine a guide identification sequence based on the first environment feature point set; A display module, configured to display at least one guidance animation based on the guidance identification sequence; Each of the guiding animations is used to guide the focus of the user's sight to move from the current guiding mark to the next guiding mark.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps of the method according to any one of claims 1 to 7 when executing the program.

Images