A scene recognition method based on AR glasses

Abstract

The application relates to the technical field of scene recognition, and specifically discloses a scene recognition method based on AR glasses, which comprises the following steps: establishing a database, storing object information and sample images of the objects in the database; acquiring a scene image in front of a user in real time, dividing the scene image into rectangular grids, obtaining each rectangular grid and numbering; confirming the rectangular grid where each object is located, acquiring the corresponding number of the rectangular grid, obtaining a number set; acquiring an eye image of the user in real time, magnifying the eye image to be consistent with the size of the scene image in proportion; dividing the eye image into a plurality of pixels, acquiring a circular area formed by a plurality of adjacent pixels with the lowest gray value, and marking the circular area as a pupil area; acquiring the number of the rectangular grid where the center point of the pupil area is located, confirming the corresponding number set, obtaining an object associated with the number set, and recognizing object information corresponding to the object according to the sample image, and displaying the object information in the field of view of the user.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of scene recognition, in particular to a scene recognition method based on AR glasses. BACKGROUND

[0002] Scene recognition is an important research direction in the field of computer vision, aiming to automatically recognize specific scenes in images or videos; for example, given a landmark image, the scene of the image can be quickly and accurately recognized, and the recognition result can be a specific geographical location, the name of the scene, or a certain same scene in the database; this process includes image classification, feature extraction, bag-of-words model, and design and implementation of ensemble learning classifier.

[0003] Scene recognition based on AR glasses is a cutting-edge technology that combines virtual and real-world scenes through augmented reality (AR) and interaction; this technology captures the environment within the viewing angle through a depth camera, and then uses image recognition, positioning analysis and Al calculation to achieve three-dimensional reconstruction of the current environment, so that the AR glasses can better understand the objects and scenes in the real environment.

[0004] In the prior art, the AR glasses can capture the scene in front of the user to obtain a scene image, and then perform image recognition on the objects in the scene image to obtain the information of the objects and display the information to the user; however, the scene image obtained by the AR glasses often contains multiple objects, and the visual effect of displaying all object information in the scene image is too redundant, and it is not determined which object the user wants to recognize in the scene image. SUMMARY

[0005] The purpose of the present application is to provide a scene recognition method based on AR glasses to solve the above technical problems.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] A scene recognition method based on AR glasses, comprising an AR glass, comprising the following steps:

[0008] Step S1: establishing a database, storing object information and sample images of objects in the database, each sample image corresponding to an object information;

[0009] Step S2: acquiring a scene image in front of the user in real time, dividing the scene image into a rectangular grid to obtain a plurality of rectangular grids, and numbering the rectangular grids;

[0010] Step S3: confirming the rectangular grid where each object is located in the scene image, obtaining the corresponding number of the rectangular grid to obtain a number set; associating the number set with the corresponding object.

[0011] Step S4: real-time acquisition of the eye image of the user, the aspect ratio of the eye image being consistent with that of the scene image, and the eye image being enlarged to be consistent with the size of the scene image;

[0012] Step S5: division of the enlarged eye image into a plurality of pixels, acquisition of a circular region composed of a plurality of adjacent pixels with the lowest gray value, denoted as a pupil region, and acquisition of the number of a rectangular grid in which a center point of the pupil region is located;

[0013] Step S6: confirmation of a corresponding number set according to the number of the rectangular grid, obtaining of an object associated with the number set, identification of object information corresponding to the object according to a sample image, and display of the object information in the field of view of the user.

[0014] As a further scheme of the present application, in step S6, the process of identifying the object information corresponding to the object according to the sample image comprises:

[0015] respectively extracting feature information of the object in the sample image and the scene image, and comparing the feature information of the object with the feature information of the sample image;

[0016] obtaining a similarity of the object to each sample image, and obtaining object information corresponding to a sample image with the highest similarity.

[0017] As a further scheme of the present application, in step S3, in the process of confirming the rectangular grid in which each object is located in the scene image, only objects with complete edges in the scene image are confirmed.

[0018] As a further scheme of the present application, in step S1, the process of establishing the database comprises:

[0019] The database comprises a plurality of storage nodes, each of the storage nodes being associated with object information and storing a plurality of sample images of the object.

[0020] As a further scheme of the present application, in step S2, the process of acquiring the scene image and the eye image comprises setting a time interval threshold, and acquiring the scene image and the eye image once every time interval threshold.

[0021] As a further scheme of the present application, in step S5, when the number of the rectangular grid in which the center point of the pupil region is located does not belong to any number set, the numbers of all the rectangular grids in which the pupil region is located are acquired to obtain a pupil number set;

[0022] a plurality of number sets intersecting with the pupil number set are acquired, and object information corresponding to the number sets is displayed in the field of view of the user.

[0023] The beneficial effects of the present application are:

[0024] In the method, a database is established to store object information and sample images, and to identify information of objects in a scene; a scene image in front of a user is acquired, the scene image is divided into a plurality of rectangular grids, each rectangular grid is numbered, and a number set of a rectangular grid in which each object is located is acquired; the process of dividing the rectangular grids and numbering facilitates subsequent confirmation of a position at which a line of sight of the user stays and an object located at a corresponding position in the scene; an eye image of the user is acquired, the eye image has a consistent aspect ratio, the eye image is enlarged in proportion to be consistent with the scene image, and a pupil region in the eye image is acquired; the position of the pupil region is used to determine the position at which the line of sight of the user stays; an object corresponding to a center point of the pupil region is obtained according to a number of a rectangular grid in which the center point is located, and information of the object is displayed in a field of view of the user. In the prior art, when there are a plurality of objects in a scene, displaying all the information of the objects is prone to causing a display effect to be excessively redundant, and in the method, the object that the user needs to understand is accurately determined according to the position at which the line of sight of the user stays, and the information of the object is displayed in the field of view of the user, so that the display effect is more concise and accurate. BRIEF DESCRIPTION OF DRAWINGS

[0025] The application will be further described below with reference to the drawings.

[0026] Figure 1 is a method schematic diagram of a scene recognition method based on AR glasses. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the application will be apparently and completely described with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the application.

[0028] Please refer to Figure 1 The application is a scene recognition method based on AR glasses, and the method comprises the following steps:

[0029] Step S1: a database is established, and object information and sample images of objects are stored in the database, each sample image corresponds to an object information;

[0030] The database establishment process comprises:

[0031] The database comprises a plurality of storage nodes, each storage node is associated with an object information, and a plurality of sample images of the object are stored in the storage node;

[0032] It is understood that the database is used to store information about objects and feature information of the objects, identify objects in a scene through feature information, and retrieve information about the objects from the database;

[0033] Step S2: acquiring a scene image in front of the user in real time, dividing the scene image into a rectangular grid, obtaining a plurality of rectangular grids, and numbering the rectangular grids;

[0034] The process of acquiring the scene image and the eye image includes setting a time interval threshold, and acquiring the scene image and the eye image once every time interval threshold;

[0035] Step S3: Identify the rectangular grid where each object is located in the scene image, obtain the number corresponding to the rectangular grid, and obtain a number set; associate the number set with the corresponding object;

[0036] In the process of confirming the rectangular grid where each object is located in the scene image, only objects with complete edges in the scene image are confirmed;

[0037] It is worth noting that for objects that do not have complete edges in the scene, such as walls, the sea, etc., such objects are not recognized; this is mainly because traditional image processing and computer vision technologies mainly rely on the edge information of objects for recognition and analysis; these methods usually locate the outline of the object by detecting the edge strength and direction in the image, so as to classify and recognize it; edge detection is a key step in image processing and computer vision. Its purpose is to find points in the image where the brightness changes significantly. These points are usually regarded as the edges or boundaries of the object; however, if the object has no obvious edges, or the edge information is not enough to distinguish different objects, then traditional edge detection methods may not be able to accurately identify these objects. For example, for some objects with complex textures or non-rigid deformations, relying solely on edge information may result in misjudgment;

[0038] Step S4: Acquire the user's eye image in real time, wherein the aspect ratio of the eye image is consistent with the scene image, and proportionally enlarge the eye image to the same size as the scene image;

[0039] It should be further explained that the above-mentioned magnification process facilitates the correspondence between the position where the user's gaze rests and the scene image, so as to confirm the object that the user needs to understand;

[0040] Step S5: Divide the magnified eye image into a number of pixels, obtain a circular area consisting of a number of adjacent pixels with the lowest grayscale value, and record it as the pupil area; and obtain the number of the rectangular grid where the center point of the pupil area is located;

[0041] It needs to be further explained that when the center point of the pupil region is in the rectangular grid whose number does not belong to any number set, the numbers of all rectangular grids where the pupil region is located are obtained to obtain the pupil number set;

[0042] The number sets intersecting with the pupil number set are obtained, and the object information corresponding to the number sets is displayed in the user's field of view;

[0043] Step S6: According to the number of the rectangular grid, the corresponding number set is confirmed, the object associated with the number set is obtained, and the object information corresponding to the object is identified according to the sample image and displayed in the user's field of view;

[0044] The process of identifying the object information corresponding to the object according to the sample image includes:

[0045] The feature information of the object in the sample image and the scene image is extracted respectively, and the feature information of the object is compared with the feature information of the sample image;

[0046] The similarity of the object and each sample image is obtained, and the object information corresponding to the sample image with the highest similarity is obtained.

[0047] The above has carried out the detailed description to one embodiment of the application, but the content described is only the preferred embodiment of the application, and cannot be considered as used for limiting the implementation range of the application. Any equivalent changes and improvements made according to the application scope should still belong to the patent coverage range of the application.

Claims

1. An AR glasses-based scene recognition method, comprising AR glasses, characterized in that, The method comprises the following steps: Step S1: establishing a database, storing object information and sample images of objects in the database, each sample image corresponding to an object information; Step S2: acquiring a scene image in front of a user in real time, dividing the scene image into a rectangular grid to obtain a plurality of rectangular grids, and numbering the rectangular grids; Step S3: confirming the rectangular grids in which the objects are located in the scene image, obtaining the numbers of the rectangular grids, and obtaining a number set; associating the number set with the corresponding objects; Step S4: acquiring an eye image of the user in real time, the aspect ratio of the eye image being consistent with that of the scene image, and enlarging the eye image to be consistent with the size of the scene image; Step S5: dividing the enlarged eye image into a plurality of pixels, obtaining a circular region composed of a plurality of adjacent pixels with the lowest gray value, denoted as a pupil region, and obtaining the number of the rectangular grid in which the center point of the pupil region is located; Step S6: confirming the corresponding number set according to the number of the rectangular grid, obtaining the objects associated with the number set, identifying the object information corresponding to the objects according to the sample images, and displaying the object information in the field of view of the user; When the number of the rectangular grid in which the center point of the pupil region is located does not belong to any number set, obtaining the numbers of all the rectangular grids in which the pupil region is located to obtain a pupil number set; Obtaining a plurality of number sets having an intersection with the pupil number set, and displaying the object information corresponding to the number sets in the field of view of the user. 2.The AR glasses-based scene recognition method of claim 1, wherein, In step S6, the process of identifying the object information corresponding to the objects according to the sample images comprises: Extracting the feature information of the objects in the sample images and the scene image respectively, comparing the feature information of the objects with the feature information of the sample images, obtaining the similarity of the objects with each sample image, and obtaining the object information corresponding to the sample image with the highest similarity. In step S3, in the process of confirming the rectangular grids in which the objects are located in the scene image, only the objects with complete edges in the scene image are confirmed. 3.The AR glasses-based scene recognition method of claim 1, wherein, In step S1, the process of establishing the database comprises: 4.The AR glasses-based scene recognition method of claim 1, wherein, The database comprises a plurality of storage nodes, each of which is associated with an object information and stores a plurality of sample images of the object. In step S2, the process of acquiring the scene image and the eye image comprises setting a time interval threshold, and acquiring the scene image and the eye image every time interval threshold. 5.The AR glasses-based scene recognition method of claim 1, wherein, ​

Images