Positioning tracking method, device and wearable device thereof

Abstract

The application provides a positioning tracking method, a system and a wearable device. The positioning tracking method is applied to a non-wearable image display device. The positioning tracking method comprises the following steps: acquiring an image containing a marker, the marker being arranged on the wearable device; identifying the marker in the image, and determining relative spatial position information between the wearable device and the image display device according to the marker; and rendering a virtual object according to the relative spatial position information, and displaying the virtual object in the image display device. In the positioning tracking method, the image display device can acquire the position information of the marker by collecting the image containing the marker integrated in the wearable device, so as to track the wearable device.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image display, and in particular to a positioning tracking method and device, and a wearable device. BACKGROUND

[0002] With the development of technology, machine intelligence and information intelligence are increasingly popular, and technologies for recognizing user images through image acquisition devices such as machine vision or virtual vision to achieve human-computer interaction are becoming more and more important. Augmented Reality (AR) technology constructs virtual objects that do not exist in the real environment by means of computer graphics technology and visualization technology, and accurately fuses the virtual objects into the real environment through image recognition positioning technology, integrates the virtual objects with the real environment by means of a display device, and displays a real sensory experience to the user. The primary technical problem to be solved by augmented reality technology is how to accurately fuse virtual objects into the real world, that is, to make the virtual objects appear at the correct position of the real scene with the correct angle and posture, thereby producing strong visual reality. In traditional technology, the presentation angle of the virtual object is usually fixed, and the display angle of the virtual object can be changed only after the user controls it through a controller, which is not convenient for the user. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a positioning tracking method, device and wearable device.

[0004] In one aspect, the embodiments of the present application provide a positioning tracking method applied to a non-wearable image display device. The positioning tracking method comprises: acquiring an image containing a marker, the marker being arranged on a wearable device; identifying the marker in the image, and determining relative spatial position information between the wearable device and the image display device according to the marker; and rendering a virtual object according to the relative spatial position information, and displaying the virtual object in the image display device.

[0005] In some embodiments, the wearable device is glasses, and the marker is arranged on the frame of the glasses. When the glasses are worn, the relative spatial position information between the glasses and the image display device is determined according to the marker, which comprises: determining the relative spatial position information between the eyes of a user wearing the glasses and the image display device according to the marker.

[0006] In some embodiments, the relative spatial position information between the eyes of a user wearing the glasses and the image display device is determined according to the marker, which comprises: determining a sub-marker contained in the marker according to the image of the marker; positioning the eye area of a user wearing the glasses according to the sub-marker, and determining the relative spatial position information between the eyes of a user wearing the glasses and the image display device.

[0007] In some embodiments, before rendering the virtual object according to the relative spatial position information, the method further comprises: capturing an eye image of the user; extracting an eye feature of the eye image, and determining the relative spatial position information between the eye of the user wearing the glasses and the image display device according to the eye feature.

[0008] In some embodiments, the determining the relative spatial position information between the eye of the user wearing the glasses and the image display device according to the eye feature comprises: comparing the eye features between adjacent frames of the eye image to obtain an eye position change of the user wearing the glasses; calculating motion data of the eye according to the eye position change; and determining the relative spatial position information between the eye of the user wearing the glasses and the image display device according to the motion data of the eye.

[0009] In some embodiments, after the determining the relative spatial position information between the eye of the user wearing the glasses and the image display device according to the eye feature, the method further comprises: correcting the estimated position information according to the calibration position information to obtain the relative spatial position information between the eye of the user wearing the glasses and the image display device; wherein the estimated position information is the relative spatial position information determined according to the eye image, and the calibration position information is the relative spatial position information determined according to the marker.

[0010] In some embodiments, the method further comprises: obtaining the eye image captured in real time through a first thread, and obtaining the estimated position information according to the eye image; obtaining the image containing the marker through a second thread, and obtaining the calibration position information according to the marker; comparing the estimated position information with the calibration position information, and correcting the estimated position information according to the calibration position information when the estimated position information is inconsistent with the calibration position information.

[0011] In another aspect, the present application also provides a positioning and tracking device, comprising: an image capturing module configured to obtain an image containing a marker, the marker being arranged on a wearable device; a position relationship determining module configured to identify the marker in the image, and determine the relative spatial position information between the wearable device and an image display device according to the marker; and a display module configured to render a virtual object according to the relative spatial position information, and display the virtual object in the image display device.

[0012] In yet another aspect, the present application also provides a wearable device for assisting positioning and tracking, comprising a frame and a marker arranged on the frame, the marker being identified by a terminal device to determine the relative position relationship between the eye of the user and the terminal device.

[0013] In some embodiments, the frame includes a left frame and a right frame arranged side by side, the marker includes a first sub-marker and a second sub-marker, and the first sub-marker and the second sub-marker are arranged on two sides of the left frame, respectively.

[0014] In some embodiments, the frame includes a left frame and a right frame arranged side by side, and the marker further includes a third sub-marker and a fourth sub-marker, and the third sub-marker and the fourth sub-marker are arranged on two sides of the right frame, respectively.

[0015] In some embodiments, the frame includes a left frame and a right frame arranged side by side, and the wearable device further includes a nose bridge connected between the left frame and the right frame, and the marker further includes a fifth sub-marker arranged on the nose bridge.

[0016] In some embodiments, the wearable device further includes a filter layer covering the marker.

[0017] In the positioning and tracking method provided by the embodiments of the present application, the non-wearable image display device can acquire the position information of the marker by collecting an image containing the marker integrated on the wearable device, so as to track the wearable device. The image display device can determine the relative spatial position relationship between the image display device and the wearable device according to the image containing the marker, and display the constructed virtual object at a corresponding viewing angle according to the relative spatial position relationship, so that the display viewing angle of the virtual object can change along with the position relationship between the wearable device and the image display device, which is beneficial to the user to conveniently observe the virtual object from multiple angles and improves the interactivity between the user and the virtual content. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is an application scenario diagram of the positioning and tracking method provided by the embodiments of the present application;

[0020] Figure 2 is a structural diagram of the wearable device provided by the embodiments of the present application;

[0021] Figure 3 is a structural diagram of the wearable device provided by another embodiment of the present application;

[0022] Figure 4 is a structural diagram of the wearable device provided by another embodiment of the present application;

[0023] Figure 5 is a flowchart of a positioning tracking method provided by an embodiment of the present application;

[0024] Figure 6 is a functional module diagram of a positioning tracking device provided by an embodiment of the present application; DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0026] It should be noted that when a component is referred to as being "fixed" to another component, it can be directly on the other component or there can be an intervening component. When a component is referred to as being "connected" to another component, it can be directly connected to the other component or there can be an intervening component. When a component is referred to as being "disposed" on another component, it can be directly disposed on the other component or there can be an intervening component.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0028] Referring to Figure 1 , a virtual content interaction system 10 provided by an embodiment of the present application is shown, which includes an image display device 300 and a wearable device 400. In the embodiment of the present application, a marker 450 (see Figure 2 ) is provided on the wearable device 400. In use, the marker 450 on the wearable device 400 can be within the field of view of the image display device 300, so that the image display device 300 can acquire an image of the marker 450 and identify the marker 450.

[0029] In the embodiments of the present application, the image display device 300 can be a mobile device such as a mobile phone or a tablet, or a desktop virtual reality / augmented reality display device. When the image display device 300 is a desktop display device, it can be an integrated display device or an external display device. For example, the image display device 300 can be externally connected to a smart terminal such as a mobile phone, i.e., the image display device 300 can be plugged into or connected to an external display device (such as a mobile phone or a tablet) to display virtual objects in the image display device 300.

[0030] In the embodiments of the present application, the image display device 300 is a desktop display device, which includes an image acquisition device 301 and a display device 303. The image acquisition device 301 and the display device 303 can jointly form the desktop display device. Specifically, the display device 303 includes a control center 3031 and a display 3033. The display 3033 can be a see-through mirror, and the control center 3031 is configured to project a virtual object onto the display 3033 so that the user can observe the virtual object in the display 3033. While the user observes the virtual object in the display 3033, the user can also observe the environment in front of the display 3033 through the display 3033, and thus the image obtained by the user's eyes is a virtual reality superimposed scene in which the virtual content and the environment in front are superimposed. The image acquisition device 301 is electrically connected to the display device 303, and the image acquisition device 301 is further configured to acquire environmental information in a field of view thereof.

[0031] The wearable device 400 is configured to be worn by the user, so that the image display device 300 can determine a display perspective of the virtual object according to position information of the wearable device 400. The marker 450 is integrated in the wearable device 400. The marker 450 can be in a field of view of the image acquisition device 301 of the image display device 300, i.e., the image acquisition device 301 can acquire an image of the marker 450. The image of the marker 450 is stored in the image display device 300, and is configured to be used by the image display device 300 to determine relative position information between the marker 450 and the image display device 300 according to the image of the marker 450, so as to render and display the virtual object.

[0032] The marker 450 can be a marker image including at least one sub-marker having a certain shape. Of course, the specific marker 450 is not limited in the embodiments of the present application, as long as the marker 450 can be recognized by the image display device 300.

[0033] As Figure 1As shown, the user can wear the wearable device 400, when the marker 450 on the wearable device 400 is located within the field of view of the image acquisition device 301 of the image display device 300, the image acquisition device 301 can acquire an image of the marker 450, and according to the acquired image containing at least one sub-marker distributed on the marker 450, the relative positional relationship and rotational relationship between the marker 450 and the image display device 300 can be determined, and other information can be determined, so as to render and display a virtual object, such as Figure 1 As shown, the building model 600 displayed by the virtual object corresponding to the marker 450, thereby the user can observe the virtual object based on the marker, and can observe different perspectives of the virtual object at different positions. As shown, Figure 1 As shown, the virtual object 600 presented by the image display device 300 is a simulation building model, when the user wearing the wearable device 400 stands at A to observe the virtual object 600 on the image display device 300, the user can observe the view of the simulation building model at the first perspective (such as a preset axonometric view), when the user moves to B, the user can observe the structure view of the simulation building model relative to the left side of the user.

[0034] For the above-mentioned scenario, the embodiment of the present application provides a positioning tracking method, the execution subject of the positioning tracking method provided by the embodiment of the present application can be the image display device, in the positioning tracking method, the image display device can acquire the position information of the marker by acquiring the image containing the marker integrated on the wearable device, so as to track the wearable device. The image display device can determine the relative spatial positional relationship between the image display device and the wearable device according to the image containing the marker, and render the virtual object at the corresponding perspective according to the relative spatial positional relationship, so that the display perspective of the virtual object can change along with the positional relationship between the wearable device and the image display device, which is beneficial for the user to conveniently observe the virtual object from multiple perspectives.

[0035] Please refer to Figure 4 In one embodiment, the present application provides a positioning tracking method applied to the above-mentioned image display device, the method comprises: S101 to S105.

[0036] Step S101: acquiring an image containing a marker, wherein the marker is arranged on a wearable device.

[0037] Step S103: identifying the marker in the image, and determining the relative spatial positional information between the wearable device and the image display device according to the marker.

[0038] Step S105: rendering a virtual object according to the relative spatial positional information and displaying the virtual object in the image display device.

[0039] In one embodiment, based on the positioning tracking method described above, the present application further provides a positioning tracking method applied to the image display device described above, which comprises steps S101-S105.

[0040] Step S101: An image containing a marker is acquired, wherein the marker is arranged on a wearable device.

[0041] Further, the image containing the marker is acquired by an image acquisition device of the image display device, wherein the marker can be integrated into the wearable device, for example, can be a pattern fixedly presented on the surface of the wearable device, or can be a pattern selectively presented on the surface of the wearable device (for example, a pattern displayed after the wearable device is powered on).

[0042] Further, the wearable device can be glasses for a user to wear. At least two markers can be arranged on the glasses, and the at least two markers are arranged on a left frame and a right frame of the glasses respectively, so as to identify the eye position of the user. In some embodiments, at least three markers can be arranged on the glasses, and the at least three markers are arranged on the left frame, the right frame and a bridge of the glasses respectively, so as to identify the plane where the frames of the glasses are located, facilitate fitting the contour of the glasses, and acquire the spatial angle and the rotation posture of the glasses. It can be understood that the markers on the wearable device can be one or more, and the wearable device can also be other devices, for example, a hat with markers, a necklace with markers, a watch, a shirt, etc., but is not limited thereto.

[0043] Step S103: The marker in the image is identified, and the relative spatial position information between the wearable device and the image display device is determined according to the marker.

[0044] In some embodiments, the wearable device can include at least one marker, and at this time, step S103 can include: identifying the at least one marker contained in the acquired image, calculating the relative position and orientation relationship between the at least one marker and the image display device, and determining the relative spatial position relationship between the image display device and the wearable device.

[0045] Further, according to the display state, size and display angle of the marker in the image, the position information and the orientation information of the marker relative to the image display device are calculated, so as to determine the relative spatial position relationship between the wearable device and the image display device. The image display device can directly take the position information and the orientation information of the marker relative to the image display device as the relative spatial position relationship between the wearable device and the image display device.

[0046] In one embodiment, the wearable device may be glasses. When the glasses are worn, step S103 may include: determining relative spatial position information between the eyes of the user wearing the glasses and the image display device according to the marker.

[0047] In some embodiments, the marker provided on the glasses may include multiple sub-markers. Each sub-marker may be a pattern having a specific shape, and each sub-marker may include one or more characteristic points. The shape of the characteristic points is not limited and may be a dot, a ring, a triangle, or other shapes. The multiple sub-markers may be provided at different locations on the frame of the glasses, and the multiple sub-markers may collectively constitute a single marker. The image display device captures an image containing the multiple sub-markers, identifies each sub-marker, obtains characteristic information for each sub-marker, and determines the arrangement and positional relationship between the sub-markers, thereby determining the relative spatial positional relationship between the glasses and the image display device.

[0048] At this time, step S103 may include:

[0049] Step S1031: determining the sub-markers contained in the marker according to the image of the marker;

[0050] Step S1032: Locate the eye area of ​​the user wearing the glasses according to the sub-marker, and determine the relative spatial position information between the eyes of the user wearing the glasses and the image display device.

[0051] Thus, by extracting sub-markers from the image of the marker and tracking the sub-markers to determine the user's eye region, the relative spatial position information between the user's eyes and the image display device can be determined more accurately. The above-mentioned sub-markers can be understood as images of markers or partial images of markers at specific locations. For example, the image or partial image of a marker outside the user's eyes can be considered a sub-marker to facilitate identification of the user's eye region; or the image or partial image of a marker between the user's eyes can be considered a sub-marker to facilitate locating the user's eye region.

[0052] Further, the plurality of sub-markers arranged on the glasses worn by the user can be the same sub-markers, or can be different markers. The different sub-markers can have different characteristic information, which can include but is not limited to the shape, color, number of included feature points, etc. of the sub-markers. The image display device identifies the plurality of sub-markers arranged on the glasses, and can obtain the arrangement position relationship between the sub-markers. The arrangement position relationship refers to the relative positions and arrangement orders between the sub-markers, so that the pose and position of the marker can be determined according to the arrangement position relationship between the sub-markers and the size of the sub-markers, so as to obtain the relative spatial position relationship between the glasses and the image display device, or the relative spatial position information between the eyes of the user wearing the glasses and the image display device.

[0053] In one embodiment, in addition to arranging a plurality of sub-markers of one marker on different positions of the frame of the glasses respectively, one complete marker including a plurality of sub-markers can also be arranged on the frame of the glasses, for example, one marker is arranged at the middle position of the glasses. A plurality of markers can also be arranged at different positions of the frame of the glasses, so as to obtain the relative spatial position relationship between the plurality of markers and the image display device, or the relative spatial position information between the eyes of the user wearing the glasses and the image display device.

[0054] In one embodiment, the relative spatial position information between the eyes of the user and the image display device can be determined according to the eye image of the user, so as to improve the speed and fluency of tracking the eyes of the user. At this time, step S103 can further include:

[0055] Step S1035: acquiring an eye image of the user;

[0056] Step S1036: extracting an eye feature of the eye image, and determining the relative spatial position information between the eyes of the user wearing the glasses and the image display device according to the eye feature.

[0057] Further, the eye region of the user can be tracked in real time according to the eye image of the user, so as to calculate the motion data of the eyes of the user and predict the motion of the eyes of the user, which is beneficial to predict the motion trend of the eyes of the user, so as to improve the efficiency of obtaining the relative spatial position information. At this time, step S1036 can include: acquiring an eye image of the user wearing the glasses in real time, comparing the eye features between adjacent frames of the eye image to obtain the change of the position of the eyes of the user wearing the glasses, calculating the motion data of the eyes according to the change of the position of the eyes; and determining the relative spatial position information between the eyes of the user wearing the glasses and the image display device according to the motion data of the eyes.

[0058] Further, by comparing the eye features between the adjacent frames of the eye images, the increase of the eye position of the user wearing the glasses is obtained, and the motion data of the eye is estimated. For example, at the current Nth frame, the coordinates of the eye position are determined as (X, Y, Z), by comparing the same eye features between the Nth frame and the N+1th frame of the eye images, the increase of the eye position change is obtained as (x1, y1, z1), and then the coordinates of the eye position at the N+1th frame can be determined as (X+x1, Y+y1, Z+z1), and the coordinates of the eye position at the Nth frame, the N+1th frame, the N+2th frame, the N+2th frame, …, the N+mth frame can be calculated in turn, so that the motion data of the eye can be calculated. Wherein, the coordinates of the eye position at the current Nth frame can be (X, Y, Z) determined by the above step S103, that is, the coordinates of the eye position can be obtained according to the relative spatial position information between the wearable device and the image display device determined by the marker, and the coordinates of the eye position are taken as the reference amount, and the increase of the coordinates of the eye position is calculated by comparing the eye features between the adjacent frames of the eye images, so that the coordinates of the eye position at the N+1th frame, the N+2th frame, the N+2th frame, …, the N+mth frame are calculated one by one. In this way, the steps of data processing can be simplified, the speed of tracking the eye of the user can be improved, and the fluency can be improved.

[0059] Step S1037: According to the relative spatial position information determined in step S1035, the relative spatial position information determined in step S1038 is corrected to obtain the accurate relative spatial position information between the eye of the user wearing the glasses and the image display device. Further, the estimated position information is corrected according to the calibration position information to obtain the relative spatial position information between the eye of the user wearing the glasses and the image display device; wherein the estimated position information is the relative spatial position information determined according to the eye image, and the calibration position information is the relative spatial position information determined according to the marker.

[0060] Specifically, in some embodiments, the above step S1037 can be executed in the form of double threads, at this time, step S1037 can include: obtaining the real-time collected eye image through the first thread, and obtaining the estimated position information according to the eye image; obtaining the image containing the marker through the second thread, and obtaining the calibration position information according to the marker; comparing the estimated position information with the calibration position information, and when the estimated position information is inconsistent with the calibration position information, correcting the estimated position information according to the calibration position information.

[0061] In one embodiment, the image display device can fuse the position information obtained by the two threads to obtain the relative spatial position information between the eye of the user and the image display device, and the fusion manner can be various and is not limited herein. The image display device can obtain the relative spatial position information (i.e., the above-mentioned estimated position information) of the latest frame through the first thread, and obtain the relative spatial position information (i.e., the above-mentioned calibrated position information) of the same frame number as the latest frame through the second thread, and fuse the two to obtain the final relative spatial position information, for example, the average of the relative spatial position information obtained by the first thread and the relative spatial position information obtained by the second thread, or weighted and calculated according to different weights, etc.

[0062] In one embodiment, since the second thread obtains the relative spatial position information at a lower frame rate and at a slower speed, it can not directly obtain the relative spatial position information of the same frame number as the latest frame, and the second thread can estimate the relative spatial position information of the same frame number as the latest frame according to the relative spatial position information obtained by the previous frame.

[0063] Therefore, by acquiring the image of the marker to determine the relative spatial position information between the eye of the user and the image display device (referred to as calibrated position information), the relative spatial position information can be more accurate, and at the same time, by acquiring the image of the eye of the user to determine the relative spatial position information between the eye of the user and the image display device (referred to as estimated position information), the speed of obtaining the relative spatial position information can be improved, and the estimated position information is calibrated according to the calibrated position information to obtain more accurate relative spatial position information, which can balance the speed and accuracy and improve the fluency of the positioning and tracking method.

[0064] It can be understood that in some specific embodiments, the execution time of the above-mentioned steps is not limited, for example, steps S1031-S1033 can be executed first, and then steps S1035-S1037 can be executed; or steps S1031-S1033 and steps S1035-S1036 can be executed simultaneously, and then step S1037 can be executed; or steps S1035-S1036 can be executed first, and then steps S1031-S1033 and S1037 can be executed.

[0065] Further, the relative spatial position information between the eye of the user and the image display device obtained by the above-mentioned steps can be, but is not limited to, relative position information, relative orientation information, relative angle information, relative rotation information, and attitude information, etc.

[0066] Step S105: rendering a virtual object according to the relative spatial position information and displaying the virtual object in the image display device.

[0067] In some embodiments, after obtaining the relative spatial position information, model rendering data corresponding to the relative spatial position information can be obtained, and a virtual object can be rendered according to the model rendering data. The model data can include rendering coordinates, color data, texture data, rendering perspective, etc. for rendering. Specifically, after the image display device obtains the relative spatial position relationship between the marker and the image display device, the rendering coordinates of the virtual object can be determined according to the relative spatial position relationship, and the virtual object can be rendered and displayed according to the rendering coordinates. The rendering coordinates can be used to represent the relative spatial position relationship between the virtual object and the image display device in the virtual space. The relative spatial position relationship can include relative position, relative orientation, etc. The image display device can convert the relative spatial position relationship in the real space into relative coordinate data in the virtual space, and calculate the rendering coordinates of the virtual object in the virtual space according to the relative coordinate data, so that the virtual object can be accurately displayed.

[0068] In some embodiments, according to the relative spatial position information, the angle relationship between the wearable device and the image display device is determined, and the display perspective of the virtual object is determined according to the angle relationship. Step S105 can include:

[0069] Step S1051: obtaining the relative spatial angle between the wearable device and the image display device according to the relative spatial position information.

[0070] Step S1052: determining the display perspective of the virtual object according to the relative spatial angle between the wearable device and the image display device and the preset corresponding rule, rendering the virtual object, and displaying the virtual object in the image display device.

[0071] The preset corresponding rule is a corresponding relationship between the display perspective of the virtual object corresponding to the relative spatial angle. When the virtual object is a three-dimensional virtual model, for example Figure 1 As shown in the figure, the virtual object 600 presented by the image display device 300 is a simulation building model. When the user wearing the wearable device 400 stands at A to observe the virtual object 600 on the image display device 300, the relative spatial angle between the wearable device 400 and the image display device 300 is a first angle at this time. According to the relative spatial angle and the preset corresponding rule, the display perspective of the virtual object is determined to be a first perspective, so that the user can observe the view of the simulation building model at the first perspective (for example, a preset axonometric view). When the user moves to B, the relative spatial angle between the wearable device 400 and the image display device 300 is a second angle. According to the relative spatial angle and the preset corresponding rule, the display perspective of the virtual object is determined to be a second perspective (for example, a preset northwest view), so that the user can observe the structural view of the simulation building model relative to the left side of the user (for example, a preset northwest view).

[0072] Further, the relative spatial angle can be calculated in real time when determining the display perspective of the virtual object and rendering the virtual object, so that the user can observe the virtual object in the corresponding perspective during the movement, which is beneficial to improve the viewing of the image display.

[0073] Further, in some embodiments, the display perspective of the virtual object can be directly switched by setting a threshold of the relative spatial angle. Specifically, for example, when the relative spatial angle between the wearable device and the image display device does not fall within the threshold range of the relative spatial angle (e.g., below the lower threshold or above the upper threshold), the display perspective of the virtual object is determined as the opposite perspective, so that the user can observe the view of the opposite side of the virtual object. This is because the display of the image display device usually has a viewing angle, such as 5-175 degrees, and if the user observes the display outside the range of 5-175 degrees, it is difficult to see the image displayed on the display, which will cause the user to observe the virtual object at most to observe the view of the left and right sides of the virtual object relative to the user, and it is difficult to observe the back view of the virtual object relative to the user. Therefore, in order to overcome this defect, the display perspective of the virtual object is directly switched by setting a threshold of the relative spatial angle, and when the relative spatial angle between the wearable device and the image display device is below the lower threshold or above the upper threshold, the display perspective of the virtual object is directly determined as the back perspective relative to the user, and the virtual object is rendered to present the virtual object in the perspective of 180 degrees rotation in front of the user, which can improve the observation experience of the user. Therefore, in short, the step S1052 of the positioning tracking method described above can include:

[0074] Step S1053: determining whether the relative spatial angle between the wearable device and the image display device falls within the threshold range of the preset relative spatial angle, if yes, performing step S1054, if no, performing step S1055.

[0075] Step S1054: determining the display perspective of the virtual object as the front perspective, and determining the specific display perspective of the virtual object according to the relative spatial angle between the wearable device and the image display device and the preset corresponding rule, rendering the virtual object, and displaying the virtual object in the image display device.

[0076] It should be understood that the front perspective is the perspective that can be observed by the user wearing the wearable device relative to the display screen of the image display device within the maximum area range in front of the display screen, that is, the perspective that can be observed by the user wearing the wearable device when moving within the viewing range of the display screen. In terms of a specific virtual object, if the virtual object is a three-dimensional virtual model (such as a virtual human model), the front perspective of the virtual object is the perspective that can be observed by the user wearing the wearable device relative to the display screen of the image display device within the maximum area range in front of the display screen, that is, the perspective that can be observed by the user wearing the wearable device when moving within the viewing range of the display screen. Figure 1the front view angle of the virtual object can be understood as: the northeast view angle, the east view angle, the southeast view angle, the south view angle, the southwest view angle, the west view angle and the northwest view angle. Correspondingly, the virtual object can have a back view angle, which is the view angle of the virtual model other than the front view angle. In terms of a specific virtual object, if the virtual object is a three-dimensional virtual model (such as a simulated building model in Figure 1 the back view angle of the virtual object can be understood as the north view angle.

[0077] Step S1055: determining that the display view angle of the virtual object is the back view angle, rendering the virtual object, and displaying the virtual object in the image display device.

[0078] It can be understood that in step S105, the angle relationship between the wearable device and the image display device is determined according to the relative spatial position information, and the display view angle of the virtual object is determined according to the angle relationship. For example, the horizontal angle between the wearable device and the image display device changes, it should be understood that in other embodiments, when the angle between the wearable device and the image display device changes in the vertical direction, the display view angle of the virtual object is determined according to the relative spatial angle between the wearable device and the image display device and the preset corresponding rule, and the virtual object is rendered, so that the user can conveniently observe the bottom view or the top view of the virtual object.

[0079] In the positioning and tracking method provided by the embodiments of the present application, the image display device can acquire the position information of the marker integrated in the wearable device by collecting the image containing the marker, to track the wearable device. The image display device can determine the relative spatial position relationship between the image display device and the wearable device according to the image containing the marker, and display the constructed virtual object at the corresponding view angle according to the relative spatial position relationship, so that the display view angle of the virtual object can change along with the position relationship between the wearable device and the image display device, which is beneficial for the user to conveniently observe the virtual object from multiple angles.

[0080] Please refer to Figure 5In one embodiment, the present application provides a virtual content positioning tracking device 100 for executing the above-mentioned positioning tracking method. The virtual content positioning tracking device 100 includes an image acquisition module 101, a position relationship determination module 103, and a display module 105. The image acquisition module 201 is used to acquire images of markers, the position relationship determination module 103 is used to determine the relative spatial position relationship between the image display device and the wearable device based on the image containing the marker, and the display module 105 is used to render and display virtual objects based on the relative spatial position relationship. It can be understood that each of the above modules can be a program module running in a computer-readable storage medium. In an embodiment of the present application, the positioning tracking device 100 is stored in the memory of the image display device 300 and is configured to be executed by one or more processors of the image display device 300. The workings of each of the above modules are as follows:

[0081] The image acquisition module 101 is used to acquire an image containing a marker, wherein the marker is set on the wearable device. Furthermore, the image acquisition module 101 acquires the image containing the marker through the image acquisition device of the image display device.

[0082] The position relationship determination module 103 is used to identify markers in the image and determine the relative spatial position information between the wearable device and the image display device based on the markers. The position relationship determination module 103 includes a first position information determination unit 1031, a second position information determination unit 1033, and a calibration unit 1035.

[0083] The first position information determination unit 1031 is used to determine the relative spatial position information between the eyes of the user wearing glasses and the image display device based on the marker. Specifically, the first position information determination unit 1031 is used to identify at least one marker contained in the captured image, calculate the relative position and orientation relationship between the at least one marker and the image display device, and determine the relative spatial position relationship between the image display device and the wearable device. The first position information determination unit 1031 is further used to determine the sub-markers contained in the marker based on the image of the marker; locate the eye area of ​​the user wearing glasses based on the sub-markers, and determine the relative spatial position information between the eyes of the user wearing glasses and the image display device.

[0084] The second position information determination unit 1033 is configured to determine the relative spatial position information between the user's eyes and the image display device based on the user's eye image, thereby improving the speed and smoothness of eye tracking. Specifically, the second position information determination unit 1033 is configured to extract eye features from the user's eye image and determine the relative spatial position information between the eyes of the user wearing glasses and the image display device based on the eye features.

[0085] The correction unit 1035 is configured to correct the relative spatial position information determined by the second position information determination unit 1033 according to the relative spatial position information determined by the first position information determination unit 1031. Specifically, the correction unit 1035 is configured to take the relative spatial position information determined according to the image of the eye region as estimated position information, and take the relative spatial position information determined according to the marker as calibration position information; correct the estimated position information according to the calibration position information to obtain the relative spatial position information between the eye of the user wearing the glasses and the image display device. Further, the correction unit 1035 is configured to compare the estimated position information with the calibration position information, and correct the estimated position information according to the calibration position information when the estimated position information is inconsistent with the calibration position information.

[0086] The display module 105 is configured to render the virtual object according to the relative spatial position information and display the virtual object in the image display device. In some embodiments, the display module 105 is configured to determine the angle relationship between the wearable device and the image display device according to the relative spatial position information, and determine the display perspective of the virtual object according to the angle relationship, and the display module 105 can include an angle determination unit 1051, a perspective determination unit 1053, a rendering unit 1055, and a display unit 1057.

[0087] In some embodiments, when the position relationship determination module 103 obtains the relative spatial position information, the rendering unit 1055 is configured to obtain model rendering data corresponding to the relative spatial position information, and render the virtual object according to the model rendering data. Specifically, when the position relationship determination module 103 obtains the relative spatial position relationship between the marker, the rendering unit 1055 is configured to determine the rendering coordinates of the virtual object according to the relative spatial position relationship, and render and display the virtual object according to the rendering coordinates, and the rendering coordinates can be used to represent the relative spatial position relationship between the virtual object and the image display device in the virtual space. The display unit 1057 is configured to display the rendered virtual object in the image display device.

[0088] Further, when the rendering unit 1055 determines the display perspective of the virtual object according to the relative spatial angle and renders the virtual object, the relative spatial angle can be calculated in real time, and the virtual object can be rendered in real time, and the display unit 1057 can display the rendered virtual object in real time, so that the user can observe the virtual object to be transformed at the corresponding perspective in the moving process, which is beneficial to improve the viewing of the image display.

[0089] The angle determination unit 1051 is configured to obtain the relative spatial angle between the wearable device and the image display device according to the relative spatial position information;

[0090] The view angle determination unit 1053 is configured to determine the display view angle of the virtual object according to the relative spatial angle between the wearable device and the image display device and a preset corresponding rule. Further, in some embodiments, the view angle determination unit 1053 is further configured to trigger the display view angle of the virtual object to directly switch by setting a threshold of the relative spatial angle.

[0091] Please refer again to Figure 2 In one embodiment, the present application provides a wearable device 400 for assisting in positioning tracking. The wearable device 400 of the present application is a pair of glasses for a user to wear, which includes a frame 410 and a marker 450 arranged on the frame 410. The marker 450 is identified by a terminal device (such as the image display device 300) to determine the relative position relationship between the eye of the user and the terminal device.

[0092] In the embodiment of the present application, the frame 410 includes a left temple 411, a right temple 413, and a frame 415 arranged between the left temple 411 and the right temple 413. In other embodiments, the wearable device 400 can be a clip-on glasses, and the wearable device 400 does not include a temple, but includes a clip connected to the frame 415, which directly clips the wearable device 400 on the myopic glasses of the user, bringing great convenience to myopic users.

[0093] The frame 415 includes a left frame 4151, a right frame 4153, and a bridge 4155 arranged side by side, the left frame 4151 is connected to the left temple 411, the right frame 4153 is connected to the right temple 413, and the bridge 4155 is connected between the left frame 4151 and the right frame 4153.

[0094] The marker 450 is arranged on the frame 415. Specifically, the marker 450 is arranged on the outer side of the frame 415, which is understood to be the side of the frame 415 facing away from the user's eye when the user wears the wearable device 400. In the embodiment, the marker 450 is a plurality of markers 450, which can be divided into a plurality of marker groups according to the arrangement, for example, the plurality of markers 450 includes a first group 451, a second group 453, and a third group 455, the first group 451 is arranged on the left frame 4151, the second group 453 is arranged on the right frame 4153, and the third group 455 is arranged on the bridge 4155. The third group 455 is different from the first group 451 and the second group 453, so as to facilitate the terminal device (such as the image display device 300) to determine the spatial position of the wearable device 400 by identifying the three markers 450. Alternatively, in other embodiments, the third group 455 is different from the first group 451 and the second group 453.

[0095] Further, in order to facilitate fitting the contour of the wearable device 400, the markers 450 can be evenly distributed on the contour of the frame 415. At this time, the first group 451 can include one or more sub-markers, for example, can include a first sub-marker 4511 and a second sub-marker 4513, which are respectively arranged on both sides of the left frame 4151. Specifically, in the embodiment shown in Figure 2 and Figure 3 , the first sub-marker 4511 and the second sub-marker 4513 are both arranged on the side of the left frame 4151 away from the nose bridge 4155, and the first sub-marker 4511 is arranged at the upper end of the left frame 4151, and the second sub-marker 4513 is arranged at the lower end of the left frame 4151. The above-mentioned upper end should be understood as the end of the frame 415 close to the user's eyebrows when the user wears the wearable device 400, and correspondingly, the above-mentioned lower end should be understood as the end of the frame 415 away from the user's eyebrows when the user wears the wearable device 400.

[0096] Correspondingly, the second group 453 can include one or more sub-markers, for example, can include a third sub-marker 4531 and a fourth sub-marker 4533, which are respectively arranged on both sides of the right frame 4153. Specifically, in the embodiment shown in Figure 2 and Figure 3 , the third sub-marker 4531 and the fourth sub-marker 4533 are both arranged on the side of the right frame 4153 away from the nose bridge 4155, and the third sub-marker 4531 is arranged at the upper end of the right frame 4153, and the fourth sub-marker 4533 is arranged at the lower end of the right frame 4153.

[0097] The third group 455 can include a fifth sub-marker 4551, which is arranged on the nose bridge 4155. The fifth sub-marker 4551 and the first sub-marker 4511, the second sub-marker 4513, the third sub-marker 4531 and the fourth sub-marker 4533, these five markers can be different from each other (as shown in Figure 2 ), can be completely the same (as shown in Figure 3 ), or at least two of them are the same (as shown in Figure 4 ).

[0098] Please refer to Figure 4In some embodiments, the first sub-marker 4511, the second sub-marker 4513, the third sub-marker 4531, and the fourth sub-marker 4533 are all identical, while the fifth sub-marker 4551 is different from the other four sub-markers, so that the five sub-markers together form the outline of the wearable device 400. In this way, by setting the four sub-markers around as the same marker, and setting the sub-marker (the fifth sub-marker 4551) approximately in the middle as a marker different from the other sub-markers, the image display device 300 is facilitated to recognize the outline of the wearable device 400, and to locate the approximate middle position of the wearable device 400, so as to facilitate the positioning and tracking of the wearable device 400. Among the first sub-marker 4511, the second sub-marker 4513, the third sub-marker 4531, and the fourth sub-marker 4533, each sub-marker includes a background 457 and one feature point 459, the feature point 459 being different from the background so as to be recognized by the image display device 300. The fifth sub-marker 4551 includes a background 427 and two feature points 459. In this embodiment, the feature points 459 are approximately circular. By setting a small number of feature points 429 on the sub-markers, the area of the feature points 429 can be relatively large (e.g., occupying 1 / 3 or more of the area of the sub-marker), so that the sub-markers are more easily recognized by the image display device 300.

[0099] In this embodiment, the marker 450 is a planar marker integrated in the frame 415, which can be a predetermined symbol or pattern. The marker 450 is used to be recognized by the image display device 300, and to determine the relative spatial position relationship between the wearable device 400 and the image display device 300. In this embodiment, the marker 450 is a physical structure provided on the display panel 430. In other embodiments, the marker 430 can also be a predetermined symbol or pattern displayed after being powered on.

[0100] It can be understood that the specific pattern displayed by the marker 450 is not limited, which can be any pattern that can be acquired by the image acquisition device 301 of the image display device 300. For example, the specific pattern of the marker 450 can be one or a combination of the following patterns: a circle, a triangle, a rectangle, an oval, a wavy line, a straight line, a curve, and the like, and is not limited to the description in the specification. It can be understood that in other embodiments, the marker 450 can be other types of patterns, and the marker 450 can be more effectively identified by the image acquisition device 301. For example, the specific pattern of the marker 450 can be a geometric pattern (such as a circle, a triangle, a rectangle, an oval, a wavy line, a straight line, a curve, and the like) that can be distinguished by the image acquisition device 301, a predetermined pattern (such as an animal head, a commonly used symbol such as a traffic sign, or the like) or other patterns to form a marker, and is not limited to the description in the specification. It can also be understood that in other embodiments, the marker 450 can be a bar code, a two-dimensional code, or the like.

[0101] Further, in some embodiments, the wearable device 400 can further include a light filtering layer (not shown in the figure), which can be stacked on the side of the marker 450 away from the frame 415.

[0102] The light filtering layer is used to filter light other than the light emitted by the illumination device of the image display device 300 to the marker 450, so as to avoid the influence of ambient light on the marker 450 when reflecting light, thereby making the marker 450 more easily identified. In some embodiments, the light filtering performance of the light filtering layer can be set according to actual needs. For example, when the marker 450 enters the field of view of the image acquisition device 301 to be identified, in order to improve the identification efficiency, the image acquisition device 301 usually acquires images with the aid of an auxiliary light source, for example, when an infrared light source is used for assistance, the light filtering layer is used to filter light other than infrared light (such as visible light, ultraviolet light, etc.), so that light other than infrared light cannot pass through the light filtering layer, and infrared light can pass through and reach the marker 450. When the auxiliary light source projects infrared light to the marker 450, the light filtering layer filters ambient light other than infrared light, so that only infrared light reaches the marker 450 and is reflected to the near-infrared image acquisition device, thereby reducing the influence of ambient light on the identification process.

[0103] In one embodiment, the present application also provides a computer readable storage medium, which stores program codes that can be called by a processor to execute the method described in the above method embodiment. It should be noted that in the embodiments provided in the specification, the above-mentioned embodiments can be combined with each other without conflict, and the features of each embodiment can also be combined with each other, and are not limited by the embodiments.

[0104] The positioning tracking method and the wearable device described above, by the positioning tracking method, display a virtual object according to a marker, so that the virtual object can be intuitively displayed in front of a user, and the virtual object can be controlled in real time by the wearable device, which is conducive to realizing the interaction between the user and the virtual object, making the information carried by the virtual object easier to obtain, and improving the user experience.

[0105] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0106] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0107] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not drive the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of position tracking, characterized by, The positioning tracking method is applied to a non-wearable image display device, and comprises the following steps: An image containing a marker is acquired, the marker being arranged on a wearable device; The marker in the image is recognized, and relative spatial position information between the wearable device and the image display device is determined according to the marker; and According to the relative spatial position information, a relative spatial angle between the wearable device and the image display device is acquired; When the angle between the wearable device and the image display device changes in a horizontal direction, if the relative spatial angle falls within a preset threshold range of the relative spatial angle, it is determined that a display perspective of a virtual object is a front perspective, the display perspective of the virtual object is determined according to the relative spatial angle and a preset corresponding rule, the virtual object is rendered according to the display perspective of the virtual object, and the virtual object is displayed in the image display device; if the relative spatial angle does not fall within the preset threshold range of the relative spatial angle, it is determined that the display perspective of the virtual object is a back perspective, the virtual object is rendered according to the back perspective, and the virtual object is displayed in the image display device, the back perspective being a perspective of the virtual object other than the front perspective; When the angle between the wearable device and the image display device changes in a vertical direction, the display perspective of the virtual object is determined according to the relative spatial angle between the wearable device and the image display device and the preset corresponding rule, and the virtual object is rendered, the display perspective of the virtual object including a bottom perspective or a top perspective, the display perspective of the virtual object changing along with the change of the relative spatial angle in the vertical direction.

2. The method of claim 1, wherein, The wearable device is glasses, and the marker is arranged on a frame of the glasses; When the glasses are worn, the relative spatial position information between the glasses and the image display device is determined according to the marker, comprising: The relative spatial position information between an eye of a user wearing the glasses and the image display device is determined according to the marker.

3. The method of claim 2, wherein, The relative spatial position information between the eye of the user wearing the glasses and the image display device is determined according to the marker, comprising: The marker contains a sub-marker according to the image of the marker; The eye region of the user wearing the glasses is located according to the sub-marker, and the relative spatial position information between the eye of the user wearing the glasses and the image display device is determined.

4. The method of claim 2, wherein, Before the virtual object is rendered according to the relative spatial position information, the method further comprises: An eye image of the user is acquired; An eye feature of the eye image is extracted, and the relative spatial position information between the eye of the user wearing the glasses and the image display device is determined according to the eye feature.

5. The method of claim 4, wherein, The relative spatial position information between the eye of the user wearing the glasses and the image display device is determined according to the eye feature, comprising: The eye feature is compared between adjacent frames of the eye image, and the position change of the eye of the user wearing the glasses is obtained. According to the eye position change, calculate motion data of the eye; and According to the motion data of the eye, determine relative spatial position information between the eye of a user wearing the glasses and the image display device.

6. The method of any one of claims 4 or 5, wherein, After the relative spatial position information between the eye of a user wearing the glasses and the image display device is determined according to the eye feature, the method further comprises: According to the calibration position information, correct the estimated position information to obtain the relative spatial position information between the eye of a user wearing the glasses and the image display device; The estimated position information is the relative spatial position information determined according to the eye image, and the calibration position information is the relative spatial position information determined according to the marker.

7. The method of claim 6, wherein, The method further comprises: Through a first thread, obtain the eye image collected in real time, and according to the eye image, obtain the estimated position information; Through a second thread, obtain the image containing the marker, and according to the marker, obtain the calibration position information, wherein the frame rate of the second thread is lower than that of the first thread, when the same frame number of calibration position information as the latest frame obtained by the first thread is not obtained through the second thread, estimate the same frame number of calibration position information as the latest frame through the calibration position information obtained by the second thread; Compare the estimated position information with the calibration position information, and when the estimated position information is inconsistent with the calibration position information, correct the estimated position information according to the calibration position information.

8. A position tracking device, characterized by The positioning and tracking device comprises: An image acquisition module, configured to obtain an image containing a marker, the marker being arranged on a wearable device; A position relationship determination module, configured to identify the marker in the image, and determine relative spatial position information between the wearable device and the image display device according to the marker; and The display module is configured to: when the angle between the wearable device and the image display device changes in a horizontal direction, acquire a relative spatial angle between the wearable device and the image display device according to the relative spatial position information; when the relative spatial angle falls within a preset threshold range of relative spatial angles, determine a display perspective of the virtual object as a front perspective, determine a display perspective of the virtual object according to the relative spatial angle and a preset corresponding rule, render the virtual object according to the display perspective of the virtual object, and display the virtual object in the image display device; when the relative spatial angle does not fall within the preset threshold range of relative spatial angles, determine the display perspective of the virtual object as a back perspective, render the virtual object according to the back perspective, and display the virtual object in the image display device, the back perspective being a perspective of the virtual object other than the front perspective; when the angle between the wearable device and the image display device changes in a vertical direction, determine the display perspective of the virtual object according to the relative spatial angle between the wearable device and the image display device and the preset corresponding rule, and render the virtual object, the display perspective of the virtual object including a bottom perspective or a top perspective, and the display perspective of the virtual object changing along with a change in the relative spatial angle in the vertical direction.

9. A wearable device for assisting in locating a tracking device, comprising a mirror frame, characterized in that, The display module is configured to: when the angle between the wearable device and the image display device changes in a horizontal direction, acquire a relative spatial angle between the wearable device and the image display device according to the relative spatial position information; when the relative spatial angle falls within a preset threshold range of relative spatial angles, determine a display perspective of the virtual object as a front perspective, determine a display perspective of the virtual object according to the relative spatial angle and a preset corresponding rule, render the virtual object according to the display perspective of the virtual object, and display the virtual object in the image display device; when the relative spatial angle does not fall within the preset threshold range of relative spatial angles, determine the display perspective of the virtual object as a back perspective, render the virtual object according to the back perspective, and display the virtual object in the image display device, the back perspective being a perspective of the virtual object other than the front perspective; when the angle between the wearable device and the image display device changes in a vertical direction, determine the display perspective of the virtual object according to the relative spatial angle between the wearable device and the image display device and the preset corresponding rule, and render the virtual object, the display perspective of the virtual object including a bottom perspective or a top perspective, and the display perspective of the virtual object changing along with a change in the relative spatial angle in the vertical direction.

10. The wearable device of claim 9, wherein, The mirror frame comprises a left frame and a right frame arranged side by side, the marker comprises a first sub-marker and a second sub-marker, and the first sub-marker and the second sub-marker are arranged on both sides of the left frame respectively; Or / and, the mirror frame comprises a left frame and a right frame arranged side by side, and the marker further comprises a third sub-marker and a fourth sub-marker, and the third sub-marker and the fourth sub-marker are arranged on both sides of the right frame respectively; Or / and, the mirror frame comprises a left frame and a right frame arranged side by side, and the wearable device further comprises a nose bridge part connected between the left frame and the right frame, and the marker further comprises a fifth sub-marker arranged on the nose bridge part.

11. The wearable device of any one of claims 9-10, wherein, Further comprising a filter layer, and the filter layer covers the marker.

Images