AR device attitude detection method, AR device attitude detection device, electronic device, and computer program

Abstract

The present application relates to the technical field of computers, particularly to the technical field of augmented reality, and provides a method, apparatus, electronic device, and storage medium for detecting the orientation of an AR device. The method includes the steps of: acquiring at least two collected images based on different shooting orientations by emitting detection rays from an AR device; determining predicted distances for at least two pairs of target image feature points that include the same target image feature based on the at least two collected images; determining target device feature points in the AR device that correspond to each of the target image feature points based on a comparison result between the actual distances between each of the device feature points in the AR device and the predicted distances; and determining orientation information of the AR device based on position information of each of the target device feature points and each of the target image feature points. In this application, the AR device only needs to reflect the detection rays, and the orientation information is obtained by simple distance comparison, thereby reducing power consumption and computational power required for detecting the orientation of the AR device.

Description

[Technical Field]

[0001] The present application relates to the technical field of computers, and in particular to the technical field of augmented reality, and provides a method, apparatus, electronic device, and storage medium for detecting the orientation of an AR device. [Background technology]

[0002] With the development of science, technologies such as Augmented Reality (AR) and Virtual Reality (VR) have gradually matured and become popular among the public. AR technology is a combination of virtual and real worlds, and can enhance the sense of reality by presenting virtual text, images, 3D models, etc. in the real world through AR devices. Orientation detection for AR devices is a very basic and important part of AR technology.

[0003] Taking a head-mounted device as an example, the orientation detection method for an AR head-mounted device is mainly realized using devices such as a camera and a depth sensor. Because these devices are all located on the head-mounted device, the operation of the camera and depth sensor requires high power from the AR head-mounted device, and the operation of the depth sensor also places a certain demand on the device's computing power, resulting in high power consumption for the head-mounted device.

[0004] Therefore, how to reduce the power consumption and computational power required for posture detection of AR devices is an urgent issue that needs to be addressed. Summary of the Invention [Problem to be solved by the invention]

[0005] Embodiments of the present application provide a method, apparatus, electronic device, storage medium, and program product for detecting the orientation of an AR device, for reducing power consumption of the orientation detection of the AR device. [Means for solving the problem]

[0006] The method for detecting the orientation of an AR device provided in the embodiment of the present application includes: A step of acquiring at least two collected images based on different shooting orientations by emitting a detection light beam to an AR device, wherein each collected image includes image feature points obtained by reflecting the detection light beam based on device feature points of the AR device; determining a predicted distance for each of at least two pairs of target image feature points that include the same target image feature point based on the at least two collected images; determining target device features in the AR device corresponding to each of the target image features based on a comparison result between the actual distances and the predicted distances between the device features in the AR device; and determining posture information of the AR device based on position information of each target device feature point and each target image feature point.

[0007] The orientation detection device for the AR device provided in the embodiment of the present application is an acquisition unit configured to acquire at least two collected images based on different photographing orientations by emitting a detection light beam to an AR device, each of the collected images including image feature points obtained by reflecting the detection light beam based on device feature points in the AR device; a prediction unit for determining a predicted distance for each of at least two target image feature pairs including the same target image feature point based on the at least two collected images; a comparison unit for determining target device feature points in the AR device corresponding to each of the target image feature points based on a comparison result between the actual distances and the predicted distances between the device feature points in the AR device; and a determination unit for determining attitude information of the AR device based on position information of each target device feature point and each target image feature point, respectively.

[0008] Optionally, each of the collected images is collected based on a collection device located at one of the photographing orientations, and the prediction unit specifically: A predicted distance for each of the at least two pairs of target image feature points is determined based on the distance between the at least two pairs of target image feature points in each collected image and an orientation parameter indicating a positional relationship between the at least two collection devices.

[0009] Optionally, the actual distances between any two device feature points are different, and the difference value between any two actual distances is greater than a first predetermined threshold.

[0010] Optionally, the comparison unit specifically comprises: comparing actual distances between the device feature points in the AR device with the predicted distances; determining at least two device feature pairs containing identical device feature points based on the comparison result, wherein a difference value between the actual distance and the corresponding predicted distance between each device feature pair is smaller than a second predetermined threshold, and the second predetermined threshold is equal to or smaller than the first predetermined threshold; A target device feature point corresponding to each of the target image feature points is determined from among the at least two device feature point pairs.

[0011] The electronic device provided in the embodiments of the present application includes a processor and a memory, and a computer program is stored in the memory. When the computer program is executed by the processor, the electronic device realizes the steps of any one of the above-described methods for detecting the posture of an AR device.

[0012] A computer-readable storage medium provided in an embodiment of the present application includes a computer program, which, when executed by a processor, causes the steps of any one of the above-described methods for detecting the attitude of an AR device to be realized.

[0013] A computer program product provided in an embodiment of the present application includes a computer program, which, when executed by a processor, realizes the steps of any one of the above-described methods for detecting the attitude of an AR device. [Effects of the Invention]

[0014] The beneficial effects of the present invention are as follows:

[0015] The present application provides a method, apparatus, electronic device, and storage medium for detecting the orientation of an AR device. In this application, a base station emits a detection beam to an AR device, and the base station acquires at least two collected images based on different shooting orientations. The base station then determines the reflection of the detection beam by device features in the AR device based on image features in the images. This means that both the detection beam emitter and the shooting device are located at the base station. This eliminates the power consumption of the emitter and the shooting device in the AR device. The base station then predicts the predicted distances in the real environment for two pairs of target image features containing the same target image feature based on each collected image. The base station then compares the predicted distances with the actual distances between the device features in the AR device to find the target device features corresponding to each of the target image features. This process only requires a simple distance calculation, i.e., distance comparison, which reduces the requirements for the device's computing power. After finding the correspondence, the base station can calculate the posture information of the AR device based on the position information of each target device feature point in the AR device coordinate system and the position information of each target image feature point in the base station coordinate system. Since the process of determining the posture of the AR device is also performed by the base station, there is no need for the AR device to provide computational support. This effectively reduces the power consumption and computational requirements for posture detection of the AR device.

[0016] Other features and advantages of the present application will be set forth in the following description, and in part will be obvious from the description, or may be learned by the practice of the present application. The objectives and other advantages of the present application may be achieved and obtained by the structure particularly pointed out in the written description, claims, and drawings. [Brief explanation of the drawings]

[0017] The drawings described herein are intended to provide a further understanding of the present application and constitute a part of the present application. The schematic examples and the description thereof are intended to aid in the interpretation of the present application and are not to be construed as an undue limitation of the present application. [Figure 1] FIG. 1 is a schematic diagram illustrating an application scenario of the AR device orientation detection method provided in the embodiment of the present application. [Figure 2] 1 is an overall flowchart of a method for detecting the orientation of an AR device provided in an embodiment of the present application; [Figure 3] 1 is a schematic diagram of an AR device provided in an example of the present application. [Figure 4] 1 is a schematic diagram of a collected image provided in an embodiment of the present application. [Figure 5] FIG. 2 is a schematic diagram of target image feature points provided in an embodiment of the present application; [Figure 6] FIG. 1 is a logic diagram of the predicted distance calculation provided in an embodiment of the present application. [Figure 7] 1 is a flowchart for determining target device feature points corresponding to each target image feature point provided in an embodiment of the present application; [Figure 8] FIG. 1 is a diagram illustrating the correspondence between target image feature points and target device feature points provided in an embodiment of the present application. [Figure 9] FIG. 10 is a logic diagram for determining device features corresponding to target image features provided in an embodiment of the present application. [Figure 10] 1 is a diagram illustrating an interaction between an AR device, a base station, and a terminal device provided in an embodiment of the present application. [Figure 11] 1 is a flowchart illustrating a specific implementation of another AR device orientation detection method provided in an embodiment of the present application; [Figure 12] FIG. 1 is a logic diagram illustrating an interaction between an AR device, a base station, and a terminal device provided in an embodiment of the present application. [Figure 13] 1 is a schematic diagram illustrating the configuration of an attitude detection device of an AR device provided in an embodiment of the present application. [Figure 14] FIG. 1 is a schematic diagram of a hardware configuration of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0018] In order to clarify the purpose, configuration, and advantages of the embodiments of the present application, the present invention will be clearly and completely described below with reference to the drawings of the embodiments of the present application. It is clear that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments that a person skilled in the art can obtain from the embodiments described in this specification without creative work fall within the scope of protection of the present invention.

[0019] Some of the concepts related to the embodiments of the present application will be introduced below.

[0020] Collected image: Obtained by the base station by photographing the AR device. The AR device has device feature points that can reflect the detection light. Accordingly, the collected image also includes image feature points obtained by reflecting the detection light based on the device feature points of the AR device, and reflects the reflection of the infrared detection light by the device feature points.

[0021] Image feature points: The base station captures device feature points on the AR device to obtain visible image feature points corresponding to each device feature point in the collected image. In this application, the image feature points are divided into target image feature points and candidate image feature points. The target image feature points are the three image feature points initially acquired by the base station, and the candidate image feature points are candidate image feature points corresponding to the candidate device feature points selected by the base station to adjust the posture information after acquiring posture information based on the target image feature points.

[0022] Device feature point: A point on an AR device, which may be made of a material that can reflect infrared light (or other detection light, but is not specifically limited in this specification). In an embodiment of the present application, in order to further improve the efficiency of the base station side in determining corresponding points of target image features and reduce errors, the distances between any two device feature points may be set to be different, and the absolute value of the difference in actual distances corresponding to any two device feature point pairs may be set to be greater than a first predetermined threshold.

[0023] Base station coordinate system: A coordinate system that has a certain point in the base station as its origin. For example, if the center point of a collection device in the base station is set as the origin of the base station coordinate system, the positional relationship of a certain point with respect to the base station can be reflected in this coordinate system.

[0024] AR device coordinate system: A coordinate system with a certain point on the AR device as its origin. For example, if the midpoint of the line connecting the centers of the two eye lenses on the AR device is set as the origin of the AR device coordinate system, the positional relationship of a certain point to the AR device can be reflected in this coordinate system.

[0025] Predicted distance: The distance between two target image feature points in multiple collected images and the device feature points corresponding to the two target image feature points calculated by the base station based on the distance between the two target image feature points and the positional relationship between the multiple collecting devices. In this application, the predicted distance is divided into a first predicted distance and a second predicted distance. Here, the first predicted distance is the predicted distance of one target image feature point pair corresponding to three target image feature points, and the second predicted distance is the predicted distance of another target image feature point pair corresponding to the three target image feature points.

[0026] The design concept of the embodiment of the present invention will be briefly introduced below.

[0027] With the development of science, technologies such as AR and VR have gradually matured and become more visible to the public. AR technology is a combination of virtual and real worlds, and can enhance the sense of reality by presenting virtual text, images, 3D models, etc. in the real world through a head-mounted AR device. Posture detection for head-mounted AR devices is a very basic and important part of AR technology.

[0028] Related technologies mainly use an outside-in approach for VR head-mounted devices. That is, the sensor that can determine the device's orientation is located outside the head-mounted device, not on the head-mounted device. However, this approach requires placing an LED array on the head-mounted device, which emits light spontaneously and the sensor detects the light to achieve positioning. This implementation process is complicated, and the need to place the LED array on the head-mounted device significantly increases the power consumption of the head-mounted device.

[0029] Meanwhile, the orientation detection method for AR head-mounted devices mainly adopts the inside-out method, that is, the camera and depth sensor that can determine the device's orientation are directly located on the head-mounted device. During orientation detection, the camera and depth sensor consume a large amount of power for the head-mounted device, and the depth sensor needs to perform a series of calculations, which places high demands on the computing power of the head-mounted device.

[0030] In view of this, embodiments of the present application provide a method, apparatus, electronic device, storage medium, and program product for detecting the orientation of an AR device. In this application, a base station emits detection light beams to an AR device, and the base station acquires at least two collected images based on different shooting orientations. The base station determines the reflection status of the detection light beams by device features in the AR device based on image features in the images. That is, both the detection light beam emitter and the shooting device are located in the base station, not in the AR device. Therefore, the AR device does not consume power from the emitter and the shooting device.

[0031] The base station then predicts the predicted distances in the real environment of two target image feature pairs containing the same target image feature based on each collected image, and compares the predicted distances with the actual distances between the device feature points in the AR device. Because the actual distances between the device feature points in the AR device are all different, the base station can find the target device feature points corresponding to each of the target image features by comparing the distances.

[0032] Finally, the base station can calculate the position information of the AR device relative to the base station, i.e., the orientation information of the AR device, based on the position information of each target device feature point in the AR device coordinate system and the position information of each target image feature point in the base station coordinate system. Since the process of determining the orientation of the AR device is also performed by the base station, there is no need for the AR device to provide computational support. This reduces the power consumption and computational requirements for detecting the orientation of the AR device.

[0033] Hereinafter, preferred embodiments of the present application will be described with reference to the drawings in the specification. It should be understood that the preferred embodiments described herein are used only for the explanation and interpretation of the present application, and are not intended to limit the present application. Furthermore, if not inconsistent, the embodiments and features of the embodiments in the present application can be combined with each other.

[0034] The configuration provided in this application can be applied to 6-DOF spatial posture detection of lightweight AR glasses to realize real-virtual fusion applications in specific scenarios (e.g., office scenarios, board game scenarios, etc.). One optional application scenario is shown in FIG. 1.

[0035] 1 is a schematic diagram of an application scenario of an embodiment of the present application. As shown in FIG. 1, the application scenario diagram includes one AR device 110, one base station 120, and one terminal device 130.

[0036] Here, the AR device includes, but is not limited to, a head-mounted AR device, a helmet-mounted AR device, and the like.

[0037] In the present embodiment, the AR device 110 is a head-mounted AR device (e.g., lightweight AR glasses). The AR device includes an AR optical display module for projecting and displaying a virtual screen (e.g., the shaded area in FIG. 1 ). The AR device further includes an advanced reduced instruction set computer (ARM SOC) and a device feature capable of reflecting light. For example, if the light emitted from the base station is infrared light, the device feature may be an infrared light reflecting point made of infrared fluorescent material. The AR device 110 connects to the hotspot of the terminal device 130 via wireless fidelity (WIFI) and performs local time synchronization with the base station 120 via Network Time Protocol (NTP).

[0038] It should be noted that other types of AR devices are equally applicable to the present application, and the above-mentioned light beam emitted from the base station and the material on which the device features are disposed are merely exemplary, and the present application does not specifically limit them.

[0039] The base station 120 is a positioning base station including a plurality of collecting devices (for example, a pair of infrared binocular cameras), a light emitting device (for example, an infrared LED light), and an ARM SOC. The base station 120 deploys an NTP local time server and connects to the Internet via the WIFI hotspot of the terminal device 130 to synchronize the global time.

[0040] The terminal device 130 provides a WIFI hotspot for the AR device, allowing the AR device 110 to network with the base station 120 and synchronize the base station local time via the NTP protocol. The terminal device 130 may include, but is not limited to, a mobile phone, a tablet computer, a laptop computer, a desktop computer, an e-reader, a smart voice interaction device, a smart home appliance, an in-vehicle terminal, and the like.

[0041] The AR device orientation detection method in each embodiment of the present application is executed by the base station 120. The base station 120 emits infrared detection light toward the AR device 110 using an infrared LED light, and simultaneously acquires two collected images obtained by reflecting the infrared detection light from device features in the AR device 110 using an infrared binocular camera including two collection devices positioned at different shooting orientations. Based on the two collected images, the base station 120 calculates predicted distances in an actual scenario for at least two target image feature pairs containing the same target image feature, and compares these predicted distances with different actual distances between the device features in the AR device 110 to determine target device features in the AR device 110 corresponding to each of the target image feature pairs. Finally, the base station 120 acquires orientation information of the AR device 110 by calculating relative position information of the AR device 110 with respect to the base station 120 based on position information of each target device feature in the AR device coordinate system and position information of each target image feature in the base station coordinate system.

[0042] Note that what is shown in FIG. 1 is merely an example, and the number of AR devices 110, base stations 120, and terminal devices 130 is not limited, and the embodiments of the present application do not specifically limit them.

[0043] Furthermore, the embodiments of the present application are applicable to various scenarios, including but not limited to scenarios such as cloud technology, artificial intelligence, smart transportation, and driver assistance.

[0044] Hereinafter, the posture detection method for an AR device provided in an exemplary embodiment of the present application will be described with reference to the drawings, taking into account the above-mentioned application scenarios. It should be noted that the above-mentioned application scenarios are provided only to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect.

[0045] 2 is a flowchart illustrating an implementation of the AR device posture detection method provided in the embodiment of the present application. The method is implemented by a base station. As shown in FIG. 2, the method may include the following specific implementation steps:

[0046] In S201, the base station emits a detection beam to the AR device to acquire at least two collected images based on different photographing directions.

[0047] Each of the collected images includes image feature points obtained by reflecting detection light rays based on device feature points in the AR device.

[0048] FIG. 3 is a schematic diagram of an AR device provided in an embodiment of the present application. As shown in FIG. 3, the AR device is a lightweight AR glass equipped with a plurality of black dots, which are device features capable of reflecting light. For example, if the light emitted from the base station is infrared light, the device feature may be an infrared light reflecting dot arranged with an infrared fluorescent material. In other words, each device feature in the AR device is made of a material capable of reflecting infrared light.

[0049] The infrared LED light in the base station can emit infrared light to the AR device, and at the same time, multiple collection devices located at different shooting orientations in the base station can capture multiple collection images by shooting the AR device.

[0050] Here, each collected image is collected based on a collection device located in one photographing direction, and the collection device may be a camera, a photographic device, etc. For example, when the collection device is a camera, each collection device may be an independent photographing device, or multiple collection devices may be located in the same photographing direction.

[0051] 4 is a schematic diagram of a collected image provided in an embodiment of the present application. As shown in FIG. 4, a collection device in a base station captures an AR device to obtain a schematic diagram of one collected image. The collected image includes multiple image feature points, and each image feature point corresponds to one device feature point in the AR device.

[0052] The multiple collection devices may be an infrared binocular camera (consisting of two collection devices positioned at different shooting orientations). The infrared binocular camera captures two collection images. The two collection images can reflect the reflection of infrared light by device features in the AR device. To reduce flicker and improve the quality of the collection images, the switch control of the infrared LED array needs to be strictly synchronized with the camera exposure.

[0053] In addition, the method in this application in which the base station emits infrared light and the AR device reflects it can be replaced by an infrared LED, i.e., by placing an infrared LED in the AR device. In the following, we will use an infrared binocular camera and infrared light as examples.

[0054] In addition, infrared light has low power consumption and is well suited to detecting infrared reflection to determine the orientation of an AR device. Other light sources that can achieve the above effects are also applicable to the present application, and will not be described in detail in this specification. The following description will be given using infrared light as an example.

[0055] Taking a real-world scenario as an example, for example, object A wears an AR device, turns it on, and prepares to project a virtual screen. The infrared LED light in the base station emits infrared light to the AR device, and at the same time, the infrared binocular camera in the base station captures the reflection of the device feature points on the AR device to obtain two collected images.

[0056] In S202, the base station determines, based on the at least two collected images, a predicted distance for each of at least two pairs of target image feature points including the same target image feature point.

[0057] In this embodiment, each captured image includes at least three image features (more than three device features in the AR device).

[0058] Taking two collected images as an example, after obtaining the collected images, the base station first randomly selects three common target image features contained in the two collected images, and determines at least two target image feature pairs based on the three target image features. "Containing the same target image feature" means that any two target image feature pairs must contain one common target image feature. For example, from image feature 1, image feature 2, and image feature 3, image feature pairs {image feature 1, image feature 2}, {image feature 1, image feature 3}, and {image feature 2, image feature 3} can be determined. Here, {image feature point 1, image feature point 2} and {image feature point 1, image feature point 3} commonly include image feature point 1, {image feature point 1, image feature point 3} and {image feature point 2, image feature point 3} commonly include image feature point 3, and {image feature point 1, image feature point 2} and {image feature point 2, image feature point 3} commonly include image feature point 2.

[0059] Furthermore, the pose detection method for an AR device provided herein can be completed by selecting two target image feature pairs from the three target image feature pairs. The selection process can be random. For example, {image feature 1, image feature 2} and {image feature 2, image feature 3} can be selected, or {image feature 1, image feature 2} and {image feature 1, image feature 3} can be selected, or {image feature 2, image feature 3} and {image feature 1, image feature 3} can be selected.

[0060] Alternatively, two target image feature point pairs having a large difference in distance between the image feature points may be selected, or the posture detection process for the AR device may be performed once for each of two arbitrary target image feature point pairs, and the average of the posture information finally obtained may be calculated. This application does not limit the specific embodiment. The following description will be mainly given using two target image feature point pairs as an example.

[0061] In addition to randomly selecting three target image features, three image features with relatively large distances may be selected as the target image features to reduce errors. For example, in one optional embodiment, if the collected image has four image features, i.e., image feature 1, image feature 2, image feature 3, and image feature 4, all image feature combinations containing three image features, i.e., (image feature 1, image feature 2, image feature 3), (image feature 1, image feature 2, image feature 4), (image feature 2, image feature 3, image feature 4), and (image feature 1, image feature 3, image feature 4), may be determined, and the one with the largest sum of distances between the three image features may be selected as the target image feature. The present application does not specifically limit the specific embodiment.

[0062] 5 is a schematic diagram of target image feature points provided in an embodiment of the present application. As shown in FIG. 5, in this embodiment, the collected image has eight image feature points (shown as black squares and hatched squares in the figure). Here, since the sum of the distances between the three image feature points shown as black squares is the largest, these three image feature points are selected and designated as target image feature point 1, target image feature point 2, and target image feature point 3, respectively.

[0063] The base station then determines predicted distances of the three image features in the real environment based on the situations of the three image features reflected in each of the at least two collected images, i.e., predicts the distances between the device features corresponding to the three image features. The distances of the image features in the collected images are two-dimensional, but the predicted distances are three-dimensional distances in the real environment.

[0064] If the number of collected images is more than two, two images may be randomly selected, or one predicted distance may be determined for every two images, and then the average value may be calculated.

[0065] In one optional embodiment, the base station determines a predicted distance for each of the at least two pairs of target image feature points based on the distances for the at least two pairs of target image feature points in each collected image and orientation parameters indicating a positional relationship between the at least two collection devices.

[0066] That is, after obtaining the distances between at least two pairs of target image features in each collected image and the orientation parameters indicating the positional relationship between each collection device, the base station may determine the three-dimensional coordinates of the three target image features in the base station coordinate system based on this information, and then determine the predicted distances corresponding to the pairs of target image features based on the three-dimensional coordinates. This step subsequently determines the device features corresponding to the target image features based on the predicted distances, that is, the purpose of determining which device features a certain target image feature in the collected image specifically corresponds to which device feature captured by the collection device.

[0067] Here, the zero point of the base station coordinate system may be set to the center point of one of the infrared binocular cameras. The present application does not specifically limit the specific embodiment.

[0068] Specifically, for example, if the base station selects two target image feature pairs, namely {image feature 1, image feature 2} and {image feature 1, image feature 3}, the base station may predict the three-dimensional coordinates of image feature 1, image feature 2, and image feature 3 in the base station coordinate system in the real environment based on the distance between image feature 1 and image feature 2 in the two collected images, the distance between image feature 1 and image feature 3 in the two collected images, and the orientation parameters of the two collection devices in the infrared binocular camera, and then calculate the predicted distance of {image feature 1, image feature 2} and the predicted distance of {image feature 1, image feature 3} based on the three-dimensional coordinates.

[0069] In the above, the orientation parameters of the two acquisition devices reflect the relative positional relationship between the two acquisition devices, for example, the distance between the two acquisition devices and the relative angle of the imaging direction.

[0070] Based on the assumption of S201, FIG. 6 is a logic diagram of the predicted distance calculation provided in the embodiment of the present application. As shown in FIG. 6, the base station selects image feature points a1, a2, and a3 from the collected images as target image feature points, and a1, a2 and a1, a3 are two target image feature point pairs. The base station calculates coordinates p1, p2, and p3 of a1, a2, and a3 in the base station coordinate system based on the distances between the three target image feature points in the two collected images and the positional relationship between the two cameras in the infrared binocular camera. Based on p1, p2, and p3, the base station can further calculate the predicted distance x1 between a1 and a2 and the predicted distance x2 between a1 and a3.

[0071] In S203, the base station determines target device feature points in the AR device corresponding to each of the target image feature points based on a comparison result between the actual distances and the predicted distances between each of the device feature points in the AR device.

[0072] In the above, the base station may invoke a position determination algorithm to identify target device features corresponding to each target image feature through spatial position analysis.

[0073] In addition, to realize a process that can identify target device features corresponding to target image features simply by comparing distances, the present application provides a method for arranging each device feature in an AR device.

[0074] Optionally, the actual distances between any two device feature points on the AR device are different, and the difference value between any two actual distances is distinguishable, i.e., the difference value between any two actual distances is greater than a first predetermined threshold. This method can speed up the identification of the target device feature and simplify the algorithm.

[0075] Additionally, the first predetermined threshold may be further set based on the resolution of the acquisition device and the orientation parameters.

[0076] It should be noted that the difference value between any two examples in this application is the absolute value of the difference, that is, the difference value is a non-negative number.

[0077] Based on this, for each predicted distance, the base station may determine a device feature pair corresponding to each target image feature pair by comparing it with each actual distance, and further determine the target image feature pair and the relationship between the target image feature and the target image feature in the corresponding device feature pair. Figure 7 is a flowchart for determining target device feature points corresponding to each target image feature point provided in an embodiment of the present application. As shown in Figure 7, the base station may specifically perform the following steps:

[0078] In S701, the base station compares the actual distance between each pair of device feature points in the AR device with each predicted distance.

[0079] Assuming there are two target image feature point pairs, each of the predicted distances includes a first predicted distance of one of the two target image feature point pairs (hereinafter referred to as the first target image feature point pair) and a second predicted distance of the other target image feature point pair (hereinafter referred to as the second target image feature point pair). The base station may sequentially determine two device feature point pairs corresponding to the two target image feature point pairs, that is, first determine the device feature point pair corresponding to one of the target image feature point pairs, and then determine the device feature point pair corresponding to the other target image feature point pair, or may simultaneously determine the two device feature point pairs corresponding to the two target image feature point pairs. The present application does not specifically limit the specific embodiment.

[0080] In S702, the base station determines, based on the comparison result, at least two device feature pairs including the same device feature, where a difference value between the actual distance corresponding to each device feature pair and the corresponding predicted distance is smaller than a second predetermined threshold.

[0081] In the above, the second predetermined threshold is equal to or less than the first predetermined threshold.

[0082] In one optional embodiment, the base station determines two device features whose difference value between the corresponding actual distance and the first predicted distance is smaller than a second predetermined threshold as a first device feature pair corresponding to the first target image feature pair.

[0083] Then, the base station searches for at least one other device feature pair in which the difference value between the corresponding actual distance and the second predicted distance is less than a second predetermined threshold and which includes the same device feature as the first device feature pair.

[0084] For example, the two target image feature point pairs obtained by the base station are {image feature point 1, image feature point 2} and {image feature point 1, image feature point 3}, the first predicted distance between image feature point 1 and image feature point 2 is d1, and the second predicted distance between image feature point 1 and image feature point 3 is d2. The base station first searches for an actual distance between each device feature point pair in the AR device that is closest to the first predicted distance d1, and finds that the actual distance D1 between device feature point 1 and device feature point 2 and the first predicted distance d1 are It is determined that the condition TIFF2025532492000002.tif7170 is satisfied. TIFF2025532492000003.tif5170 is the difference between the actual distance D1 and the first predicted distance d1, TIFF2025532492000004.tif5170 is the second predetermined threshold.

[0085] In the embodiment of the present application, assuming that the second predetermined threshold value<<the first predetermined threshold value, it is often possible to determine only one actual distance whose difference value between the first predicted distance and the actual distance is smaller than the second predetermined threshold value.

[0086] If the second predetermined threshold is less than or equal to the first predetermined threshold and the difference between the two is not large, there may be one or more actual distances whose difference value with respect to the first predicted distance is smaller than the second predetermined threshold.

[0087] In the above, if there are multiple actual distances whose difference value from the first predicted distance is smaller than the second predetermined threshold, the device feature pair corresponding to the one actual distance with the smallest difference value may be selected as the device feature pair corresponding to the target image feature pair {image feature 1, image feature 2}.

[0088] For example, the difference value between the actual distance between device feature point 1 and device feature point 2 and the first predicted distance is less than the second predetermined threshold, i.e., 0.05 cm, and the difference value between the actual distance between device feature point 5 and device feature point 6 and the first predicted distance is less than the second predetermined threshold, i.e., 0.1 cm. In this case, {device feature point 1, device feature point 2} is selected as the device feature point pair corresponding to the target image feature point pair {image feature point 1, image feature point 2}.

[0089] Similarly, the base station searches for at least one actual distance close to the second predicted distance d2 from among the actual distances between each device feature pair in the AR device, and all of these actual distances satisfy the condition that the difference value between them and the second predicted distance is smaller than a second predetermined threshold. The device feature pair corresponding to each of these actual distances is the other device feature pair, and each other device feature pair must include device feature 1 or device feature 2. Finally, the base station needs to determine a second device feature pair corresponding to the second target image feature pair from among the at least one other device feature pair.

[0090] In S703, the base station determines target device feature points corresponding to each target image feature point from among the at least two device feature point pairs.

[0091] Specifically, the base station determines that identical device features in two device feature pairs are target device features corresponding to identical target image features in two corresponding target image feature pairs, determines that device features other than the identical device features in a first device feature pair are target device features corresponding to image features other than the identical image features in the first target image feature pair, and determines that device features other than the identical device features in a second device feature pair are target device features corresponding to image features other than the identical image features in the second target image feature pair.

[0092] That is, the base station needs to find a second device feature pair from at least one other device feature pair, and set it as a device feature pair corresponding to the target image feature pair {image feature 1, image feature 3}. For example, if the base station selects {device feature 1, device feature 3} as a device feature pair corresponding to {image feature 1, image feature 3}, the actual distance D2 between device feature 1 and device feature 3 and the second predicted distance d2 must also be Meets the condition TIFF2025532492000005.tif7170.

[0093] Then, since both of the two determined device feature point pairs include device feature point 1 and both of the two target image feature point pairs include image feature point 1, device feature point 1 is set as the target device feature point corresponding to image feature point 1, device feature point 2 in the device feature point pair {device feature point 1, device feature point 2} is set as the target device feature point corresponding to image feature point 2 in the corresponding target image feature point pair {image feature point 1, image feature point 2}, and device feature point 3 in the device feature point pair {device feature point 1, device feature point 3} is set as the target device feature point corresponding to image feature point 3 in the corresponding target image feature point pair {image feature point 1, image feature point 3}.

[0094] In other words, the first target image feature point pair is set to include a first target image feature point and a second target image feature point, and the second target image feature point pair is set to include a first target image feature point and a third target image feature point, and if the same device feature point in the two device feature point pairs is the first device feature point in the first device feature point pair, the first device feature point is set to the first target device feature point in the AR device that corresponds to the first target image feature point, the second device feature point in the first device feature point pair is set to the second target device feature point in the AR device that corresponds to the second target image feature point, and the third device feature point in the second device feature point pair is set to the third target device feature point in the AR device that corresponds to the third target image feature point.

[0095] To summarize, in the above example, device feature point 1 is the first device feature point, device feature point 2 is the second device feature point, device feature point 3 is the third device feature point, image feature point 1 corresponds to device feature point 1, image feature point 2 corresponds to device feature point 2, and image feature point 3 corresponds to device feature point 3.

[0096] 8 is a correspondence diagram between target image features and target device features provided in an embodiment of the present application. As shown in FIG. 8, through the above method, the base station can determine that target image feature 1, target image feature 2, and target image feature 3 in the collected image correspond to device feature 1, device feature 2, and device feature 3 in the AR device, respectively.

[0097] For example, if the base station selects {device feature 2, device feature 4} as a device feature pair corresponding to {image feature 1, image feature 3}, the actual distance D3 between the device feature 2 and the device feature 4 and the second predicted distance d2 must also be Meets the condition TIFF2025532492000006.tif7170.

[0098] Finally, since both of the two determined device feature point pairs include device feature point 2 and both of the two target image feature point pairs include image feature point 1, device feature point 2 is set as the target device feature corresponding to image feature point 1, device feature point 1 in the device feature point pair {device feature point 1, device feature point 2} is set as the target device feature corresponding to image feature point 2 in the corresponding target image feature point pair {image feature point 1, image feature point 2}, and device feature point 4 in the device feature point pair {device feature point 2, device feature point 4} is set as the target device feature corresponding to image feature point 3 in the corresponding target image feature point pair {image feature point 1, image feature point 3}.

[0099] In other words, when the same device feature point in two device feature point pairs is set to be the second device feature point in the first device feature point pair, the second device feature point is set to be the second target device feature point corresponding to the first target image feature point in the AR device, the first device feature point in the first device feature point pair is set to be the first target device feature point corresponding to the second target image feature point in the AR device, and the third device feature point in the second device feature point pair is set to the third target device feature point corresponding to the third target image feature point in the AR device.

[0100] To summarize, in the above example, device feature point 1 is the first device feature point, device feature point 2 is the second device feature point, device feature point 4 is the third device feature point, image feature point 1 corresponds to device feature point 2, image feature point 2 corresponds to device feature point 1, and image feature point 3 corresponds to device feature point 4.

[0101] In addition, in the above process, the base station needs to find one other device feature pair from at least one other device feature pair to be the second device feature pair corresponding to the second target image feature pair. This process may be divided into the following two cases:

[0102] Case 1: If the number of other device feature pairs is one, the other device feature pair is set as a second device feature pair corresponding to the second target image feature pair.

[0103] Case 2: If there are multiple other device feature pairs, one of the multiple other device feature pairs is selected based on the magnitude relationship of the actual distances between the multiple other device feature pairs, and is set as the second device feature pair corresponding to the second target image feature pair.

[0104] In case 1, the base station only needs to directly set this one other device feature pair as the second device feature pair corresponding to the second target image feature pair {image feature 1, image feature 3}.

[0105] In Case 2, since there are multiple other device feature pairs, these other device feature pairs may all include the first device feature, may all include the second device feature, or may partially include the first device feature and partially include the second device feature.

[0106] In either case of Case 2 above, there is only one device feature pair corresponding to the second target image feature pair, and considering that the smaller the difference value between the actual distance and the second predicted distance corresponding to this device feature pair, the better, in one optional embodiment, the base station determines another device feature pair having the smallest difference value between the corresponding actual distance and the second predicted distance as the second device feature pair corresponding to the second target image feature pair.

[0107] Thereafter, for example, the base station finds two other device feature pairs, where the first other device feature pair includes the first device feature and the third device feature, and the second other device feature pair includes the second device feature and the fourth device feature.

[0108] If the actual distance corresponding to the first other device feature point pair is smaller than the actual distance corresponding to the second other device feature point pair, the first device feature point is determined as the corresponding point of the first target image feature point, the second device feature point in the first device feature point pair corresponding to the previously determined first target image feature point pair is determined as the corresponding point of the second target image feature point, and the third device feature point in the first other device feature point pair is determined as the corresponding point of the third target image feature point.

[0109] If the actual distance corresponding to the second other device feature point pair is smaller than the actual distance corresponding to the first other device feature point pair, the second device feature point is determined as the corresponding point of the first target image feature point, the first device feature point in the previously determined first device feature point pair is determined as the corresponding point of the second target image feature point, and the fourth device feature point in the second other device feature point pair is determined as the corresponding point of the third target image feature point.

[0110] For example, after determining that {device feature 1, device feature 2} corresponds to {image feature 1, image feature 2}, the base station finds two other device feature pairs that satisfy the condition that the difference value between {image feature 1, image feature 3} and the distance d2 is smaller than a second predetermined threshold. Assume that the two other device feature pairs are {device feature 1, device feature 3} and {device feature 2, device feature 4}, respectively, and that the distance corresponding to {device feature 1, device feature 3} is D2 and the distance corresponding to {device feature 2, device feature 4} is D3. TIFF2025532492000007.tif7170{Device feature point 1, Device feature point 3} is a device feature point pair corresponding to {Image feature point 1, Image feature point 3}, TIFF2025532492000008.tif7170{Device feature point 2, device feature point 4} is set as a device feature point pair corresponding to {Image feature point 1, image feature point 3}.

[0111] There is another method for determining a target device feature point corresponding to a target image feature point. After determining that {device feature point 1, device feature point 2} corresponds to {image feature point 1, image feature point 2}, the base station searches for one device feature point among the other device feature points in the AR device for which the difference value between the distance from device feature point 1 and the second predicted distance is smaller than a second predetermined threshold. If there are multiple device feature points, the base station selects the one with the smallest difference value. For example, the base station finally determines that the difference value between the distance from device feature point 3 to device feature point 1 and the second predicted distance is smallest and smaller than the second predetermined threshold. Furthermore, the base station searches for one device feature point for which the difference value between the distance from device feature point 2 and the second predicted distance is smaller than the second predetermined threshold. If there are multiple device feature points, the base station selects the one with the smallest difference value. For example, the base station finally determines that the difference value between the distance from device feature point 4 to device feature point 2 and the second predicted distance is smallest and smaller than the second predetermined threshold.

[0112] The subsequent process is the same as the method described above, that is, by comparing the magnitude of the difference value between the distance between device feature point 3 and device feature point 1 and the second predicted distance, and the magnitude of the difference value between the distance between device feature point 4 and device feature point 2 and the second predicted distance, it is determined whether the device feature pair corresponding to {image feature point 1, image feature point 3} is actually {device feature point 1, device feature point 3} or {device feature point 2, device feature point 4}.

[0113] This process determines that {device feature 1, device feature 2} actually correspond to {image feature 1, image feature 2}, and then further assumes whether device feature 1 corresponds to image feature 1 or whether device feature 1 corresponds to image feature 2. Finally, the target device feature corresponding to each target image feature (i.e., image feature 1, image feature 2, image feature 3) is finally determined based on the magnitude of the difference value between the actual distance corresponding to each of {device feature 1, device feature 3} and {device feature 2, device feature 4} and the second predicted distance.

[0114] Note that the phrase "at least two device feature pairs including the same device feature" in S702 of the present application refers to the possibility that there are more than two device feature pairs that satisfy the condition that the difference value between the corresponding actual distance and the predicted distance is smaller than the second predetermined threshold. For example, after determining one device feature pair corresponding to one target image feature pair, when determining other device feature pairs corresponding to other target image feature pairs, multiple other device feature pairs are found that satisfy the condition that the difference value between the corresponding actual distance and the second predicted distance is smaller than the second predetermined threshold. Each of the other device feature pairs and the first device feature pair corresponding to the first target image feature pair each include the same device feature, but the "same device feature" in these other device feature pairs may be the same or different.

[0115] For example, if a first device feature pair is {device feature 1, device feature 2}, any of the other device feature pairs may include device feature 1 in the first device feature pair, any of the other device feature pairs may include device feature 2 in the first device feature pair, or some other device feature pairs may include device feature 1 in the first device feature pair, and some other other device feature pairs may include device feature 2 in the first device feature pair.

[0116] Based on the assumption of S202, Fig. 9 is a logic diagram for determining device feature points corresponding to target image feature points provided in an embodiment of the present application. As shown in Fig. 9, the base station calculates a predicted distance x1 between a1 and a2 and a predicted distance x2 between a1 and a3, and then determines one device feature point pair {P1, P2} based on the predicted distance x1 between a1 and a2 and the distances between each device feature point pair. Compared with the actual distances between any other two device feature points, the actual distance y1 between device feature points P1 and P2 in the AR device is closest to x1, TIFF2025532492000009.tif7170Therefore, the device feature pair {P1, P2} corresponds to the target image feature pair {a1, a2}.

[0117] Then, the base station determines one device feature point P3 whose distance from the device feature point P1 is closest to x2 (e.g., the difference value between x2 and the distance y2 between P1 and P3 is smaller than the second predetermined threshold), and determines one device feature point P4 whose distance from the device feature point P2 is closest to x2 (e.g., the difference value between x2 and the distance y3 between P2 and P4 is also smaller than the second predetermined threshold). The base station compares the difference value between y2 and x2 with the difference value between y3 and x2 and determines that the difference value between y2 and x2 is smaller. Therefore, the device feature point pair {P1, P3} corresponds to the target image feature point pair {a1, a3}.

[0118] Furthermore, the base station determines that the target image feature point a1 corresponds to the device feature point P1, the target image feature point a2 corresponds to the device feature point P2, and the target image feature point a3 corresponds to the device feature point P3.

[0119] In S204, the base station determines posture information of the AR device based on the position information of each target device feature point and each target image feature point.

[0120] Based on the position information of each target device feature point in the AR device coordinate system and the position information of each target image feature point in the base station coordinate system, the relative position information of the AR device with respect to the base station is determined, and the relative position information is used as the attitude information of the AR device.

[0121] After determining the target device feature points corresponding to each target image feature point, the base station may obtain the three-dimensional coordinates of each target device feature point in the AR device coordinate system, and determine the three-dimensional orientation T0 of the AR device coordinate system relative to the base station coordinate system based on the three-dimensional coordinates of each target image feature point in the base station coordinate system. This three-dimensional orientation T0 is the orientation information of the AR device, and its specific format is as follows: TIFF2025532492000010.tif10170, where R is the rotation matrix, reflecting the angular relationship between the AR device coordinate system and the base station coordinate system, TIFF2025532492000011.tif7170 is the translation matrix, which reflects the distance between the AR device coordinate system and the base station coordinate system, and the rotation matrix must satisfy the following conditions: TIFF2025532492000012.tif8170Here, I is the identity matrix.

[0122] Apparently, the attitude information T0 of the AR device is six-degree-of-freedom attitude data.

[0123] Then, if each acquired image contains more than three image feature points, i.e., if each acquired image contains three or more image feature points, T0 may be adjusted using candidate image feature points other than the target image feature point. The final pose information T1 is obtained by performing nonlinear optimization using T0 as the initial pose information.

[0124] In one optional embodiment, the base station predicts reference position information of each of the at least one candidate device feature points in the base station coordinate system based on position information and posture information of each of the at least one candidate device feature points other than each target device feature point in the AR device coordinate system, predicts predicted position information of each of the at least one candidate device feature points in the AR device coordinate system based on the reference position information, orientation parameters for indicating the positional relationship between each collection device, and posture information, and finally adjusts the posture information based on a difference between the position information of each of the at least one candidate device feature points in the AR device coordinate system and the corresponding predicted position information.

[0125] In the above, for each candidate device feature point other than the target device feature point in the AR device, the base station can inversely calculate the coordinates of the candidate device feature point in the base station coordinate system (i.e., reference location information) based on T0 and the coordinates of the candidate device feature point in the AR device coordinate system, and then predict the coordinates of the candidate device feature point in the AR device coordinate system (i.e., predicted location information) based on the reference location information, the orientation parameters of the infrared binocular camera, and T0.

[0126] As one optional method for obtaining predicted location information, the base station predicts the feature point position of each of at least one candidate device feature point in the collected image based on reference location information and orientation parameters indicating the positional relationship between each collecting device, and for each feature point position, sets an image feature point whose distance from the feature point position is within a predetermined range as a candidate image feature point corresponding to the candidate device feature point, and predicts predicted location information of the corresponding candidate device feature point in the AR device coordinate system based on the position information and posture information of each of the at least one candidate image feature point in the base station coordinate system.

[0127] In S202, the base station calculates the three-dimensional coordinates of the three target image features in the base station coordinate system based on the orientation parameters and the distances between at least two pairs of target image features in each collected image. This calculation process can now be reversed, that is, for each candidate device feature, the position of the candidate device feature in the collected image, i.e., the feature point position, can be calculated based on its reference position information and the orientation parameters. This corresponds to estimating the approximate location of the corresponding candidate image feature in the image, assuming that the candidate device feature was captured in the collected image. The purpose of this step is to subsequently find the candidate image feature corresponding to the candidate device feature in the collected image.

[0128] The base station then projects the estimated feature point position onto the collected image, finds the image feature point closest to the feature point position, and determines it as the candidate image feature point corresponding to the candidate device feature point. The true position of the candidate image feature point in the collected image should satisfy the condition that the distance between the feature point position and the candidate image feature point is < 5 pixels.

[0129] The base station then determines to re-predict the predicted position information of the corresponding candidate device feature in the AR device coordinate system based on the coordinates of the candidate image feature in the base station coordinate system and the posture information.

[0130] The base station may determine one final pose information T1 based on each difference between each predicted position information and the true coordinates of the corresponding candidate device feature in the AR device coordinate system. The pose information T1 may minimize the sum of the difference between the predicted position information of each candidate device feature and the true coordinates of the candidate device feature in the AR device coordinate system, and the difference between the predicted position information of the target device feature and the true coordinates of the target device feature in the AR device coordinate system. Specifically, the following formula may be used: TIFF2025532492000013.tif26170

[0131] The above formula is calculated using a minimum quadratic optimization algorithm. TIFF2025532492000014.tif7170 is the rotation matrix, TIFF2025532492000015.tif7170 is the translation matrix, TIFF2025532492000016.tif7170AR are the true coordinates of device feature point i in the device coordinate system, TIFF2025532492000017.tif7170 are the coordinates of image feature point i corresponding to device feature point i in the base station coordinate system, TIFF2025532492000018.tif7170 is predicted location information for device feature point i. Device feature point i is either a candidate device feature point or a target device feature point.

[0132] Furthermore, in the above formula, when the posture information is T0, T0 is calculated based on the coordinates of the target image feature points corresponding to the target device feature points in the base station coordinate system and the true coordinates of the target device feature points in the AR device, so in this case, it can be seen that the difference between the predicted position information of the target device feature points and the true coordinates of the target device feature points in the AR device is 0.

[0133] Following the hypothesis of S203, the base station determines that the target image feature point a1 corresponds to the device feature point P1, the target image feature point a2 corresponds to the device feature point P2, and the target image feature point a3 corresponds to the device feature point P3, and then calculates the posture information T0 of the AR device based on the coordinates of the target image feature points a1, a2, and a3 in the base station coordinate system and the coordinates of the device feature points P1, P2, and P3 in the AR device coordinate system.

[0134] The base station also finds candidate image feature points a5 and a6 corresponding to candidate device feature points P5 and P6 other than device feature points P1, P2, and P3. Candidate image feature points a5 and a6 are visible in the collected image. Based on the coordinates of the candidate image feature points a5 and a6 in the base station coordinate system and the orientation parameters of the infrared binocular camera, predicted position information of candidate device feature points P5 and P6 in the AR device coordinate system corresponding to the candidate image feature points a5 and a6 predicted by the base station is obtained. Each predicted position information is compared with the true coordinates of the corresponding candidate device feature points in the AR device coordinate system to adjust T0 and finally obtain posture information T1. The posture information T1 can minimize the sum of the difference between the predicted position information of the candidate device feature points P5 and P6 and the true coordinates of the candidate device feature points P5 and P6 in the AR device coordinate system, and the difference values ​​between the predicted position information of the device feature points P1, P2, and P3 and the true coordinates of the device feature points P1, P2, and P3 in the AR device coordinate system.

[0135] FIG. 10 illustrates an interaction diagram between the AR device, the base station, and the terminal device provided in the present application. After acquiring the attitude information T1, the base station may timestamp the 6-DOF attitude information and send the time-stamped attitude information T1 to the terminal device via Wi-Fi. The AR device itself may measure its own inertial measurement unit (IMU) data and send the time-stamped IMU data to the terminal device. The terminal device performs image rendering based on the attitude information T1 and the IMU data and sends the rendered image to the AR device. The terminal device also needs to synchronize its time with the base station, and the base station simultaneously synchronizes its time with the AR device.

[0136] The IMU may be a six-axis sensor that may include a three-axis gyroscope and a three-axis accelerometer. The gyroscope measures the angular velocity (rotational degrees of freedom) along each axis, i.e., the number of degrees that can pass through one second when rotating along this axis, and the accelerometer measures the acceleration (translational degrees of freedom) along each axis. The IMU data can be used to predict the orientation of the AR device at the next point in time based on the device's current movement speed, acceleration, and other conditions.

[0137] In addition, to further reduce communication power consumption, the Bluetooth Low Energy (BLE) protocol may be used to transmit the IMU data from the AR device to the terminal device.

[0138] The specific process of rendering on the terminal device may involve performing data fusion between the posture information uploaded from the base station and the IMU data uploaded from the AR device, estimating the rendering time and image transmission time, and predicting the virtual screen at the next time point based on the current time point and the estimated rendering time and image transmission time, rendering the correct 3D posture, and transmitting the rendering result to the AR device via WIFI for display.

[0139] The method provided herein significantly reduces the computational power and power consumption requirements for AR devices. While a typical head-mounted AR device's inside-out posture detection method consumes 1.5 watts or more of power, the present invention only requires power consumed by Wi-Fi transmission, resulting in an optimized average power consumption of approximately 200 to 500 mW. Furthermore, the AR device of the present invention can be lightweight AR glasses, eliminating the need for additional electronic components or circuits, simplifying system design.

[0140] 11 is a flowchart showing a specific implementation of another AR device orientation detection method provided in an embodiment of the present application. The method may include the following specific implementation steps:

[0141] In S1101, the base station emits a detection ray (e.g., an illumination ray) to the AR device and takes at least two collection images.

[0142] In S1102, the base station determines whether three target image feature points can be acquired from the collected image, and if three target image feature points can be acquired from the collected image, executes S1103, and if three target image feature points cannot be acquired from the collected image, executes S1104.

[0143] In S1103, the base station determines the coordinates p1, p2, and p3 of three target image feature points a1, a2, and a3 in the base station coordinate system.

[0144] In S1104, the posture detection fails.

[0145] In S1105, the base station obtains two target image feature point pairs {a1, a2} and {a1, a3} based on the three target image feature points, and calculates the predicted distances d1 = |p1p2|, d2 = |p1p3| of the two target image feature point pairs, respectively.

[0146] In S1106, the base station determines, from among the device feature points, two device feature points P1 and P2 that satisfy the condition that the distance between P1 and P2 is closest to d1.

[0147] In S1107, the base station determines one device feature point P3 whose distance to P1 is closest to d2.

[0148] In S1108, the base station determines one device feature point P4 whose distance to P2 is closest to d2.

[0149] When specifically implemented, the difference value between the distance between P1 and P3 in S1107 and d2 may be smaller than the second predetermined threshold, and the difference value between the distance between P2 and P4 in S1108 and d2 may be smaller than the second predetermined threshold.

[0150] If neither S1107 nor S1108 is able to find a device feature pair that satisfies the condition that the difference value from d2 is smaller than the second predetermined threshold, then location determination fails. If only one of S1107 and S1108 succeeds in finding a device feature pair that satisfies the condition that the difference value from d2 is smaller than the second predetermined threshold, then it is directly determined that the device feature pair corresponds to the target image feature pair {a1, a3}.

[0151] In S1109, the base station compares the magnitude relationship between ||P1P3|-d2| and ||P2P4|-d2|, and if ||P1P3|-d2| is smaller than ||P2P4|-d2|, executes S1110, and if ||P1P3|-d2| is equal to or greater than ||P2P4|-d2|, executes S1111.

[0152] In S1110, the base station determines that the target image feature points a1, a2, and a3 correspond to the target device feature points P1, P2, and P3, and obtains posture information of the AR device based on the coordinates of P1, P2, and P3 in the AR device coordinate system and the coordinates of a1, a2, and a3 in the base station coordinate system.

[0153] In S1111, the base station determines that the target image feature points a1, a2, and a3 correspond to the target device feature points P2, P1, and P4, and obtains the posture information of the AR device based on the coordinates of P2, P1, and P4 in the AR device coordinate system and the coordinates of a1, a2, and a3 in the base station coordinate system.

[0154] The flowcharts listed above are merely examples, and the order of steps S1107 and S1108 may be reversed or may be performed simultaneously. The present application does not limit the specific implementation form.

[0155] FIG. 12 is a logic diagram illustrating the interaction between an AR device, a base station, and a terminal device provided in an embodiment of the present application. As shown in FIG. 12, the base station emits a light beam to the AR device, and the device feature points on the AR device reflect the light beam. At the same time, the base station captures the device feature points to obtain multiple captured images. The base station then determines a predicted distance for each of at least two pairs of target image feature points that include the same target image feature point, and determines a target device feature point corresponding to each target image feature point based on the predicted distance and the actual distance between each device feature point. Finally, the base station determines attitude information of the AR device based on the position information of the target image feature points and the position information of the corresponding target device feature points, and transmits the attitude information to the terminal device. The AR device also acquires its own inertial measurement unit data and transmits it to the terminal device. The terminal device renders a virtual screen based on the attitude information of the AR device and the inertial measurement unit data. After the rendering is completed, the rendered screen is transmitted to the AR device for display.

[0156] Based on the same inventive concept, the present embodiment further provides a posture detection device for an AR device. Fig. 13 is a schematic diagram of the structure of the posture detection device for an AR device. As shown in Fig. 13, the device: An acquisition unit 1301 that acquires at least two collected images based on different shooting orientations by emitting a detection light beam to an AR device, and each collected image includes image feature points obtained by reflecting the detection light beam based on device feature points in the AR device; a prediction unit 1302 for determining, based on at least two acquired images, a predicted distance for each of at least two target image feature pairs including the same target image feature; a comparison unit 1303 for determining target device feature points in the AR device corresponding to each of the target image feature points based on a comparison result between the actual distances and the predicted distances between the device feature points in the AR device; and a determining unit 1304 for determining attitude information of the AR device based on position information of each target device feature point and each target image feature point, respectively.

[0157] Optionally, each of the collected images is collected based on a collection device positioned at one photographing orientation. A predicted distance for each of the at least two pairs of target image feature points is determined based on the distances between the at least two pairs of target image feature points in each of the collected images and the orientation parameters for indicating the positional relationship between the collection devices.

[0158] Optionally, the actual distances between any two device feature points are different, and the difference value between any two actual distances is greater than a first predetermined threshold.

[0159] Optionally, the comparison unit 1303 specifically comprises: comparing the actual distances between the device feature points in the AR device with the predicted distances; Based on the comparison result, at least two device feature pairs containing the same device feature are determined, and from among the at least two device feature pairs, target device feature points corresponding to each target image feature point are determined, wherein a difference value between the actual distance between each device feature pair and the corresponding predicted distance is smaller than a second predetermined threshold, and the second predetermined threshold is equal to or smaller than the first predetermined threshold.

[0160] Optionally, each predicted distance includes a first predicted distance for a first pair of target image feature points and a second predicted distance for a second pair of target image feature points of the at least two pairs of target image feature points.

[0161] Specifically, the comparison unit 1303 Two device features having a difference value between the corresponding actual distance and the first predicted distance that is smaller than a second predetermined threshold are determined as a first device feature pair corresponding to the first target image feature pair; searching for at least one other device feature pair, the difference value between the corresponding actual distance and the second predicted distance being less than a second predetermined threshold and including the same device feature as the first device feature pair; A target device feature point corresponding to each of the target image features is determined from among the device features included in each of the one device feature pair and at least one other device feature pair.

[0162] Optionally, the comparison unit 1303 specifically comprises: determining a second pair of device feature points from among the at least one other pair of device feature points that corresponds to the second pair of target image feature points; The same device feature points are set as target device feature points corresponding to the same target image feature points; determining a device feature point other than the identical device feature point in the first device feature point pair as a target device feature point corresponding to an image feature point other than the identical image feature point in the first target image feature point pair; A device feature point other than the identical device feature point in the second device feature point pair is set as a target device feature point corresponding to an image feature point other than the identical image feature point in the second target image feature point pair.

[0163] Optionally, the comparison unit 1303 specifically comprises: if the number of other device feature pairs is one, the other device feature pair is a second device feature pair corresponding to the second target image feature pair; If there are a plurality of other device feature point pairs, one of the plurality of other device feature point pairs is selected based on the magnitude relationship of the actual distances between the plurality of other device feature point pairs, and is set as the second device feature point pair corresponding to the second target image feature point pair.

[0164] Optionally, the comparison unit 1303 specifically comprises: Another device feature pair having the smallest difference value between the corresponding actual distance and the second predicted distance is determined as a second device feature pair corresponding to the second target image feature pair.

[0165] Optionally, the determining unit 1304 specifically: determining relative position information of the AR device with respect to the base station based on the position information of each target device feature point in the AR device coordinate system and the position information of each target image feature point in the base station coordinate system; The relative position information is used as the orientation information of the AR device.

[0166] Optionally, each collected image is collected based on a collecting device positioned at one photographing orientation, and the apparatus further includes an adjustment unit 1305.

[0167] The adjusting unit 1305 predicts reference position information of each of the at least one candidate device feature points in the base station coordinate system based on the position information of each of the at least one candidate device feature points other than each target device feature point in the AR device coordinate system and the posture information; predicting predicted position information of each of the at least one candidate device feature point in the AR device coordinate system based on reference position information, orientation parameters indicating a positional relationship between the at least two collecting devices, and posture information; The pose information is adjusted based on a difference between the position information of each of the at least one candidate device feature points in the AR device coordinate system and the corresponding predicted position information.

[0168] Optionally, the adjusting unit 1305 specifically: predicting a feature point position of each of the at least one candidate device feature point in the collected image based on the reference position information and an orientation parameter indicating a positional relationship between the at least two collection devices; For each feature point position, image feature points whose distance between the feature point positions is within a predetermined range are determined as candidate image feature points corresponding to the candidate device feature points; Based on the position information of at least one candidate image feature point in the base station coordinate system and the posture information, predicted position information of a corresponding candidate device feature point in the AR device coordinate system is predicted.

[0169] Optionally, each device feature in the AR device is made of a material that reflects infrared light.

[0170] For convenience of explanation, the above-mentioned parts are described as modules (or units) according to their functions. Of course, when implementing the present application, the functions of each module (or unit) may be realized by the same or multiple pieces of software or hardware.

[0171] After introducing the method and apparatus for detecting the posture of an AR device according to an exemplary embodiment of the present application, an electronic device according to another exemplary embodiment of the present application will be introduced next.

[0172] Those skilled in the art will appreciate that various aspects of the present application may be embodied as a system, method, or program product. Thus, various aspects of the present application may be specifically embodied as an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software. These embodiments may be collectively referred to herein as a "circuit," "module," or "system."

[0173] Based on the same inventive concept as the above-described method embodiment, an embodiment of the present application further provides an electronic device. In one embodiment, the electronic device may be a terminal device 130 as shown in FIG. 1. In this embodiment, the electronic device may include components such as a communication component 1410, a memory 1420, a display unit 1430, a camera 1440, a sensor 1450, an audio circuit 1460, a Bluetooth module 1470, and a processor 1480, as shown in FIG.

[0174] The communication component 1410 is used to communicate with the server. In some embodiments, the communication component 1410 may include a Wireless Fidelity (WiFi) module, which is a short-range wireless transmission technology. The WiFi module can help the electronic device to send and receive information of an object (e.g., a user).

[0175] The memory 1420 can be used to store software programs and data. The processor 1480 executes various functions and data processing of the terminal device 130 by executing the software programs or data stored in the memory 1420. The memory 1420 stores an operating system on which the terminal device 130 can operate. The memory 1420 in the present application can store the operating system and various application programs, and can also store a computer program that executes the AR device orientation detection method of the embodiment of the present application.

[0176] The display unit 1430 may also be used to display information input by or provided to an object, and a graphical user interface (GUI) of various menus of the terminal device 130. Specifically, the display unit 1430 may include a display screen 1432 provided on the front of the terminal device 130. The display unit 1430 may be used to display a rendering interface or the like in the embodiments of the present application.

[0177] The display unit 1430 can also be used to accept input numeric and character information and generate signal inputs related to object settings and function control of the terminal device 130. Specifically, the display unit 1430 may include a touch screen 1431 provided on the front side of the terminal device 130. The touch screen 1431 can collect touch operations of objects on or near it (e.g., clicking a button or dragging a scroll box).

[0178] Here, the touch screen 1431 may cover the display screen 1432, or the touch screen 1431 and the display screen 1432 may be integrated to realize the input and output functions of the terminal device 130, and the integrated one may be called a touch display screen. The display unit 1430 in this application may display application programs and corresponding operating procedures.

[0179] The camera 1440 can be used to capture still images, and the object can publish the images captured by the camera 1440 in an application. There can be one or more cameras 1440. An optical image of the object is generated by a lens and projected onto a photosensitive element. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to the processor 1480 for conversion into a digital image signal.

[0180] The terminal device may include at least one type of sensor 1450, such as an acceleration sensor 1451, a distance sensor 1452, a fingerprint sensor 1453, or a temperature sensor 1454. The terminal device may also be equipped with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0181] The audio circuit 1460 , the speaker 1461 , and the microphone 1462 can provide an audio interface between the object and the terminal device 130 .

[0182] The Bluetooth module 1470 is used to exchange information with other Bluetooth devices that have Bluetooth modules via a Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (e.g., a smart watch) that also has a Bluetooth module via the Bluetooth module 1470 for data exchange.

[0183] The processor 1480 is the control center of the terminal device. It connects various parts of the entire terminal using various interfaces and lines, executes or performs software programs stored in the memory 1420, and accesses data stored in the memory 1420 to perform various functions of the terminal device and process data. In some embodiments, the processor 1480 may include one or more processing units. The processor 1480 may include an application processor that mainly processes the operating system, object interface, and application programs, and a baseband processor that mainly processes wireless communication. As can be understood, the baseband processor does not have to be integrated into the processor 1480. The processor 1480 in this application may execute the operating system, application programs, object interface display and touch response, as well as the AR device orientation detection method of the embodiment of this application. The processor 1480 is also coupled to the display unit 1430.

[0184] In some possible embodiments, each aspect of the AR device orientation detection method provided herein may be realized in the form of a program product including a computer program. When the program product is executed by a processor, the steps of the AR device orientation detection method according to various exemplary embodiments of the present application described herein are realized. For example, the electronic device may perform the steps illustrated in FIG. 2.

[0185] As used herein, a readable storage medium may be any tangible medium that contains or stores a program, which may be used by or in combination with an instruction execution system, apparatus, or device. A readable signal medium may include a propagated data signal, in baseband or as part of a carrier, carrying a readable computer program. Such a propagated data signal may take various forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. A readable signal medium may be any readable medium other than a readable storage medium, which may transmit, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.

[0186] The computer program contained in the readable medium may be transmitted over any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0187] Computer programs for carrying out the operations of the present application can be written in any combination of one or more programming languages. The computer programs can run entirely on the target electronic device, partially on the target electronic device, as a separate software package, partially on the target electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the target electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (e.g., via the Internet using an Internet Service Provider).

[0188] It should be noted that although the above detailed description refers to several units or sub-units of the device, such division is not mandatory but merely exemplary. In fact, according to embodiments of the present application, the features and functions of two or more units described above may be embodied in one unit. Conversely, the features and functions of one unit described above may be further divided so as to be embodied by multiple units.

[0189] Furthermore, although the figures may depict the operations of the methods of the present application in a particular order, this does not require or imply that the operations must be performed in that particular order, or that all of the illustrated operations must be performed to achieve desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or a single step may be broken down into multiple steps.

[0190] The present application has been described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block of the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine. The instructions, executed by the processor of the computer or other programmable data processing device, thereby generate an apparatus for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0191] These computer program instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture that includes an instruction apparatus that implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0192] These computer program instructions may be loaded into a computer or other programmable data processing device to execute a series of operational steps on the computer or other programmable device to generate a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0193] It should be noted that the terms "first," "second," "third," "fourth," "1," "two," etc. (when present) in the specification, claims, and drawings are used to distinguish between similar objects and are not necessarily used to describe a particular order or priority. It should be understood that the data so used can be interchanged, where appropriate, so that the embodiments of the present invention described herein can be practiced in orders other than those shown or textually described.

[0194] Although the preferred embodiments of the present application have been described, those skilled in the art, once they understand the basic creative concept, can make other changes and modifications to these embodiments. Therefore, it is intended that the appended claims be interpreted to include the preferred embodiments and all changes and modifications that fall within the scope of the present application.

Claims

1. A method for detecting an orientation of an augmented reality (AR) device, comprising: A step of acquiring at least two collected images based on different shooting orientations by emitting detection light rays to an AR device, wherein each collected image includes image feature points obtained by reflecting the detection light rays based on device feature points in the AR device; determining a predicted distance for each of at least two pairs of target image feature points that include the same target image feature point based on the at least two acquired images; determining target device feature points in the AR device corresponding to each of the target image feature points based on a comparison result between the actual distances and the predicted distances between the device feature points in the AR device; determining posture information of the AR device based on position information of each target device feature point and each target image feature point, 10. A method for detecting the orientation of an AR device, comprising:

2. Each of the acquired images is acquired based on an acquisition device positioned at one of the imaging orientations, The step of determining a predicted distance for each of at least two pairs of target image feature points including the same target image feature point based on the at least two acquired images includes: determining a predicted distance for each of the at least two pairs of target image feature points based on the distances for the at least two pairs of target image feature points in each collected image and on orientation parameters indicating a positional relationship between the at least two collection devices; The method for detecting the orientation of an AR device according to claim 1 .

3. The actual distances between any two device feature points are different, and the difference value between the actual distances is greater than a first predetermined threshold value; The method for detecting the orientation of an AR device according to claim 1 .

4. determining target device feature points in the AR device corresponding to each of the target image feature points based on a comparison result between the actual distances and the predicted distances between the device feature points in the AR device, comparing an actual distance between each pair of device feature points in the AR device with each of the predicted distances; determining at least two device feature pairs containing the same device feature based on the comparison result, wherein a difference value between the actual distance and the corresponding predicted distance between each device feature pair is smaller than a second predetermined threshold, and the second predetermined threshold is equal to or smaller than the first predetermined threshold; determining target device feature points corresponding to each of the target image feature points from among the at least two device feature point pairs; The method for detecting the orientation of an AR device according to any one of claims 1 to 3.

5. There are two pairs of target image feature points, and the predicted distances include a first predicted distance for a first pair of target image feature points and a second predicted distance for a second pair of target image feature points; The step of determining at least two device feature pairs containing the same device feature includes: determining two device features having a difference value between the corresponding actual distance and the first predicted distance that is smaller than the second predetermined threshold as a first device feature pair corresponding to the first target image feature pair; and searching for at least one other device feature pair in which a difference value between the corresponding actual distance and the second predicted distance is smaller than the second predetermined threshold and which includes the same device feature as the first device feature pair. The method for detecting the orientation of an AR device according to claim 4 .

6. The step of determining target device feature points corresponding to each of the target image feature points from among at least two device feature point pairs comprises: determining a second pair of device features from among at least one other pair of device features that corresponds to the second pair of target image features; determining the identical device feature as a target device feature corresponding to the identical target image feature; determining a device feature other than the identical device feature point in the first device feature point pair as a target device feature point corresponding to an image feature other than the identical image feature point in the first target image feature point pair; and setting a device feature other than the identical device feature in the second device feature pair as a target device feature corresponding to an image feature other than the identical image feature in the second target image feature pair. The method for detecting the orientation of an AR device according to claim 5 .

7. determining a second pair of device features corresponding to the second pair of target image features from among at least one other pair of device features, if the number of the other device feature pairs is one, setting the other device feature pair as a second device feature pair corresponding to the second target image feature pair; and if the number of the other device feature point pairs is plural, selecting one from the plurality of other device feature point pairs based on a magnitude relationship between the actual distances of the plurality of other device feature point pairs, and setting the selected one as a second device feature point pair corresponding to the second target image feature point pair. The method for detecting the orientation of an AR device according to claim 6 .

8. the step of selecting one of the other device feature point pairs as a second device feature point pair corresponding to the second target image feature point pair based on a magnitude relationship between the actual distances of the other device feature point pairs, determining another pair of device feature points, which has a minimum difference value between the corresponding actual distance and the second predicted distance, as a second pair of device feature points corresponding to the second pair of target image feature points; The method for detecting the orientation of an AR device according to claim 7 .

9. The step of determining posture information of the AR device based on position information of each target device feature point and each target image feature point includes: determining relative position information of the AR device with respect to the base station based on position information of each target device feature point in the AR device coordinate system and position information of each target image feature point in the base station coordinate system; and setting the relative position information as attitude information of the AR device. The method for detecting the orientation of an AR device according to any one of claims 1 to 3.

10. Each of the acquired images is acquired based on an acquisition device positioned at one of the imaging orientations, and the method includes: predicting reference position information of each of the at least one candidate device feature points in a base station coordinate system based on position information of each of the at least one candidate device feature points other than each of the target device feature points in an AR device coordinate system and the posture information; predicting predicted position information of each of the at least one candidate device feature point in the AR device coordinate system based on the reference position information, orientation parameters indicating a positional relationship between the at least two collection devices, and the pose information; adjusting the pose information based on a difference between position information of each of the at least one candidate device feature point in the AR device coordinate system and corresponding predicted position information. The method for detecting the orientation of an AR device according to any one of claims 1 to 3.

11. The step of predicting predicted position information of each of the at least one candidate device feature point in the AR device coordinate system based on the reference position information, orientation parameters indicating a positional relationship between the at least two collection devices, and the posture information includes: predicting a feature point location of each of the at least one candidate device feature point in the collected image based on the reference position information and the orientation parameters; a step of determining, for each feature point position, an image feature point whose distance from the feature point position is within a predetermined range as a candidate image feature point corresponding to the candidate device feature point; and predicting predicted position information of a corresponding candidate device feature point in the AR device coordinate system based on position information of each of at least one candidate image feature point in the base station coordinate system and the pose information. The method for detecting the orientation of an AR device according to claim 10.

12. Each device feature point in the AR device is made of a material that reflects infrared light. The method for detecting the orientation of an AR device according to any one of claims 1 to 3.

13. An attitude detection device for an AR device, an acquisition unit that acquires at least two collected images based on different shooting orientations by emitting a detection light beam to an AR device, and each collected image includes image feature points obtained by reflecting the detection light beam based on device feature points in the AR device; a prediction unit for determining a predicted distance for each of at least two target image feature pairs including the same target image feature point based on the at least two acquired images; a comparison unit for determining target device feature points in the AR device corresponding to each of the target image feature points based on a comparison result between the actual distances and the predicted distances between the device feature points in the AR device; a determination unit for determining posture information of the AR device based on position information of each target device feature point and each target image feature point, An orientation detection device for an AR device, characterized by:

14. 13. An electronic device comprising: a processor and a memory; a computer program stored in the memory; and, when the computer program is executed by the processor, the electronic device realizing the steps of the method for detecting the orientation of an AR device according to claim 1.

15. 13. A computer-readable storage medium storing a computer program, the computer program causing a processor to execute steps of the method for detecting the orientation of an AR device according to claim 1.

16. 13. A computer program product including a computer program, the computer program causing, when executed by a processor, the computer program to implement the steps of the method for detecting the attitude of an AR device according to claim 1.

Images