A method for detecting the attitude of an AR device, an AR device attitude detection apparatus, an electronic device, and a computer program

Abstract

The present application relates to the technical field of computers, particularly to the technical field of augmented reality, and provides a method, apparatus, electronic device, and storage medium for detecting the orientation of an AR device. The method includes the steps of: acquiring at least two collected images based on different shooting orientations by emitting detection rays from an AR device; determining predicted distances for at least two pairs of target image feature points that include the same target image feature based on the at least two collected images; determining target device feature points in the AR device that correspond to each of the target image feature points based on a comparison result between the actual distances between each of the device feature points in the AR device and the predicted distances; and determining orientation information of the AR device based on position information of each of the target device feature points and each of the target image feature points. In this application, the AR device only needs to reflect the detection rays, and the orientation information is obtained by simple distance comparison, thereby reducing power consumption and computational power required for detecting the orientation of the AR device.

Description

Technical Field

[0001] This application relates to the technical field of computers, and particularly to the technical field of augmented reality, and provides a method and apparatus for detecting the pose of an AR device, an electronic device, and a storage medium.

Background Art

[0002] With the development of science, technologies such as augmented reality (AR) and virtual reality (VR) have gradually matured and come to the attention of the general public. Among them, AR technology is a technology that combines virtual and reality, and by presenting virtual characters, images, three-dimensional models, etc. to the real world through an AR device, the sense of reality can be enhanced. The pose detection of an AR device is a very basic and important part of AR technology.

[0003] Taking a head-mounted device as an example, the method for detecting the pose of an AR head-mounted device is mainly realized by using devices such as a camera and a depth sensor. Since all these devices are arranged on the head-mounted device, high power needs to be provided from the AR head-mounted device for the operation of the camera and the depth sensor, and there is also a certain requirement for the computing power of the device for the operation of the depth sensor. As a result, the power consumption of the head-mounted device becomes high.

[0004] Therefore, how to reduce the power consumption and computing power requirements for the pose detection of an AR device should be urgently solved.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Embodiments of this application provide a method and apparatus for detecting the pose of an AR device, an electronic device, a storage medium, and a program product for reducing the power consumption of pose detection of an AR device.

Means for Solving the Problems

[0006] The posture detection method for an AR device provided in the embodiment of this application is: A step of obtaining at least two collected images based on different shooting orientations by emitting a detection ray to an AR device, wherein each collected image includes image feature points obtained by reflecting the detection ray based on device feature points in the AR device; The steps include determining the predicted distances of at least two pairs of target image feature points that contain the same target image feature point, based on the at least two collected images, The steps include determining the target device feature point corresponding to each target image feature point in the AR device based on the comparison result between the actual distance and each predicted distance between each device feature point in the AR device, The process includes the step of determining the orientation information of the AR device based on the positional information of each target device feature point and each target image feature point.

[0007] The posture detection device for the AR device provided in the embodiment of the present application is An acquisition unit that acquires at least two collected images based on different shooting orientations by emitting a detection ray to an AR device, wherein each collected image includes image feature points obtained by reflecting the detection ray based on device feature points in the AR device, A prediction unit that determines the predicted distance of each of at least two pairs of target image feature points that contain the same target image feature point, based on the at least two collected images, A comparison unit that determines the target device feature point corresponding to each target image feature point in the AR device based on the comparison result between the actual distance and the predicted distance between each device feature point in the AR device, The system includes a determination unit that determines the orientation information of the AR device based on the positional information of each target device feature point and each target image feature point.

[0008] Selectively, each acquired image is acquired based on an acquisition device located in one of the aforementioned imaging orientations, and the prediction unit specifically, The distance between the at least two pairs of target image feature points in each collected image and , small The predicted distances of each of the at least two pairs of target image feature points are determined based on orientation parameters that indicate the positional relationship between the two acquisition devices, or at least on the same parameters.

[0009] Selectively, the actual distances between any two device feature points are different, and the difference between any two actual distances is greater than a first predetermined threshold.

[0010] Optionally, the comparison unit is, The actual distance between each feature point in the AR device is compared with the predicted distance between each feature point. Based on the comparison results, at least two pairs of device feature points containing the same device feature point are determined, wherein the difference between the actual distance between each pair of device feature points and the corresponding predicted distance is smaller than a second predetermined threshold, and the second predetermined threshold is less than or equal to the first predetermined threshold. From the aforementioned pair of at least two device feature points, a target device feature point corresponding to each of the target image feature points is determined.

[0011] The electronic device provided in the embodiment of the present invention comprises a processor and a memory, the memory storing a computer program, and when the computer program is executed by the processor, it realizes one of the steps of the above-described method for detecting the attitude of an AR device.

[0012] The computer-readable storage medium provided in the embodiment of the present application includes a computer program which, when executed by a processor, enables the implementation of any one of the steps of the above-described method for detecting the attitude of an AR device.

[0013] The computer program product provided in the embodiment of the present application includes a computer program which, when executed by a processor, enables the implementation of any one of the steps of the above-described method for detecting the attitude of an AR device. [Effects of the Invention]

[0014] The beneficial effects of this application are as follows:

[0015] Embodiments of the present invention provide an attitude detection method, apparatus, electronic device, and storage medium for an AR device. In this invention, a base station emits a detection ray to the AR device, and the base station acquires at least two collected images based on different shooting orientations. The reflection of the detection ray by device feature points in the AR device is determined by the image feature points in the images, i.e., both the detection ray emitter and the shooting device are located at the base station. Therefore, the AR device does not consume power from the emitter and shooting device. Subsequently, based on each collected image, the base station predicts the predicted distance in the real environment for each pair of target image feature points that contain the same target image feature point, and finds the target device feature point corresponding to each target image feature point by comparing the actual distance between each device feature point in the AR device with each predicted distance. This process only requires distance comparison, which is a simple distance calculation, and thus has a lower computational power requirement for the device. Once a correspondence is found, the base station can calculate the AR device's attitude information based on the positional information of each target device feature point in the AR device coordinate system and the positional information of each target image feature point in the base station coordinate system. Since the process of determining the AR device's attitude is also performed by the base station, there is no need for the AR device to provide computational support. This effectively reduces the power consumption and computational power requirements for AR device attitude detection.

[0016] Other features and advantages of this application are described in the following specification, are partially evident from the specification, or are understood by practicing this application. The objectives and other advantages of this application may be achieved and obtained by the configurations specifically shown in the description, claims, and drawings. [Brief explanation of the drawing]

[0017] The drawings described herein are provided to further understand the present application and constitute part of the present application. The schematic embodiments and descriptions thereof are for interpretive purposes and do not constitute an inappropriate limitation of the present application. [Figure 1] It is a schematic diagram of an application scenario of a method for detecting the posture of an AR device provided in an embodiment of the present application. [Figure 2] It is an overall flowchart of a method for detecting the posture of an AR device provided in an embodiment of the present application. [Figure 3] It is a schematic diagram of an AR device provided in an embodiment of the present application. [Figure 4] It is a schematic diagram of a collected image provided in an embodiment of the present application. [Figure 5] It is a schematic diagram of target image feature points provided in an embodiment of the present application. [Figure 6] It is a logic diagram of predicted distance calculation provided in an embodiment of the present application. [Figure 7] It is a flowchart for determining target device feature points corresponding to each of the target image feature points provided in an embodiment of the present application. [Figure 8] It is a correspondence diagram between target image feature points and target device feature points provided in an embodiment of the present application. [Figure 9] It is a logic diagram for determining device feature points corresponding to target image feature points provided in an embodiment of the present application. [Figure 10] It is an interaction diagram between an AR device, a base station, and a terminal device provided in an embodiment of the present application. [Figure 11] It is a specific implementation flowchart of a method for detecting the posture of another AR device provided in an embodiment of the present application. [Figure 12] It is a logic diagram of the interaction between an AR device, a base station, and a terminal device provided in an embodiment of the present application. [Figure 13] It is a schematic diagram of the configuration of a device for detecting the posture of an AR device provided in an embodiment of the present application. <​​​​​​​​​To further clarify the purpose, structure, and advantages of the embodiments of this application, the present invention will be described clearly and completely below with reference to the drawings of the embodiments. As will be obvious, the embodiments described are some, but not all, embodiments of the present invention. All other embodiments that a person skilled in the art can derive from the embodiments described herein without creative work are within the scope of the present invention.

[0019] The following describes some of the concepts related to the embodiments of this application.

[0020] The collected images are obtained when a base station photographs an AR device. The AR device has device feature points that can reflect detection rays. Accordingly, the collected images also include image feature points obtained by reflecting detection rays based on the device feature points of the AR device, and reflect the reflection of infrared detection rays by the device feature points.

[0021] Image Feature Points: By capturing device feature points on an AR device, the base station obtains visible image feature points corresponding to each device feature point in the collected image. In this application, image feature points are divided into target image feature points and candidate image feature points. Target image feature points are the three image feature points initially acquired by the base station, while candidate image feature points are candidate image feature points corresponding to candidate device feature points selected by the base station to adjust the orientation information after acquiring orientation information based on the target image feature points.

[0022] Device feature point: A point in an AR device, which may be made of a material capable of reflecting infrared light (or other detection light, but not specifically limited herein). In embodiments of this application, in order to further improve the efficiency of the base station's determination of corresponding points of target image feature points and reduce errors, the distances between any two device feature points may be different, and the absolute value of the difference in the actual distances corresponding to any pair of device feature points may be set to be greater than a first predetermined threshold.

[0023] Base station coordinate system: This is a coordinate system with a specific point on the base station as its origin. For example, if the center point of a data collection device on the base station is taken as the origin of the base station coordinate system, then the positional relationship of a certain point relative to the base station can be reflected in this coordinate system.

[0024] AR device coordinate system: This is a coordinate system with a specific point on the AR device as its origin. For example, if the origin of the AR device coordinate system is the midpoint of the line connecting the centers of the two lenses on the AR device, then the positional relationship of a certain point to the AR device can be reflected in this coordinate system.

[0025] Predicted distance: This is the distance between device feature points corresponding to two target image feature points, calculated by the base station based on the distance between two target image feature points in multiple acquired images and the positional relationship between multiple acquisition devices. In this application, the predicted distance is divided into a first predicted distance and a second predicted distance. Here, the first predicted distance is the predicted distance for one pair of target image feature points corresponding to three target image feature points, and the second predicted distance is the predicted distance for another pair of target image feature points corresponding to three target image feature points.

[0026] The following briefly introduces the design concept of the embodiment of this application.

[0027] With the advancement of science, technologies such as AR and VR have gradually matured and become accessible to the public. Among these, AR technology combines the virtual and real worlds, enhancing the sense of reality by presenting virtual text, images, and 3D models into the real world using head-mounted AR devices. Pose detection in head-mounted AR devices is a very basic and important part of AR technology.

[0028] In related technologies, the outside-in method is mainly employed for VR head-mounted devices. That is, the sensors that can determine the orientation of the device are located outside the head-mounted device, not on it. However, this method requires placing an LED array on the head-mounted device, and position determination is achieved by the LED array spontaneously emitting light and the sensors detecting the light rays. The implementation process is complex, and the need to place the LED array on the head-mounted device significantly increases the power consumption of the head-mounted device.

[0029] On the other hand, the orientation detection method for AR head-mounted devices mainly employs an inside-out method. That is, the camera and depth sensor that can determine the device's orientation are located directly on the head-mounted device. During orientation detection, the camera and depth sensor consume a large amount of power from the head-mounted device, and the depth sensor needs to perform a series of calculations, thus placing a high demand on the computing power of the head-mounted device.

[0030] In view of this, embodiments of the present application provide an attitude detection method, apparatus, electronic device, storage medium, and program product for an AR device. In the present application, a base station emits a detection ray to the AR device, and the base station acquires at least two collected images based on different shooting orientations. The reflection of the detection ray by device feature points in the AR device is determined by the image feature points in the images. That is, both the detection ray emitter and the shooting device are located at the base station, not at the AR device. Therefore, the AR device does not consume power from the emitter and the shooting device.

[0031] Subsequently, the base station predicts the predicted distance in the real environment for each pair of target image feature points containing the same target image feature point, based on each collected image, and compares the predicted distance with the actual distance between each device feature point in the AR device. Since the actual distances between each device feature point in the AR device are all different, the distance comparison allows the target device feature point corresponding to each target image feature point to be found.

[0032] Finally, the base station can calculate the position information of the AR device relative to the base station, i.e., the attitude information of the AR device, based on the position information of each target device feature point in the AR device coordinate system and the position information of each target image feature point in the base station coordinate system. Since the process of determining the attitude of the AR device is also performed by the base station, there is no need for the AR device to provide computing power support. This reduces the power consumption and computing power requirements for AR device attitude detection.

[0033] Preferred embodiments of the present application will be described below with reference to the drawings of the specification. It should be understood that the preferred embodiments described herein are for the purpose of describing and interpreting the present application only and are not intended to limit the present application. Furthermore, where there is no contradiction, the embodiments and features described herein can be combined with each other.

[0034] The configuration provided in this application can be applied to 6-degree-of-freedom spatial pose detection of lightweight AR glasses to realize a blend of reality and fiction in specific scenarios (e.g., office scenario, board game scenario, etc.). One optional application scenario is shown in Figure 1.

[0035] Figure 1 is a schematic diagram of an application scenario of an embodiment of the present invention. As shown in Figure 1, this application scenario diagram includes one AR device 110, one base station 120, and one terminal device 130.

[0036] Here, AR devices include, but are not limited to, head-mounted AR devices and helmet-mounted AR devices.

[0037] The embodiments of this invention will be described using the case where the AR device 110 is a head-mounted AR device (for example, it may be lightweight AR glasses). The AR device includes an AR optical display module for projecting and displaying a virtual screen (for example, the shaded area in Figure 1) onto reality. The AR device further includes an Advanced RISC Machines System On Chip (ARM SOC) and device feature points capable of reflecting light rays. For example, if the light rays emitted from the base station are infrared rays, the device feature points may be infrared light reflection points arranged with infrared fluorescent material. The AR device 110 connects to the hotspot of the terminal device 130 via wireless fidelity (WIFI) and synchronizes local time with the base station 120 via Network Time Protocol (NTP).

[0038] Furthermore, this application is also applicable to other types of AR devices, and the light rays emitted from the base station and the materials on which the device feature points are placed are merely illustrative examples; this application does not specifically limit them.

[0039] Base station 120 is a location-determining base station that includes multiple data acquisition devices (which may be a pair of infrared binocular cameras), a light emission device (which may be an infrared LED light), and an ARM SOC. Base station 120 places an NTP local time server and connects to the internet via the terminal device 130's WIFI hotspot to synchronize global time.

[0040] Terminal device 130 provides a Wi-Fi hotspot to the AR device. This allows the AR device 110 to connect to the base station 120 and synchronize with the base station's local time via the NTP protocol. Terminal device 130 includes, but is not limited to, devices such as mobile phones, tablet computers, laptop computers, desktop computers, e-readers, smart voice interaction devices, smart home appliances, and in-car terminals.

[0041] In each embodiment of this application, the AR device attitude detection method is performed by the base station 120. The base station 120 emits infrared detection rays to the AR device 110 using infrared LED lights, and at the same time acquires two acquired images obtained by an infrared binocular camera that includes two acquisition devices located in different shooting directions, which reflect the infrared detection rays from the device feature points on the AR device 110. Based on the two acquired images, the base station 120 calculates the predicted distance in each actual scenario for at least two pairs of target image feature points that contain the same target image feature point, and determines the target device feature point on the AR device 110 that corresponds to each pair of target image feature points. Finally, the base station 120 acquires the attitude information of the AR device 110 by calculating the relative position information of the AR device 110 with respect to the base station 120, based on the position information of each target device feature point in the AR device coordinate system and the position information of each target image feature point in the base station coordinate system.

[0042] It should be noted that Figure 1 is merely illustrative, and the number of AR devices 110, base stations 120, and terminal devices 130 is not limited, and the embodiments of this application do not specifically limit them.

[0043] Furthermore, the embodiments of this application are applicable to a variety of scenarios, including but not limited to those involving cloud technology, artificial intelligence, smart transportation, and driver assistance.

[0044] The pose detection method for an AR device provided in exemplary embodiments of the present application will be described below with reference to the drawings, with reference to the application scenarios described above. It should be noted that the application scenarios described above are provided solely to facilitate understanding of the spirit and principles of the present application, and embodiments of the present application are not limited in this respect.

[0045] Figure 2 is an implementation flowchart of the attitude detection method for an AR device provided in an embodiment of the present invention. The method takes the case where the base station is the implementing entity as an example. As shown in Figure 2, the method may include the following specific implementation flow.

[0046] In S201, the base station acquires at least two collected images based on different shooting orientations by emitting detection beams to the AR device.

[0047] Each of the collected images above includes image feature points obtained by reflecting detection rays based on device feature points in the AR device.

[0048] Figure 3 is a schematic diagram of an AR device provided in an embodiment of the present invention. As shown in Figure 3, this AR device is a lightweight pair of AR glasses with multiple black dots, which are device feature points capable of reflecting light rays. For example, if the light rays emitted from the base station are infrared rays, the device feature points may be infrared light reflection points arranged with an infrared fluorescent material. In other words, each device feature point in the AR device is made of a material capable of reflecting infrared light.

[0049] Infrared LED lights at the base station can emit infrared rays to AR devices. Simultaneously, multiple acquisition devices located at the base station in different shooting directions acquire multiple images by photographing the AR devices.

[0050] Here, each collected image is acquired based on a collection device located in a single shooting orientation, and the collection device may be a camera, a photographic device, or the like. Taking the case where the collection device is a camera as an example, each collection device may be an independent shooting device, or multiple collection devices may be located on the same shooting device.

[0051] Figure 4 is a schematic diagram of the collected image provided in the embodiment of the present application. As shown in Figure 4, a collection device at a base station photographs an AR device to collect one image. The statue The collected images contain multiple image feature points, each of which corresponds to a single device feature point on the AR device.

[0052] Multiple acquisition devices may include an infrared binocular camera (composed of two acquisition devices positioned at different orientations). The infrared binocular camera captures two acquired images. These two acquired images can reflect the reflection of infrared light by device feature points in the AR device. To reduce flicker and improve the quality of the acquired images, the switch control of the infrared LED array must be strictly synchronized with the camera's exposure.

[0053] Furthermore, the method in this application in which the base station emits infrared light and the AR device reflects the infrared light can be replaced by using infrared LEDs, that is, by placing infrared LEDs on the AR device. In the following, all examples will be infrared binocular cameras and infrared light.

[0054] Furthermore, infrared light has low power consumption and is well applicable to methods that detect infrared reflection to determine the orientation of AR devices. All other types of light that can achieve the above effects are also applicable to this invention and will not be described individually in this specification. The following explanation will use infrared light as an example.

[0055] To illustrate with a real-world scenario, for example, object A wears and turns on an AR device, preparing to project a virtual screen. At the same time, an infrared LED light at the base station emits infrared rays towards the AR device, and an infrared binocular camera at the base station captures the reflection of device feature points on the AR device, resulting in two collected images.

[0056] In S202, the base station determines the predicted distance for each of at least two pairs of target image feature points that contain the same target image feature point, based on at least two acquired images.

[0057] In this embodiment, each collected image contains at least three image feature points (more than three device feature points in an AR device).

[0058] Taking two collected images as an example, after obtaining the collected images, the base station first randomly selects three common target image feature points included in the two collected images, and then determines at least two pairs of target image feature points based on these three feature points. "Containing the same target image feature points" means that any two pairs of target image feature points must contain one common target image feature point. For example, from image feature point 1, image feature point 2, and image feature point 3, the image feature point pairs {image feature point 1, image feature point 2}, {image feature point 1, image feature point 3}, and {image feature point 2, image feature point 3} can be determined. Here, {image feature point 1, image feature point 2} and {image feature point 1, image feature point 3} both contain image feature point 1, {image feature point 1, image feature point 3} and {image feature point 2, image feature point 3} both contain image feature point 3, and {image feature point 1, image feature point 2} and {image feature point 2, image feature point 3} both contain image feature point 2.

[0059] Furthermore, by selecting two pairs of target image feature points from the three pairs of target image feature points described above, the pose detection method for the AR device provided in this application can be completed. The selection process may be random selection. For example, {image feature point 1, image feature point 2} and {image feature point 2, image feature point 3} may be selected, or {image feature point 1, image feature point 2} and {image feature point 1, image feature point 3} may be selected, or {image feature point 2, image feature point 3} and {image feature point 1, image feature point 3} may be selected.

[0060] Alternatively, two pairs of target image feature points with a large difference in distance between them may be selected, or the pose detection process of the AR device may be performed once for each of any two pairs of target image feature points, and the average value of the pose information obtained at the end may be taken. This application does not specifically limit the embodiments. Below, we will mainly explain using two pairs of target image feature points as examples.

[0061] In addition to randomly selecting three target image feature points, to reduce errors, three image feature points that are relatively far apart may be selected as target image feature points. For example, in one optional embodiment, if the acquired image has four image feature points, namely image feature point 1, image feature point 2, image feature point 3, and image feature point 4, all combinations of image feature points that include three image feature points, namely (image feature point 1, image feature point 2, image feature point 3), (image feature point 1, image feature point 2, image feature point 4), (image feature point 2, image feature point 3, image feature point 4), and (image feature point 1, image feature point 3, image feature point 4), may be determined, and the one among them where the sum of the distances between the three image feature points is maximized may be selected as the target image feature point. This application does not specifically limit the specific embodiments.

[0062] Figure 5 is a schematic diagram of the target image feature points provided in the embodiment of the present invention. As shown in Figure 5, in this embodiment, the acquired image has eight image feature points (shown as black squares and diagonal squares in the figure). Here, since the sum of the distances between the three image feature points shown as black squares is the maximum, these three image feature points are selected and designated as target image feature point 1, target image feature point 2, and target image feature point 3, respectively.

[0063] Subsequently, the base station determines the predicted distance in the true environment of the three image feature points based on the situation of the three image feature points reflected in at least two of the acquired images, i.e., predicts the distance between the device feature points corresponding to the three image feature points. The distances of the image feature points in the acquired images are two-dimensional, but the predicted distances are three-dimensional distances in the true environment.

[0064] If there are more than two images to collect, two images may be randomly selected, or the predicted distance may be determined once for every two images, and the average value may be taken at the end. This application does not specifically limit the specific embodiments.

[0065] In one optional embodiment, the base station determines the predicted distance of each of the at least two target image feature point pairs based on the distances between the at least two target image feature point pairs in each acquired image and orientation parameters to indicate the positional relationship between at least two acquisition devices.

[0066] In other words, after obtaining the distances between at least two pairs of target image feature points in each acquired image and orientation parameters to indicate the positional relationship between each acquisition device, the base station may determine the three-dimensional coordinates of the three target image feature points in the base station coordinate system based on this information, and further determine the predicted distance corresponding to the pair of target image feature points based on these three-dimensional coordinates. This step aims to subsequently determine the device feature points corresponding to the target image feature points based on the predicted distances, that is, to determine which device feature point a particular target image feature point in the acquired image is specifically captured by the acquisition device.

[0067] Here, the zeros of the base station coordinate system may be set to the center point of one of the infrared binocular cameras. This application does not specifically limit the specific embodiments.

[0068] Specifically, for example, if a base station selects two target image feature point pairs, namely {image feature point 1, image feature point 2} and {image feature point 1, image feature point 3}, the base station may predict the three-dimensional coordinates of image feature point 1, image feature point 2, and image feature point 3 in the base station coordinate system in a true environment based on the distance between image feature point 1 and image feature point 2 in the two acquired images, the distance between image feature point 1 and image feature point 3 in the two acquired images, and the orientation parameters of the two acquisition devices in the infrared binocular camera. Then, based on these three-dimensional coordinates, the predicted distance of {image feature point 1, image feature point 2} and the predicted distance of {image feature point 1, image feature point 3} may be calculated.

[0069] In the above, the orientation parameters of the two acquisition devices reflect the relative positional relationship between the two acquisition devices, such as the distance between the two acquisition devices and the relative angle of the imaging direction.

[0070] Following the hypothesis of S201, Figure 6 is a logic diagram of the predicted distance calculation provided in the embodiment of the present invention. As shown in Figure 6, the base station selects image feature points a1, a2, and a3 from the acquired images as target image feature points, and uses a1, a2 and a1, a3 as two target image feature point pairs. Based on the distances between the three target image feature points in the two acquired images and the positional relationship between the two cameras in the infrared binocular camera, the base station calculates the coordinates p1, p2, and p3 of a1, a2, and a3 in the base station coordinate system. Based on p1, p2, and p3, the base station can further calculate the predicted distance x1 between a1 and a2 and the predicted distance x2 between a1 and a3.

[0071] In S203, the base station determines the target device feature point corresponding to each target image feature point in the AR device, based on the comparison between the actual distance and the predicted distance between each device feature point in the AR device.

[0072] In the above, the base station may call a positioning algorithm and identify target device feature points corresponding to each target image feature point by spatial position analysis.

[0073] Furthermore, in order to realize a process that allows for the identification of target device feature points corresponding to target image feature points simply by comparing distances, this invention provides a method for arranging each device feature point in an AR device.

[0074] Selectively, the actual distances between any two device feature points in an AR device are different, and the difference between any two actual distances is identifiable, i.e., the difference between any two actual distances is greater than a first predetermined threshold. This method can expedite the identification of target device feature points and simplify the algorithm.

[0075] Furthermore, the first predetermined threshold may be set based on the resolution and localization parameters of the acquisition device.

[0076] Furthermore, the difference value of any two examples in this application is the absolute value of the difference, that is, the difference value is a non-negative number.

[0077] Based on this, for each predicted distance, the base station determines the device feature point pair corresponding to each target image feature point pair by comparing it with each actual distance, and further, the target image feature points and target in the target image feature point pair and the corresponding device feature point pair. device The relationship with feature points may be determined. Figure 7 is a flowchart for determining the target device feature points corresponding to each of the target image feature points provided in the embodiments of the present application. As shown in Figure 7, the base station may specifically perform the following steps.

[0078] In S701, the base station compares the actual distance between each pair of device feature points in the AR device with each predicted distance.

[0079] Assuming there are two pairs of target image feature points, each of the above predicted distances includes a first predicted distance for one of the two pairs of target image feature points (hereinafter referred to as the first pair of target image feature points) and a second predicted distance for the other pair of target image feature points (hereinafter referred to as the second pair of target image feature points). The base station may determine two device feature point pairs corresponding to the two pairs of target image feature points in sequence, that is, it may first determine the device feature point pair corresponding to one of the two pairs of target image feature points and then determine the device feature point pair corresponding to the other pair of target image feature points, or it may determine the two device feature point pairs corresponding to the two pairs of target image feature points simultaneously. This application does not specifically limit the specific embodiments.

[0080] In S702, the base station determines at least two pairs of device feature points that contain the same device feature point, based on the comparison results. Here, the difference between the actual distance corresponding to each pair of device feature points and the corresponding predicted distance is smaller than a second predetermined threshold.

[0081] In the above, the second predetermined threshold is less than or equal to the first predetermined threshold.

[0082] In one optional embodiment, the base station uses two device feature points whose difference value between the corresponding actual distance and the first predicted distance is less than a second predetermined threshold as the first device feature point pair corresponding to the first target image feature point pair.

[0083] Subsequently, the base station searches for at least one other pair of device feature points in which the difference between the corresponding actual distance and the second predicted distance is smaller than a second predetermined threshold, and which includes the same device feature points as the first pair of device feature points.

[0084] For example, the two target image feature point pairs obtained by the base station are {image feature point 1, image feature point 2} and {image feature point 1, image feature point 3}, the first predicted distance between image feature point 1 and image feature point 2 is d1, and the second predicted distance between image feature point 1 and image feature point 3 is d2. The base station first searches for an actual distance close to the first predicted distance d1 from the actual distances between each device feature point pair in the AR device, and the actual distance D1 between device feature point 1 and device feature point 2, and the first predicted distance d1 are It is determined that the condition TIFF0007842942000001.tif7170 is met. Here, TIFF0007842942000002.tif5170 is the difference between the actual distance D1 and the first predicted distance d1. TIFF0007842942000003.tif5170 is the second predetermined threshold.

[0085] In the embodiments of this application, assuming that the second predetermined threshold << the first predetermined threshold, it is often possible to determine only one actual distance in which the difference value between the first predicted distance and the actual distance is less than the second predetermined threshold.

[0086] If the second predetermined threshold is less than or equal to the first predetermined threshold, and the difference between the two is not large, there may be one or more actual distances where the difference value between the first predicted distance and the actual distance is less than the second predetermined threshold.

[0087] In the above case, if there are multiple actual distances where the difference value between the first predicted distance and the actual distance is less than the second predetermined threshold, the device feature point pair corresponding to the one actual distance with the smallest difference value may be selected as the device feature point pair corresponding to the target image feature point pair {image feature point 1, image feature point 2}.

[0088] For example, the actual distance between device feature point 1 and device feature point 2 is such that the difference between it and the first predicted distance is less than the second predetermined threshold, and the difference is 0.05 cm. The actual distance between device feature point 5 and device feature point 6 is such that the difference between it and the first predicted distance is less than the second predetermined threshold, and the difference is 0.1 cm. In this case, {device feature point 1, device feature point 2} is selected as the device feature point pair corresponding to the target image feature point pair {image feature point 1, image feature point 2}.

[0089] Similarly, the base station searches for at least one actual distance from among the actual distances between each pair of device feature points in the AR device that is close to the second predicted distance d2, and all of these actual distances satisfy the condition that the difference between them and the second predicted distance is less than a second predetermined threshold. The device feature point pairs corresponding to each of these actual distances are the other device feature point pairs, and each of these other device feature point pairs must contain either device feature point 1 or device feature point 2. Finally, the base station needs to determine the second device feature point pair corresponding to the second target image feature point pair from among at least one other device feature point pair.

[0090] In S703, the base station determines the target device feature point corresponding to each target image feature point from at least two pairs of device feature points.

[0091] Specifically, the base station sets identical device feature points in two pairs of device feature points as target device feature points corresponding to identical target image feature points in two corresponding pairs of target image feature points, sets device feature points other than the identical device feature points in the first pair of device feature points as target device feature points corresponding to image feature points other than the identical target image feature points in the first pair of target image feature points, and sets device feature points other than the identical device feature points in the second pair of device feature points as target device feature points corresponding to image feature points other than the identical target image feature points in the second pair of target image feature points.

[0092] In other words, the base station needs to find a second pair of device feature points from at least one other pair of device feature points and make it the device feature point pair corresponding to the target image feature point pair {image feature point 1, image feature point 3}. For example, if the base station selects {device feature point 1, device feature point 3} as the device feature point pair corresponding to {image feature point 1, image feature point 3}, then the actual distance D2 between device feature point 1 and device feature point 3, and the second predicted distance d2 must also be The condition TIFF0007842942000004.tif7170 is met.

[0093] Subsequently, since device feature point 1 is included in both of the two determined pairs of device feature points, and image feature point 1 is included in both of the two pairs of target image feature points, device feature point 1 is set as the target device feature point corresponding to image feature point 1, device feature point 2 in the device feature point pair {device feature point 1, device feature point 2} is set as the target device feature point corresponding to image feature point 2 in the corresponding target image feature point pair {image feature point 1, image feature point 2}, and device feature point 3 in the device feature point pair {device feature point 1, device feature point 3} is set as the target device feature point corresponding to image feature point 3 in the corresponding target image feature point pair {image feature point 1, image feature point 3}.

[0094] In other words, the first pair of target image feature points is set to include the first and second target image feature points, and the second pair of target image feature points is set to include the first and third target image feature points. If the same device feature point in two pairs of device feature points is the first device feature point in the first pair of device feature points, then the first device feature point is set to be the first target device feature point in the AR device that corresponds to the first target image feature point. The second device feature point in the first pair of device feature points is set to be the second target device feature point in the AR device that corresponds to the second target image feature point. The third device feature point in the second pair of device feature points is set to be the third target device feature point in the AR device that corresponds to the third target image feature point.

[0095] To summarize, in the above example, device feature point 1 is the first device feature point, device feature point 2 is the second device feature point, device feature point 3 is the third device feature point, image feature point 1 corresponds to device feature point 1, image feature point 2 corresponds to device feature point 2, and image feature point 3 corresponds to device feature point 3.

[0096] Figure 8 is a diagram showing the correspondence between target image feature points and target device feature points provided in the embodiment of the present invention. As shown in Figure 8, the base station can determine by the method described above that target image feature point 1, target image feature point 2, and target image feature point 3 in the acquired image correspond to device feature point 1, device feature point 2, and device feature point 3 in the AR device, respectively.

[0097] For example, if a base station selects {device feature point 2, device feature point 4} as a device feature point pair corresponding to {image feature point 1, image feature point 3}, then the actual distance D3 and the second predicted distance d2 between device feature point 2 and device feature point 4 will also be The condition TIFF0007842942000005.tif7170 is met.

[0098] Finally, since both of the two determined pairs of device feature points contain device feature point 2, and both of the two pairs of target image feature points contain image feature point 1, device feature point 2 is set as the target device feature point corresponding to image feature point 1, device feature point 1 in the device feature point pair {device feature point 1, device feature point 2} is set as the target device feature point corresponding to image feature point 2 in the corresponding target image feature point pair {image feature point 1, image feature point 2}, and device feature point 4 in the device feature point pair {device feature point 2, device feature point 4} is set as the target device feature point corresponding to image feature point 3 in the corresponding target image feature point pair {image feature point 1, image feature point 3}.

[0099] In other words, if the same device feature point in two pairs of device feature points is set to be the second device feature point in the first pair of device feature points, then the second device feature point is set to be the second target device feature point in the AR device corresponding to the first target image feature point, the first device feature point in the first pair of device feature points is set to be the first target device feature point in the AR device corresponding to the second target image feature point, and the third device feature point in the second pair of device feature points is set to be the third target device feature point in the AR device corresponding to the third target image feature point.

[0100] To summarize, in the example above, device feature point 1 is the first device feature point, device feature point 2 is the second device feature point, device feature point 4 is the third device feature point, image feature point 1 corresponds to device feature point 2, image feature point 2 corresponds to device feature point 1, and image feature point 3 corresponds to device feature point 4.

[0101] Furthermore, in the process described above, the base station needs to find one other device feature point pair from among at least one other device feature point pair and make it the second device feature point pair corresponding to the second target image feature point pair. This process may be divided into the following two cases.

[0102] Case 1: If there is only one other device feature point pair, that other device feature point pair is designated as the second device feature point pair corresponding to the second target image feature point pair.

[0103] Case 2: If there are multiple pairs of other device feature points, one pair is selected from among them based on the relative actual distances of the multiple pairs of other device feature points, and this is designated as the second device feature point pair corresponding to the second target image feature point pair.

[0104] In Case 1, the base station simply needs to directly use this one other pair of device feature points as the second pair of device feature points corresponding to the second target image feature point pair {image feature point 1, image feature point 3}.

[0105] In Case 2, since there are multiple other pairs of device feature points, each of these other pairs of device feature points may contain the first device feature point, each may contain the second device feature point, or each may partially contain the first device feature point and partially contain the second device feature point.

[0106] In either case of Case 2 above, there is only one device feature point pair corresponding to the second target image feature point pair, and considering that it is desirable for the difference between the actual distance and the second predicted distance corresponding to this device feature point pair to be small, as one optional embodiment, the base station sets the other device feature point pair that minimizes the difference between the corresponding actual distance and the second predicted distance as the second device feature point pair corresponding to the second target image feature point pair.

[0107] Subsequently, for example, the base station found two other pairs of device feature points. Here, the first pair of other device feature points includes the first and third device feature points, and the second pair of other device feature points includes the second and fourth device feature points.

[0108] If the actual distance corresponding to the first pair of other device feature points is smaller than the actual distance corresponding to the second pair of other device feature points, the first device feature point is set as the corresponding point to the first target image feature point, the second device feature point in the first pair of device feature points corresponding to the previously determined first pair of target image feature points is set as the corresponding point to the second target image feature point, and the third device feature point in the first pair of other device feature points is set as the corresponding point to the third target image feature point.

[0109] If the actual distance corresponding to the second pair of other device feature points is smaller than the actual distance corresponding to the first pair of other device feature points, the second device feature point is set as the corresponding point to the first target image feature point, the first device feature point in the previously determined first pair of device feature points is set as the corresponding point to the second target image feature point, and the fourth device feature point in the second pair of other device feature points is set as the corresponding point to the third target image feature point.

[0110] For example, after determining that {device feature point 1, device feature point 2} corresponds to {image feature point 1, image feature point 2}, the base station finds two other pairs of device feature points that satisfy the condition that the difference value between {image feature point 1, image feature point 3} and the distance d2 is less than a second predetermined threshold. Assume that these two other pairs of device feature points are {device feature point 1, device feature point 3} and {device feature point 2, device feature point 4}, respectively, and that the distance corresponding to {device feature point 1, device feature point 3} is D2, and the distance corresponding to {device feature point 2, device feature point 4} is D3. TIFF0007842942000006.tif7170{device feature point 1, device feature point 3} are treated as device feature point pairs corresponding to {image feature point 1, image feature point 3}, TIFF0007842942000007.tif7170{device feature point 2, device feature point 4} are considered the device feature point pair corresponding to {image feature point 1, image feature point 3}.

[0111] In addition, there is another method for determining target device feature points that correspond to target image feature points. After the base station determines that {device feature point 1, device feature point 2} corresponds to {image feature point 1, image feature point 2}, it searches among the other device feature points in the AR device for one device feature point whose distance from device feature point 1 and the difference value between the distance and the second predicted distance is less than a second predetermined threshold. If there are multiple such points, it selects the one with the smallest difference value. For example, it ultimately determines that the difference value between the distance from device feature point 3 and device feature point 1 and the second predicted distance is smallest and also less than the second predetermined threshold. Furthermore, it searches for one device feature point whose distance from device feature point 2 and the difference value between the distance and the second predicted distance is less than the second predetermined threshold. If there are multiple such points, it selects the one with the smallest difference value. For example, it ultimately determines that the difference value between the distance from device feature point 4 and device feature point 2 and the second predicted distance is smallest and also less than the second predetermined threshold.

[0112] The subsequent process is the same as the method described above; that is, by comparing the magnitude of the difference between the distance between device feature point 3 and device feature point 1 and the second predicted distance, and the difference between the distance between device feature point 4 and device feature point 2 and the second predicted distance, it is determined whether the device feature point pair corresponding to {image feature point 1, image feature point 3} is {device feature point 1, device feature point 3} or {device feature point 2, device feature point 4}.

[0113] This process first determines that {device feature point 1, device feature point 2} actually correspond to {image feature point 1, image feature point 2}, and then makes assumptions about whether device feature point 1 corresponds to image feature point 1 or image feature point 2. Finally, based on the magnitude of the difference between the actual distance and the second predicted distance corresponding to {device feature point 1, device feature point 3} and {device feature point 2, device feature point 4} respectively, the target device feature point corresponding to each target image feature point (i.e., image feature point 1, image feature point 2, image feature point 3) is finally determined.

[0114] In addition, in S702 of this application, "at least two pairs of device feature points containing the same device feature point" means that there may be more than two device feature points that satisfy the condition that the difference between the corresponding actual distance and the predicted distance is less than the second predetermined threshold. For example, after determining one pair of device feature points corresponding to one pair of target image feature points, when determining other pairs of device feature points corresponding to other pairs of target image feature points, multiple other pairs of device feature points that satisfy the condition that the difference between the corresponding actual distance and the second predicted distance is less than the second predetermined threshold are found. Each of these other pairs of device feature points and the first pair of device feature points corresponding to the first pair of target image feature points contains the same device feature point, but these "same device feature points" in these other pairs of device feature points may be the same or different.

[0115] For example, if the first pair of device feature points is {device feature point 1, device feature point 2}, then any other pair of device feature points may contain device feature point 1 in the first pair, any other pair of device feature points may contain device feature point 2 in the first pair, some other pairs of device feature points may contain device feature point 1 in the first pair, and some other pairs of device feature points may contain device feature point 2 in the first pair.

[0116] Following the hypothesis of S202, Figure 9 is a logic diagram for determining device feature points corresponding to target image feature points provided in the embodiment of the present invention. As shown in Figure 9, the base station calculates the predicted distance x1 between a1 and a2 and the predicted distance x2 between a1 and a3, and then determines one device feature point pair {P1, P2} based on the predicted distance x1 between a1 and a2 and the distance between each device feature point pair. Compared with the actual distance between any other two device feature points, the actual distance y1 between device feature points P1 and P2 in the AR device is closest to x1, TIFF0007842942000008.tif7170 Therefore, the device feature point pair {P1,P2} corresponds to the target image feature point pair {a1,a2}.

[0117] Subsequently, the base station determines one device feature point P3 that is closest to x2 in distance from device feature point P1 (for example, the difference between the distance y2 between P1 and P3 and x2 is smaller than a second predetermined threshold), and determines one device feature point P4 that is closest to x2 in distance from device feature point P2 (for example, the difference between the distance y3 between P2 and P4 and x2 is also smaller than a second predetermined threshold). The base station compares the difference between y2 and x2 with the difference between y3 and x2 and determines that the difference between y2 and x2 is smaller. Therefore, the device feature point pair {P1,P3} corresponds to the target image feature point pair {a1,a3}.

[0118] Furthermore, the base station determines that target image feature point a1 corresponds to device feature point P1, target image feature point a2 corresponds to device feature point P2, and target image feature point a3 corresponds to device feature point P3.

[0119] In S204, the base station determines the attitude information of the AR device based on the positional information of each target device feature point and each target image feature point.

[0120] Based on the positional information of each target device feature point in the AR device coordinate system and the positional information of each target image feature point in the base station coordinate system, the relative position information of the AR device with respect to the base station is determined, and this relative position information is used as the attitude information of the AR device.

[0121] After determining the target device feature points corresponding to each target image feature point, the base station may obtain the 3D coordinates of each target device feature point in the AR device coordinate system and determine the 3D orientation T0 of the AR device coordinate system relative to the base station coordinate system based on the 3D coordinates of each target image feature point in the base station coordinate system. This 3D orientation T0 is, in other words, the orientation information of the AR device, and its specific form is: The filename is TIFF0007842942000009.tif10170. Here, R is a rotation matrix that reflects the angular relationship between the AR device coordinate system and the base station coordinate system. TIFF0007842942000010.tif7170 is a translation matrix that reflects the distance between the AR device coordinate system and the base station coordinate system, and the rotation matrix must satisfy the following conditions. TIFF0007842942000011.tif8170 Here, I is the identity matrix.

[0122] Clearly, the pose information T0 of the AR device is 6-degree-of-freedom pose data.

[0123] Subsequently, if each collected image contains more than three image feature points, that is, if each collected image contains three or more image feature points, T0 may be adjusted using candidate image feature points other than the target image feature points. By performing nonlinear optimization with T0 as the initial pose information, the final pose information T1 is obtained.

[0124] In one optional embodiment, the base station predicts the reference position information of each of the at least one candidate device feature points in the base station coordinate system based on the position information of each of the at least one candidate device feature points other than each target device feature point in the AR device coordinate system, and the attitude information; predicts the predicted position information of each of the at least one candidate device feature points in the AR device coordinate system based on the reference position information, orientation parameters indicating the positional relationship between each acquisition device, and the attitude information; and finally adjusts the attitude information of each of the at least one candidate device feature points in the AR device coordinate system based on the difference between the position information of each candidate device feature point and the corresponding predicted position information.

[0125] In the above, for each candidate device feature point other than the target device feature point in the AR device, the base station can inversely calculate the coordinates of the candidate device feature point in the base station coordinate system (i.e., reference position information) based on T0 and the coordinates of the candidate device feature point in the AR device coordinate system. Then, based on the reference position information, the orientation parameters of the infrared binocular camera, and T0, the base station predicts the coordinates of the candidate device feature point in the AR device coordinate system (i.e., predicted position information).

[0126] As one optional method for acquiring predicted location information, the base station predicts the location of each feature point of at least one candidate device feature point in the acquired image based on reference location information and orientation parameters that indicate the positional relationship between each acquisition device. For each feature point location, image feature points that are within a predetermined distance from the feature point location are designated as candidate image feature points corresponding to the candidate device feature point. Based on the location information and orientation information of each of the at least one candidate image feature points in the base station coordinate system, the base station predicts the predicted location information of the corresponding candidate device feature point in the AR device coordinate system.

[0127] In S202, the base station calculates the three-dimensional coordinates of three target image feature points in the base station coordinate system based on the orientation parameters and the distance between at least two pairs of target image feature points in each acquired image. Currently, this calculation process can be applied in reverse; that is, for each candidate device feature point, the position of that candidate device feature point in the acquired image, i.e., the feature point position, can be calculated based on its reference position information and orientation parameters. This is equivalent to roughly estimating the position of the corresponding candidate image feature point in the image, assuming that the candidate device feature point has been captured in the acquired image. The purpose of this step is to subsequently find the candidate image feature point corresponding to the candidate device feature point in the acquired image.

[0128] Subsequently, the base station projects the estimated feature point location onto the collected image, searches for the image feature point closest to that location, and determines it as the candidate image feature point corresponding to the candidate device feature point. The true location of the candidate image feature point in the collected image should satisfy the condition that the distance from the feature point location is < 5 pixels.

[0129] Subsequently, the base station decides to predict the predicted position information of the corresponding candidate device feature point in the AR device coordinate system again, based on the coordinates of the candidate image feature point in the base station coordinate system and the orientation information.

[0130] The base station may determine a single final attitude information T1 based on the difference between each predicted position information and the true coordinates of the corresponding candidate device feature point in the AR device coordinate system. The attitude information T1 can minimize the sum of the difference between the predicted position information of each candidate device feature point and the true coordinates of the candidate device feature point in the AR device coordinate system, and the difference between the predicted position information of the target device feature point and the true coordinates of the target device feature point in the AR device coordinate system. Specifically, the following formula can be used. TIFF0007842942000012.tif26170

[0131] The above formula is calculated using a minimum quadratic optimization algorithm. TIFF0007842942000013.tif7170 is a rotation matrix, TIFF0007842942000014.tif7170 is a translation matrix, TIFF0007842942000015.tif7170The true coordinates of device feature point i in the AR device coordinate system, TIFF0007842942000016.tif7170 These are the coordinates of the image feature point i corresponding to the device feature point i in the base station coordinate system. TIFF0007842942000017.tif7170 is the predicted location information for device feature point i. Device feature point i is either a candidate device feature point or a target device feature point.

[0132] Furthermore, in the above formula, if the orientation information is T0, T0 is calculated based on the coordinates of the target image feature points corresponding to the target device feature points in the base station coordinate system and the true coordinates of the target device feature points in the AR device. Therefore, in this case, the difference between the predicted position information of the target device feature points and the true coordinates of the target device feature points in the AR device is 0.

[0133] Following the hypothesis of S203, the base station determines that target image feature point a1 corresponds to device feature point P1, target image feature point a2 corresponds to device feature point P2, and target image feature point a3 corresponds to device feature point P3. Then, based on the coordinates of target image feature points a1, a2, and a3 in the base station coordinate system and the coordinates of device feature points P1, P2, and P3 in the AR device coordinate system, it calculates the attitude information T0 of the AR device.

[0134] Furthermore, the base station searches for candidate image feature points a5 and a6 that correspond to candidate device feature points P5 and P6 other than device feature points P1, P2, and P3. Candidate image feature points a5 and a6 are visible in the acquired image. Based on the coordinates of candidate image feature points a5 and a6 in the base station coordinate system and the orientation parameters of the infrared binocular camera, the predicted position information of candidate device feature points P5 and P6 corresponding to the candidate image feature points a5 and a6 predicted by the base station is obtained in the AR device coordinate system. By comparing each predicted position information with the true coordinates of the corresponding candidate device feature points in the AR device coordinate system, T0 is adjusted, and finally, pose information T1 is obtained. The orientation information T1 can minimize the sum of the differences between the predicted position information of candidate device feature points P5 and P6 and the actual coordinates of candidate device feature points P5 and P6 in the AR device coordinate system, and the differences between the predicted position information of device feature points P1, P2, and P3 and the actual coordinates of device feature points P1, P2, and P3 in the AR device coordinate system.

[0135] Figure 10 shows a diagram illustrating the interaction between the AR device provided in this application, a base station, and a terminal device. After acquiring attitude information T1, the base station may assign a timestamp to this 6-degree-of-freedom attitude information and transmit the timestamped attitude information T1 to the terminal device via Wi-Fi. The AR device itself may measure its own inertial measurement unit (IMU) data and transmit the timestamped IMU data to the terminal device. The terminal device performs image rendering based on the attitude information T1 and the IMU data and transmits the rendered image to the AR device. The terminal device also needs to synchronize its time with the base station, and the base station simultaneously synchronizes its time with the AR device.

[0136] The above IMU may also be a 6-axis sensor that may include a 3-axis gyroscope and a 3-axis accelerometer. The gyroscope measures the angular velocity (rotational degrees of freedom) on each axis, i.e., the number of degrees that can be covered in one second if rotating in this direction, and the accelerometer measures the acceleration (movement degrees of freedom) on each axis. IMU data can be used to predict the attitude of the AR device at the next moment, based on the current conditions of the AR device, such as its movement speed and acceleration.

[0137] Furthermore, to further reduce power consumption during communication, the Bluetooth Low Energy (BLE) protocol may be used to transmit IMU data from the AR device to the terminal device.

[0138] As a specific rendering process for the terminal device, data fusion may be performed between attitude information uploaded from the base station and IMU data uploaded from the AR device to estimate rendering time and image transmission time. Before the next time point arrives, the virtual screen at the next time point may be predicted based on the current time point and the estimated rendering time and image transmission time, the correct 3D attitude may be rendered, and the rendering result may be transmitted to the AR device via Wi-Fi for display.

[0139] The method provided in this application significantly reduces the computing power and power consumption requirements for AR devices. While typical inside-out attitude detection methods for head-mounted AR devices consume 1.5W or more, this application requires only power consumption for Wi-Fi transmission, resulting in an optimized average power consumption of approximately 200-500mW. Furthermore, the AR device of this application may be lightweight AR glasses, eliminating the need for additional electronic components or circuits and simplifying system design.

[0140] Figure 11 shows a specific implementation flowchart of the posture detection method for another AR device provided in the embodiment of the present application. This method may include the following specific implementation flowchart.

[0141] In S1101, the base station emits a detection ray (e.g., an illumination ray) to the AR device to capture at least two collected images.

[0142] In S1102, the base station determines whether it can obtain three target image feature points from the collected image. If it can obtain three target image feature points from the collected image, it executes S1103. If it cannot obtain three target image feature points from the collected image, it executes S1104.

[0143] In S1103, the base station determines the coordinates p1, p2, and p3 of the three target image feature points a1, a2, and a3 in the base station coordinate system.

[0144] Attitude detection fails in S1104.

[0145] In S1105, the base station obtains two target image feature point pairs {a1, a2} and {a1, a3} based on three target image feature points, and calculates the predicted distances d1=|p1p2| and d2=|p1p3| for each of the two target image feature point pairs.

[0146] In S1106, the base station determines two device feature points P1 and P2 from among the device feature points that satisfy the condition that the distance between P1 and P2 is closest to d1.

[0147] In S1107, the base station determines one device feature point P3 that is closest to d2 in distance from P1.

[0148] In S1108, the base station determines one device feature point P4 that is closest to d2 in distance from P2.

[0149] In concrete implementation, the difference between the distance between P1 and P3 in S1107 and d2 may be smaller than the second predetermined threshold, and the difference between the distance between P2 and P4 in S1108 and d2 may be smaller than the second predetermined threshold.

[0150] If neither S1107 nor S1108 can find a pair of device feature points that satisfies the condition that the difference value with d2 is less than the second predetermined threshold, the position determination fails. If only one of S1107 or S1108 succeeds in finding a pair of device feature points that satisfies the condition that the difference value with d2 is less than the second predetermined threshold, it is directly determined that the pair of device feature points corresponds to the target image feature point pair {a1, a3}.

[0151] In S1109, the base station compares the magnitude relationship between ||P1P3|-d2| and ||P2P4|-d2|. If ||P1P3|-d2| is smaller than ||P2P4|-d2|, S1110 is executed. If ||P1P3|-d2| is greater than or equal to ||P2P4|-d2|, S1111 is executed.

[0152] In S1110, the base station determines that target image feature points a1, a2, and a3 correspond to target device feature points P1, P2, and P3, and acquires AR device attitude information based on the coordinates of P1, P2, and P3 in the AR device coordinate system and the coordinates of a1, a2, and a3 in the base station coordinate system.

[0153] In S1111, the base station determines that target image feature points a1, a2, and a3 correspond to target device feature points P2, P1, and P4, and acquires AR device attitude information based on the coordinates of P2, P1, and P4 in the AR device coordinate system and the coordinates of a1, a2, and a3 in the base station coordinate system.

[0154] The flowchart listed above is merely illustrative, and steps S1107 and S1108 may be performed in any order, or simultaneously. This invention does not limit its specific implementation.

[0155] Figure 12 is a logic diagram illustrating the interaction between an AR device, a base station, and a terminal device provided in an embodiment of the present invention. As shown in Figure 12, the base station emits a light ray to the AR device, and as the device feature points on the AR device reflect the light ray, the base station simultaneously captures the device feature points to obtain multiple collected images. Subsequently, the base station determines the predicted distance for each of at least two pairs of target image feature points that contain the same target image feature point, and determines the target device feature point corresponding to each target image feature point based on the predicted distance and the actual distance between each device feature point. Finally, the base station determines the attitude information of the AR device based on the position information of the target image feature points and the position information of the corresponding target device feature points, and transmits the attitude information to the terminal device. The AR device also acquires its own inertial measurement unit data and transmits it to the terminal device. The terminal device renders a virtual screen based on the attitude information and inertial measurement unit data of the AR device, and after rendering is complete, transmits the rendered screen to the AR device for display.

[0156] Based on a similar inventive concept, embodiments of the present application further provide an attitude detection device for an AR device. Figure 13 is a schematic diagram of the configuration of the attitude detection device for an AR device. As shown in Figure 13, the device is An acquisition unit 1301 that acquires at least two collected images based on different shooting orientations by emitting a detection ray to an AR device, wherein each collected image includes image feature points obtained by reflecting the detection ray based on device feature points in the AR device, and A prediction unit 1302 determines the predicted distance for each of at least two pairs of target image feature points that contain the same target image feature point, based on at least two collected images. A comparison unit 1303 determines the target device feature point corresponding to each target image feature point in the AR device based on the comparison result between the actual distance and each predicted distance between each device feature point in the AR device, The system may also include a determination unit 1304 that determines the orientation information of the AR device based on the positional information of each target device feature point and each target image feature point.

[0157] Selectively, each acquired image is collected based on an acquisition device located in one of the imaging orientations. The prediction unit 1302 specifically... Based on the distance between at least two pairs of target image feature points in each acquired image and orientation parameters indicating the positional relationship between each acquisition device, the predicted distance for each of the at least two pairs of target image feature points is determined.

[0158] Selectively, the actual distances between any two device feature points are different, and the difference between any two actual distances is greater than a first predetermined threshold.

[0159] Optionally, the comparison unit 1303 is, specifically, The actual distance between feature points on each AR device is compared with the predicted distance between them. Based on the comparison results, at least two pairs of device feature points containing the same device feature point are determined, and from among the at least two pairs of device feature points, a target device feature point corresponding to each target image feature point is determined, where the difference between the actual distance between each pair of device feature points and the corresponding predicted distance is smaller than a second predetermined threshold, and the second predetermined threshold is less than or equal to the first predetermined threshold.

[0160] Optionally, each predicted distance includes a first predicted distance for a first pair of target image feature points and a second predicted distance for a second pair of target image feature points among at least two pairs of target image feature points.

[0161] The comparison unit 1303 is, specifically, Two device feature points whose difference value between the corresponding actual distance and the first predicted distance is less than the second predetermined threshold are defined as the first device feature point pair corresponding to the first target image feature point pair. The system searches for at least one other pair of device feature points that contains the same device feature points as the first pair of device feature points, where the difference between the corresponding actual distance and the second predicted distance is smaller than a second predetermined threshold, From among the device feature points included in one pair of device feature points and at least one other pair of device feature points, the target device feature point corresponding to each target image feature point is determined.

[0162] Optionally, the comparison unit 1303 is, specifically, From at least one other pair of device feature points, determine the second pair of device feature points that corresponds to the second pair of target image feature points. The same device feature points are treated as target device feature points corresponding to the same target image feature points. Device feature points other than the same device feature point in the first pair of device feature points, and the same device feature points in the first pair of target image feature points. target Target device feature points corresponding to image feature points other than image feature points, Device feature points other than the same device feature point in the second pair of device feature points, and the same device feature points in the second pair of target image feature points. target These are designated as target device feature points corresponding to image feature points other than image feature points.

[0163] Optionally, the comparison unit 1303 is, specifically, If there is only one other device feature point pair, that other device feature point pair will be designated as the second device feature point pair corresponding to the second target image feature point pair. If there are multiple pairs of other device feature points, one pair is selected from among them based on the relative actual distances between them, and this is designated as the second device feature point pair corresponding to the second target image feature point pair.

[0164] Optionally, the comparison unit 1303 is, specifically, The other pair of device feature points that minimizes the difference between the corresponding actual distance and the second predicted distance is designated as the second pair of device feature points corresponding to the second pair of target image feature points.

[0165] Optionally, the decision unit 1304 specifically: Based on the positional information of each target device feature point in the AR device coordinate system and the positional information of each target image feature point in the base station coordinate system, the relative position information of the AR device with respect to the base station is determined. Relative position information is used as the orientation information of the AR device.

[0166] Selectively, each acquired image is collected based on an acquisition device located in one of the imaging orientations, and the apparatus further includes an adjustment unit 1305.

[0167] The adjustment unit 1305 predicts the reference position information of at least one candidate device feature point in the base station coordinate system based on the position information and attitude information of at least one candidate device feature point other than each target device feature point in the AR device coordinate system. Based on the reference position information, orientation parameters indicating the positional relationship between the at least two acquisition devices, and orientation information, the predicted position information for each of the at least one candidate device feature points in the AR device coordinate system is predicted. The orientation information is adjusted based on the difference between the positional information of at least one candidate device feature point in the AR device coordinate system and the corresponding predicted positional information.

[0168] Optionally, the adjustment unit 1305 is, specifically, Based on reference position information and orientation parameters indicating the positional relationship between at least two acquisition devices, the feature point positions of at least one candidate device feature point in the acquired image are predicted. For each feature point location, image feature points whose distance from other feature point locations is within a predetermined range are designated as candidate image feature points corresponding to candidate device feature points. Based on the positional information and orientation information of at least one candidate image feature point in the base station coordinate system, the predicted positional information of the corresponding candidate device feature point in the AR device coordinate system is predicted.

[0169] Optionally, each feature point in an AR device is composed of a material that reflects infrared light.

[0170] For the sake of explanation, the above parts are described separately by function into modules (or units). Of course, when implementing this invention, the functions of each module (or unit) may be realized with the same or multiple software or hardware.

[0171] After introducing an exemplary embodiment of the AR device attitude detection method and apparatus of the present application, we will now introduce an electronic device according to another exemplary embodiment of the present application.

[0172] Those skilled in the art will understand that various embodiments of the present application can be implemented as systems, methods, or program products. Therefore, these various embodiments may be implemented specifically as complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software. These embodiments may collectively be referred to here as “circuits,” “modules,” or “systems.”

[0173] Based on the same inventive concept as the embodiments of the method described above, embodiments of the present application further provide an electronic device. In one embodiment, the electronic device may be a terminal device 130 as shown in Figure 1. In this embodiment, the configuration of the electronic device may include components such as a communication component 1410, a memory 1420, a display unit 1430, a camera 1440, a sensor 1450, an audio circuit 1460, a Bluetooth module 1470, and a processor 1480, as shown in Figure 14.

[0174] The communication component 1410 is used to communicate with the server. In some embodiments, Communication component 1410 is, The circuit may include a Wireless Fidelity (WiFi) module, which belongs to short-range wireless transmission technology. The WiFi module can help the electronic device transmit and receive information about an object (e.g., a user).

[0175] Memory 1420 can be used to store software programs and data. The processor 1480 executes various functions and data processing of the terminal device 130 by executing the software programs or data stored in memory 1420. Memory 1420 stores an operating system on which the terminal device 130 can operate. Memory 1420 in this application can store an operating system and various application programs, and can also store a computer program that executes the attitude detection method of the AR device of the embodiment of this application.

[0176] The display unit 1430 can also be used to display information input by an object, information provided to an object, and the graphical user interface (GUI) of various menus of the terminal device 130. Specifically, the display unit 1430 may include a display screen 1432 provided on the front of the terminal device 130. The display unit 1430 can be used to display the rendering interface in the embodiment of the present application, etc.

[0177] The display unit 1430 can also be used to receive input numbers and characters and generate signal inputs related to object settings and function control of the terminal device 130. Specifically, the display unit 1430 may include a touchscreen 1431 located on the front of the terminal device 130. The touchscreen 1431 can collect touch operations on or near objects (for example, clicking buttons or dragging scroll boxes).

[0178] Here, the touchscreen 1431 may cover the display screen 1432, or the touchscreen 1431 and the display screen 1432 may be integrated to realize the input and output functions of the terminal device 130, and the integrated unit can be called a touch display screen. The display unit 1430 in this application can display an application program and corresponding operating procedures.

[0179] Camera 1440 can be used to capture still images, and the object can display the images captured by camera 1440 in the application. There may be one or more cameras 1440. The object generates an optical image through a lens, which is projected onto a photosensitive element. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to processor 1480 to be converted into a digital image signal.

[0180] The terminal device may include at least one type of sensor 1450, for example, an accelerometer 1451, a distance sensor 1452, a fingerprint sensor 1453, and a temperature sensor 1454. The terminal device may also be equipped with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0181] The audio circuit 1460, speaker 1461, and microphone 1462 can provide an audio interface between the object and the terminal device 130.

[0182] The Bluetooth module 1470 is used to exchange information with other Bluetooth devices that also have a Bluetooth module via the Bluetooth protocol. For example, a terminal device can establish a Bluetooth connection with a wearable electronic device (e.g., a smartwatch) that also has a Bluetooth module via the Bluetooth module 1470 for data exchange.

[0183] The processor 1480 is the control center of the terminal device, connecting various parts of the entire terminal using various interfaces and lines, executing or performing software programs stored in memory 1420, and retrieving data stored in memory 1420 to perform various functions of the terminal device and process data. In some embodiments, the processor 1480 may include one or more processing units. The processor 1480 may incorporate an application processor that mainly processes the operating system, object interface, and application programs, and a baseband processor that mainly processes wireless communication. To make it clear, the baseband processor does not have to be incorporated into the processor 1480. The processor 1480 in this application can execute the operating system, application programs, object interface display and touch response, and the attitude detection method of the AR device in the embodiment of this application. The processor 1480 is also coupled to the display unit 1430.

[0184] In some possible embodiments, each aspect of the AR device attitude detection method provided herein may be implemented in the form of a program product comprising a computer program. When the program product is executed by a processor, the steps of the AR device attitude detection method according to various exemplary embodiments of the application described herein are implemented. For example, an electronic device may perform the steps shown in Figure 2.

[0185] In this specification, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device. A readable signaling medium may include data signals propagated in the baseband or as part of a carrier, which may contain a readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signaling medium may be any readable medium other than a readable storage medium, which may transmit, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.

[0186] Computer programs contained in a readable medium may be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, or any suitable combination thereof.

[0187] A computer program for performing the operations of this invention can be created in any combination of one or more programming languages. The computer program may be executed entirely on the target electronic device, partially on the target electronic device, executed as an independent software package, partially executed on the target electronic device and partially on a remote electronic device, or fully executed on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device may be connected to the target electronic device via any type of network, including a local area network (LAN) or wide area network (WAN), or it may be connected to an external electronic device (for example, connected via the Internet using an Internet service provider).

[0188] It should be noted that while the detailed description above refers to some units or subunits of the apparatus, such divisions are illustrative and not mandatory. In practice, according to embodiments of the present application, the features and functions of two or more units described above may be embodied in a single unit. Conversely, the features and functions of a single unit described above may be further divided so as to be embodied by multiple units.

[0189] Furthermore, although the drawings depict the operation of the method of the present application in a specific order, this does not require or suggest that these operations must be performed in this specific order, or that all illustrated operations must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one, and / or one step may be broken down into multiple steps.

[0190] This application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of this application. It should be understood that computer program instructions may implement each flow and / or block in the flowchart and / or block diagram, as well as combinations of flows and / or blocks in the flowchart and / or block diagram. These computer program instructions may be provided to a processor of a general-purpose computer, a dedicated computer, an embedded processor, or another programmable data processing device to generate a machine. The instructions executed by the computer or other programmable data processing device processor then generate a device for implementing one or more flows in the flowchart and / or one or more blocks in the block diagram.

[0191] These computer program instructions may be stored in computer-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner. This allows the instructions stored in the computer-readable memory to generate a product including an instruction unit, which implements the functions specified in one or more flows of a flowchart and / or one or more blocks of a block diagram.

[0192] These computer program instructions may be loaded onto a computer or other programmable data processing device to execute a series of operational steps on the computer or other programmable device to generate processing to be implemented on the computer. The instructions executed on the computer or other programmable device then provide steps to implement a function specified in one or more flows of a flowchart and / or one or more blocks of a block diagram.

[0193] Furthermore, terms such as "1st," "2nd," "3rd," "4th," "1," and "2" (if any) in the specification, claims, and drawings of this application are used to distinguish similar subjects and are not necessarily used to describe a specific order or priority. It should be understood that they are used in this manner. term These components are interchangeable, where appropriate, so that the embodiments of the present application described herein can be carried out in an order other than that shown in the figures or described in text.

[0194] While preferred embodiments of the present application have been described, those skilled in the art, once they understand the basic creative concepts, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be construed as including the preferred embodiments, as well as all changes and modifications that fall within the scope of the present application.

Claims

1. A method for detecting the attitude of an augmented reality (AR) device, which is performed by a base station, A step of acquiring at least two collected images based on different shooting orientations by emitting a detection ray to an AR device, wherein each collected image includes image feature points obtained by reflecting the detection ray based on device feature points in the AR device; The steps include determining the predicted distances of at least two pairs of target image feature points that contain the same target image feature point, based on the at least two collected images, The steps include determining the target device feature point corresponding to each target image feature point in the AR device based on the comparison result between the actual distance and the predicted distance between each device feature point in the AR device, The process includes the step of determining the orientation information of the AR device based on the positional information of each target device feature point and each target image feature point. A method for detecting the posture of an AR device, characterized by the following:

2. Each collected image was acquired based on a collection device located in one of the aforementioned shooting orientations. The step of determining the predicted distances of at least two pairs of target image feature points that contain the same target image feature point, based on the at least two acquired images, The process includes determining the predicted distance of each of the at least two target image feature point pairs based on the distance between the at least two target image feature point pairs in each acquired image and a orientation parameter indicating the positional relationship between the at least two acquisition devices. The method for detecting the posture of an AR device according to feature 1.

3. If the actual distances between any two device feature points are different, and the difference between the two actual distances is greater than a first predetermined threshold, The method for detecting the posture of an AR device according to feature 1.

4. The step of determining the target device feature point corresponding to each target image feature point in the AR device, based on the comparison result between the actual distance and each predicted distance between each device feature point in the AR device, is as follows: The steps include comparing the actual distance between each pair of device feature points in the AR device with each predicted distance, A step of determining at least two pairs of device feature points that include the same device feature point based on the comparison results, wherein the difference between the actual distance between each pair of device feature points and the corresponding predicted distance is smaller than a second predetermined threshold, and the second predetermined threshold is less than or equal to the first predetermined threshold. The process includes the step of determining a target device feature point corresponding to each target image feature point from among the at least two pairs of device feature points. The method for detecting the posture of an AR device according to feature 3.

5. There are two pairs of target image feature points, and each predicted distance includes a first predicted distance for the first pair of target image feature points and a second predicted distance for the second pair of target image feature points. The step of determining at least two pairs of device feature points that contain the same device feature point is: The steps include: setting two device feature points whose difference value between the corresponding actual distance and the first predicted distance is less than the second predetermined threshold as the first device feature point pair corresponding to the first target image feature point pair; The process includes the step of finding at least one other pair of device feature points in which the difference between the corresponding actual distance and the second predicted distance is smaller than the second predetermined threshold, and which includes the same device feature points as the first pair of device feature points. The method for detecting the posture of an AR device according to feature 4.

6. The step of determining a target device feature point corresponding to each of the target image feature points from at least two pairs of device feature points is: The steps include determining a second pair of device feature points corresponding to the second pair of target image feature points from at least one other pair of device feature points, The steps include: setting the same device feature point to a target device feature point corresponding to the same target image feature point; The steps include: setting a device feature point other than the identical device feature point in the aforementioned pair of device feature points as a target device feature point corresponding to an image feature point other than the identical target image feature point in the aforementioned pair of target image feature points; The step of making a device feature point other than the identical device feature point in the two device feature point pair the target device feature point that corresponds to the image feature point other than the identical target image feature point in the two target image feature point pair, The method for detecting the posture of an AR device according to feature 5.

7. The step of determining a second pair of device feature points corresponding to the second pair of target image feature points from at least one other pair of device feature points is: If the number of the aforementioned other device feature point pairs is one, the step is to make the aforementioned other device feature point pairs the second device feature point pairs corresponding to the second target image feature point pairs. If there are multiple pairs of other device feature points, the step of selecting one of the multiple pairs of other device feature points based on the relative actual distances of the multiple pairs of other device feature points to be designated as the second device feature point pair corresponding to the second target image feature point pair, is included. The method for detecting the posture of an AR device according to feature 6.

8. The step of selecting one of the multiple other device feature point pairs based on the relative actual distances of the multiple other device feature point pairs to make it the second device feature point pair corresponding to the second target image feature point pair is: The process includes the step of selecting the other pair of device feature points that minimizes the difference between the corresponding actual distance and the second predicted distance as the second pair of device feature points corresponding to the second pair of target image feature points. The method for detecting the posture of an AR device according to feature 7.

9. The step of determining the orientation information of the AR device based on the positional information of each target device feature point and each target image feature point is as follows: A step of determining the relative position information of the AR device with respect to the base station based on the position information of each target device feature point in the AR device coordinate system and the position information of each target image feature point in the base station coordinate system. The step of using the relative position information as orientation information of the AR device is included. The method for detecting the posture of an AR device according to feature 1.

10. Each acquired image is collected based on an acquisition device located in one of the aforementioned shooting orientations, and the method is, A step of predicting the reference position information of each of the at least one candidate device feature points in the base station coordinate system based on the position information of each of the at least one candidate device feature points other than each of the target device feature points in the AR device coordinate system and the attitude information, A step of predicting the predicted position information of each of the at least one candidate device feature points in the AR device coordinate system based on the reference position information, orientation parameters for indicating the positional relationship between the at least two acquisition devices, and orientation information. The further step includes adjusting the attitude information based on the difference between the position information of each of the at least one candidate device feature points in the AR device coordinate system and the corresponding predicted position information. The method for detecting the posture of an AR device according to feature 1.

11. The step of predicting the predicted position information of each of the at least one candidate device feature points in the AR device coordinate system based on the reference position information, orientation parameters for indicating the positional relationship between the at least two acquisition devices, and orientation information is as follows: A step of predicting the feature point positions of each of the at least one candidate device feature points in the acquired image based on the reference position information and the orientation parameters, For each feature point location, the step of selecting an image feature point that is within a predetermined distance from the feature point location as a candidate image feature point corresponding to the candidate device feature point, The method includes the step of predicting the predicted position information of the corresponding candidate device feature point in the AR device coordinate system based on the position information of at least one candidate image feature point in the base station coordinate system and the attitude information of the respective candidate image feature point, The method for detecting the posture of an AR device according to feature 10.

12. Each feature point in the aforementioned AR device is made of a material that reflects infrared light. The method for detecting the posture of an AR device according to feature 1.

13. An attitude detection device for an AR device, An acquisition unit that acquires at least two collected images based on different shooting orientations by emitting a detection ray to an AR device, wherein each collected image includes image feature points obtained by reflecting the detection ray based on device feature points in the AR device, A prediction unit that determines the predicted distance of each of at least two pairs of target image feature points that contain the same target image feature point, based on the at least two collected images, A comparison unit that determines the target device feature point corresponding to each target image feature point in the AR device based on the comparison result between the actual distance and the predicted distance between each device feature point in the AR device, A determination unit that determines the orientation information of the AR device based on the positional information of each target device feature point and each target image feature point, is included. An attitude detection device for an AR device characterized by the following.

14. An electronic device comprising a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it realizes the steps of the AR device attitude detection method described in any one of claims 1 to 12.

15. A computer program characterized by causing a computer to execute the posture detection method for an AR device described in any one of claims 1 to 12.

Images