Posture detection method and apparatus for ar device, and electronic device, storage medium and program product
Through the base station transmitting detection light and using image feature points to determine the attitude information of the AR device, the high power consumption and high computing power problems caused by attitude detection of existing AR devices are solved, and efficient attitude detection of lightweight AR devices is realized.
Patent Information
- Application Number
- PCT/CN2024/086911
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-04
- Filing Date
- 2024-04-10
- Publication Date
- 2024-11-07
AI Technical Summary
The attitude detection methods of existing AR devices lead to high power consumption and high computing power requirements, especially the use of cameras and depth sensors in head-mounted devices lead to high power consumption and computing requirements for the device.
The base station transmits detection light to the AR device, acquires collected images of different shooting orientations, uses image feature points to determine the reflection of the device feature points, and determines the correspondence relationship between the target image feature points by comparing the predicted distance and the actual distance, and calculates the AR device pose information avoids complex calculations and light source configuration on AR devices.
It effectively reduces the power consumption and computing power requirements of AR equipment attitude detection, simplifies equipment design, and realizes 6-degree-of-free space attitude detection of lightweight AR glasses, which is suitable for a variety of application scenarios.
Smart Images

Figure CN2024086911_07112024_PF_FP_ABST
Abstract
Description
Method, device, electronic device, storage medium, and program product for detecting posture of AR device Technical Field
[0001] The present application relates to the field of computer technology, in particular to the field of augmented reality technology, and provides a posture detection method, apparatus, electronic device, and storage medium for an AR device.
[0002] Background of the Invention
[0003] With the advancement of science, technologies such as augmented reality (AR) and virtual reality (VR) have gradually matured and entered the public eye. AR technology, which combines virtual reality with reality, can be used to present virtual text, images, 3D models, and other objects in the real world through AR devices, thereby enhancing the sense of reality. Posture detection in AR devices is a fundamental and crucial component of AR technology.
[0004] Taking head-mounted devices as an example, the posture detection method of AR head-mounted devices is mainly implemented using devices such as cameras and depth sensors. These devices are all configured on the head-mounted devices. Therefore, the cameras and depth sensors require the AR head-mounted devices to provide higher power when running. The operation of the depth sensor also has certain requirements on the computing power of the device, which ultimately results in high power consumption of the head-mounted devices.
[0005] Therefore, how to reduce the power consumption and computing power requirements for AR device posture detection is an urgent problem to be solved.
[0006] Summary of the Invention
[0007] Embodiments of the present application provide a method, apparatus, electronic device, storage medium, and program product for detecting the posture of an AR device, to reduce power consumption for detecting the posture of the AR device.
[0008] An embodiment of the present application provides a posture detection method for an AR device, including:
[0009] Acquire at least two captured images based on different shooting directions by emitting a detection light toward the AR device; each captured image includes: image feature points obtained by reflecting the detection light based on device feature points on the AR device;
[0010] Determining, based on the at least two acquired images, respective predicted distances of at least two pairs of target image feature points containing the same target image feature point;
[0011] Determining, based on a comparison result between an actual distance between each device feature point on the AR device and each predicted distance, a target device feature point corresponding to each target image feature point on the AR device;
[0012] The posture information of the AR device is determined based on the position information of each target device feature point and each target image feature point.
[0013] An embodiment of the present application provides a posture detection device for an AR device, comprising:
[0014] an acquisition unit, configured to acquire at least two captured images based on different shooting directions by emitting a detection light toward the AR device; each captured image comprising: image feature points obtained by reflecting the detection light based on device feature points on the AR device;
[0015] a prediction unit, configured to determine, based on the at least two acquired images, respective predicted distances of at least two pairs of target image feature points containing the same target image feature point;
[0016] a comparing unit, configured to determine, based on a comparison result between an actual distance between each device feature point on the AR device and each predicted distance, a target device feature point corresponding to each target image feature point in the AR device;
[0017] The determining unit is configured to determine the posture information of the AR device based on the position information of each target device feature point and each target image feature point.
[0018] Optionally, each captured image is captured by a capture device located at one of the shooting positions; and the prediction unit is specifically configured to:
[0019] Based on the distances of the at least two target image feature point pairs in each acquired image and calibration parameters used to represent the positional relationship between the at least two acquisition devices, the predicted distances of the at least two target image feature point pairs are determined.
[0020] Optionally, the actual distances between any two device feature points are different, and the difference between any two actual distances is greater than a first preset threshold.
[0021] Optionally, the comparison unit is specifically used for:
[0022] Comparing the actual distance between each device feature point on the AR device with each predicted distance;
[0023] Based on the comparison result, determining at least two device feature point pairs containing the same device feature point, wherein a difference between an actual distance and a corresponding predicted distance between each device feature point pair is less than a second preset threshold; wherein the second preset threshold is not greater than the first preset threshold;
[0024] The target device feature points corresponding to the respective target image feature points are determined from the at least two device feature point pairs.
[0025] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of any of the above-mentioned posture detection methods for AR devices are implemented.
[0026] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for detecting the posture of an AR device are implemented.
[0027] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for detecting the posture of an AR device.
[0028] The beneficial effects of this application are as follows:
[0029] Embodiments of the present application provide a method, apparatus, electronic device, and storage medium for detecting the posture of an AR device. This application utilizes a base station to transmit a probe beam to the AR device and obtain at least two captured images based on different shooting orientations, thereby determining the reflection of the probe beam by device feature points on the AR device using image feature points in the images. That is, the probe beam transmitting device and the camera are both located on the base station, eliminating the need for power consumption by the transmitting device and the camera. The base station then predicts the predicted distances between two pairs of target image feature points containing the same target image feature point in a real-world environment based on each captured image. The base station then compares the actual distances between each device feature point on the AR device with each predicted distance to identify the target device feature point corresponding to each target image feature point. This process requires only a simple distance calculation, or distance comparison, and requires less computing power from the device. After finding the corresponding relationship, the base station can calculate the posture information of the AR device based on the position information of each target device feature point in the AR device coordinate system and the position information of each target image feature point in the base station coordinate system. Since the process of determining the posture of the AR device is also performed by the base station, computing power support from the AR device is not required. Based on this, the power consumption and computing power requirements for AR device posture detection can be effectively reduced.
[0030] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.
[0031] BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0033] FIG1 is a schematic diagram of an application scenario of a posture detection method for an AR device provided in an embodiment of the present application;
[0034] FIG2 is an overall flow chart of a posture detection method for an AR device provided in an embodiment of the present application;
[0035] FIG3 is a schematic diagram of an AR device provided in an embodiment of the present application;
[0036] FIG4 is a schematic diagram of an image acquisition method according to an embodiment of the present application;
[0037] FIG5 is a schematic diagram of feature points of a target image provided by an embodiment of the present application;
[0038] FIG6 is a logic diagram for calculating a predicted distance provided by an embodiment of the present application;
[0039] FIG7 is a flow chart of determining target device feature points corresponding to respective target image feature points according to an embodiment of the present application;
[0040] FIG8 is a correspondence diagram between target image feature points and target device feature points provided by an embodiment of the present application;
[0041] FIG9 is a logic diagram of determining a device feature point corresponding to a target image feature point according to an embodiment of the present application;
[0042] FIG10 is an interaction diagram of an AR device, a base station, and a terminal device provided in an embodiment of the present application;
[0043] FIG11 is a flowchart of a specific implementation of another method for detecting the posture of an AR device provided in an embodiment of the present application;
[0044] FIG12 is a logic diagram of interaction among an AR device, a base station, and a terminal device provided in an embodiment of the present application;
[0045] FIG13 is a schematic diagram of the structure of a posture detection device for an AR device provided in an embodiment of the present application;
[0046] FIG14 is a schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present application.
[0047] Modes for Carrying Out the Invention
[0048] To make the purpose, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solutions of the present application, but not all of them. Based on the embodiments described in this application document, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the technical solutions of the present application.
[0049] The following is an introduction to some concepts involved in the embodiments of this application.
[0050] Captured image: This image is captured by a base station capturing an AR device. The AR device contains device feature points that reflect the detection light. Accordingly, the captured image also contains image feature points based on the reflection of the detection light from the device feature points on the AR device. These points reflect the reflection of the infrared detection light from the device feature points.
[0051] Image feature points: The base station captures the device feature points on the AR device and obtains visible image feature points corresponding to each device feature point in the captured image. In this application, image feature points are divided into target image feature points and candidate image feature points. The target image feature points are the three image feature points initially acquired by the base station, and the candidate image feature points are the candidate image feature points corresponding to the candidate device feature points selected by the base station in order to adjust the posture information after obtaining the posture information of the target image feature points.
[0052] Device Feature Points: Points on an AR device, which can be made of a material that reflects infrared light (or other detection light, not specifically limited herein). In the embodiments of the present application, to further improve the efficiency of the base station side in determining the corresponding points of the target image feature points and reduce errors, the distances between any two device feature points can be set to be different, and the absolute value of the difference in the actual distances corresponding to any two device feature points is greater than a first preset threshold.
[0053] Base station coordinate system: A coordinate system with a point on the base station as its origin. For example, using the center point of a data acquisition device on the base station as the origin of the base station coordinate system can reflect the positional relationship of a point relative to the base station in this coordinate system.
[0054] AR device coordinate system: A coordinate system with a point on the AR device as its origin. For example, the midpoint of the line connecting the centers of the two eyepieces on the AR device is used as the origin of the AR device coordinate system. This coordinate system can reflect the position of a point relative to the AR device.
[0055] Predicted distance: The distance between device feature points corresponding to two target image feature points, calculated by the base station based on the distance between the two target image feature points in multiple acquired images and the positional relationship between multiple acquisition devices. In this application, this is divided into a first predicted distance and a second predicted distance. The first predicted distance is the predicted distance between one target image feature point pair corresponding to three target image feature points, and the second predicted distance is the predicted distance between another target image feature point pair corresponding to three target image feature points.
[0056] The following is a brief introduction to the design concept of the embodiment of this application:
[0057] With the development of science, technologies such as AR and VR have gradually matured and entered the public eye. AR technology, which combines virtual reality with reality, can present virtual text, images, 3D models, and other objects in the real world through head-mounted AR devices, thereby enhancing the sense of reality. Gesture detection in head-mounted AR devices is a fundamental and important part of AR technology.
[0058] In related technologies, VR head-mounted devices mainly adopt an outside-in approach, that is, the sensor that can determine the device posture is not located on the head-mounted device, but outside the head-mounted device. However, this method requires the configuration of an LED array on the head-mounted device, relying on the LED array to actively emit light and the sensor to sense the light to achieve positioning. The implementation process is complicated, and the LED array needs to be configured on the head-mounted device, which greatly increases the power consumption of the head-mounted device.
[0059] The posture detection method of AR head-mounted devices mainly adopts an inside-out approach, that is, the camera and depth sensor that can determine the device posture are directly located on the head-mounted device. During posture detection, the camera and depth sensor will consume a lot of power of the head-mounted device, and the depth sensor needs to perform a series of calculations, which places high demands on the computing power of the head-mounted device.
[0060] Based on this, the embodiments of the present application provide a posture detection method, device, electronic device, storage medium and program product for an AR device. Since the present application transmits detection light to the AR device through a base station, and obtains at least two captured images based on different shooting orientations through the base station, the reflection of the device feature points on the AR device to the detection light is determined through the image feature points in the image; that is, the emission device and the camera device of the detection light are located on the base station instead of the AR device, so there will be no power consumption of the emission device and the camera device on the AR device.
[0061] Afterwards, the base station predicts the predicted distances between two target image feature point pairs containing the same target image feature point in the real environment based on each captured image; and compares the predicted distances with the actual distances between each device feature point on the AR device. Since the actual distances between each device feature point on the AR device are different, the target device feature points corresponding to each target image feature point can be found through distance comparison.
[0062] Finally, the base station calculates the AR device's relative position to the base station—that is, its posture—based on the position information of each target device's feature points in the AR device's coordinate system and the position information of each target image's feature points in the base station's coordinate system. Because the base station also performs the AR device's posture determination, the AR device doesn't need to provide computing power. This reduces the power consumption and computing power required for AR device posture detection.
[0063] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.
[0064] The solution proposed in this application can be applied to 6-DOF spatial posture detection of lightweight AR glasses to realize virtual-reality fusion applications in some specific scenarios (e.g., office scenes, desktop gaming scenes, etc.). An optional application scenario is shown in Figure 1.
[0065] FIG1 is a schematic diagram of an application scenario of an embodiment of the present application. As shown in FIG1 , the application scenario includes an AR device 110 , a base station 120 , and a terminal device 130 .
[0066] Among them, AR devices include but are not limited to head-mounted AR devices, helmet-type AR devices, etc.
[0067] The embodiment of the present application is explained by taking the AR device 110 as a head-mounted AR device as an example, for example, it can be lightweight AR glasses. The AR device is provided with an AR optical display module for projecting and displaying a virtual image in reality, as shown in the shaded portion in Figure 1. In addition, the AR device is also provided with an Advanced RISC Machines System On Chip (ARM SOC) and device feature points that can reflect light. For example, when the light emitted by the base station is infrared light, the device feature points can be infrared reflective points arranged with infrared fluorescent materials. The AR device 110 is connected to the hotspot of the terminal device 130 via a wireless fidelity (WIFI) network, and performs local time synchronization with the base station 120 via the Network Time Protocol (NTP).
[0068] It should be noted that other types of AR devices are also applicable to this application, and the light emitted by the above-mentioned base station and the materials used to arrange the device feature points are only examples, and this application does not make specific limitations.
[0069] Base station 120 is a positioning base station. It includes multiple data collection devices, such as a set of infrared binocular cameras; light emitting devices, such as infrared LEDs; and an ARM SoC. Base station 120 deploys an NTP local time server and synchronizes global time with the internet via the Wi-Fi hotspot of terminal device 130.
[0070] Terminal device 130 provides a Wi-Fi hotspot for AR devices, enabling network connectivity between AR device 110 and base station 120. The device also synchronizes the base station's local time using the NTP protocol. Terminal devices 130 include, but are not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, intelligent voice interaction devices, smart appliances, and in-vehicle terminals.
[0071] It should be noted that the AR device posture detection method in each embodiment of the present application is performed by base station 120. Base station 120 transmits infrared detection light to AR device 110 via an infrared LED. Simultaneously, using two infrared binocular cameras, each containing a capture device located at different shooting positions, it captures two captured images, obtained by reflecting the infrared detection light off device feature points on AR device 110. Based on the two captured images, base station 120 calculates the predicted distances between at least two pairs of target image feature points containing the same target image feature point in the actual scene. This distance is then compared with the actual distances between the device feature points on AR device 110 to determine the corresponding target device feature point on AR device 110 for each pair of target image feature points. Finally, based on the position information of each target device feature point in the AR device coordinate system and the position information of each target image feature point in the base station coordinate system, base station 120 calculates the relative position information of AR device 110 relative to base station 120, thereby obtaining posture information of AR device 110.
[0072] It should be noted that what is shown in FIG1 is only an example, and the number of AR devices 110 , base stations 120 , and terminal devices 130 is not limited and is not specifically limited in the embodiments of the present application.
[0073] In addition, the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving and other scenarios.
[0074] The following describes the posture detection method of an AR device provided by an exemplary embodiment of the present application in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect.
[0075] FIG2 is a flowchart of an implementation method for detecting a posture of an AR device provided in an embodiment of the present application. Taking a base station as an example, the method may include the following specific implementation process:
[0076] S201: The base station transmits a detection light to the AR device to obtain at least two captured images based on different shooting directions.
[0077] In the above, each captured image includes: image feature points obtained by reflecting the detection light based on the device feature points on the AR device.
[0078] Figure 3 is a schematic diagram of an AR device provided in an embodiment of the present application. As shown in Figure 3, the AR device is a lightweight pair of AR glasses with multiple black dots on them. These black dots are device feature points that can reflect light. For example, when the light emitted by the base station is infrared light, the device feature points can be infrared reflective dots arranged with infrared fluorescent material. In other words, each device feature point on the AR device is composed of a material that can reflect infrared light.
[0079] The infrared LED light on the base station can emit infrared light to the AR device. At the same time, multiple acquisition devices on the base station located at different shooting directions shoot the AR device to obtain multiple captured images.
[0080] Each captured image is captured by a capture device located at a shooting position, and the capture device may be a camera, etc. Taking the capture device as an example, each capture device may be an independent camera device, or multiple capture devices may be located on the same camera device.
[0081] Figure 4 is a schematic diagram of a captured image provided in an embodiment of the present application. As shown in Figure 4, a capture device on the base station shoots the AR device to obtain a captured image schematic diagram. The captured image contains multiple image feature points, and each image feature point corresponds to a device feature point on the AR device.
[0082] Multiple acquisition devices can be infrared binocular cameras (composed of two acquisition devices, each located in different shooting positions). The infrared binocular camera captures two images, which reflect the reflection of infrared light by the device's feature points on the AR device. To reduce flicker and improve image quality, the on / off control of the infrared LED array must be strictly synchronized with the camera's exposure.
[0083] In addition, the method in which the base station emits infrared light and the AR device reflects infrared light can be replaced by infrared LEDs, that is, infrared LEDs are configured on the AR device. The following uses infrared binocular cameras and infrared light as examples.
[0084] It should be noted that infrared light consumes little power and can be well used in the method of determining the posture of the AR device by detecting infrared reflections. In addition, other light that can achieve the above effects are also applicable to this application and will not be described in detail in this article. The following uses infrared light as an example to explain:
[0085] Taking a real-world scenario as an example, for example, subject A wears and turns on an AR device to prepare for virtual image projection. The infrared LED light on the base station emits infrared light to the AR device. At the same time, the infrared binocular camera on the base station captures the reflection of the device feature points on the AR device to obtain two captured images.
[0086] S202: The base station determines, based on at least two collected images, respective predicted distances of at least two pairs of target image feature points containing the same target image feature point.
[0087] In this embodiment, each captured image contains at least three image feature points (the number of device feature points on the AR device is greater than 3).
[0088] Taking two acquired images as an example, after obtaining the acquired images, the base station first randomly selects three common target image feature points contained in the two acquired images. Based on these three target image feature points, at least two target image feature point pairs are determined. "Including the same target image feature point" means that any two target image feature point pairs must contain the same target image feature point. For example, image feature point 1, image feature point 2, and image feature point 3 can be determined as the image feature point pairs {image feature point 1, image feature point 2}, {image feature point 1, image feature point 3}, and {image feature point 2, image feature point 3}. Among them, {image feature point 1, image feature point 2} and {image feature point 1, image feature point 3} both contain image feature point 1; {image feature point 1, image feature point 3} and {image feature point 2, image feature point 3} both contain image feature point 3; and {image feature point 1, image feature point 2} and {image feature point 2, image feature point 3} both contain image feature point 2.
[0089] In addition, the method for detecting the posture of an AR device proposed in this application can be completed by selecting two target image feature point pairs from the three target image feature point pairs mentioned above. The selection process can be random selection. For example, {image feature point 1, image feature point 2} and {image feature point 2, image feature point 3} can be selected, or {image feature point 1, image feature point 2} and {image feature point 1, image feature point 3} can be selected, or {image feature point 2, image feature point 3} and {image feature point 1, image feature point 3} can be selected.
[0090] Alternatively, two target image feature point pairs with a large difference in distance between them can be selected; or the AR device posture detection process can be performed on any two target image feature point pairs, and the resulting posture information can be averaged. This application does not specifically limit the specific implementation method. The following mainly uses two target image feature point pairs as an example for explanation.
[0091] In addition to randomly selecting three target image feature points, three image feature points with relatively large distances can also be selected as target image feature points to reduce errors. For example, an optional implementation is: if there are 4 image feature points in the captured image: image feature point 1, image feature point 2, image feature point 3 and image feature point 4, all image feature point combinations containing three image feature points can be determined (image feature point 1, image feature point 2, image feature point 3), (image feature point 1, image feature point 2, image feature point 4) (image feature point 2, image feature point 3, image feature point 4) (image feature point 1, image feature point 3, image feature point 4), and the one with the largest sum of distances between the three image feature points is used as the target image feature point. This application does not specifically limit the specific implementation methods.
[0092] FIG5 is a schematic diagram of target image feature points provided in an embodiment of the present application. As shown in FIG5 , in this embodiment, there are eight image feature points in the captured image, represented by black squares and slashed squares. The image feature points represented by the three black squares with the largest sum of distances between them are selected as target image feature point 1, target image feature point 2, and target image feature point 3, respectively.
[0093] The base station then determines the predicted distance between the three image feature points in the real environment based on the three image feature points reflected in each of the at least two captured images. In other words, it predicts the distance between the device feature points corresponding to the three image feature points. The distance between the image feature points in the captured images is two-dimensional, while the predicted distance is a three-dimensional distance in the real environment.
[0094] If more than two images are collected, two images can be randomly selected, or the predicted distance can be determined for every two images, and the average value is finally taken. This application does not make any specific limitations on the specific implementation methods.
[0095] An optional implementation is: the base station determines the predicted distance of at least two target image feature point pairs based on the distance of at least two target image feature point pairs in each acquired image and the calibration parameters used to represent the positional relationship between at least two acquisition devices.
[0096] That is, after obtaining the distances between at least two pairs of target image feature points in each captured image, as well as calibration parameters representing the positional relationships between the capture devices, the base station can use this information to determine the three-dimensional coordinates of the three target image feature points in the base station coordinate system. Furthermore, based on these three-dimensional coordinates, it can determine the predicted distances corresponding to the target image feature point pairs. This step is intended to subsequently determine the device feature points corresponding to the target image feature points based on the predicted distances. Specifically, it determines which device feature point captured by the capture device corresponds to a specific target image feature point in the captured image.
[0097] Among them, the zero point of the base station coordinate system can be set at the center point of one of the infrared binocular cameras. This application does not make any specific limitations on the specific implementation method.
[0098] Specifically, for example, the base station selects two target image feature point pairs, namely {image feature point 1, image feature point 2} and {image feature point 1, image feature point 3}. The base station can predict the three-dimensional coordinates of image feature point 1, image feature point 2 and image feature point 3 in the real environment and the base station coordinate system based on the distance between image feature point 1 and image feature point 2 on the two acquired images, the distance between image feature point 1 and image feature point 3 on the two acquired images, and the calibration parameters of the two acquisition devices on the infrared binocular camera, and then calculate the predicted distance of {image feature point 1, image feature point 2} and the predicted distance of {image feature point 1, image feature point 3} based on the three-dimensional coordinates.
[0099] In the above, the calibration parameters of the two acquisition devices reflect the relative position relationship between the two acquisition devices, such as the distance between the two acquisition devices and the relative angle of the shooting directions.
[0100] Following the assumptions in S201, FIG6 is a logic diagram for calculating the predicted distance provided by an embodiment of the present application. As shown in FIG6, the base station selects image feature points a1, a2, and a3 in the captured image as target image feature points, and uses a1, a2 and a1, a3 as two target image feature point pairs. The base station calculates the coordinates p1, p2, and p3 of a1, a2, and a3 in the base station coordinate system based on the distances between the three target image feature points in the two captured images and the positional relationship between the two cameras in the infrared binocular camera. Based on p1, p2, and p3, the base station can further calculate the predicted distance x1 between a1 and a2 and the predicted distance x2 between a1 and a3.
[0101] S203: The base station determines the target device feature points corresponding to the target image feature points in the AR device based on a comparison result between the actual distances between the device feature points on the AR device and the predicted distances.
[0102] In the above, the base station can call a positioning algorithm and use spatial position analysis to identify target device feature points corresponding to each target image feature point.
[0103] In addition, in order to realize the process of confirming that the target image feature point corresponds to the target device feature point by comparing the distance, this application proposes a method for arranging the feature points of each device on the AR device:
[0104] Optionally, the actual distances between any two device feature points on the AR device are different, and the difference between the two actual distances is distinguishable, that is, the difference between the two actual distances is greater than a first preset threshold. This method can accelerate the confirmation of target device feature points and simplify the algorithm.
[0105] In addition, the first preset threshold value can be further set according to the resolution and calibration parameters of the acquisition device.
[0106] It should be noted that the difference between any two examples in this application is the absolute value of the difference, that is, the difference is a non-negative number.
[0107] Based on this, for each predicted distance, the base station can compare it with each actual distance to determine the device feature point pair corresponding to the corresponding target image feature point pair, and further determine the relationship between the target image feature point pair and the corresponding device feature point pair. Figure 7 is a flowchart of determining the target device feature point corresponding to each target image feature point provided by an embodiment of the present application. As shown in Figure 7, the base station can specifically perform the following steps:
[0108] S701: The base station compares the actual distance between each device feature point pair on the AR device with each predicted distance.
[0109] Assuming there are two target image feature point pairs, the above-mentioned predicted distances include: a first predicted distance for one of the two target image feature point pairs (hereinafter referred to as the first target image feature point pair), and a second predicted distance for the other target image feature point pair (hereinafter referred to as the second target image feature point pair). The base station can successively determine the two device feature point pairs corresponding to each of the two target image feature point pairs, that is, first determine the device feature point pair corresponding to one of the target image feature point pairs, and then determine the other; or it can simultaneously determine the two device feature point pairs corresponding to each of the two target image feature point pairs. This application does not specifically limit the specific implementation method.
[0110] S702: The base station determines, based on the comparison result, at least two device feature point pairs containing the same device feature point, wherein the difference between the actual distance corresponding to each device feature point pair and the corresponding predicted distance is less than a second preset threshold.
[0111] In the above, the second preset threshold is not greater than the first preset threshold.
[0112] An optional implementation is that the base station uses two device feature points whose corresponding actual distance and the first predicted distance have a difference smaller than a second preset threshold as the first device feature point pair corresponding to the first target image feature point pair.
[0113] Afterwards, the base station searches for at least one other device feature point pair whose difference between the corresponding actual distance and the second predicted distance is smaller than a second preset threshold and contains the same device feature point as the first device feature point pair.
[0114] For example, the base station obtains two target image feature point pairs as {image feature point 1, image feature point 2} and {image feature point 1, image feature point 3}, and the first predicted distance between image feature point 1 and image feature point 2 is d1, and the second predicted distance between image feature point 1 and image feature point 3 is d2. The base station first searches for an actual distance close to the first predicted distance d1 in the actual distances between each device feature point pair on the AR device, and determines that the actual distance D1 between device feature point 1 and device feature point 2 and the first predicted distance d1 satisfy the condition |D1-d1|<δ, where |D1-d1| is the difference between the actual distance D1 and the first predicted distance d1, and δ is the second preset threshold.
[0115] In the embodiment of the present application, under the premise that the second preset threshold value is less than the first preset threshold value, it is often possible to determine that the difference between an actual distance and the first predicted distance is less than the second preset threshold value.
[0116] If the second preset threshold is less than or equal to the first preset threshold, and the difference between the two is not large, then there may be one or more actual distances whose difference with the first predicted distance is smaller than the second preset threshold.
[0117] In the above, if there are multiple actual distances whose difference with the first predicted distance is less than the second preset threshold, the device feature point pair corresponding to the actual distance with the smallest difference can be selected as the device feature point pair corresponding to the target image feature point pair {image feature point 1, image feature point 2}.
[0118] For example, the difference between the actual distance between device feature point 1 and device feature point 2 and the first predicted distance is less than the second preset threshold, which is 0.05 cm; the difference between the actual distance between device feature point 5 and device feature point 6 and the first predicted distance is less than the second preset threshold, which is 0.1 cm; then {device feature point 1, device feature point 2} is selected as the device feature point pair corresponding to the target image feature point pair {image feature point 1, image feature point 2}.
[0119] Similarly, the base station searches for at least one actual distance close to the second predicted distance d2 among the actual distances between each device feature point pair on the AR device. These actual distances all satisfy the condition that the difference between them and the second predicted distance is less than a second preset threshold. The device feature point pairs corresponding to each of these actual distances are other device feature point pairs, and each other device feature point pair must contain device feature point 1 or device feature point 2. Finally, the base station needs to determine the second device feature point pair corresponding to the second target image feature point pair from the at least one other device feature point pair.
[0120] S703: The base station determines the target device feature points corresponding to each target image feature point from at least two device feature point pairs.
[0121] Specifically, the base station uses the same device feature point in the two device feature point pairs as the target device feature point corresponding to the same target image feature point in the corresponding two target image feature point pairs; and uses the device feature points other than the same device feature point in the first device feature point pair as the target device feature points corresponding to the image feature points other than the same image feature point in the first target image feature point pair; and uses the device feature points other than the same device feature point in the second device feature point pair as the target device feature points corresponding to the image feature points other than the same image feature point in the second target image feature point pair.
[0122] That is, the base station needs to find a second device feature point pair from at least one other device feature point pair to serve as the device feature point pair corresponding to the target image feature point pair {image feature point 1, image feature point 3}. For example, if the base station selects {device feature point 1, device feature point 3} as the device feature point pair corresponding to {image feature point 1, image feature point 3}, the actual distance D2 between device feature point 1 and device feature point 3 and the second predicted distance d2 must also satisfy the condition |D2-d2|<δ.
[0123] Afterwards, since the two determined device feature point pairs both contain device feature point 1, and the two target image feature point pairs both contain image feature point 1, device feature point 1 is used as the target device feature point corresponding to image feature point 1; device feature point 2 in the device feature point pair {device feature point 1, device feature point 2} is used as the target device feature point corresponding to image feature point 2 in the corresponding target image feature point pair {image feature point 1, image feature point 2}; device feature point 3 in the device feature point pair {device feature point 1, device feature point 3} is used as the target device feature point corresponding to image feature point 3 in the corresponding target image feature point pair {image feature point 1, image feature point 3}.
[0124] That is, the first target image feature point pair is set to include the first target image feature point and the second target image feature point, and the second target image feature point pair is set to include the first target image feature point and the third target image feature point; if the same device feature point in the two device feature point pairs is the first device feature point in the first device feature point pair, then the first device feature point is used as the first target device feature point corresponding to the first target image feature point in the AR device; the second device feature point in the first device feature point pair is used as the second target device feature point corresponding to the second target image feature point in the AR device; and the third device feature point in the second device feature point pair is used as the third target device feature point corresponding to the third target image feature point in the AR device.
[0125] In summary, the device feature point 1 in the above example is the first device feature point, the device feature point 2 is the second device feature point, the device feature point 3 is the third device feature point, the image feature point 1 corresponds to the device feature point 1, the image feature point 2 corresponds to the device feature point 2, and the image feature point 3 corresponds to the device feature point 3.
[0126] Figure 8 shows a correspondence diagram between target image feature points and target device feature points, provided in an embodiment of the present application. As shown in Figure 8 , the base station can use the above method to determine that target image feature points 1, 2, and 3 in the captured image correspond to device feature points 1, 2, and 3 on the AR device, respectively.
[0127] For example, the base station selects {device feature point 2, device feature point 4} as the device feature point pair corresponding to {image feature point 1, image feature point 3}, and the actual distance D3 between device feature point 2 and device feature point 4 and the second predicted distance d2 must also satisfy the condition |D3-d2|<δ.
[0128] Finally, the two determined device feature point pairs both contain device feature point 2, and the two target image feature point pairs both contain image feature point 1. Therefore, device feature point 2 is used as the target device feature point corresponding to image feature point 1; device feature point 1 in the device feature point pair {device feature point 1, device feature point 2} is used as the target device feature point corresponding to image feature point 2 in the corresponding target image feature point pair {image feature point 1, image feature point 2}; device feature point 4 in the device feature point pair {device feature point 2, device feature point 4} is used as the target device feature point corresponding to image feature point 3 in the corresponding target image feature point pair {image feature point 1, image feature point 3}.
[0129] That is, the same device feature point in the two device feature point pairs is set as the second device feature point in the first device feature point pair, and the second device feature point is used as the second target device feature point corresponding to the first target image feature point in the AR device; the first device feature point in the first device feature point pair is used as the first target device feature point corresponding to the second target image feature point in the AR device; and the third device feature point in the second device feature point pair is used as the third target device feature point corresponding to the third target image feature point in the AR device.
[0130] In summary, the device feature point 1 in the above example is the first device feature point, the device feature point 2 is the second device feature point, the device feature point 4 is the third device feature point, the image feature point 1 corresponds to the device feature point 2, the image feature point 2 corresponds to the device feature point 1, and the image feature point 3 corresponds to the device feature point 4.
[0131] In addition, in the above process, the base station needs to find one other device feature point pair from at least one other device feature point pair as the second device feature point pair corresponding to the second target image feature point pair. This process can be divided into the following two cases:
[0132] Case 1: If the number of the other device feature point pairs is one, the other device feature point pair is used as the second device feature point pair corresponding to the second target image feature point pair.
[0133] Case 2: If there are multiple other device feature point pairs, one is selected from the multiple other device feature point pairs based on the actual distance relationship between the multiple other device feature point pairs as the second device feature point pair corresponding to the second target image feature point pair.
[0134] For case 1, the base station only needs to directly use this other device feature point pair as the second device feature point pair corresponding to the second target image feature point pair {image feature point 1, image feature point 3}.
[0135] For the second case, since there are multiple other device feature point pairs, these other device feature point pairs may all contain the first device feature point, may all contain the second device feature point, or may partially contain the first device feature point and partially contain the second device feature point.
[0136] Regardless of which of the above situations is true, considering that there can only be one device feature point pair corresponding to the second target image feature point pair, and the smaller the difference between the actual distance corresponding to the device feature point pair and the second predicted distance, the better, therefore, an optional implementation method is: the base station uses the other device feature point pair with the smallest difference between the corresponding actual distance and the second predicted distance as the second device feature point pair corresponding to the second target image feature point pair.
[0137] Afterwards, for example, the base station finds two other device feature point pairs, wherein the first other device feature point pair includes the first device feature point and the third device feature point, and the second other device feature point pair includes the second device feature point and the fourth device feature point.
[0138] If the actual distance corresponding to the first other device feature point pair is smaller than the actual distance corresponding to the second other device feature point pair, the first device feature point is used as the corresponding point of the first target image feature point; the second device feature point of the first device feature point pair previously determined to correspond to the first target image feature point pair is used as the corresponding point of the second target image feature point; the third device feature point of the first other device feature point pair is used as the corresponding point of the third target image feature point.
[0139] If the actual distance corresponding to the second other device feature point pair is smaller than the actual distance corresponding to the first other device feature point pair, the second device feature point is used as the corresponding point of the first target image feature point; the first device feature point in the previously determined first device feature point pair is used as the corresponding point of the second target image feature point; and the fourth device feature point in the second other device feature point pair is used as the corresponding point of the third target image feature point.
[0140] For example, after determining that {device feature point 1, device feature point 2} corresponds to {image feature point 1, image feature point 2}, the base station finds two other device feature point pairs that both satisfy the condition that the difference in distance d2 from {image feature point 1, image feature point 3} is less than a second preset threshold. Assuming that the two other device feature point pairs are {device feature point 1, device feature point 3} and {device feature point 2, device feature point 4}, and the distance corresponding to {device feature point 1, device feature point 3} is D2, and the distance corresponding to {device feature point 2, device feature point 4} is D3, if |D2-d2|<|D3-d2|<δ, then {device feature point 1, device feature point 3} is used as the device feature point pair corresponding to {image feature point 1, image feature point 3}. If |D3-d2|<|D2-d1|<δ, then {device feature point 2, device feature point 4} is used as the device feature point pair corresponding to {image feature point 1, image feature point 3}.
[0141] In addition, there is another way to determine the target device feature point corresponding to the target image feature point. After the base station determines that {device feature point 1, device feature point 2} corresponds to {image feature point 1, image feature point 2}, among the other device feature points on the AR device, find a device feature point whose distance from device feature point 1 and the difference between the second predicted distance are less than the second preset threshold. If there are multiple device feature points, select the one with the smallest difference. For example, it is finally determined that the difference between the distance between device feature point 3 and device feature point 1 and the second predicted distance is the smallest and less than the second preset threshold. Further find a device feature point whose distance from device feature point 2 and the difference between the second predicted distance are less than the second preset threshold. If there are multiple device feature points, select the one with the smallest difference. For example, it is finally determined that the difference between the distance between device feature point 4 and device feature point 2 and the second predicted distance is the smallest and less than the second preset threshold.
[0142] The subsequent process is the same as the above method, that is, comparing the difference between the distance between device feature point 3 and device feature point 1 and the second predicted distance, as well as the difference between the distance between device feature point 4 and device feature point 2 and the second predicted distance, to determine whether the device feature point pair corresponding to {image feature point 1, image feature point 3} is {device feature point 1, device feature point 3}, or {device feature point 2, device feature point 4}.
[0143] This process actually determines the correspondence between {device feature point 1, device feature point 2} and {image feature point 1, image feature point 2}, and then further assumes whether device feature point 1 corresponds to image feature point 1 or device feature point 1 corresponds to image feature point 2. Finally, based on the difference between the actual distance corresponding to {device feature point 1, device feature point 3} and {device feature point 2, device feature point 4} and the second predicted distance, the target device feature point corresponding to each target image feature point (i.e., image feature point 1, image feature point 2, image feature point 3) is finally determined.
[0144] It should be noted that the "at least two device feature point pairs containing the same device feature point" described in S702 of this application means that there may be more than two device feature point pairs that meet the condition that the difference between the corresponding actual distance and the predicted distance is less than the second preset threshold. For example, after determining a device feature point pair corresponding to a target image feature point pair, when determining another device feature point pair corresponding to another target image feature point pair, multiple other device feature point pairs that meet the condition that the difference between the corresponding actual distance and the second predicted distance is less than the second preset threshold are found. Each of the other device feature point pairs and the first device feature point pair corresponding to the above-mentioned first target image feature point pair will each contain the same device feature point, but the "same device feature point" among these other device feature points may be the same or different.
[0145] For example, the first device feature point pair is {device feature point 1, device feature point 2}. It is possible that all other device feature point pairs include device feature point 1 in the first device feature point pair; it is also possible that all other device feature point pairs include device feature point 2 in the first device feature point pair; it is also possible that some other device feature point pairs include device feature point 1 in the first device feature point pair, and another part of other device feature point pairs include device feature point 2 in the first device feature point pair.
[0146] Following the assumptions in S202, FIG9 is a logic diagram for determining device feature points corresponding to target image feature points provided by an embodiment of the present application. As shown in FIG9 , after the base station calculates the predicted distance x1 between a1 and a2 and the predicted distance x2 between a1 and a3, it determines a device feature point pair {P1, P2} based on the predicted distance x1 between a1 and a2 and the distance between each device feature point pair. Compared with the actual distance between any other two device feature points, the actual distance y1 of the device feature points P1 and P2 on the AR device is closest to x1, and satisfies |x1-y1|<δ, where δ is the second preset threshold. Then the device feature point pair {P1, P2} corresponds to the target image feature point pair {a1, a2}.
[0147] The base station then determines a device feature point P3 whose distance to device feature point P1 is closest to x2, for example, the difference between the distances y2 and x2 between P1 and P3 is less than a second preset threshold. The base station also determines a device feature point P4 whose distance to device feature point P2 is closest to x2, for example, the difference between the distances y3 and x2 between P2 and P4 is also less than a second preset threshold. The base station compares the difference between y2 and x2 and the difference between y3 and x2 and determines that the difference between y2 and x2 is smaller. Therefore, the device feature point pair {P1, P3} corresponds to the target image feature point pair {a1, a3}.
[0148] Further, the base station determines that the target image feature point a1 corresponds to the device feature point P1, the target image feature point a2 corresponds to the device feature point P2, and the target image feature point a3 corresponds to the device feature point P3.
[0149] S204: The base station determines the posture information of the AR device based on the position information of each target device feature point and each target image feature point.
[0150] Based on the position information of each target device feature point in the AR device coordinate system and the position information of each target image feature point in the base station coordinate system, the relative position information of the AR device relative to the base station is determined; and the relative position information is used as the posture information of the AR device.
[0151] After determining the target device feature points corresponding to each target image feature point, the base station can obtain the 3D coordinates of each target device feature point in the AR device coordinate system. Based on the 3D coordinates of each target image feature point in the base station coordinate system, the base station can determine the 3D pose T0 of the AR device coordinate system relative to the base station coordinate system. The 3D pose T0 is also the pose information of the AR device, and its specific form is:
[0152] Where R is the rotation matrix, which reflects the angle between the AR device coordinate system and the base station coordinate system. L is the translation matrix, which reflects the distance between the AR device coordinate system and the base station coordinate system. The rotation matrix must meet the following conditions: R'R=I,|R|=1
[0153] Where I is the identity matrix.
[0154] Obviously, the posture information T0 of the AR device is 6-DOF posture data.
[0155] After this, if each acquired image contains more than three image feature points, then T0 can be adjusted using candidate image feature points other than the target image feature points. Nonlinear optimization is performed using T0 as the initial pose information to obtain the final pose information T1.
[0156] An optional implementation is: the base station predicts the reference position information of each of the at least one candidate device feature points in the base station coordinate system based on the position information of each of the at least one candidate device feature points other than the target device feature points in the AR device coordinate system, as well as the posture information; and predicts the predicted position information of each of the at least one candidate device feature points in the AR device coordinate system based on the reference position information, the calibration parameters used to represent the positional relationship between the various acquisition devices, and the posture information; finally, the base station adjusts the posture information based on the difference between the position information of each of the at least one candidate device feature points in the AR device coordinate system and the corresponding predicted position information.
[0157] In the above, for each candidate device feature point on the AR device except the target device feature point, the base station can reversely calculate the coordinates of the candidate device feature point in the base station coordinate system (i.e., reference position information) based on T0 and the coordinates of the candidate device feature point in the AR device coordinate system, and then predict the coordinates of the candidate device feature point in the AR device coordinate system (i.e., predicted position information) based on the reference position information, the calibration parameters of the infrared binocular camera and T0.
[0158] An optional way to obtain predicted position information is: the base station predicts the feature point position of at least one candidate device feature point in the captured image based on the reference position information and the calibration parameters used to represent the positional relationship between the acquisition devices; for each feature point position, the image feature point whose distance to the feature point position is within a preset range is used as the candidate image feature point corresponding to the candidate device feature point; based on the position information and posture information of at least one candidate image feature point in the base station coordinate system, the predicted position information of the corresponding candidate device feature point in the AR device coordinate system is predicted.
[0159] In S202, the base station calculates the three-dimensional coordinates of the three target image feature points in the base station coordinate system based on the calibration parameters and the distances between at least two pairs of target image feature points in each captured image. Now, the calculation process can be applied in reverse. That is, for each candidate device feature point, the position of the candidate device feature point in the captured image can be calculated based on its reference position information and calibration parameters. This is equivalent to estimating the approximate position of the corresponding candidate image feature point in the captured image if the candidate device feature point is captured in the captured image. The purpose of this step is to subsequently find the candidate image feature point corresponding to the candidate device feature point in the captured image.
[0160] Afterwards, the base station projects the estimated feature point position into the captured image, searches for the image feature point closest to the feature point position, and determines it as the candidate image feature point corresponding to the candidate device feature point. The actual position of the candidate image feature point on the captured image should satisfy the condition that the distance between it and the feature point position is less than 5pix.
[0161] Afterwards, the base station determines to predict the predicted position information of the corresponding candidate device feature point in the AR device coordinate system based on the coordinates and posture information of the candidate image feature point in the base station coordinate system.
[0162] The base station can determine a final posture information T1 based on the differences between each predicted position information and the actual coordinates of the corresponding candidate device feature point in the AR device coordinate system. The posture information T1 can minimize the sum of the differences between the predicted position information of each candidate device feature point and the actual coordinates of the candidate device feature point in the AR device coordinate system, as well as the difference between the predicted position information of the target device feature point and the actual coordinates of the target device feature point in the AR device coordinate system. Specifically, the following formula can be used:
[0163] in, And R1'R1=I, |R1|=1.
[0164] The above formula is calculated using the least quadratic optimization algorithm, R1 is the rotation matrix, L1 is the translation matrix, P i is the real coordinate of the device feature point i in the AR device coordinate system, p i T1p is the coordinate of the image feature point i corresponding to the device feature point i in the base station coordinate system, i is the predicted location information of device feature point i. Device feature point i is a candidate device feature point or a target device feature point.
[0165] In addition, it can be seen that in the above formula, when the posture information is T0, since T0 is calculated based on the coordinates of the target image feature point corresponding to the target device feature point in the base station coordinate system and the real coordinates of the target device feature point under the AR device, the difference between the predicted position information of the target device feature point and the real coordinates of the target device feature point under the AR device is 0.
[0166] Following the assumption in S203, the base station determines that the target image feature point a1 corresponds to the device feature point P1, the target image feature point a2 corresponds to the device feature point P2, and the target image feature point a3 corresponds to the device feature point P3. Then, based on the coordinates of the target image feature points a1, a2, and a3 in the base station coordinate system and the coordinates of the device feature points P1, P2, and P3 in the AR device coordinate system, the AR device posture information T0 is calculated.
[0167] In addition, the base station further finds candidate image feature points a5 and a6 corresponding to candidate device feature points P5 and P6 in addition to device feature points P1, P2, and P3. Candidate image feature points a5 and a6 are visible in the captured image. Based on the coordinates of candidate image feature points a5 and a6 in the base station coordinate system and the calibration parameters of the infrared binocular camera, the predicted position information of candidate device feature points P5 and P6 corresponding to candidate image feature points a5 and a6 predicted by the base station in the AR device coordinate system is obtained. Each predicted position information is compared with the actual coordinates of the corresponding candidate device feature point in the AR device coordinate system to adjust T0, and finally obtain posture information T1. Posture information T1 can minimize the sum of the differences between the predicted position information of candidate device feature points P5 and P6 and the actual coordinates of candidate device feature points P5 and P6 in the AR device coordinate system, as well as the differences between the predicted position information of device feature points P1, P2, and P3 and the actual coordinates of device feature points P1, P2, and P3 in the AR device coordinate system.
[0168] As shown in Figure 10, it is an interaction diagram of an AR device, a base station and a terminal device provided in an embodiment of the present application. After obtaining the posture information T1, the base station can timestamp the 6-degree-of-freedom posture information and send the posture information T1 with the timestamp to the terminal device via WIFI. The AR device itself can also measure its own inertial measurement unit (IMU) data and send the IMU data with the timestamp to the terminal device. The terminal device renders the image based on the posture information T1 and the IMU data, and sends the rendered image to the AR device. In addition, the terminal device also needs to maintain time synchronization with the base station, and the base station also maintains time synchronization with the AR device.
[0169] The IMU mentioned above can be a 6-axis sensor, which may include a 3-axis gyroscope and a 3-axis accelerometer. The gyroscope measures the angular velocity (rotational degree of freedom) on each axis, that is, the degree of rotation per second along this direction of rotation; the accelerometer measures the acceleration (motional degree of freedom) on each axis. IMU data can be used to predict the AR device's posture at the next moment based on the device's current movement speed, acceleration, and other conditions.
[0170] In addition, the AR device can use the Bluetooth Low Energy (BLE) protocol to send IMU data to the terminal device to further reduce the power consumption of communication.
[0171] The specific process of terminal device rendering can be: fusing the posture information uploaded by the base station and the IMU data uploaded by the AR device, and estimating the rendering time and image transmission time. Before the next moment arrives, based on the current moment, the estimated rendering time and image transmission time, the virtual picture of the next moment is forward predicted, the correct three-dimensional posture is rendered, and the rendering result is sent to the AR device via WIFI for display.
[0172] Based on the method proposed in this application, the computing power and power consumption requirements of AR devices can be greatly reduced. Usually, the power consumption of the posture detection solution from the inside to the outside of the head-mounted AR device is more than 1.5W, while this application only requires the power consumption brought by WIFI transmission. After optimization, the average power consumption is about 200-500mW. In addition, the AR device in this application can be a lightweight AR glasses, which does not require additional electronic components and circuits, simplifying the system design.
[0173] Refer to Figure 11, which is a flowchart of a specific implementation of another method for detecting the posture of an AR device provided in an embodiment of the present application. The method may include the following specific implementation process:
[0174] S1101: The base station transmits a detection light, such as an illumination light, to the AR device and captures at least two captured images.
[0175] S1102: The base station determines whether three target image feature points can be obtained in the captured image. If so, execute S1103; otherwise, execute S1104.
[0176] S1103: The base station determines the coordinates p1, p2, and p3 of the three target image feature points a1, a2, and a3 in the base station coordinate system.
[0177] S1104: Posture detection failed.
[0178] S1105: The base station obtains two target image feature point pairs {a1, a2} and {a1, a3} based on the three target image feature points, and calculates the predicted distances d1 = |p1p2| and d2 = |p1p3| of the two target image feature point pairs.
[0179] S1106: The base station determines two device feature points P1 and P2 from among the device feature points, such that the distance between P1 and P2 is closest to d1.
[0180] S1107: The base station determines a device feature point P3 whose distance to P1 is closest to d2.
[0181] S1108: The base station determines a device feature point P4 whose distance to P2 is closest to d2.
[0182] In specific implementation, the difference between the distance between P1 and P3 in S1107 and d2 may be smaller than the second preset threshold, and the difference between the distance between P2 and P4 in S1108 and d2 may be smaller than the second preset threshold.
[0183] If both S1107 and S1108 cannot find a device feature point pair that satisfies the condition that the difference with d2 is less than the second preset threshold, positioning fails. If only one of S1107 and S1108 successfully finds a device feature point pair that satisfies the condition that the difference with d2 is less than the second preset threshold, the device feature point pair is directly determined to be the device feature point pair corresponding to the target image feature point pair {a1, a3}.
[0184] S1109: The base station compares ||P1P3|-d2| with ||P2P4|-d2|. If ||P1P3|-d2| is smaller than ||P2P4|-d2|, execute S1110; otherwise, execute S1111.
[0185] S1110: The base station determines that the target image feature points a1, a2, and a3 correspond to the target device feature points P1, P2, and P3, and obtains the posture information of the AR device based on the coordinates of P1, P2, and P3 in the AR device coordinate system and the coordinates of a1, a2, and a3 in the base station coordinate system.
[0186] S1111: The base station determines that the target image feature points a1, a2, and a3 correspond to the target device feature points P2, P1, and P4, and obtains the posture information of the AR device based on the coordinates of P2, P1, and P4 in the AR device coordinate system and the coordinates of a1, a2, and a3 in the base station coordinate system.
[0187] It should be noted that the flowcharts listed above are just simple examples. The order of the relevant S1107 and S1108 can be swapped or performed simultaneously. This application does not limit its specific implementation method.
[0188] FIG12 is an interaction logic diagram of an AR device, a base station, and a terminal device provided in an embodiment of the present application. As shown in FIG12 , the base station transmits light to the AR device, and the device feature points on the AR device reflect the light. At the same time, the base station shoots the device feature points to obtain multiple captured images. Afterwards, the base station determines the predicted distances of at least two target image feature points containing the same target image feature point, and determines the target device feature points corresponding to each target image feature point based on the predicted distance and the actual distance between each device feature point; finally, the base station determines the posture information of the AR device based on the position information of the target image feature point and the position information of the corresponding target device feature point, and sends the posture information to the terminal device. The AR device also obtains its own inertial measurement unit data and sends it to the terminal device. The terminal device renders a virtual screen based on the posture information and inertial measurement unit data of the AR device, and transmits the rendered screen to the AR device for display after the rendering is completed.
[0189] Based on the same inventive concept, an embodiment of the present application further provides a posture detection device for an AR device. FIG13 is a schematic structural diagram of the posture detection device for an AR device. As shown in FIG13 , the device may include:
[0190] An acquisition unit 1301 is configured to acquire at least two captured images based on different shooting directions by transmitting a detection light to the AR device; each captured image includes image feature points obtained by reflecting the detection light based on device feature points on the AR device;
[0191] A prediction unit 1302 is configured to determine, based on at least two acquired images, respective predicted distances of at least two pairs of target image feature points containing the same target image feature point;
[0192] A comparing unit 1303 is configured to determine a target device feature point corresponding to each target image feature point on the AR device based on a comparison result between the actual distance between each device feature point on the AR device and each predicted distance;
[0193] The determining unit 1304 is configured to determine the posture information of the AR device based on the position information of each target device feature point and each target image feature point.
[0194] Optionally, each captured image is captured by a capture device located at a shooting orientation. The prediction unit 1302 is specifically configured to:
[0195] Based on the distances of at least two pairs of target image feature points in each acquired image and calibration parameters for representing the positional relationship between each acquisition device, the predicted distances of the at least two pairs of target image feature points are determined.
[0196] Optionally, the actual distances between any two device feature points are different, and the difference between any two actual distances is greater than a first preset threshold.
[0197] Optionally, the comparing unit 1303 is specifically configured to:
[0198] Compare the actual distance between each device feature point on the AR device with each predicted distance;
[0199] Based on the comparison results, at least two device feature point pairs containing the same device feature point are determined, wherein the difference between the actual distance and the corresponding predicted distance between each device feature point pair is less than a second preset threshold; from the at least two device feature point pairs, the target device feature point corresponding to each target image feature point is determined; wherein the second preset threshold is not greater than the first preset threshold.
[0200] Optionally, each predicted distance includes: a first predicted distance of a first target image feature point pair among the at least two target image feature point pairs, and a second predicted distance of a second target image feature point pair;
[0201] The comparison unit 1303 is specifically used for:
[0202] Two device feature points whose corresponding actual distances have a difference with the first predicted distance that is less than a second preset threshold are used as a first device feature point pair corresponding to the first target image feature point pair;
[0203] Searching for at least one other device feature point pair whose difference between the corresponding actual distance and the second predicted distance is less than a second preset threshold and contains the same device feature point as the first device feature point pair;
[0204] Target device feature points corresponding to respective target image feature points are determined from the device feature points respectively included in one device feature point pair and at least one other device feature point pair.
[0205] Optionally, the comparing unit 1303 is specifically configured to:
[0206] Determining, from the at least one other device feature point pair, a second device feature point pair corresponding to the second target image feature point pair;
[0207] The same device feature point is used as the target device feature point corresponding to the same target image feature point;
[0208] The device feature points other than the same device feature points in the first device feature point pair are used as target device feature points corresponding to the image feature points other than the same image feature points in the first target image feature point pair;
[0209] The device feature points other than the same device feature point in the second device feature point pair are used as target device feature points corresponding to the image feature points other than the same image feature point in the second target image feature point pair.
[0210] Optionally, the comparing unit 1303 is specifically configured to:
[0211] If the number of the other device feature point pairs is one, the other device feature point pair is used as the second device feature point pair corresponding to the second target image feature point pair;
[0212] If there are multiple other device feature point pairs, one is selected from the multiple other device feature point pairs based on the size relationship of the actual distances between the multiple other device feature point pairs and is used as the second device feature point pair corresponding to the second target image feature point pair.
[0213] Optionally, the comparing unit 1303 is specifically configured to:
[0214] The other device feature point pair with the smallest difference between the corresponding actual distance and the second predicted distance is used as the second device feature point pair corresponding to the second target image feature point pair.
[0215] Optionally, the determining unit 1304 is specifically configured to:
[0216] Determine the relative position of the AR device relative to the base station based on the position information of each target device feature point in the AR device coordinate system and the position information of each target image feature point in the base station coordinate system.
[0217] The relative position information is used as the posture information of the AR device.
[0218] Optionally, each captured image is captured by a capture device located at one shooting position; the apparatus further includes:
[0219] An adjustment unit 1305 is configured to predict reference position information of each of the at least one candidate device feature points in the base station coordinate system based on the position information and posture information of each of the at least one candidate device feature points other than each target device feature point in the AR device coordinate system.
[0220] Predicting predicted position information of each of at least one candidate device feature point in the AR device coordinate system based on the reference position information, the calibration parameters used to represent the positional relationship between the at least two acquisition devices, and the posture information;
[0221] The posture information is adjusted based on a difference between position information of each of the at least one candidate device feature points in the AR device coordinate system and the corresponding predicted position information.
[0222] Optionally, the adjusting unit 1305 is specifically configured to:
[0223] Predicting a feature point position of each of at least one candidate device feature points in the captured image based on the reference position information and a calibration parameter used to represent a positional relationship between at least two capture devices;
[0224] For each feature point position, an image feature point whose distance to the feature point position is within a preset range is used as a candidate image feature point corresponding to the candidate device feature point;
[0225] Based on the position information and posture information of at least one candidate image feature point in the base station coordinate system, the predicted position information of the corresponding candidate device feature point in the AR device coordinate system is predicted.
[0226] Optionally, each device feature point on the AR device is made of a material that reflects infrared light.
[0227] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.
[0228] After introducing the posture detection method and apparatus of an AR device according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.
[0229] Those skilled in the art will appreciate that various aspects of the present application can be implemented as systems, methods, or program products. Therefore, various aspects of the present application can be specifically implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."
[0230] Based on the same inventive concept as the above-mentioned method embodiment, an electronic device is also provided in an embodiment of the present application. In one embodiment, the electronic device may be the terminal device 130 shown in Figure 1 . In this embodiment, the structure of the electronic device may be as shown in Figure 14 , including: a communication component 1410, a memory 1420, a display unit 1430, a camera 1440, a sensor 1450, an audio circuit 1460, a Bluetooth module 1470, a processor 1480, and other components.
[0231] The communication component 1410 is used to communicate with the server. In some embodiments, it may include a wireless fidelity (WiFi) module. The WiFi module is a short-range wireless transmission technology. Electronic devices can help objects (such as users) send and receive information through the WiFi module.
[0232] The memory 1420 can be used to store software programs and data. The processor 1480 executes various functions and data processing of the terminal device 130 by running the software programs or data stored in the memory 1420. The memory 1420 stores the operating system that enables the terminal device 130 to operate. In this application, the memory 1420 can store the operating system and various application programs, and can also store a computer program that executes the posture detection method of the AR device in the embodiment of the application.
[0233] The display unit 1430 may also be used to display information input by or provided to an object, as well as a graphical user interface (GUI) of various menus of the terminal device 130. Specifically, the display unit 1430 may include a display screen 1432 disposed on the front of the terminal device 130. The display unit 1430 may be used to display the rendering interface, etc., in the embodiments of the present application.
[0234] The display unit 1430 can also be used to receive input digital or character information and generate signal input related to the object setting and function control of the terminal device 130. Specifically, the display unit 1430 may include a touch screen 1431 set on the front of the terminal device 130, which can collect touch operations of objects on or near it, such as clicking a button, dragging a scroll box, etc.
[0235] The touch screen 1431 can be covered on the display screen 1432, or the touch screen 1431 and the display screen 1432 can be integrated to realize the input and output functions of the terminal device 130. The integrated display screen can be simply called a touch screen. In this application, the display unit 1430 can display applications and corresponding operation steps.
[0236] Camera 1440 can be used to capture still images. Subjects can publish images captured by camera 1440 through applications. Camera 1440 can be one or more. The subject's optical image is projected onto a photosensitive element through a lens. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to processor 1480 for conversion into a digital image signal.
[0237] The terminal device may further include at least one sensor 1450, such as an acceleration sensor 1451, a distance sensor 1452, a fingerprint sensor 1453, and a temperature sensor 1454. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.
[0238] The audio circuit 1460 , the speaker 1461 , and the microphone 1462 may provide an audio interface between the object and the terminal device 130 .
[0239] The Bluetooth module 1470 is used to exchange information with other Bluetooth devices having a Bluetooth module through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1470 to exchange data.
[0240] The processor 1480 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1420 and calling data stored in the memory 1420. In some embodiments, the processor 1480 may include one or more processing units; the processor 1480 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, object interface and application programs, etc., and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1480. In this application, the processor 1480 can run the operating system, application programs, object interface display and touch response, as well as the posture detection method of the AR device in the embodiment of the present application. In addition, the processor 1480 is coupled to the display unit 1430.
[0241] In some possible implementations, various aspects of the posture detection method for an AR device provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is executed by a processor, the steps of the posture detection method for an AR device according to various exemplary implementations of the present application described above in this specification are implemented. For example, the electronic device may execute the steps shown in Figure 2.
[0242] As used herein, a readable storage medium may be any tangible medium that contains or stores a program that can be used by or in combination with a command execution system, device, or component. A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in combination with a command execution system, device, or component.
[0243] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0244] The computer program for performing the operations of the present application can be written in any combination of one or more programming languages. The computer program can be executed entirely on the subject electronic device, partially on the subject electronic device, as a stand-alone software package, partially on the subject electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the subject electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).
[0245] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.
[0246] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0247] The present application is described with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program commands. These computer program commands can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the command executed by the processor of the computer or other programmable data processing device produces a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.
[0248] These computer program commands may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer-readable memory produce a manufactured product including a command device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0249] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0250] It should be noted that the terms "first," "second," "third," "fourth," "1," "2," etc. (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.
[0251] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.
Claims
1. A posture detection method for an augmented reality (AR) device, characterized in that: The method comprises: By emitting a detection light to the AR device, at least two captured images based on different shooting directions are acquired; each captured image includes: image feature points obtained by reflecting the detection light based on device feature points on the AR device; Determine, based on the at least two acquired images, respective predicted distances of at least two pairs of target image feature points that include the same target image feature point; Determine, based on a comparison result between an actual distance between each device feature point on the AR device and each predicted distance, a target device feature point corresponding to each target image feature point in the AR device; Based on the position information of each target device feature point and each target image feature point, the posture information of the AR device is determined.
2. The method according to claim 1, characterized in that Each captured image is captured based on a capture device located at one of the shooting positions; The step of determining respective predicted distances of at least two pairs of target image feature points including the same target image feature point based on the at least two acquired images comprises: Based on the distances of the at least two target image feature point pairs in each acquired image and the calibration parameters used to represent the positional relationship between the at least two acquisition devices, the predicted distances of the at least two target image feature point pairs are determined.
3. The method according to claim 1, characterized in that The actual distances between any two device feature points are different, and the difference between any two actual distances is greater than a first preset threshold.
4. The method according to any one of claims 1 to 3, characterized in that: The determining, based on the comparison result between the actual distance between each device feature point on the AR device and each predicted distance, the target device feature point corresponding to each target image feature point in the AR device, includes: Comparing the actual distances between the device feature point pairs on the AR device with the predicted distances respectively; Based on the comparison result, determining at least two device feature point pairs containing the same device feature point, wherein the difference between the actual distance between each device feature point pair and the corresponding predicted distance is less than a second preset threshold; wherein the second preset threshold is not greater than the first preset threshold; From the at least two device feature point pairs, target device feature points corresponding to respective target image feature points are determined.
5. The method according to claim 4, characterized in that There are two target image feature point pairs, and each predicted distance includes: a first predicted distance of a first target image feature point pair, and a second predicted distance of a second target image feature point pair; The determining of at least two device feature point pairs containing the same device feature point includes: Using two device feature points whose corresponding actual distances have a difference with the first predicted distance that is less than the second preset threshold as the first device feature point pair corresponding to the first target image feature point pair; Search for at least one other device feature point pair whose difference between the corresponding actual distance and the second predicted distance is less than the second preset threshold and contains the same device feature point as the first device feature point pair.
6. The method according to claim 5, characterized in that The step of determining, from at least two pairs of device feature points, target device feature points corresponding to the respective target image feature points comprises: Determine, from at least one other device feature point pair, a second device feature point pair corresponding to the second target image feature point pair; Using the same device feature point as the target device feature point corresponding to the same target image feature point; Using the device feature points other than the same device feature point in the first device feature point pair as target device feature points corresponding to the image feature points other than the same image feature point in the first target image feature point pair; The device feature points other than the same device feature point in the second device feature point pair are used as target device feature points corresponding to the image feature points other than the same image feature point in the second target image feature point pair.
7. The method according to claim 6, characterized in that The determining, from at least one other device feature point pair, a second device feature point pair corresponding to the second target image feature point pair comprises: If the number of the other device feature point pairs is one, use the other device feature point pair as the second device feature point pair corresponding to the second target image feature point pair; If there are multiple other device feature point pairs, then the actual distances of the multiple other device feature point pairs are calculated. relationship, selecting one from the multiple other device feature point pairs and using it as the second device feature point pair corresponding to the second target image feature point pair.
8. The method according to claim 7, characterized in that According to the size relationship of the actual distances of the plurality of other device feature point pairs, selecting one from the plurality of other device feature point pairs as the second device feature point pair corresponding to the second target image feature point pair, comprises: The other device feature point pair with the smallest difference between the corresponding actual distance and the second predicted distance is used as the second device feature point pair corresponding to the second target image feature point pair.
9. The method according to any one of claims 1 to 3, characterized in that: The determining the posture information of the AR device based on the position information of each target device feature point and each target image feature point includes: Determine relative position information of the AR device with respect to the base station based on position information of each target device feature point in the AR device coordinate system and position information of each target image feature point in the base station coordinate system; The relative position information is used as the posture information of the AR device.
10. The method according to any one of claims 1 to 3, characterized in that: Each captured image is captured based on a capture device located at one of the shooting positions; the method further includes: Based on at least one candidate device feature point other than the target device feature points, respective position information in the AR device coordinate system, and the posture information, predicting reference position information of each of the at least one candidate device feature point in the base station coordinate system; Based on the reference position information, the calibration parameters used to represent the positional relationship between the at least two acquisition devices, and the posture information, predict the predicted position information of each of the at least one candidate device feature points in the AR device coordinate system; The posture information is adjusted based on the difference between the position information of each of the at least one candidate device feature points in the AR device coordinate system and the corresponding predicted position information.
11. The method according to claim 10, characterized in that The predicting, based on the reference position information, the calibration parameters for representing the positional relationship between the at least two acquisition devices, and the posture information, the predicted position information of each of the at least one candidate device feature points in the AR device coordinate system includes: Based on the reference position information and the calibration parameters, predicting the feature point position of each of the at least one candidate device feature points in the acquired image; For each feature point position, an image feature point whose distance to the feature point position is within a preset range is used as a candidate image feature point corresponding to the candidate device feature point; Based on the position information of at least one candidate image feature point in the base station coordinate system and the posture information, the predicted position information of the corresponding candidate device feature point in the AR device coordinate system is predicted.
12. The method according to any one of claims 1 to 3, characterized in that: Each device feature point on the AR device is made of a material that reflects infrared light.
13. A posture detection device for an AR device, characterized in that: include: An acquisition unit, configured to acquire at least two captured images based on different shooting directions by emitting a detection light beam to the AR device; Each collected image includes: image feature points obtained by reflecting the detection light based on device feature points on the AR device; A prediction unit, configured to determine, based on the at least two acquired images, respective predicted distances of at least two pairs of target image feature points containing the same target image feature point; A comparison unit, configured to determine, based on a comparison result between an actual distance between each device feature point on the AR device and each predicted distance, a target device feature point corresponding to each target image feature point in the AR device; The determination unit is used to determine the posture information of the AR device based on the position information of each target device feature point and each target image feature point.
14. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of any method described in claims 1 to 12 are implemented.
15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.
16. A computer program product, comprising a computer program; characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.
Citation Information
Patent Citations
Target posture measuring method based on binocular vision under double mediums
CN102607526A
Depth information determination method and device, terminal and storage medium
CN113689484A
Position measurement apparatus for measuring position of object having reflective surface in the three-dimensional space
US20170221224A1
Method, system and apparatus for determining a pose for an object
US20180374237A1
Posture Estimation System, Posture Estimation Device and Range Image Camera
US20190102613A1
