Low-power visual tracking system

Abstract

Systems, methods, and non-transient media for low-power visual tracking systems are provided. An example method may include: receiving one or more images captured by each image sensor system of a first device's image sensor system set, the one or more images capturing a pattern set on a second device, wherein the first device has lower power requirements compared to the second device, and the pattern set has a predetermined configuration on the second device; determining a set of pixels corresponding to the pattern set on the second device based on the one or more images captured by each image sensor system; determining the position and relative pose of each pattern in space based on the pixel set corresponding to the pattern set; and determining the pose of the first device relative to the second device based on the position and relative pose of each pattern.

Description

Technical Field

[0001] This disclosure generally relates to visual tracking systems for extended reality and other applications.

[0002] background

[0003] Pose estimation can be used in a variety of applications, such as extended reality (e.g., virtual reality, augmented reality, mixed reality, etc.), computer vision, and robotics, to determine the position and orientation of objects and / or devices relative to targets (such as scenes, humans, other objects, and / or other devices). Pose information can be used to manage interactions between objects / devices and scenes, humans, other objects, and / or other devices. For example, a robot's pose (e.g., position and orientation) can be used to allow the robot to manipulate objects or avoid collisions with objects while moving around in a scene. As another example, the relative pose of a user-worn device to its controller can be used to provide an extended reality experience to the user, where the pose and movement of the controller are tracked and rendered on the user-worn device. However, the computational complexity of pose estimation systems can impose significant power and resource requirements and can be a limiting factor in various applications. The computational complexity of pose estimation can also limit the performance and scalability of tracking and localization applications that rely on pose information.

[0004] Brief Overview

[0005] Systems, methods, and computer-readable media for low-power visual tracking for extended reality and other applications are disclosed. According to at least one example, a method for low-power visual tracking for extended reality and other applications is provided. The method may include: receiving one or more images captured by at least one image sensor system on a first device, the one or more images capturing a pattern set having a predetermined configuration on a second device, wherein the first device has lower power requirements compared to the second device; determining a set of pixels corresponding to the pattern set on the second device based on the one or more images captured by the at least one image sensor system; determining the position and relative pose in space of each pattern from the pattern set based on the set of pixels corresponding to the pattern set on the second device; and determining the pose of the first device relative to the second device based on the position and relative pose in space of each pattern from the pattern set.

[0006] According to at least one example, an apparatus for low-power visual tracking for extended reality and other applications is provided. The apparatus may include: a memory; and one or more processors coupled to the memory, the processors being configured to: receive one or more images captured by at least one image sensor system on the apparatus, the images capturing a pattern set having a predetermined configuration on a device, wherein the apparatus has lower power requirements compared to the device; determine a set of pixels corresponding to the pattern set on the device based on the one or more images captured by the at least one image sensor system; determine the position and relative pose in space of each pattern from the pattern set based on the set of pixels corresponding to the pattern set on the device; and determine the pose of the apparatus relative to the device based on the position and relative pose in space of each pattern from the pattern set.

[0007] According to at least one example, a non-transient computer-readable medium is provided for low-power visual tracking for extended reality and other applications. The non-transient computer-readable medium may include instructions stored thereon that, when executed by one or more processors, cause the processors to: receive one or more images captured by at least one image sensor system on a first device, the images capturing a pattern set with a predetermined configuration on a second device, wherein the first device has lower power requirements than the second device; determine a set of pixels corresponding to the pattern set on the second device based on the one or more images captured by the at least one image sensor system; determine the position and relative pose in space of each pattern from the pattern set based on the set of pixels corresponding to the pattern set on the second device; and determine the pose of the first device relative to the second device based on the position and relative pose in space of each pattern from the pattern set.

[0008] According to at least one example, an apparatus for low-power visual tracking for extended reality and other applications is provided. The apparatus may include means for: receiving one or more images captured by at least one image sensor system on a first device, the one or more images capturing a pattern set having a predetermined configuration on a second device, wherein the first device has lower power requirements compared to the second device; determining a set of pixels corresponding to the pattern set on the second device based on the one or more images captured by the at least one image sensor system; determining the position and relative pose in space of each pattern from the pattern set based on the pixel set corresponding to the pattern set on the second device; and determining the pose of the first device relative to the second device based on the position and relative pose in space of each pattern from the pattern set.

[0009] In some examples of the methods, apparatus (equipment) and non-transient computer-readable storage media described above, the at least one image sensor system may include an image sensor system set, and determining the orientation of the first device relative to the second device (or the device relative to the device) is further based on a predetermined relative position and orientation of the image sensor system set.

[0010] In some aspects, the methods, apparatus (equipment) and non-transient computer-readable storage media described above may include: determining the three-dimensional (3D) coordinates of one or more points in space associated with a scene captured by an image sensor system assembly; and determining the relative 3D pose of the image sensor system assembly on a first device (or apparatus) based on a predetermined relative position and orientation of the image sensor system assembly and the 3D coordinates of the one or more points in space, wherein the pose of the first device (or apparatus) relative to a second device is further based on the relative 3D pose of the image sensor system assembly on the first device (or apparatus).

[0011] In some examples, the attitude of the first device (or apparatus) relative to the second device includes a six-degree-of-freedom (6DoF) attitude, and the predetermined configuration of the pattern set includes: the relative position of each pattern on the second device, the relative orientation of each pattern on the second device, the shape of each pattern, the size of each pattern, the characteristics of each pattern, and / or the arrangement of the pattern set.

[0012] In some cases, determining the position and relative pose of each pattern from a pattern set in space includes: determining the 3D orientation of the pattern set by rotating the 3D coordinates in space corresponding to the pixel set corresponding to the pattern set, the 3D coordinates being rotated relative to a reference 3D coordinate set; and determining the 3D position of the pattern set by translating the 3D coordinates in space corresponding to the pixel set corresponding to the pattern set, the 3D coordinates being translated relative to the reference 3D coordinate set. In some examples, the relative pose of each pattern may be based on the 3D orientation and 3D position of the pattern set, and the position of each pattern includes a corresponding 3D position from the 3D position of the pattern set.

[0013] In some aspects, determining the attitude of the first device (or apparatus) relative to the second device further includes determining the attitude of the second device relative to the first device (or apparatus), wherein the attitude of the first device (or apparatus) includes a first 6DoF attitude and the attitude of the second device includes a second 6DoF attitude.

[0014] In some examples, the first device (or apparatus) may include a hand controller device and the second device may include a head-mounted display device.

[0015] In some examples, the at least one image sensor system may include a low-power image sensor system, and each pattern in the pattern set is visible in the infrared and / or visible light spectrum.

[0016] In some aspects, at least one pattern in the pattern set includes encoded machine-readable information, which includes: location information associated with the at least one pattern, an identifier associated with the at least one pattern, a unique code, settings, and / or information about a user account associated with an extended reality application in main memory on a first device (or apparatus) and / or a second device.

[0017] In some aspects, determining the pixel set corresponding to the pattern set on the second device includes: detecting each pattern from the pattern set on the second device based on one or more images captured by the at least one image sensor system; and identifying one or more points in each pattern from the pattern set, the one or more points corresponding to one or more pixels from the pixel set, wherein determining the position and relative orientation of each pattern in space is based on the one or more points in each pattern.

[0018] In some aspects, determining the set of pixels corresponding to the set of patterns on the second device includes: detecting each pattern from the set of patterns on the second device based on one or more images captured by the at least one image sensor system; for each pattern, detecting a smaller inner pattern; and identifying one or more points in each smaller inner pattern, the one or more points corresponding to one or more pixels from the set of pixels, wherein determining the position and relative orientation of each pattern in space is based on the one or more points in each smaller inner pattern.

[0019] In some examples, determining the pose of the first device (or apparatus) relative to the second device includes determining the 6DoF pose of the first device (or apparatus) relative to the second device and the 6DoF pose of the second device relative to the first device (or apparatus). In some aspects, the methods, apparatus (equipment), and non-transient computer-readable storage media described above may include: detecting one or more additional patterns on an object captured in one or more additional images captured by the at least one image sensor system; determining one or more pixels in the one or more additional images corresponding to the one or more additional patterns on the object; and determining, based on the one or more pixels, an additional 6DoF pose of the object relative to the first device (or apparatus) and the 6DoF pose of the second device. In some examples, the object may include a wall, a display device, a video game console, furniture, appliances, or household items.

[0020] In some aspects, the aforementioned device (equipment) may include one or more sensors. In some aspects, the aforementioned device (equipment) may include a mobile device. In some examples, the aforementioned device (equipment) may include a hand controller, a mobile phone, a wearable device, a display device, a mobile computer, a head-mounted device, and / or a camera.

[0021] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. This subject matter should be understood in conjunction with the appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0022] The foregoing, as well as other features and embodiments, will become more apparent from the following description, claims and accompanying drawings. Brief description of the attached diagram

[0024] In order to describe the various advantages and features of this disclosure, a more specific description of the above principles will be presented by referring to specific embodiments of the principles illustrated in the accompanying drawings. It should be understood that these drawings depict only exemplary embodiments of this disclosure and should not be considered as limiting its scope. The principles herein are described and explained with additional specificity and detail using the drawings, in which:

[0025] Figure 1 This is a simplified block diagram illustrating an example environment for low-power visual tracking based on some examples of this disclosure;

[0026] Figure 2 Example views of an auxiliary system and a main processing system in an example use case of using an image sensor system on an auxiliary system to detect the attitude of the auxiliary system relative to a main processing system, according to some examples of this disclosure;

[0027] Figure 3A This is an illustration illustrating example use cases for tracking the six degrees of freedom (6DoF) attitude of an auxiliary system and / or a main processing system, according to some examples of this disclosure;

[0028] Figure 3B This is an illustration illustrating another example use case for tracking attitude information of the auxiliary system 150 and / or the main processing system, according to some examples of this disclosure;

[0029] Figure 3C This is an illustration illustrating example use cases for tracking the 6DoF attitude of an auxiliary system and / or a main processing system, according to some examples of this disclosure;

[0030] Figure 3DThese are illustrations illustrating example use cases for tracking marked objects based on benchmark markers, according to some examples of this disclosure;

[0031] Figure 4A -C describes example configurations of benchmark markers based on some examples of this disclosure;

[0032] Figure 5 Examples of reference lines projected between reference markers and used for attitude transitions between a reference pattern and an observed pattern, according to some examples of this disclosure, are explained;

[0033] Figure 6 It is a diagram illustrating an example pose transition between a reference pattern and an observed pattern according to some examples of this disclosure;

[0034] Figure 7 This is a flowchart illustrating an example method for estimating the relative attitude of an auxiliary system and a main processing system, based on some examples of this disclosure; and

[0035] Figure 8 An example computing device architecture based on some examples of this disclosure is explained.

[0036] Detailed description

[0037] Certain aspects and embodiments of this disclosure are provided below. Some of these aspects and embodiments may be applied independently, and some may be combined, as will be apparent to those skilled in the art. Specific details are set forth in the following description for purposes of explanation to provide a thorough understanding of the various embodiments of this application. However, it will be apparent that the embodiments may be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0038] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of the exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing the exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0039] As previously mentioned, the computational complexity of attitude estimation systems can impose significant power and resource requirements and can be a limiting factor in various applications. The computational complexity of attitude estimation can also limit the performance and scalability of tracking and localization applications that rely on attitude information. For example, the computational complexity of visual and inertial tracking, localization, and attitude estimation algorithms can impose high power and resource requirements on devices used in applications involving tracking, localization, and attitude estimation, such as extended reality (e.g., virtual reality, augmented reality, mixed reality, etc.), gaming, and robotics. Such power and resource requirements are exacerbated by the recent trend of implementing such technologies in mobile and wearable devices and making these devices smaller, lighter, and more comfortable (e.g., by reducing the heat emitted by the device) for longer periods of wear.

[0040] To illustrate, extended reality (XR) devices (such as head-mounted devices, e.g., head-mounted displays (HMDs), smart glasses, etc.) generally rely on high-power components to track their pose (e.g., their relative three-dimensional (3D) position and orientation) relative to other items (e.g., other devices, objects, humans, obstacles, etc.). Such tracking can include 6-DoF tracking, which involves tracking translational movements (e.g., forward / backward or surge, up / down or heave, and left / right or sway) and rotational movements along the x, y, and z axes (e.g., pitch, yaw, and turn), and can be computationally intensive. Furthermore, high-power components for 6DoF tracking can include, for example, high-resolution cameras, high-power processors, and bright light-emitting diode (LED) illuminators in the visible and / or infrared (IR) spectrum.

[0041] The higher power consumption and computational intensity of 6DoF tracking can limit the battery life of devices used in 6DoF tracking applications. Furthermore, higher-powered devices used in 6DoF applications (including XR devices such as head-mounted displays) generally consume more power than auxiliary devices (such as hand controllers used in conjunction with higher-powered devices). This is at least in part due to the display and graphics computations typically performed by the higher-powered devices. As a result, the batteries of higher-powered devices often run out faster than the batteries of their auxiliary devices.

[0042] This disclosure describes systems, methods, and computer-readable media for low-power visual tracking in XR and other applications. The low-power visual tracking techniques described herein can reduce battery consumption and improve battery life in devices used in 6DoF tracking applications, such as XR devices. In some examples, the techniques described herein can offload some or all of the 6DoF tracking tasks, typically performed by higher-power devices, to an auxiliary device used with the higher-power device. The auxiliary device can utilize low-power vision / camera and processing components to perform 6DoF tracking tasks at a lower power rate, thereby improving the battery life of the higher-power device, which typically depletes faster than the auxiliary device's battery life.

[0043] For example, in some scenarios, instead of using a higher-power LED assembly on the auxiliary device and a high-resolution camera on the higher-power device to calculate the relative pose of the auxiliary and higher-power devices, the auxiliary device can implement a lower-power camera to perform 6DoF tracking and offload the associated power consumption from the higher-power device. The higher-power device can implement a unique pattern printed on the device, which the lower-power camera on the auxiliary device can use to identify the relative pose of the auxiliary and higher-power devices. The auxiliary device can leverage the lower-power capabilities of the lower-power camera and a hardware-accelerated object detection framework to detect the unique pattern printed on the higher-power device.

[0044] The auxiliary device can utilize a unique pattern on the higher-power device as a reference marker for 6DoF tracking. For example, a lower-power camera on the auxiliary device can detect the pattern and use it as a reference marker to identify the relative attitude of the auxiliary device and the higher-power device. In some cases, the auxiliary device can perform 3D triangulation on a sparse set of reference points detected from the unique pattern on the higher-power device to identify the relative attitude of the auxiliary device and the higher-power device. This allows the auxiliary device to identify the relative attitude of the auxiliary device and the higher-power device at a lower power rate and reduces the power usage of the higher-power device. In some examples, 6DoF tracking can be performed entirely (or almost entirely) on the auxiliary device, thereby limiting the amount of information transmitted between the auxiliary device and the higher-power device and thus reducing transmission costs and bandwidth requirements.

[0045] The low-power visual tracking technique described in this paper can be implemented in a variety of use cases and applications. For example, it can be implemented in XR applications, robotics applications, autonomous system applications, gaming applications, and more. To illustrate, in some examples, low-power visual tracking can be implemented by an autonomous robotic vacuum cleaner to perform path planning and localization based on relative posture calculations; it can be implemented by autonomous vehicles to achieve higher tracking, mapping, and planning performance in real-time (or near real-time); it can be implemented by a game controller connected to a TV-based console; it can be implemented by a hand controller connected to a head-mounted display (HMD), and so on.

[0046] In a non-limiting illustrative example, low-power visual tracking techniques can be implemented in 6DoF or 3DoF XR applications. The term Extended Reality (XR) can encompass Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR), and others. Each of these forms of XR allows users to experience or interact with immersive virtual environments or content. To provide a realistic XR experience, XR technologies generally aim to integrate virtual content with the physical world and often involve matching the relative pose and movement of objects and devices. This can involve computing the relative pose of devices, objects, and / or maps of the real-world environment to match the relative position and movement of devices, objects, and / or the real-world environment, and to convincingly anchor the content to the real-world environment. Relative pose information can be used to match virtual content with motion perceived by the user and the spatiotemporal state of devices, objects, and the real-world environment.

[0047] The low-power visual tracking technique described below will be presented in the context of XR. However, it should be noted that, as explained above, the low-power visual tracking technique can be implemented in a wide variety of other applications, such as, for example, robotics applications, autonomous driving or navigation applications, gaming systems, and controllers. Accordingly, for illustrative purposes, XR is provided throughout the paper as a non-limiting example application of the low-power visual tracking technique.

[0048] The techniques of the present invention will be described in the following disclosure. The discussion begins with a description of example systems and techniques for providing low-power visual tracking and pose estimation, such as... Figures 1 to 6 The following is a description of example methods for providing low-power visual tracking and pose estimation, such as... Figure 7 As explained in the text. The discussion concludes with a description of an example computing device architecture, including example hardware components suitable for performing low-power visual tracking and associated operations, such as... Figure 8 As explained in the text. This disclosure now turns to Figure 1 .

[0049] Figure 1 This is an illustration of an example environment for low-power visual tracking. In this example, the environment may include a main processing system 100 and an auxiliary system 150. In some cases, the main processing system 100 may include a higher-power system that includes higher-power components and / or implements higher-power and / or more complex operations, while the auxiliary system 150 may include a lower-power system that includes lower-power components and / or implements lower-power and / or more complex operations.

[0050] In some cases, the main processing system 100 and the auxiliary system 150 may include separate devices for integrating relative pose information (e.g., the position and orientation of the main processing system 100 and the auxiliary system 150 relative to each other) for use in vision tracking applications or scenarios. For example, the main processing system 100 and the auxiliary system 150 may include separate devices that are used in combination in vision tracking applications to provide one or more functionalities utilizing the relative pose of the main processing system 100 and the auxiliary system 150. By way of example and not limitation, vision tracking applications may include XR applications, robotics applications, autonomous driving or navigation applications, gaming applications, etc.

[0051] In some examples, the main processing system 100 may include electronic devices configured to provide one or more functionalities using information about the relative attitude of the main processing system 100 and the auxiliary system 150, such as XR functionality, gaming functionality, autonomous driving or navigation functionality, computer vision functionality, robotic functionality, etc. For example, in some cases, the main processing system 100 may be an XR device (e.g., a head-up display, a head-up display device, smart glasses, a smart TV system, etc.) and / or a game console, while the auxiliary system 150 may be a controller (e.g., a hand controller, a remote controller, an input device, an external control device, etc.) for interacting with the main processing system 100 and / or the content provided by the main processing system 100.

[0052] As another example, in some cases, the main processing system 100 may be a robotic device (e.g., a robot, autonomous system, robotic tool, or component), while the auxiliary system 150 may be an external device (e.g., a controller, tracking node, etc.) for generating visual tracking information implemented by the main processing system 100. For illustrative purposes, the main processing system 100 and the auxiliary system 150 will be described in the following disclosure as an XR device and a hand controller, respectively. However, as mentioned above, in other examples, the main processing system 100 and the auxiliary system 150 may include other types of devices.

[0053] exist Figure 1In the illustrative example shown, the main processing system 100 may include one or more computing components 110, reference markers 120A-N (collectively referred to as "120"), an image sensor 125, an extended reality engine 130, one or more computer vision models 132, an image processing engine 134, and a rendering engine 136. In some examples, the main processing system 100 may also include other sensors and / or components, as examples and not limitations, such as gyroscopes, accelerometers, inertial measurement units (IMUs), radar, light detection and ranging (LIDAR) sensors, audio sensors, light-emitting diode (LED) devices, storage devices, caches, communication interfaces, displays, memory devices, etc.

[0054] In addition, Figure 1 In the illustrative example shown, the assistive system 150 may include an image sensor system 152, an accelerometer 154, a gyroscope, a tracking engine 158, and a network 160. In some examples, the assistive system 150 may also include other sensors and / or components, as examples and not limitations, such as an IMU, radar, LiDAR, audio sensor, LED device, storage device, cache, communication interface, memory device, etc. Reference will be made below. Figure 8 This section further describes an example architecture and example hardware components that can be implemented by the main processing system 100 and / or the auxiliary system 150. It should be noted that... Figure 1 The components shown with respect to the main processing system 100 and the auxiliary system 150 are merely illustrative examples provided for purposes of illustration, and in other examples, the main processing system 100 and / or the auxiliary system 150 may include more than Figure 1 The components shown may have more or fewer components.

[0055] The main processing system 100 may be a single computing device or part of or implemented by a single computing device or a plurality of computing devices. In some examples, the main processing system 100 may be part of an electronic device (or a plurality of electronic devices), such as a camera system (e.g., a digital camera, IP camera, video camera, security camera, etc.), a telephone system (e.g., a smartphone, cellular phone, conferencing system, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a smart TV, a display device, a game console, an XR device (such as an HMD, a drone, a computer in a vehicle), an IoT (Internet of Things) device, a smart wearable device, or (such as) any other suitable electronic device. In some implementations, one or more computing components 110, a reference marker 120, an image sensor 125, an extended reality engine 130, one or more computer vision models 132, an image processing engine 134, and a rendering engine 136 may be part of the same computing device.

[0056] For example, in some implementations, one or more computing components 110, reference markers 120, image sensors 125, extended reality engines 130, one or more computer vision models 132, image processing engines 134, and rendering engines 136 may be integrated into a camera system, smartphone, laptop computer, tablet computer, smart wearable device, XR device (such as HMD), IoT device, gaming system, and / or any other computing device. However, in some implementations, one or more of the computing components 110, reference markers 120, image sensors 125, extended reality engines 130, one or more computer vision models 132, image processing engines 134, and / or rendering engines 136 may be part of or implemented by two or more separate computing devices.

[0057] Similarly, the assistance system 150 may be part of or implemented by a single computing device or multiple computing devices. In some examples, the assistance system 150 may be part of electronic devices (or multiple electronic devices), such as smartphones, laptops or notebooks, tablets, controllers (e.g., hand controllers, remote controllers, external control devices, input devices, etc.), IoT devices, smart wearable devices, or (such as) any other suitable electronic devices. In some implementations, the image sensor system 152, accelerometer 154, gyroscope 156, tracking engine 158, and computer vision model 160 may be part of the same computing device.

[0058] For example, in some implementations, the image sensor system 152, accelerometer 154, gyroscope 156, tracking engine 158, and computer vision model 160 may be integrated into a controller, smartphone, laptop computer, tablet computer, smart wearable device, IoT device, and / or any other computing device. However, in some implementations, one or more of the image sensor system 152, accelerometer 154, gyroscope, tracking engine 158, and / or one or more computer vision models 160 may be part of or implemented by two or more separate devices.

[0059] By way of example and not limitation, one or more computing components 110 of the main processing system 100 may include a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, a digital signal processor (DSP) 116, and / or an image signal processor (ISP) 118. The main processing system 100 may use one or more computing components 110 to perform various computational operations, such as, for example, extended reality operations (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, etc.), image / video processing, graphics rendering, machine learning, data processing, modeling, computation, and / or any other operation. Figure 1 In the example shown, one or more computing components 110 implement an extended reality (XR) engine 130, an image processing engine 134, and a rendering engine 136. In other examples, one or more computing components 110 may also implement one or more other processing engines. Furthermore, the XR engine 130 may implement one or more computer vision models 132 configured to perform XR operations such as tracking, localization, pose estimation, mapping, content anchoring, etc.

[0060] The operation of the XR engine 130, one or more computer vision models 132, image processing engine 134, and rendering engine 136 (and any other processing engines) can be implemented by any computing component of one or more computing components 110. In one illustrative example, the operation of the rendering engine 136 can be implemented by the GPU 114, and the operation of the XR engine 130, one or more computer vision models 132, image processing engine 134, and / or one or more other processing engines can be implemented by the CPU 112, DSP 116, and / or ISP 118. In some examples, the operation of the XR engine 130, one or more computer vision models 132, and image processing engine 134 can be implemented by the ISP 118. In other examples, the operation of the XR engine 130, one or more computer vision models 132, and / or image processing engine 134 can be implemented by the CPU 112, DSP 116, ISP 118, and / or a combination of the CPU 112, DSP 116, and ISP 118.

[0061] In some cases, one or more computing components 110 may include other electronic circuitry or hardware, computer software, firmware, or any combination thereof to perform any of the operations described herein. Furthermore, in some examples, one or more computing components 110 may include... Figure 1 The computing components shown may include more or fewer computing components. In fact, CPU 112, GPU 114, DSP 116, and ISP 118 are merely illustrative examples provided for purposes of explanation.

[0062] The main processing system 100 may include reference markers 120A to 120N (collectively, “120”) printed, displayed, etched, configured, attached, and / or provided on the exterior / outer side of the main processing system 100. In some cases, the reference markers 120 may be located at certain known locations on the main processing system 100 and / or at certain known distances relative to each other. In some cases, the reference markers 120 may have one or more of the same or different predetermined sizes, shapes, and / or configurations. The number, relative positions, relative distances, sizes, shapes, and / or configurations of the reference markers 120 may vary in different examples, as further described below. Furthermore, the reference markers 120 may include patterns, codes, encoded data, and / or objects that can be detected and analyzed by the image sensor 152 on the auxiliary system 150 to identify attitude information, as further described below. In some cases, the reference markers 120 may be visible in the visible light spectrum and / or infrared spectrum.

[0063] Reference marker 120 may include a specific pattern directly on the main processing system 100 or (e.g., attached, pasted, provided, etc.) on an object or material on the main processing system 100. For example, in some cases, reference marker 120 may include a patterned sticker (or adhesive element or material). As another example, reference marker 120 may include a pattern etched, designed, overlaid, or printed on the main processing system 100. As yet another example, reference marker 120 may include a patterned film, object, overlay, or material.

[0064] Image sensor 125 may include any image and / or video sensor or capture device, such as a digital camera sensor, a video camera sensor, a smartphone camera sensor, or an image / video capture device on an electronic device (such as a television or computer, camera, etc.). In some cases, image sensor 125 may be part of a camera or computing device (such as a digital camera, video camera, IP camera, smartphone, smart TV, gaming system, etc.). Furthermore, in some cases, image sensor 125 may include multiple image sensors, such as rear and front sensor devices, and may be part of a dual-camera or other multi-camera assembly (e.g., including two cameras, three cameras, four cameras, or other numbers of cameras).

[0065] In some examples, image sensor 125 may represent or include one or more low-power image sensor systems, such as image sensor system 152 described below with respect to auxiliary system 150. Image sensor 125 may capture image and / or video data (e.g., raw image and / or video data) that can be processed by image sensor 125, one or more computing components 110, and / or one or more other components. In some cases, image sensor 125 may detect or identify encoded information in a pattern or object, as further described below with respect to image sensor system 152 on auxiliary system 150.

[0066] In some examples, image sensor 125 may capture image data and generate frames based on the image data and / or provide the image data or frames to XR engine 130, image processing engine 1334, and / or rendering engine 136 for processing. Frames may include video frames from a video sequence or still images. Frames may include an array of pixels representing a scene. For example, a frame may be a red-green-blue (RGB) frame with red, green, and blue components per pixel; a luminance, red chrominance, blue chrominance (YCbCr) frame with a luminance component and two chrominance (red chrominance and blue chrominance) components per pixel; or any other suitable type of color or monochrome image.

[0067] In some examples, the XR engine 130 and / or one or more computer vision models 132 may perform XR processing operations based on data from one or more sensors (such as IMU, accelerometer, gyroscope, etc.) on the image sensor 125, image sensor system 152, accelerometer 154, gyroscope 156, tracking engine 158, and / or main processing system 100. For example, in some cases, the XR engine 130 and / or one or more computer vision models 132 may perform tracking, localization, pose estimation, mapping, and / or content anchoring operations.

[0068] In some examples, image processing engine 134 may perform image processing operations based on data from image sensor 125 and / or image sensor system 152. In some cases, image processing engine 134 may perform image processing operations such as, for example, filtering, depigmentation, scaling, color correction, color transformation, segmentation, noise reduction filtering, spatial filtering, artifact correction, etc. Rendering engine 136 may acquire image data generated and / or processed by computing component 110, image sensor 125, XR engine 130, one or more computer vision models 132, and / or image processing engine 134, and render video and / or image frames for presentation on a display device.

[0069] In some cases, the image sensor system 152 on the auxiliary system 150 may include a low-power imaging system or a normally open computer vision camera system. For example, each image sensor system may include an image sensor and one or more low-power processors for processing image data captured by the image sensor. In some cases, each image sensor system may include one or more image processing, computer vision, and / or other processing algorithms. For example, in some cases, the image sensor system 132 may include or implement a tracking engine 158 and / or a computer vision model 160 and perform pose estimation, as further described herein. As further described herein, in some examples, the image sensor system 152 may detect a reference marker 120 on the main processing system 100 and estimate the relative pose of the auxiliary system 150 and the main processing system 100 based on the detected reference marker 120.

[0070] Tracking engine 158 may implement one or more algorithms for tracking and estimating the relative pose of auxiliary system 150 and main processing system 100. In some examples, tracking engine 158 may receive image data captured by image sensor system 152 and perform pose estimation based on the received image data to calculate the relative pose of auxiliary system 150 and main processing system 100. In some cases, tracking engine 158 may implement computer vision model 160 to calculate the relative pose of auxiliary system 150 and main processing system 100. In some examples, tracking engine 158 and / or computer vision model 160 may be implemented by image sensor system 152. For example, each image sensor system may implement a tracking engine and a computer vision model for performing pose estimation as described herein. In other examples, tracking engine 158 and / or computer vision model 160 may be implemented by auxiliary system 150, separate from image sensor system 152.

[0071] Accelerometer 154 detects the acceleration of auxiliary system 150 and generates an acceleration measurement based on the detected acceleration. Gyroscope 156 detects and measures the orientation and angular velocity of auxiliary system 150. For example, gyroscope 156 can be used to measure the pitch, roll, and yaw of auxiliary system 150. In some examples, image sensor system 152 and / or tracking engine 158 can use the measurements obtained by accelerometer 154 and gyroscope 156 to calculate the relative attitude of auxiliary system 150 and main processing system 100, as further described herein. For example, image sensor system 152 can detect the position of reference marker 120 and main processing system 100, and image sensor system 152 and / or tracking engine 158 can use the detected position of reference marker 120 and measurements from accelerometer 154 and gyroscope 156 to calculate the attitude of auxiliary system 150 relative to main processing system 100.

[0072] Although the main processing system 100 and the auxiliary system 150 are shown to include certain components, those skilled in the art will appreciate that the main processing system 100 and the auxiliary system 150 may include more than Figure 1 The components shown may include more or fewer components. For example, in some instances, the main processing system 100 and / or auxiliary system 150 may also include one or more other memory devices (e.g., RAM, ROM, cache, etc.), one or more networking interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, caches, storage devices, and / or Figure 1 Other hardware or processing devices not shown. References will be made below. Figure 8 This is an illustrative example describing computing devices and hardware components that can be implemented by the main processing system 100 and / or the auxiliary system 150.

[0073] Figure 2 An example view 200 of the auxiliary system 150 and the main processing system 100 is illustrated in an example use case for detecting the posture of the auxiliary system 150 relative to the main processing system 100 using an image sensor system 152 on the auxiliary system 150. In this example, the auxiliary system 150 represents a hand controller (such as a ring controller) used in conjunction with the main processing system 100 to provide an XR experience. Furthermore, in this example, the main processing system 100 represents a wearable XR device (such as an HMD). However, it should be noted that the hand controller and wearable device are used as illustrative examples herein for illustrative purposes, and in other examples, the auxiliary system 150 and / or the main processing system 100 may include or represent other devices, such as, for example, other types of controllers, gaming systems, smart wearable devices, IoT devices, sensor systems, smart TVs, etc.

[0074] Similarly, for explanatory and illustrative purposes, this article refers to Figures 3A to 7 The described auxiliary system 150 and main processing system 100 will be described in the following disclosure as including or representing an example hand controller and an example wearable XR device (e.g., HMD), respectively. However, it should be noted that the hand controller and wearable XR device are merely illustrative examples for illustrative purposes, and in other examples, reference is made herein to... Figures 3A to 7 The auxiliary system 150 and / or the main processing system 100 described may include or represent other devices, such as, for example, other types of controllers, gaming systems, smart wearable devices, IoT devices, sensor systems, smart TVs, etc.

[0075] As explained, the auxiliary system 150 may include image sensor systems 152A-152N (collectively referred to as "152") that can be used to detect reference markers (e.g., 120) on the main processing system 100 when they are within the field of view (FoV) of the image sensor systems 152. The auxiliary system 150 may use the image sensor systems 152A-152N to detect the reference markers, identify the relative position of one or more of the reference markers, and use this information to estimate the pose of the auxiliary system 150 relative to the main processing system 100.

[0076] For example, when a reference marker is within the FoV of at least one image sensor system (e.g., image sensor systems 152A, 152B, 152C, 152D, 152E, 152F, 152G to 152N), the at least one image sensor system can capture an image of the reference marker and use the captured image to detect the position / location of the one or more reference markers. Similarly, when multiple reference markers are within the FoV of one or more image sensor systems (e.g., image sensor systems 152A, 152B, 152C, 152D, 152E, 152F, 152G to 152N), the one or more image sensor systems can capture images of the multiple reference markers and use the captured images to detect the relative position / location of the multiple reference markers.

[0077] In some examples, image sensor system 152 and / or auxiliary system 150 can use the relative position / location of a reference marker detected on main processing system 100 to calculate the pose of auxiliary system 150 relative to main processing system 100. For example, when a reference marker on main processing system 100 is within the FoV of an image sensor system (e.g., image sensor systems 152A, 152B, 152C, 152D, 152E, 152F, 152G to 152N) on auxiliary system 150, those image sensor systems can capture images of the reference marker. Auxiliary system 150 and / or image sensor systems can use the captured images to detect the reference marker and determine their relative position / location in 3D space. Image sensor system and / or auxiliary system 150 can use the relative position / location of the reference marker to calculate the pose of auxiliary system 150 relative to main processing system 100.

[0078] In some examples, when the reference marker is within the FoV of multiple image sensor systems, the auxiliary system 150 can track the 6DoF state of the auxiliary system 150 relative to the main processing system 100. In some cases, when the reference marker is within the FoV of only one image sensor system, the auxiliary system 150 can track a subset of the 6DoF state of the auxiliary system 150 relative to the main processing system 100. In some cases, when the reference marker is not within the FoV of any image sensor system, the auxiliary system 150 can use accelerometer 154 and gyroscope 156 to obtain inertial measurements of the auxiliary system 150 and use these inertial measurements to track the state of the auxiliary system 150.

[0079] By using an image sensor system 152 (which may have low power consumption) on the auxiliary system 150 to track the attitude of the auxiliary system 150 based on a reference marker on the main processing system 100, instead of using higher power components on the main processing system 100, it is possible to track the attitude of the auxiliary system 150 with lower power consumption. Therefore, such a strategy can save power in the main processing system 100, which typically has higher power components and thus higher power requirements and / or more limited battery life.

[0080] Although example view 200 illustrates a single auxiliary system 150, it should be noted that other examples may include multiple auxiliary systems. For instance, in some cases, two auxiliary systems (e.g., including auxiliary system 150 and another auxiliary system) (such as two hand controllers) may be used in combination with the main processing system 100 to track the corresponding pose of each auxiliary system relative to the main processing system 100. This document provides a single auxiliary system 150 as an illustrative example for purposes of illustration only.

[0081] Figure 3A This is an illustration of an example use case 300 for tracking the 6DoF pose of an auxiliary system 150 and / or a main processing system 100. In this example, reference markers 320A and 320B on the main processing system 100 are within the FoV of image sensor systems 152A, 152B to 152N (where N is a value greater than or equal to 0), and image sensor systems 152A, 152B to 152N can capture image data (e.g., one or more images or frames) of reference markers 320A and 320B on the main processing system 100. Image sensor systems 152A, 152B to 152N and / or auxiliary system 150 can perform object detection to detect reference markers 320A and 320B within the FoV of image sensor systems 152A, 152B to 152N.

[0082] Each of reference markers 320A and 320B may include a specific pattern (multiple patterns) that can be detected by image sensor systems 152A, 152B to 152N. The detected patterns of reference markers 320A and 320B can be used to identify the positioning of reference markers 320A and 320B. When both of the image sensor systems 152A, 152B to 152N detect the same pattern (e.g., reference markers 320A and 320B), the auxiliary system 150 can triangulate the three-dimensional (3D) position of reference markers 320A and 320B. The auxiliary system 150 can use the 3D positions of reference markers 120A to 120N and the relative pose of reference markers 320A and 320B to identify the 6DoF pose of the auxiliary system 150 relative to the main processing system 100. In some examples, the auxiliary system 150 may use the 3D positions of reference markers 320A and 320B and the relative orientation of reference markers 320A and 320B to identify the 6DoF orientation of the main processing system 100 relative to the auxiliary system 150.

[0083] Furthermore, in some examples, the calculated attitude information can be more accurate and robust as more reference markers are detected and / or as more image sensor systems detect the same reference markers (or multiple reference markers). For example, if the auxiliary image sensor system is able to detect one or more of reference markers 320A and 320B and / or if image sensor systems 152A, 152B to 152N (and / or the auxiliary image sensor system) are able to detect additional reference markers, the auxiliary system 150 can use the additional reference marker data to increase the accuracy of the calculated attitude information.

[0084] exist Figure 3A In this embodiment, reference markers 320A and 320B ON are printed on the front of the main processing system 100 and are shown in a specific circular configuration. However, it should be noted that other examples may implement different numbers, arrangements, shapes, sizes, structures, designs, and / or configurations of the reference markers. For example, in some cases, the main processing system 100 may have more or fewer reference markers printed on the front, top, and / or sides of the main processing system 100, and the reference markers may have the same or different shapes, sizes, patterns, structures, designs, and / or configurations. In some cases, the reference markers printed on the main processing system 100 may be oriented in a specific configuration that allows for left-right positioning in cases where the auxiliary system 150 or the main processing system 100 is upside down, twisted, or otherwise rotated.

[0085] Figure 3BThis is an illustration of another example use case 320 for tracking attitude information of the auxiliary system 150 and / or the main processing system 100. In this example, reference markers 320A and 320B are only within the FoV of the image sensor system 152A. The image sensor system 152A may capture image data (e.g., one or more images or frames) of the reference markers 320A and 320B, and the auxiliary system 150 and / or the image sensor system 152A may perform object detection to detect the reference markers 320A and 320B within the FoV of the image sensor system 152A.

[0086] As mentioned above, each of the reference markers 320A and 320B may include a specific pattern that can be detected by the image sensor system 152A. When the image sensor system 152A detects the reference markers 320A and 320B, the auxiliary system 150 (or the image sensor system 152A) may triangulate the 3D position of the reference markers 320A and 320B. The auxiliary system 150 (or the image sensor system 152A) may then use the 3D position and relative orientation of the reference markers 320A and 320B to identify at least a portion or subset of the 6DoF state of the auxiliary system 150 relative to the main processing system 100. In some examples, the auxiliary system 150 (or the image sensor system 152A) may similarly use the 3D position and relative orientation of the reference markers 320A and 320B to identify at least a portion or subset of the 6DoF state of the main processing system 100 relative to the auxiliary system 150.

[0087] Figure 3C This is an illustration of an example use case 340 for tracking the 6DoF attitude of the auxiliary system 150 and / or the main processing system 100. In this example, reference markers 320C to 320F (in...) Figure 3C The reference markers 320C to 320F are implemented on the auxiliary system 150, while the image sensor systems 152A and 152N are implemented on the main processing system 100. Furthermore, the reference markers 320C to 320F are within the FoV of the image sensor systems 152A, 152B to 152N on the main processing system 100. The image sensor systems 152A, 152B to 152N can capture image data (e.g., one or more images or frames) of the reference markers 320C to 320F, and the main processing system 100 and / or the image sensor systems 152A and / or 152B can perform object detection to detect the reference markers 320C to 320F within the FoV of the image sensor systems 152A, 152B to 152N.

[0088] Image sensor systems 152A, 152B to 152N can detect specific patterns depicted in reference markers 320C to 320F to identify the relative position / location of the reference markers 320C to 320F. When both of the image sensor systems 152A, 152B to 152N detect the same pattern (e.g., reference markers 320C to 320F), the main processing system 100 can triangulate the 3D position of the reference markers 320C to 320F. The main processing system 100 can then use the 3D position of the reference markers 320C to 320F and their relative orientation to identify the 6DoF orientation of the auxiliary system 150 relative to the main processing system 100. In some examples, the main processing system 100 can also use the 3D position of the reference markers 320C to 320F and their relative orientation to identify the 6DoF orientation of the auxiliary system 150 relative to the main processing system 100.

[0089] In some cases, the number of reference markers on the auxiliary system 150 and the number of image sensor systems on the main processing system 100 that detect the same reference markers can affect the accuracy and robustness of the attitude information calculated by the main processing system 100, as it can increase or decrease the amount of reference data points used to calculate the attitude information and / or the amount of data points related to calculating the attitude information. Furthermore, in some cases, the reference markers printed on the auxiliary system 150 can be oriented in a specific configuration that allows for distinguishing left and right positioning even when the auxiliary system 150 or the main processing system 100 is upside down, twisted, or otherwise rotated.

[0090] Figure 3D This illustration depicts an example use case 360 ​​for tracking a tagged object based on reference markers 320G and 320N. In this example, reference markers 320G and 320N are printed on a target 362 for identification and tracking during an XR experience, game, or other tracking-based application. In some cases, the user may place or print reference markers 320G and 320N on the target 362. In other cases, the target 362 may be designed with or implement reference markers 320G and 320N. For example, reference markers 320G and 320N may be etched, printed, or configured on the target 362 during configuration or manufacturing.

[0091] Target 362 may include any object or item to be tracked. For example, Target 362 may include game consoles, televisions, environmental objects, furniture, walls, doors, computers, equipment, devices, appliances, tools, etc. In addition, Target 362 may include any number, size, shape, arrangement, structure, design and / or configuration of reference markers.

[0092] Reference markers 320G and 320N on target 362 are shown within the FoV of image sensor systems 152A, 152B to 152N on main processing system 100. Image sensor systems 152A, 152B to 152N can capture image data (e.g., one or more images or frames) of reference markers 320G and 320N, and main processing system 100 and / or image sensor systems 152A and / or 152B can perform object detection to detect reference markers 320G and 320N within the FoV of image sensor systems 152A, 152B to 152N.

[0093] Image sensor systems 152A, 152B to 152N can detect specific patterns on reference markers 320G and 320N to identify the positioning of reference markers 320G and 320N. When two of the image sensor systems 152A, 152B to 152N detect the same pattern (e.g., reference markers 320G and 320N), the main processing system 100 can triangulate the 3D position of reference markers 320G and 320N. The main processing system 100 can then use the 3D position of reference markers 320G and 320N and their relative pose to identify the 6DoF pose of target 362 relative to the main processing system 100. In some examples, the main processing system 100 can also use the 3D position of reference markers 320G and 320N and their relative pose to identify the 6DoF pose of the main processing system 100 relative to target 362.

[0094] Figure 4A An example configuration 400 of the reference markers has been explained. In this example, the reference markers 420A and 420B are patterned rings. However, it should be noted that in other examples, the reference markers 420A and 420B may have any other shape, such as, for example, squares, triangles, rectangles, octagons, etc.

[0095] The pattern within the ring can have any shape, configuration, arrangement, characteristics, and / or design. Furthermore, the pattern within the reference markers 420A and 420B (e.g., the ring) can be based on any feature, element, characteristic, or item visible in the IR and / or visible light spectrum. For example, these patterns may include colors, lines, letters, symbols, codes, textures, etchings, non-uniformity, substances, images, illumination (e.g., in the visible or IR spectrum), backlighting (e.g., in the visible or IR spectrum), illumination from external or ambient sources (e.g., IR floodlight in the environment), etc.

[0096] Image sensor system 152 can analyze image data capturing reference markers 420A and 420B, and detect reference markers 420A and 420B within the image data based on the pattern (and / or one or more points in the pattern) of reference markers 420A and 420B. In some examples, object detection by image sensor system 152 can provide bounding boxes 402A and 402B around reference markers 420A and 420B. Bounding boxes 402A and 402B can represent and / or be used to identify the relative positioning of reference markers 420A and 420B.

[0097] Figure 4B Another example configuration 410 of the reference markers is explained. In this example, reference markers 420C to 420G are machine-readable barcodes, such as Quick Response (QR) codes with square patterns 412, 414, and 416. Each of the machine-readable barcodes can encode information such as location information, orientation information (e.g., left relative to right), identifiers, trackers, user information, device information, game information, XR information, application information, metadata, text, video game information (e.g., player information, game elements, etc.), setup information, and / or any other type of information. For example, a machine-readable barcode can encode a unique code for each player in a multiplayer game setup.

[0098] An image sensor system (e.g., 152) on the auxiliary system (e.g., 150) can detect reference markers to detect their relative positioning. In some examples, the image sensor system on the auxiliary system can read machine-readable barcodes in the reference markers 420C to 420G to detect information encoded in the machine-readable barcodes to determine the positioning of the reference markers and / or any other data from the encoded information. In some examples, the image sensor system can detect square patterns 412, 414, and 416 in the reference markers 420C to 420G at a distance greater than the machine-readable barcodes to allow the image sensor system to calculate the positioning of the reference markers and / or information associated with the reference markers even when the machine-readable barcodes cannot be read.

[0099] Figure 4C Another example configuration of the reference marker 420N has been explained. In this example, the reference marker 420N includes an outer pattern 430 and an inner pattern 432. In this example, the outer pattern 430 is a torus and the inner pattern 432 is an inner torus. However, it should be noted that in other examples, the outer pattern 430 and / or the inner pattern 432 may have any other shape, such as, for example, a square, a triangle, a rectangle, an octagon, etc.

[0100] In some examples, the outer pattern 430 and / or the inner pattern 432 may have any shape, configuration, arrangement, characteristics, and / or design. Furthermore, the outer pattern 430 and / or the inner pattern 432 may be based on any feature, element, or item visible in the IR and / or visible light spectrum. For example, the outer pattern 430 and / or the inner pattern 432 may include colors, lines, letters, symbols, codes, textures, etchings, non-uniformity, substances, images, illumination (e.g., in the visible or IR spectrum), backlighting (e.g., in the visible or IR spectrum), etc.

[0101] In some cases, the internal pattern 432 can be used to provide more accurate positioning of the reference marker 420N. For example, the internal pattern 432 can be used as a landmark for repeatedly and / or precisely positioning specific points on the reference marker 420N. In some cases, the image sensor system (e.g., 152) used to detect the reference marker 420N can implement adaptive stride to improve its efficiency. As a result, the bounding box around the reference marker 420N detected by the image sensor system may not always be precisely or accurately centered on the reference marker 420N, which may affect the accuracy of triangulation. For example, the bounding box around the reference marker 420N may be far away from or offset by several pixels, which may affect the accuracy of triangulation.

[0102] In some examples, to improve the accuracy of keypoint localization within a reference marker, the image sensor system (e.g., 152) can perform fast and coarse object detection over the entire pattern (e.g., outer pattern 430). After detecting the outer pattern 430 based on the fast and coarse object detection, the image sensor system can perform a fine-grained search on the inner pattern 432 within a detection window associated with the outer pattern 430. This also improves the localization accuracy of the inner pattern 432 relative to the image sensor system in larger orientations. In some examples, this two-stage localization algorithm can be supported by robust training for larger orientations. In some cases, this two-stage localization algorithm can also utilize failure branching mechanisms (e.g., conditional branching in machine learning object detection) for multi-object detection in the image sensor system.

[0103] Figure 5An example reference line 502, projected between reference markers 520A and 520B and used for attitude transitions between a reference pattern and an observed pattern, is explained. Reference line 502 may connect two or more points on reference markers 520A and 520B (and / or may be projected across these two or more points). In this example, reference line 502 connects the centers of reference markers 520A and 520B. The reference position of reference line 502 may be calculated based on a reference coordinate system in 3D space (e.g., an X, Y, Z coordinate system) (such as a world or homogeneous coordinate system). Furthermore, the observed position of reference line 502 may be determined based on an image captured by image sensor system 152. The observed position may be determined relative to an image coordinate system (e.g., a coordinate system associated with the image captured by image sensor system 152).

[0104] In some examples, the reference position of reference line 502 can be compared with the observed position of reference line 502 to calculate the attitude of reference markers 520A and 520B and / or to improve attitude estimation accuracy when calculating the attitude of reference markers 520A and 520B. In some examples, the reference position of reference line 502 and the observed position of reference line 502 can be used to transform the observed attitude of reference markers 520A and 520B relative to the reference attitude of reference markers 520A and 520B, as described below. Figure 6 Further described. In some examples, the observed pose of reference line 502 can be translated relative to the reference pose of reference line 502 using the relative distance and / or angle between coordinate planes (e.g., X, Y, Z coordinate planes) projected from the reference position of reference line 502 and the observed position of reference line 502, and / or projected between the reference position of reference line 502 and the observed position of reference line 502. The translated pose of reference line 502 can be used to determine the relative pose of the pattern associated with reference line 502.

[0105] Figure 6 This is an illustration of an example pose transition 600 between a reference pattern (e.g., reference markers 520A and 520B) and an observed pattern (e.g., observed markers 520A and 520B). In some cases, the observed pattern may include a pattern detected in an image captured via the image sensor system 152, and the reference pattern may include a pattern determined relative to a reference coordinate system (e.g., world coordinates or homogeneous coordinates).

[0106] exist Figure 6In the example shown, a translation transformation 602 is calculated between the observed position 606 of a reference line (e.g., reference line 502) and a reference position 604 of the reference line to determine the actual, absolute, and / or estimated positioning of the reference line. The translation transformation 602 can be calculated based on the reference position 604 and the observed position 606 of the reference line. In some cases, the translation transformation 602 can be based on projections from the reference position 604 and the observed position 606 of the reference line along the X, Y, and Z axes. In some examples, the projections can reflect the difference between the reference position 604 and the observed position 606 of the reference line.

[0107] For example, projections from reference position 604 (or one or more points in reference position 604) and observed position 606 (or one or more points in observed position 606) along the X, Y, and Z axes can be compared to translate or restore the position of the reference line. In some cases, projection can be used to perform a translation transformation 602 along the X, Y, and Z axes between the reference position 604 and the observed position 606 of the reference line.

[0108] In some examples, the reference position 604 of the reference line can be a known or calculated position of the reference line (and / or one or more points of the reference line). In some cases, the reference position 604 of the reference line (plotted along the X-axis) can be calculated by triangulation of one or more points of the reference line from a reference frame and / or coordinate system. In some cases, such triangulation can utilize known information about the relative configuration and / or position of patterns associated with the reference line (such as reference patterns and observed patterns (e.g., reference markers 520A and 520B)). In some cases, the observed position 606 of the reference line can be the position of the reference line (and / or one or more points of the reference line) observed and / or calculated based on images(e.g., images obtained via one or more image sensor systems 152) capturing the reference line.

[0109] Furthermore, the rotational transformation 610 between the reference orientation 612 (each depicted along the Y-axis, X-axis, and Z-axis) of the reference line and the observed orientation of the reference line can be calculated by recovering and estimating the rotation angles 620, 622, and 624 of the reference line along the X-axis, Y-axis, and Z-axis. The rotation angles 620, 622, and 624 can be calculated based on the observed orientation of the reference line and / or the projections 614, 616, and 618 generated along the X-axis, Y-axis, and Z-axis of the reference orientation 612 of the reference line. For example, the rotation angle 620 of the reference line along the X-axis can be calculated based on the projection 614 of the reference orientation 612 along the X-axis. Similarly, the rotation angle 622 of the reference line along the Y-axis can be calculated based on the projection 616 of the reference orientation 612 along the Y-axis. Finally, the rotation angle 624 of the reference line along the Z-axis can be calculated based on the projection 618 of the reference orientation 612 along the Z-axis.

[0110] In some examples, rotation angles 620, 622, and 624 can be used to estimate or restore the orientation of the reference line. Additionally, in some examples, points (multiple points) connecting projections 614, 616, and 618 along the X, Y, and Z axes can be used to calculate the rotation angles 620, 622, and 624 of the reference line and / or the orientation of the reference line.

[0111] Translation transformation 602 and rotation transformation 610 can be used to perform attitude transformation 600 between the attitude of a reference pattern associated with a reference line and the observed attitude of an observed pattern associated with a reference line. In some examples, attitude transformation 600 can be used to estimate the relative attitude of the reference pattern (e.g., reference markers 520A and 520B) associated with a reference line and the observed attitude of the observed pattern associated with a reference line (e.g., the observed pattern of reference markers 520A and 520B), which may reflect the relative position and orientation of the reference pattern (e.g., reference markers 520A and 520B) and the observed pattern. In some examples, the relative 6DoF attitude of the auxiliary system 150 and the main processing system 100 may be calculated based on the relative attitude of the reference pattern and the observed pattern.

[0112] In some examples, the relative 6DoF pose of the auxiliary system 150 and the main processing system 100 can be calculated by triangulation of the 3D position of a group of patterns (e.g., reference markers 520A and 520B) observed by the image sensor system 152, the relative pose of the group of patterns, and the calibrated relative position of the image sensor system 152. In some cases, the image sensor system 152 on the auxiliary system 150 can capture images or frames of reference lines (and / or associated patterns and / or points on them), and these images or frames can be used to estimate the relative 6DoF pose and / or motion of the auxiliary system 150 and the main processing system 100 by calculating the relative transformation between the pose of the auxiliary system 150 and the pose of the main processing system 100 (and / or the patterns or reference markers 520A and 520B). In some examples, the pose of the main processing system 100 can be calculated based on pose transformation 600, as described above.

[0113] In some examples, two or more image sensor systems 152 on the auxiliary system 150 may capture images or frames of reference markers (e.g., two or more of the reference markers) on the main processing system 100, and the relative 6DoF pose of the auxiliary system 150 and the main processing system 100 may be calculated using pixel positions in the images or frames corresponding to points in the reference markers captured in these images or frames. For example, the 6DoF poses of the two or more image sensor systems 152 may be calibrated relative to each other. The two or more image sensor systems 152 may capture images or frames of the same set of reference markers (e.g., 520A and 520B) on the main processing system 100, and detect pixels corresponding to points in the reference markers captured in these images or frames. The positions of points in the reference markers may be triangulated based on those pixels corresponding to points in the reference markers. The triangulated positions of the points corresponding to pixels may be used to calculate the 3D position and relative pose of the reference markers. The 3D position and relative pose of the reference markers may reflect the 3D position and pose of the main processing system 100 associated with the reference markers. Therefore, the 3D position and relative orientation of the reference marker can be used together with the relative orientation of two or more image sensor systems 152 to estimate the relative 6DoF orientation of the auxiliary system 150 and the main processing system 100.

[0114] Having already disclosed example systems, components, and concepts, this disclosure now turns to example method 700 for estimating the relative pose of an auxiliary system (e.g., 150) and a main processing system (e.g., 100), such as Figure 7 As shown. The steps outlined in this article are non-limiting examples provided for illustrative purposes and can be implemented in any combination thereof, including removing, adding, or modifying certain combinations of steps.

[0115] At block 702, method 700 may include receiving one or more images captured by each image sensor system of a set of image sensor systems (e.g., 152) on a first device (e.g., auxiliary system 150). The one or more images may capture a pattern set (e.g., 120A to 120N) on a second device (e.g., main processing system 100). In some examples, the pattern set may be a reference marker containing one or more patterns. Furthermore, the pattern set may be visible in the infrared or visible light spectrum.

[0116] The pattern set may have a predetermined configuration on the second device. By way of example, and not limitation, the predetermined configuration of the pattern set may include: the relative position of each pattern on the second device, the relative orientation of each pattern on the second device, the shape of each pattern, the size of each pattern, the characteristics of each pattern, and / or the arrangement of the pattern set. In some examples, one or more patterns from the pattern set may include encoded machine-readable information, such as... Figure 4B As shown. The encoded machine-readable information in the pattern may include location information associated with the pattern, an identifier associated with the pattern, a unique code, settings, and information about the user account associated with the XR application on the main memory of the second device (e.g., a player in a game, a user profile, etc.).

[0117] In some cases, the first device may have lower power requirements compared to the second device. For example, the first device may consume less power and / or have lower power components compared to the second device. In some examples, the first device may be a hand controller device and the second device may be a head-mounted display (HMD). In other examples, the first device may be any other type of controller, auxiliary device, or lower power device, while the second device may be any other type of higher power device, wearable XR device, etc. Furthermore, in some cases, each image sensor system in the image sensor system set may be a low-power image sensor system, as described above.

[0118] In some cases, each image sensor system may capture at least one image of at least one pattern. In some cases, two or more image sensor systems may each capture one or more images of the same pattern. In some examples, the image sensor system set may represent all image sensor systems on the first device. In other examples, the image sensor system set may represent a subset of all image sensor systems on the first device. For example, the image sensor system set may represent a subset of image sensor systems capable of capturing images of one or more patterns in a pattern set from their specific poses and / or angles. In other words, the image sensor system set may represent a subset of image sensor systems on the first device that have one or more patterns within their FoV when capturing one or more images.

[0119] In some cases, the number of image sensor systems capturing images of one or more patterns can vary with changes in the relative attitude of the image sensor systems and the pattern set on the second device, as such changes in relative attitude can add to or remove one or more patterns from the FoV of the image sensor systems. In practice, sometimes one or more patterns from the pattern set on the second device may be within the FoV of all image sensor systems on the first device, allowing all image sensor systems on the first device to capture images of one or more patterns. At other times, some or all of the pattern set may be within the FoV of only one, two or more, or even none of the image sensor systems on the first device. In some cases, if no pattern in the pattern set is within the FoV of any image sensor system on the first device, the auxiliary device and / or the main processing system can use inertial measurements from the respective sensors (e.g., IMU, gyroscope, accelerometer, etc.) to calculate or track attitude information.

[0120] At block 704, method 700 may include determining a set of pixels corresponding to a set of patterns on a second device based on one or more images captured by each image sensor system. For example, each image captured by the image sensor system may capture one or more patterns. Thus, the image may include pixels corresponding to one or more points in one or more patterns. The image sensor system may detect one or more patterns and one or more points in the one or more patterns. The image sensor system may then identify one or more pixels (and their pixel locations within the image) that correspond (e.g., represent, capture, depict, etc.) to one or more points in the one or more patterns. The one or more pixels (and their pixel locations) in the image may be used to estimate the position and / or orientation of one or more patterns associated with the one or more points corresponding to the pixels, as further described herein.

[0121] At block 706, method 700 may include determining the position and relative pose of each pattern from the pattern set in space (e.g., in 3D space) based on a set of pixels corresponding to a pattern set on a second device. For example, as described above, each image captured by an image sensor system may capture one or more patterns. Pixels in the image corresponding to one or more points in one or more patterns may be identified and used to estimate the position and / or pose of the one or more patterns. In some cases, the position of a pixel within the image may be used to triangulate the point in the pattern captured by the image that corresponds to that pixel. As another example, the 3D position and orientation of each pattern from the pattern set may be estimated by comparing, triangulating, and / or using the positions of each pixel within the image captured by the image sensor system set that correspond to each point in the pattern set. The 3D position and orientation of each pattern may also be used to calculate the relative 6DoF pose of each pattern.

[0122] In some examples, the position of each pixel corresponding to a point in the pattern (e.g., the coordinates and / or point associated with the pixel) can be projected onto the point in the pattern along one or more planes (e.g., along one or more axes in the image coordinate system) and compared with the projection of that point in the pattern from a reference coordinate system (e.g., a world or homogeneous coordinate system) to determine the 3D location of that point in the pattern. In some cases, the 3D position of the point in the pattern can be determined by translating the position of the point in the pattern within the image (and / or image coordinate system) relative to the position of the point in the reference coordinate system.

[0123] Furthermore, in some cases, the orientation of points in the pattern within the image (and / or image coordinate system) can be rotated relative to the orientation of points in a reference coordinate system to determine the 3D orientation of points in the pattern. The 3D positions and orientations of points in the pattern can be used to estimate the 6DoF pose of the pattern. Additionally, the 3D positions and orientations of individual points in the pattern set can be used to estimate the relative 6DoF pose of the pattern set. The relative 6DoF pose of the pattern set can also be used to determine the 6DoF pose of the main processing system 100 and / or the auxiliary system 150, as further described herein.

[0124] In some cases, determining the position and relative orientation of each pattern from a pattern set in space may include: determining the 3D orientation of the pattern set by rotating the 3D coordinates (e.g., X, Y, Z coordinates) in space corresponding to the pixel set corresponding to the pattern set; and determining the 3D position of the pattern set by translating the 3D coordinates in space corresponding to the pixel set corresponding to the pattern set. In some examples, the 3D coordinates may be rotated relative to a reference 3D coordinate system, and the 3D coordinates may be translated relative to that reference 3D coordinate system. Furthermore, in some examples, the relative orientation of each pattern may be based on the 3D orientation and the 3D position of the pattern set. Similarly, the position of each pattern may include, for example, a corresponding 3D position from the 3D position of the pattern set.

[0125] In some aspects, determining the pixel set corresponding to the pattern set may include: detecting each pattern of the pattern set from the second device based on one or more images captured by each image sensor, and identifying one or more points in each pattern of the pattern set. The one or more points identified in each pattern may correspond to one or more pixels from the pixel set. Furthermore, as mentioned above, the position and relative orientation of each pattern in space may be determined based on the one or more points in each pattern.

[0126] In other aspects, determining the pixel set corresponding to the pattern set may include: detecting each pattern from the pattern set on the second device based on one or more images captured by each image sensor, detecting smaller inner patterns for each pattern, and identifying one or more points in each smaller inner pattern. These one or more points may correspond to one or more pixels from the pixel set, and the position and relative orientation of each pattern in space may be based on these one or more points in each smaller inner pattern.

[0127] At box 708, method 700 may include determining the pose of the first device relative to the second device based on the position and relative pose of each pattern from the pattern set in space. For example, the 6DoF pose of each pattern may be used to determine the 6DoF pose of the second device (e.g., main processing system 100) and / or the relative 6DoF pose of the first device (e.g., auxiliary system 150). In some cases, the 6DoF poses of the second device (e.g., main processing system 100) and the first device (e.g., auxiliary system 150) may be calculated relative to each other to obtain the relative 6DoF pose of the first and second devices. In some examples, the relative 6DoF pose of the first and second devices may be used to coordinate and / or relate the movement, pose, interaction, representation, etc., of the first and second devices in an XR experience, such as an XR game running on the second device, an XR shopping app running on the second device, an XR modeling app running on the second device, an XR medical app running on the second device, etc. In some examples, the relative 6DoF pose of the first and second devices can be used to perform a variety of tracking and localization-based operations, such as, for example, collision avoidance, navigation, mapping, interaction with real-world objects, autonomous device operation, or tasks.

[0128] In some cases, the orientation of the first device relative to the second device can be further determined based on the predetermined relative positions and orientations of the image sensor system set on the first device. For example, the image sensor systems on the first device can be configured on the first device at specific distances, orientations, positions, separation angles, etc. This information can be known and used to determine their relative orientation at the first device. The relative orientation of the image sensor systems on the first device can be used as additional parameters for calculating the position and relative orientation of each pattern from the pattern set in space and / or the orientation of the first device relative to the second device.

[0129] In some examples, method 700 may include determining the 3D coordinates of one or more points in space associated with a scene captured by the image sensor system assembly (e.g., a scene in one or more images), and determining the relative 3D pose of the image sensor system assembly on a first device based on a predetermined relative position and orientation of the image sensor system assembly and the 3D coordinates of the one or more points in space. In some examples, the pose of the first device relative to a second device may be further based on the relative 3D pose of the image sensor system assembly on the first device. Furthermore, in some cases, the relative 3D pose of the image sensor system assembly may be, for example, a relative 6DoF pose in 3D space.

[0130] In some aspects, determining the attitude of the first device relative to the second device may include determining the attitude of the second device relative to the first device. The attitudes of the first device and the second device may be, for example, relative 6DoF attitudes.

[0131] In some aspects, method 700 may include: detecting one or more additional patterns on an object captured in one or more additional images captured by one or more image sensor systems from a set of image sensors; determining one or more pixels in the one or more additional images corresponding to the one or more additional patterns on the object; and determining an additional 6DoF pose of the object relative to a first device and / or a second device based on the one or more pixels. In some cases, the object may be, for example, a wall, a display device, a video game console, furniture, appliances, household items, etc.

[0132] In some examples, method 700 may be performed by one or more computing devices or apparatuses. In an illustrative example, method 700 may be performed by... Figure 1 The main processing system 100 and / or auxiliary system 150 shown are and / or have Figure 8 One or more computing devices, as illustrated in the computing device architecture 800, may be used to perform the actions. In some cases, such computing devices or apparatuses may include processors, microprocessors, microcomputers, or other components of the apparatus configured to perform the steps of method 700. In some examples, such computing devices or apparatuses may include one or more sensors configured to capture image data. For example, the computing device may include a smartphone, head-mounted display, mobile device, or other suitable device. In some examples, such computing devices or apparatuses may include cameras configured to capture one or more images or videos. In some cases, such computing devices may include a display for displaying images. In some examples, one or more sensors and / or cameras are separate from the computing device, in which case the computing device receives the sensed data. Such computing devices may further include a network interface configured to transmit data.

[0133] The components of a computing device can be implemented using a circuit system. For example, each component may include and / or may be implemented using electronic circuitry or other electronic hardware (which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry)), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The computing device may further include a display (as an example of or supplement to an output device), a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive Internet Protocol (IP)-based data or other types of data.

[0134] Method 700 is interpreted as a logic flowchart, which represents a sequence of operations that can be implemented by hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that performs the described operation when executed by one or more processors. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of described operations can be combined and / or performed in parallel in any order to implement the process.

[0135] Additionally, method 700 can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, executed by hardware or a combination thereof. As mentioned above, the code can be stored on a computer-readable or machine-readable storage medium, for example in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transient.

[0136] Figure 8 An example computing device architecture 800 is described, illustrating an example computing device capable of implementing the various technologies described herein. For example, computing device architecture 800 can implement... Figure 1The main processing system 100 shown comprises at least some portions and performs ToF signal processing operations as described herein. Components of the computing device architecture 800 are shown to be in electrical communication with each other via a connection 805 (such as a bus). The example computing device architecture 800 includes a processing unit (CPU or processor) 810 and computing device connections 805 that couple various computing device components, including computing device memories 815 (such as read-only memory (ROM) 820 and random access memory (RAM) 825), to the processor 810.

[0137] The computing device architecture 800 may include a cache of high-speed memory that is directly connected to, adjacent to, or integrated into the processor 810. The computing device architecture 800 may copy data from memory 815 and / or storage device 830 to cache 812 for fast access by the processor 810. In this way, the cache provides a performance boost, preventing latency for the processor 810 while waiting for data. These and other modules may control or be configured to control the processor 810 to perform various actions. Other computing device memory 815 may also be available. Memory 815 may include various different types of memory with different performance characteristics. The processor 810 may include any general-purpose processor and hardware or software services stored in storage device 830 and configured to control the processor 810, as well as dedicated processors (where software instructions are incorporated into the processor design). The processor 810 may be a self-contained system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0138] To enable user interaction with the computing device architecture 800, input device 845 can represent any number of input mechanisms, such as a microphone for voice, a touchscreen for gesture or graphic input, a keyboard, a mouse, motion input, voice input, etc. Output device 835 can also be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, projector, television, speaker device. In some instances, multimodal computing devices enable users to provide multiple types of input to communicate with computing device architecture 800. Communication interface 840 generally manages and controls user input and computing device output. There are no limitations on operation on any particular hardware arrangement, and therefore the underlying features here can be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0139] Storage device 830 is a non-volatile memory and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as magnetic tape, flash memory cards, solid-state storage devices, digital multifunction disks, cassette disks, random access memory (RAM) 825, read-only memory (ROM) 820, or mixtures thereof. Storage device 830 may include software, code, firmware, etc., for controlling processor 810. Other hardware or software modules are envisioned. Storage device 830 may be connected to computing device connection 805. In one aspect, a hardware module performing a specific function may include software components stored in a computer-readable medium connected to necessary hardware components (such as processor 810, connection 805, output device 835, etc.) to perform the function.

[0140] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transient media in which data can be stored and excluding transient electronic signals propagated via carrier waves and / or wirelessly or via wired connections. Examples of non-transient media include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as compact discs (CDs) or digital multifunction (DVDs)), flash memory, memory, or memory devices. Computer-readable media may have code and / or machine-executable instructions stored thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or transmitted via any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0141] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transient computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0142] Specific details are provided in the foregoing description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that these embodiments can be practiced without these specific details. For clarity, in some instances, the technology of the invention may be presented as including various functional blocks that include devices, device components, steps or routines in methods implemented in software or a combination of hardware and software. Additional components may be used in addition to those shown in the drawings and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring these embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without the need for unnecessary detail to avoid obscuring the embodiments.

[0143] The various embodiments described above may be process or method, depicted as flowcharts, diagrams, data flow graphs, structural diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many operations may be performed in parallel or concurrently. Furthermore, the order of operations may be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the figures. A process may correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0144] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise available from a computer-readable medium. These instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. Parts of the computer resources used are accessible via a network. The computer-executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, or source code. Examples of computer-readable media that can be used to store instructions, information used during the methods according to the described examples, and / or information created include disks or optical discs, flash memory, USB devices provided with non-volatile memory, networked storage devices, etc.

[0145] Devices implementing the various processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include: laptop computers, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mount devices, self-standing devices, etc. The functionality described herein may also be implemented using peripheral devices or plug-in cards. As a further example, such functionality may also be implemented on a circuit board within different chips or different processes executed on a single device.

[0146] Instructions, media for conveying these instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0147] In the foregoing description, aspects of this application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative embodiments of this application have been described in detail herein, it is to be understood that the various inventive concepts may be implemented and employed in a variety of other ways, and the appended claims are not intended to be construed as including these variations unless limited by prior art. Various features and aspects of the foregoing applications may be used individually or in combination. Furthermore, the embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Accordingly, this specification and the accompanying drawings should be considered illustrative rather than limiting. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in a different order than described.

[0148] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by the less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of this specification.

[0149] When the components are described as being “configured” to perform certain operations, such configurations can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors, or other suitable electronic circuits), or any combination thereof.

[0150] The phrase “coupled to” means that any component is physically connected directly or indirectly to another component, and / or that any component is in communication with another component directly or indirectly (e.g., connected to that other component via a wired or wireless connection and / or other suitable communication interface).

[0151] The language of the claims or other languages ​​that use "at least one" and / or "one or more" in a set of statements indicate that one or more members of that set (in any combination) satisfy the claim. For example, the claim language that states "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, the claim language that states "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language of the set "at least one" and / or the set "one or more" does not limit the set to the items listed in that set. For example, the claim language that states "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0152] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the examples disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0153] The techniques described herein can also be implemented using electronic hardware, computer software, firmware, or any combination thereof. These techniques can be implemented using any of a variety of devices, such as general-purpose computers, wireless communication handsets, or multi-purpose integrated circuit devices, including applications in wireless communication handsets and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code, including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium can form part of a computer program product and may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. These technologies may additionally or alternatively be implemented, at least in part, by computer-readable communication media carrying or conveying program code in the form of instructions or data structures that can be accessed, read, and / or executed by a computer, such as propagated signals or waves.

[0154] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration. Accordingly, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein.

Claims

1. A method for visual tracking, comprising: Receive one or more images captured by at least one image sensor system on a first device, the one or more images depicting a pattern set with a predetermined configuration on a second device and at least one reference line between the pattern set, wherein the first device has lower power requirements compared to the second device; A set of pixels corresponding to the pattern set on the second device is determined based on one or more images captured by the at least one image sensor system; The reference position of at least one reference line between the pattern sets is determined based on a reference coordinate system; Determine the observed position of the at least one reference line drawn in the one or more images relative to the image coordinate system associated with the one or more images; Based on the pixel set corresponding to the pattern set on the second device, the position and relative pose of each pattern in the pattern set in space are determined, wherein determining the position and relative pose of each pattern in the pattern set in space includes: The 3D position of the pattern set is determined at least in part by translating the three-dimensional 3D coordinates of the pixel set corresponding to the pattern set based on a comparison between the reference position of the at least one reference line and the observed position of the at least one reference line; and The 3D orientation of the pattern set is determined at least in part by rotating the 3D coordinates of the pixel set corresponding to the pattern set based on a comparison between the reference position of the at least one reference line and the observed position of the at least one reference line; and The attitude of the first device relative to the second device is determined based on the position and relative orientation of each pattern in the pattern set in space.

2. The method of claim 1, wherein the at least one image sensor system comprises an image sensor system set, and wherein determining the pose of the first device relative to the second device is further based on a predetermined relative position and orientation of the image sensor system set on the first device.

3. The method of claim 2, further comprising: Determine the 3D coordinates of one or more points in space associated with the scene captured by the image sensor system suite; as well as The relative 3D pose of the image sensor system assembly on the first device is determined based on the predetermined relative position and orientation of the image sensor system assembly and the 3D coordinates of the one or more points in space, wherein the pose of the first device relative to the second device is further based on the relative 3D pose of the image sensor system assembly on the first device.

4. The method of claim 1, wherein the attitude of the first device relative to the second device includes a six-degree-of-freedom (6DoF) attitude, and wherein the predetermined configuration of the pattern set includes at least one of the following: the relative position of each pattern on the second device, the relative orientation of each pattern on the second device, the shape of each pattern, the size of each pattern, and the arrangement of the pattern set.

5. The method of claim 1, wherein determining the attitude of the first device relative to the second device further comprises determining the attitude of the second device relative to the first device, wherein the attitude of the first device includes a first 6DoF attitude and the attitude of the second device includes a second 6DoF attitude.

6. The method of claim 1, wherein the first device includes a hand controller device and the second device includes a head-mounted display device.

7. The method of claim 1, wherein the at least one image sensor system comprises a low-power image sensor system, and wherein each pattern in the pattern set is visible in at least one of the infrared spectrum and the visible light spectrum.

8. The method of claim 1, wherein at least one pattern in the pattern set includes encoded machine-readable information, the encoded machine-readable information including at least one of: location information associated with the at least one pattern, an identifier associated with the at least one pattern, a unique code, application settings, and information about a user account associated with an extended reality application on main memory of at least one of the first device and the second device.

9. The method of claim 1, wherein determining the pixel set corresponding to the pattern set on the second device comprises: Each pattern from the pattern set on the second device is detected based on one or more images captured by the at least one image sensor system; as well as Identify one or more points from each pattern in the pattern set, wherein the one or more points correspond to one or more pixels from the pixel set. The determination of the 3D position and relative orientation of each pattern in space is based on one or more points in each pattern.

10. The method of claim 1, wherein determining the pixel set corresponding to the pattern set on the second device comprises: Each pattern from the pattern set on the second device is detected based on the one or more images captured by the at least one image sensor system; For each pattern, detect the smaller internal patterns; as well as Identify one or more points in each smaller internal pattern, wherein the one or more points correspond to one or more pixels from the pixel set. The determination of the 3D position and relative orientation of each pattern in space is based on one or more points in each smaller internal pattern.

11. The method of claim 1, wherein determining the attitude of the first device relative to the second device includes determining the 6DoF attitude of the first device relative to the second device and the 6DoF attitude of the second device relative to the first device, the method further comprising: Detect one or more additional patterns on objects captured in the one or more additional images based on one or more additional images captured by the at least one image sensor system; Determine one or more pixels in the one or more additional images that correspond to the one or more additional patterns on the object; as well as An additional 6DoF pose of the object relative to at least one of the 6DoF poses of the first device and the 6DoF pose of the second device is determined based on the one or more pixels.

12. The method of claim 11, wherein the object includes a wall, a display device, a video game console, furniture, or an appliance.

13. An apparatus comprising: Memory; as well as One or more processors coupled to the memory, the one or more processors being configured to: Receive one or more images captured by at least one image sensor system on the device, the one or more images depicting a pattern set having a predetermined configuration on a device and at least one reference line between the pattern set, wherein the device has lower power requirements compared to the device; A set of pixels corresponding to the pattern set on the device is determined based on one or more images captured by the at least one image sensor system; The reference position of at least one reference line between the pattern sets is determined based on a reference coordinate system; Determine the observed position of the at least one reference line drawn in the one or more images relative to the image coordinate system associated with the one or more images; Based on the pixel set corresponding to the pattern set on the device, the spatial position and relative pose of each pattern in the pattern set are determined, wherein the one or more processors configured to determine the spatial position and relative pose of each pattern in the pattern set are further configured to: The 3D position of the pattern set is determined at least in part by translating the three-dimensional 3D coordinates of the pixel set corresponding to the pattern set based on a comparison between the reference position of the at least one reference line and the observed position of the at least one reference line. as well as The 3D orientation of the pattern set is determined at least in part by rotating the 3D coordinates of the pixel set corresponding to the pattern set based on a comparison between the reference position of the at least one reference line and the observed position of the at least one reference line. as well as The orientation of the device relative to the equipment is determined based on the position and relative orientation of each pattern in the pattern set in space.

14. The apparatus of claim 13, wherein the at least one image sensor system comprises an image sensor system set, and wherein determining the orientation of the apparatus relative to the device is further based on a predetermined relative position and orientation of the image sensor system set on the apparatus.

15. The apparatus of claim 14, wherein the one or more processors are configured to: Determine the 3D coordinates of one or more points in space associated with the scene captured by the image sensor system assembly; and The relative 3D pose of the image sensor system assembly is determined based on a predetermined relative position and orientation of the image sensor system assembly and the 3D coordinates of one or more points in space, wherein the pose of the device relative to the equipment is further based on the relative 3D pose of the image sensor system assembly.

16. The apparatus of claim 13, wherein the attitude of the apparatus relative to the device comprises a six-degree-of-freedom (6DoF) attitude, and wherein the predetermined configuration of the pattern set comprises at least one of the following: the relative position of each pattern on the device, the relative orientation of each pattern on the device, the shape of each pattern, the size of each pattern, and the arrangement of the pattern set.

17. The apparatus of claim 13, wherein determining the attitude of the apparatus relative to the device further comprises determining the attitude of the device relative to the apparatus, wherein the attitude of the apparatus includes a first 6DoF attitude and the attitude of the device includes a second 6DoF attitude.

18. The apparatus of claim 13, wherein the apparatus is a hand controller device and the apparatus is a head-mounted display device.

19. The apparatus of claim 13, wherein the at least one image sensor system comprises a low-power image sensor system, and wherein each pattern in the pattern set is visible in at least one of the infrared spectrum and the visible light spectrum.

20. The apparatus of claim 13, wherein at least one pattern in the pattern set includes encoded machine-readable information, the encoded machine-readable information including at least one of: location information associated with the at least one pattern, an identifier associated with the at least one pattern, a unique code, application settings, and information about a user account associated with an extended reality application on main memory of at least one of the apparatus and the device.

21. The apparatus of claim 13, wherein determining the pixel set corresponding to the pattern set on the device comprises: Each pattern from the pattern set on the device is detected based on one or more images captured by the at least one image sensor system; as well as Identify one or more points from each pattern in the pattern set, wherein the one or more points correspond to one or more pixels from the pixel set. The determination of the 3D position and relative orientation of each pattern in space is based on one or more points in each pattern.

22. The apparatus of claim 13, wherein determining the pixel set corresponding to the pattern set on the device comprises: Each pattern from the pattern set on the device is detected based on one or more images captured by the at least one image sensor system; For each pattern, detect the smaller internal patterns; as well as Identify one or more points in each smaller internal pattern, wherein the one or more points correspond to one or more pixels from the pixel set. The determination of the 3D position and relative orientation of each pattern in space is based on one or more points in each smaller internal pattern.

23. The apparatus of claim 13, wherein determining the attitude of the apparatus relative to the device includes determining the 6DoF attitude of the apparatus relative to the device and the 6DoF attitude of the device relative to the apparatus, wherein the one or more processors are configured to: Detect one or more additional patterns on objects captured in the one or more additional images based on one or more additional images captured by the at least one image sensor system; Determine one or more pixels in the one or more additional images that correspond to the one or more additional patterns on the object; as well as An additional 6DoF pose of the object relative to the device and at least one of the 6DoF poses of the device are determined based on the one or more pixels.

24. The apparatus of claim 23, wherein the object includes a wall, a display device, a video game console, furniture, or an appliance.

25. The apparatus of claim 13, wherein the apparatus is a mobile device.

26. A non-transient computer-readable storage medium, comprising: Instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: Receive one or more images captured by at least one image sensor system on a first device, the one or more images depicting a pattern set with a predetermined configuration on a second device and at least one reference line between the pattern set, wherein the first device has lower power requirements compared to the second device; A set of pixels corresponding to the pattern set on the second device is determined based on one or more images captured by the at least one image sensor system; The reference position of at least one reference line between the pattern sets is determined based on a reference coordinate system; Determine the observed position of the at least one reference line drawn in the one or more images relative to the image coordinate system associated with the one or more images; Based on the pixel set corresponding to the pattern set on the second device, the position and relative orientation of each pattern in the pattern set in space are determined, wherein instructions executed by the one or more processors that cause the one or more processors to determine the position and relative orientation of each pattern in the pattern set in space further cause the one or more processors to: The 3D position of the pattern set is determined at least in part by translating the three-dimensional 3D coordinates of the pixel set corresponding to the pattern set based on a comparison between the reference position of the at least one reference line and the observed position of the at least one reference line. as well as The 3D orientation of the pattern set is determined at least in part by rotating the 3D coordinates of the pixel set corresponding to the pattern set based on a comparison between the reference position of the at least one reference line and the observed position of the at least one reference line. as well as The attitude of the first device relative to the second device is determined based on the position and relative orientation of each pattern in the pattern set in space.

Images