Tracking system and method

Abstract

The invention provides a tracking system and method. The head-mounted display device is suitable for being worn on the head of a user and comprises a camera and a processor. The camera obtains a body image including a body part of a user. The processor is configured to: determine a relative position between a body motion sensor of the body tracker and a body part of a user based on the body image; and performing target tracking of the body part based on the relative position and sensor data of a body motion sensor of the body tracker. The body tracker is adapted to be worn on a body part of a user and includes a body motion sensor. The body motion sensor obtains sensor data. The sensor data is indicative of movement and / or rotation of the body part.

Description

Technical Field

[0001] The present disclosure relates to a tracking system; in particular, the present disclosure relates to a tracking system and a tracking method. Background Art

[0002] In order to bring an immersive experience to users, technologies related to extended reality (XR), such as augmented reality (AR), virtual reality (VR) and mixed reality (MR), are being developed. AR technology enables users to bring virtual elements into the real world. VR technology enables users to enter a new virtual world to experience a different life. MR technology merges the real world with the virtual world. In addition, in order to bring a complete immersive experience to users, visual content, audio content or other sensory content can be provided through one or more devices. Summary of the invention

[0003] The present disclosure relates to a tracking system and a tracking method so as to more accurately track a user's body parts.

[0004] In the present disclosure, a tracking system is provided. The tracking system includes a head-mounted display device and a body tracker. The head-mounted display device is suitable for being worn on the head of a user and includes a camera and a processor. The camera is configured to obtain a body image including a body part of the user. The processor is configured to: determine the relative position between a body motion sensor of a body tracker and the body part of the user based on the body image; and perform target tracking of the body part based on the relative position and sensor data of the body motion sensor of the body tracker. The body tracker is suitable for being worn on the body part of the user and includes a body motion sensor. The body motion sensor is configured to obtain the sensor data. The sensor data is configured to indicate movement and / or rotation of the body part.

[0005] In the present disclosure, a tracking method is provided. The tracking method includes: obtaining a body image including a body part of a user; determining a relative position between a body motion sensor of a body tracker and the body part of the user based on the body image, wherein the body tracker is suitable for being worn on the body part of the user; obtaining sensor data of the body motion sensor of the body tracker, wherein the sensor data is configured to indicate movement and / or rotation of the body part; and performing target tracking of the body part based on the relative position and the sensor data.

[0006] Based on the above, according to the tracking system and tracking method, even if the body tracker is not correctly worn on the user's body part, the processor can still accurately track the body part based on the relative position between the body motion sensor and the body part, thereby improving the user experience.

[0007] In order to make the foregoing content easier to understand, several embodiments attached with the drawings will be described in detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present disclosure includes accompanying drawings to provide a further understanding of the present disclosure, and the accompanying drawings are incorporated in and constitute a part of this specification. The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure.

[0009] Figure 1A is a schematic diagram of a tracking system according to an embodiment of the present disclosure.

[0010] Figure 1B is a schematic diagram of a tracking system according to an embodiment of the present disclosure.

[0011] Figure 2 is a schematic diagram of a tracking scenario according to an embodiment of the present disclosure.

[0012] Figure 3A is a schematic diagram of a tracking scenario according to an embodiment of the present disclosure.

[0013] Figure 3B is a schematic diagram of a tracking scenario according to an embodiment of the present disclosure.

[0014] Figure 4A is a schematic diagram of a tracking scenario according to an embodiment of the present disclosure.

[0015] Figure 4B is a schematic diagram of a tracking scenario according to an embodiment of the present disclosure.

[0016] Figure 4C is a schematic diagram of a tracking scenario according to an embodiment of the present disclosure.

[0017] Figure 5A is a schematic diagram of a tracking scenario according to an embodiment of the present disclosure.

[0018] Figure 5B is a schematic diagram of a tracking scenario according to an embodiment of the present disclosure.

[0019] Figure 5C is a schematic diagram of a tracking scenario according to an embodiment of the present disclosure.

[0020] Figure 6 is a schematic flowchart of a tracking method according to an embodiment of the present disclosure.

[0021] [Explanation of Symbols]

[0022] 100A, 100B: Tracking system

[0023] 110: Head-mounted display device

[0024] 112: Display

[0025] 114, 124: Camera

[0026] 116: Processor

[0027] 118: Head motion sensor

[0028] 120: Body Tracker

[0029] 122: Body motion sensor

[0030] 200, 300A, 300B, 400A, 400B, 400C, 500A, 500B, 500C: Tracking scenarios

[0031] 201, 202, 203, 204, 205: Mobile

[0032] 600: Tracking Methods

[0033] BD: Body Parts

[0034] D1, D2, D3: horizontal distance

[0035] L1, L2, L_LA, L_O, L_RA: Length

[0036] L_S: Shoulder width

[0037] P_A: Actual wearing position

[0038] P_I: Ideal wearing position

[0039] S610, S620, S630, S640: Steps

[0040] T1, T2: thickness

[0041] U: User

[0042] VW, VW_L, VW_R: Virtual walls DETAILED DESCRIPTION

[0043] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and the description to refer to the same or like components.

[0044] Certain terms are used throughout the specification and the appended claims of the present disclosure to refer to specific components. It will be understood by those skilled in the art that electronic device manufacturers may refer to the same components by different names. It is not intended herein to distinguish between those components having the same functions but different names. In the following description and claims, words such as "comprise" and "include" are open-ended terms and should be interpreted as "including but not limited to...". The terms "first", "second" and similar terms mentioned throughout the entire specification (including the appended claims) of the present application are only used to name discrete elements or to distinguish between different embodiments or scopes. Therefore, the terms should not be regarded as limiting the upper or lower limits of the number of elements and should not be used to limit the order of arrangement of elements.

[0045] Using a body tracking device, a user's body parts can be tracked. For example, the body tracking device may include an accelerometer and / or a gyroscope. The user can wear the body tracking device on the body part so that the body tracking device can track the body part based on the detection results of the accelerometer and / or the gyroscope. However, when the user wears the body tracking device, the body tracking device may not be worn correctly. For example, the body tracking device may be worn incorrectly or may fall off during an activity. In addition, personal factors (such as the thickness of the wrist, ankle, or clothing) or wearing differences (due to different designs of wristbands and bracelets) may also lead to improper wearing of the body tracking device. Therefore, the detection results may be inaccurate, thereby reducing the user experience. For example, the avatar representing the user may twist and turn unnaturally. Therefore, a technician in the field seeks to provide a tracking method for accurately tracking a user's body parts.

[0046] Figure 1A is a schematic diagram of a tracking system according to an embodiment of the present disclosure. Figure 1A The tracking system 100A may include a head-mounted display (HMD) device 110 and a body tracker 120. The HMD device 110 may include a display 112, a camera 114, and a processor 116. The body tracker 120 may include a body motion sensor 122.

[0047] The HMD device 110 may be adapted to be worn on a user's head. The body tracker 120 may be adapted to be worn on a body part of the user. The body motion sensor 122 may be configured to obtain sensor data. The sensor data may be configured to indicate movement and / or rotation of a body part of the user. The display 112 may be configured to display a virtual world. The camera 114 may be configured to obtain a body image including a body part of the user. The processor 116 may be configured to determine a relative position between the body motion sensor 122 of the body tracker 120 and the body part of the user based on the body image. In addition, the processor 116 may be configured to perform target tracking of the body part based on the relative position and the sensor data of the body motion sensor 122 of the body tracker 120.

[0048] In one embodiment, the processor 116 may be configured to perform object detection using an object detection algorithm to identify the body part and the body motion sensor 122 in the body image. That is, a pre-stored image including the shape of the body part and / or the body motion sensor 122 may be pre-stored in the memory of the HMD device 110. The processor 116 may be configured to determine the relative position between the body motion sensor 122 of the body tracker 120 and the body part of the user based on the body image and the pre-stored image.

[0049] In one embodiment, since the relative position has been determined, the processor 116 can be configured to perform target tracking without a body image. That is, the processor 116 can be configured to determine the movement of the body part based on the sensor data of the body motion sensor 122. Therefore, when the body part is not in the field of view (FOV) of the camera 114, the processor 116 can still perform target tracking based on the relative position and sensor data without a body image.

[0050] Figure 1B is a schematic diagram of a tracking system according to an embodiment of the present disclosure. Figure 1B , the tracking system 100B may include an HMD device 110 and a body tracker 120. The HMD device 110 may include a display 112, a camera 114, a processor 116, and a head motion sensor 118. The body tracker 120 may include a body motion sensor 122 and a camera 124. The camera 114 of the HMD device 110 may also be referred to as a device camera, and the camera 124 of the body tracker 120 may also be referred to as a tracker camera.

[0051] In one embodiment, the processor 116 may be configured to obtain a simultaneous localization and mapping (SLAM) map of the user's surroundings. The processor 116 may generate a SLAM map based on the image obtained by the camera 114 using a SLAM algorithm. On the other hand, the SLAM map may be obtained from other devices or pre-stored in the memory of the HMD device 110. That is, the present disclosure does not limit how the processor 116 obtains the SLAM map. In addition, the processor 116 may be configured to determine the device coordinates of the HMD device 110 in the SLAM map.

[0052] In addition, the camera 124 of the body tracker 120 may be configured to obtain a feature image including the surrounding environment of the body part. The processor 116 may be configured to determine the tracker coordinates of the body tracker 120 and the sensor coordinates of the body motion sensor 122 of the body tracker 120 in the SLAM map based on the feature image. That is, the HMD device 110 and the body tracker 120 may be configured to share the SLAM map so that the body tracker 120 can know its own position in the SLAM map. In addition, the processor 116 may be configured to determine the relative position between the body motion sensor 122 of the body tracker 120 and the body part of the user based on the tracker coordinates and the sensor coordinates.

[0053] In one embodiment, since the relative position has been determined, the processor 116 can be configured to perform target tracking without a body image. That is, the processor 116 can be configured to determine the movement of the body part based on the sensor data of the body motion sensor 122 (also referred to as the tracker sensor data) or the feature image of the camera 124. Therefore, when the body part is not in the field of view (FOV) of the camera 114, the processor 116 can still perform target tracking based on the relative position and sensor data without a body image. Similarly, the processor 116 can be configured to determine the movement of the user's head based on the sensor data of the head motion sensor 118 (also referred to as the device sensor data).

[0054] In one embodiment, the HMD device 110 may be configured to display AR, VR, MR, or XR content. The HMD device 110 may include, for example, a headset, wearable glasses (e.g., AR / VR goggles), other similar devices suitable for AR, VR, MR, XR, or other reality-related technologies, or a combination of these devices. However, the present disclosure is not limited thereto.

[0055] In one embodiment, the display 112 may include, for example, an organic light-emitting diode (OLED) display device, a mini LED display device, a micro LED display device, a quantum dot (QD) LED display device, a liquid crystal display (LCD) display device, a tiled or foldable display device, or an electronic paper display (EPD). However, the present disclosure is not limited thereto.

[0056] In one embodiment, the camera 114 and / or the camera 124 may include, for example, a complementary metal oxide semiconductor (CMOS) camera, a charge coupled device (CCD) camera, a light detection and ranging (LiDAR) device, a radar, an infrared sensor, an ultrasonic sensor, other similar devices, or a combination of these devices. However, the present disclosure is not limited thereto.

[0057] In one embodiment, the body tracker 120 may include a tracker processor (not shown). The tracker processor and / or the processor 116 may include, for example, a microcontroller unit (MCU), a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a programmable controller, a programmable logic device (PLD), other similar devices, or a combination of these devices. In addition, in an embodiment, the functions of the processor 116 may be implemented as multiple program codes. The program code is stored in a memory and executed by the processor 116. Alternatively, in an embodiment, the functions of the processor 116 may be implemented as one or more circuits. The present disclosure does not limit whether software or hardware is used to implement the functions of the processor 116.

[0058] In one embodiment, the head motion sensor 118 and / or the body motion sensor 122 may include, for example, a gyroscope, an accelerometer, other similar devices, or a combination of these devices. However, the present disclosure is not limited thereto. In one embodiment, the head motion sensor 118 and / or the body motion sensor 122 may be a gyroscope, and the sensor data may include three angular velocities in three degrees of freedom (DOF). The three angular velocities may include a roll angular velocity around the X-axis, a pitch angular velocity around the Y-axis, and a yaw angular velocity around the Z-axis. In one embodiment, the head motion sensor 118 and / or the body motion sensor 122 may be an accelerometer, and the sensor data may include three linear acceleration values ​​in the three degrees of freedom. The three linear acceleration values ​​may include a first acceleration value along the X-axis, a second acceleration value along the Y-axis, and a third acceleration value along the Z-axis. In one embodiment, the head motion sensor 118 and / or the body motion sensor 122 may include a gyroscope and an accelerometer, and the sensor data may include changes in six degrees of freedom. The changes in the six degrees of freedom include three angular velocities and three linear acceleration values ​​corresponding to three perpendicular axes (eg, X-axis, Y-axis, and Z-axis).

[0059] In one embodiment, each of the HMD device 110 and the body tracker 120 may include a network module, and the network module may include, for example, a wired network module, a wireless network module, a Bluetooth module, an infrared module, a radio frequency identification (RFID) module, a Zigbee network module, or a near field communication (NFC) network module, but the present disclosure is not limited thereto. That is, the HMD device 110 may be configured to communicate with the body tracker 120 via wired communication or wireless communication.

[0060] In some embodiments, the HMD device 110 and / or the body tracker 120 may further include a memory. In one embodiment, the memory may include, for example, a NAND flash memory core, a NOR flash memory core, a static random access memory (SRAM) core, a dynamic random access memory (DRAM) core, a magnetoresistive random access memory (MRAM) core, a phase change memory (PCM) core, a resistive random access memory (ReRAM) core, a three-dimensional (3D) XPoint memory core, a ferroelectric random access memory (FeRAM) core, and other types of memory cores suitable for storing data. However, the present disclosure is not limited thereto.

[0061] Figure 2 is a schematic diagram of a tracking scene according to an embodiment of the present disclosure. Figure 1A to Figure 2 , the tracking scene 200 depicts some key nodes of the body of the user U. The key nodes may be the right wrist, the left wrist, the waist, the right ankle, and the left ankle. However, the present disclosure is not limited thereto. By detecting the movement 201 to 205 of the key nodes, the movement of the user U may be determined.

[0062] In one embodiment, the body tracker 120 may be suitable for being worn on or near one of the key nodes. That is, a body part may be one of the key nodes. In other words, the body tracker 120 may include, for example, a wristband, a glove, a gauntlet, a ring, a palmband, a belt, an ankle brace, a sock, other similar devices, or a combination of these devices. The processor 116 may be able to collect information about the key nodes of the body of the user U based on the sensor data of the body tracker 120.

[0063] In this way, when the user U is moving in the real world, the avatar in the virtual world displayed by the HMD device 110 may move based on the sensor data of the body tracker 120. That is, the display 112 of the HMD device 110 may be configured to display the virtual body parts of the avatar of the user U based on target tracking. Therefore, the avatar may reflect the movement of the user U in the real world.

[0064] Figure 3A is a schematic diagram of a tracking scenario according to an embodiment of the present disclosure. Figure 3Bis a schematic diagram of a tracking scene according to an embodiment of the present disclosure. Figures 1A to 3B , tracking scenes 300A and 300B illustrate how the body tracker 120 may be worn on a body part BD of a user U.

[0065] In one embodiment, the body tracker 120 may be suitable for being worn on a body part BD of the user U. Ideally, when the body tracker 120 is worn on the body part BD, the actual wearing position P_A of the body motion sensor 122 should be aligned with the ideal wearing position P_I on the body part BD. For example, the body part BD may be the wrist of the user U, and the body tracker 120 may be a wristband. The ideal wearing position P_I may be the center point of the wrist (e.g., the intersection of the wrist and the extension line of the middle finger) or the surface of the wrist joint. However, the present disclosure is not limited to this. The ideal wearing position P_I on the body part BD may be pre-stored in the memory of the body tracker 120 or determined by the user U. That is, the processor 116 may be configured to obtain the ideal wearing position P_I on the body part BD from the memory.

[0066] like Figure 3A As shown in , the actual wearing position P_A of the body motion sensor 122 is aligned with the ideal wearing position P_I on the body part BD. In one embodiment, the processor 116 may be configured to determine the actual wearing position P_A based on the body image. Therefore, the sensor data of the body motion sensor 122 may accurately reflect the movement of the body part BD. On the other hand, Figure 3B As shown in , the actual wearing position P_A of the body motion sensor 122 is not aligned with the ideal wearing position P_I on the body part BD. Therefore, the sensor data of the body motion sensor 122 may not accurately reflect the movement of the body part BD. In other words, due to the relative position between the actual wearing position P_A and the ideal wearing position P_I, the processor 116 may not accurately perform target tracking of the body part BD.

[0067] In order to accurately implement the target tracking of the body part BD, the processor 116 may be configured to determine the relative position between the actual wearing position P_A and the ideal wearing position P_I based on the body image captured by the camera 114. In this way, even if the body tracker 120 is not correctly worn on the body part BD of the user U, the processor 116 can still accurately track the body part BD based on the relative position between the body motion sensor 122 and the body part BD. Therefore, the user experience is improved.

[0068] Figure 4A is a schematic diagram of a tracking scene according to an embodiment of the present disclosure. Figures 1A to 4A, the tracking scene 400A depicts an L-shaped gesture as a calibration gesture for determining the relative position between the body motion sensor 122 and the body part BD.

[0069] like Figure 4A As shown in , for the purpose of calibration, the user U may stretch his hands forward to make an L-shaped pose, so that the body shape of the user U looks like an inverted L. That is, the processor 116 may be configured to determine the relative position based on the calibration pose of the user U. By the user U making the L-shaped pose, the body motion sensor 122 of the body tracker 120 (i.e., wristband) and the body part BD (i.e., wrist) may be in the FOV of the camera 114 of the HMD device 110. Therefore, the processor 116 is able to determine the relative position between the body motion sensor 122 and the body part BD based on the body image captured by the camera 114.

[0070] In addition, when the user U uses both hands to make a calibration pose, some body parameters may be obtained at the same time. For example, when the user U makes an L-shaped pose, the wrist and arm may be in the FOV of the camera 114 of the HMD device 110. Therefore, the processor 116 may be configured to determine the arm length of the user U and the distance between the two arms of the user U based on the body image captured by the camera 114. Based on the distance between the two arms of the user U, the shoulder width may be obtained. After obtaining the body parameters, the avatar of the user U may be able to reflect the actions of the user U in a more realistic manner.

[0071] Figure 4B is a schematic diagram of a tracking scenario according to an embodiment of the present disclosure. Figure 4C is a schematic diagram of a tracking scene according to an embodiment of the present disclosure. Figure 4B and Figure 4C , the tracking scene 400B depicts a side view of the user U posing an L-shaped posture in the virtual world as a calibration posture, and the tracking scene 400C depicts a top view of the user U posing an L-shaped posture in the virtual world as a calibration posture. Figure 4B and Figure 4C In the example, the display 112 may display a virtual wall VW in the virtual world. The function of the virtual wall VW is to help the user U to make an L-shaped pose as a calibration pose. Figure 4B and Figure 4C A virtual wall VW and a user U are depicted together, and the user U is actually in the real world, and the virtual wall VW is actually in the virtual world.

[0072] Reference Figure 4B, a virtual wall VW may be displayed to the user U, and the user U may be guided to push the virtual wall VW. The user U may use both hands to make a pushing motion in the real world, and the avatar of the user U may push the virtual wall VW in the virtual world. In order to make it easier for the user U to push and make more accurate measurements, the initial position of the virtual wall VW may be close to the user U, or more precisely, close to the body tracker 120. For example, if the initial position of the virtual wall VW is too far from the user U, the user U may not be able to touch the virtual wall VW and the calibration may fail. If the initial position of the virtual wall VW is closer, the user U is more likely to touch the virtual wall VW and push the virtual wall VW away from the user U. In one embodiment, a virtual palm may be displayed on the virtual wall VW, and the user U may put his hand on the virtual palm. Based on the height of the user U (e.g., one sixth of the height of the user U), the position (e.g., height) of the virtual palm on the virtual wall VW may be adjusted. In one embodiment, the size of the virtual palm or the relative position to the user U may be adjusted based on the size of the hand of the user U or the relative distance to the user U during the push.

[0073] In one embodiment, when the user U wears the HMD device 110, the camera 114 may be configured to obtain an image, and the processor 116 may be configured to determine, based on the image, the height of the eyes of the user U or the height of the user U. The height of the eyes of the user U may be obtained based on the height of the HMD device 110. However, the present disclosure is not limited thereto.

[0074] When the user U pushes the virtual wall VW, the body motion sensor 122 may be configured to obtain sensor data. The processor 116 may be configured to determine the horizontal distance between the body motion sensor 122 and the HMD device 110 based on the sensor data of the body motion sensor 122. In addition, the processor 116 may be configured to determine the arm length of the user based on the horizontal distance between the body motion sensor 122 and the HMD device 110. For example, the user U may be instructed to push the virtual wall VW away from the HMD device 110 with his hand, and the body motion sensor 122 may be configured to obtain sensor data. When the horizontal distance between the body motion sensor 122 and the HMD device 110 is the farthest, the processor 116 may be configured to determine the arm length of the user U based on the sensor data of the body motion sensor 122. It should be noted that the user U may push the virtual wall VW farther if he uses only one hand, so it is more accurate to use both hands to push the virtual wall VW to determine the arm length of the user U. In addition, the user U may use the wrist instead of the palm to push the virtual wall VW. Therefore, it may be determined based on the body image captured by the camera 114 whether the user U is not pushing the virtual wall VW with the palm, and the body motion sensor 122 may be configured not to obtain sensor data simultaneously.

[0075] Similarly, refer to Figure 4C , the user U may wear a body tracker 120 (also referred to as a first body tracker) on the left wrist and a body tracker 120 (also referred to as a second body tracker) on the right wrist. The processor 116 may be configured to determine the distance between the two arms of the user U or the shoulder width L_S of the user U based on sensor data from the body motion sensors 122 in the body tracker 120 on the left wrist and the body tracker 120 on the right wrist. At the same time, the body motion sensor 122 may be configured to obtain sensor data. When the horizontal distances between all the body motion sensors 122 and the HMD device 110 reach the farthest, the processor 116 may be configured to determine the shoulder width L_S based on the sensor data of the body motion sensors 122.

[0076] Figure 5A is a schematic diagram of a tracking scene according to an embodiment of the present disclosure. Figures 1A to 5A , the tracking scene 500A depicts a T-pose as a calibration pose for determining the relative position between the body motion sensor 122 and the body part BD.

[0077] like Figure 5A As shown in , for the purpose of calibration, the user U may place both hands on both sides of the user U to pose in a T-shape, so that the body shape of the user U looks like the letter T. Therefore, when the user U looks at the left hand or the right hand, the body motion sensor 122 of the body tracker 120 (i.e., the wristband) and the body part BD (i.e., the wrist) may be in the FOV of the camera 114 of the HMD device 110. Therefore, the processor 116 is able to determine the relative position between the body motion sensor 122 and the body part BD based on the body image captured by the camera 114.

[0078] In addition, when the user U poses in the calibration posture, some body parameters can be obtained at the same time. For example, when the user U poses in the T-pose, the processor 116 can be configured to determine the arm length of the user U and the distance between the two wrists of the user U based on the body image captured by the camera 114. Based on the distance between the two wrists of the user U, the length of the open arm can be obtained. In addition, the body parameters can be refined based on data collected from different calibration postures (such as the L-pose and the T-pose). For example, the shoulder width L_S can be obtained based on the L-pose, and the length of the open arm can be obtained based on the T-pose. By subtracting the shoulder width L_S from the length of the open arm and then dividing the quotient by 2, the length of a single arm can be obtained.

[0079] Figure 5B is a schematic diagram of a tracking scenario according to an embodiment of the present disclosure. Figure 5Cis a schematic diagram of a tracking scene according to an embodiment of the present disclosure. Figure 5B and Figure 5C , tracking scene 500B depicts a side view of the user U posing a T-shaped posture in the virtual world as a calibration posture, and tracking scene 500C depicts a top view of the user U posing a T-shaped posture in the virtual world as a calibration posture. Figure 5B and Figure 5C In the example, the display 112 may display a virtual wall VW_L and a virtual wall VW_R in the virtual world. The function of the virtual wall VW_L and the virtual wall VW_R is to help the user U to take a T-shaped posture as a calibration posture. Figure 5B and Figure 5C A virtual wall VW_L, a virtual wall VW_R, and a user U are depicted together, and the user U is actually in the real world, and the virtual wall VW_L and the virtual wall VW_R are actually in the virtual world.

[0080] Reference Figure 5B , the virtual wall VW_L and the virtual wall VW_R may be displayed to the user U, and the user U may be guided to push the virtual wall VW_L and the virtual wall VW_R. In order to make it easier for the user U to push and make more accurate measurements, the initial positions of the virtual wall VW_L and the virtual wall VW_R may be close to the user U, or more precisely, close to the body tracker 120. In one embodiment, a virtual palm may be displayed on the virtual wall VW_L and the virtual wall VW_R, and the user U may place his hand on the virtual palm. Based on the height of the user U, the position of the virtual palm may be adjusted.

[0081] When the user U pushes the virtual wall VW_L and the virtual wall VW_R, the body motion sensor 122 of the first body tracker on the left wrist and the body motion sensor 122 of the second body tracker on the right wrist may be configured to obtain sensor data. The processor 116 may be configured to determine the horizontal distance D1 between the body motion sensor 122 on the left wrist and the body motion sensor 122 on the right wrist as the initial distance based on the sensor data of the body motion sensor 122. For example, the user U may be guided to look at the left hand so that the left hand may be in the FOV of the camera 114 of the HMD device 110. The processor 116 may be configured to determine the length L1 from the left shoulder to the left wrist and the thickness T1 of the left palm based on the image captured by the camera 114. In addition, the processor 116 may be configured to determine the horizontal distance D2 by calculating the sum of the horizontal distance D1 and the thickness T1. Similarly, the user U may be guided to look at the right hand so that the right hand may be in the FOV of the camera 114 of the HMD device 110. The processor 116 may be configured to determine the length L2 from the right shoulder to the right wrist and the thickness T2 of the right palm based on the images captured by the camera 114 .

[0082] It should be noted that reference Figure 5B and Figure 5C , the processor 116 may be configured to determine the length L_O of the open arm (i.e., the horizontal distance D3) by calculating the sum of the horizontal distance D1, the thickness T1 of the left palm, and the thickness T2 of the right palm. That is, the processor 116 may be configured to determine the length L_O of the open arm of the user U based on the sensor data of the first body tracker and the sensor data of the second body tracker. For another example, while the user U is looking at the left hand or the right hand, the head motion sensor 118 may be configured to obtain head motion sensor data. For example, when the line of sight of the user U is parallel to the arm, the processor 116 may be configured to determine the center position of the user U based on the head motion sensor data. In addition, while instructing the user U to move his arms to push the virtual wall VW_L and the virtual wall VW_R, the body motion sensor 122 may be configured to obtain sensor data. When the horizontal distances between all the body motion sensors 122 and the HMD device 110 (i.e., the center position) are the farthest, the processor 116 may be configured to determine the length L_O of the open arm, the length L_LA of the left arm, and the length L_RA of the right arm based on the sensor data of the body motion sensors 122. In one embodiment, when the horizontal distance D3 decreases, the processor 116 may be configured to guide the user on how to correctly pose for the calibration posture (e.g., keep the arms open and as straight as possible).

[0083] Figure 6 is a schematic flow chart of a tracking method according to an embodiment of the present disclosure. Figures 1A to 6 The tracking method 600 may include step S610, step S620, step S630 and step S640.

[0084] In step S610, a body image may be obtained through the camera 114, and the body image includes a body part BD of the user U. In step S620, a relative position between the body motion sensor 122 of the body tracker 120 and the body part BD of the user U may be determined based on the body image. The body tracker 120 may be adapted to be worn on the body part BD of the user U. In step S630, sensor data of the body motion sensor 122 of the body tracker 120 may be obtained. The sensor data may be configured to indicate movement and / or rotation of the body part BD. In step S640, target tracking of the body part BD may be performed based on the relative position and the sensor data.

[0085] In addition, the implementation details of the tracking method can be found in Figure 1A to Figure 5C The present invention provides sufficient instructions, suggestions and implementation examples, and the details will not be repeated here one by one.

[0086] In summary, according to the tracking system 100A, the tracking system 100B and the tracking method 600, even if the body tracker 120 is not correctly worn on the user's body part, the processor 116 can still accurately track the body part based on the relative position between the body motion sensor 122 and the body part. In addition, when the body part is not in the field of view (FOV) of the camera 114, the processor 116 can still perform target tracking based on the relative position and sensor data without a body image. Therefore, the user experience is improved.

[0087] It is obvious to those skilled in the art that various modifications and variations can be made to the disclosed embodiments without departing from the scope or spirit of the present disclosure. In summary, the present disclosure is intended to cover modifications and variations within the scope of the following claims and their equivalents.

Claims

1. A tracking system comprising: A head mounted display device, adapted to be worn on a user's head and comprising: a camera configured to obtain a body image including a body part of the user; and The processor is configured to: determining a relative position between a body motion sensor of a body tracker and the body part of the user based on the body image; and performing target tracking of the body part based on the relative position and sensor data of the body motion sensor; and The body tracker is adapted to be worn on the body part of the user and comprises: The body motion sensor is configured to obtain the sensor data, wherein the sensor data is configured to indicate movement and / or rotation of the body part.

2. The tracking system of claim 1, wherein the processor is further configured to: Get simultaneous positioning and mapping; determining tracker coordinates of the body tracker based on the synchronized positioning and mapping map; determining sensor coordinates of the body motion sensor based on the simultaneous localization and mapping map; and The relative position is determined based on the tracker coordinates and the sensor coordinates.

3. The tracking system of claim 1 , wherein the processor is further configured to: performing object detection using an object detection algorithm to identify the body part and the body motion sensor in the body image; and Based on the object detection, the relative position is determined.

4. The tracking system of claim 1 , wherein the processor is further configured to: obtaining an ideal wearing position on the body part; determining an actual wearing position of the body motion sensor based on the body image; and The relative position is determined based on the ideal wearing position and the actual wearing position. 5 . The tracking system of claim 1 , wherein the processor is further configured to determine the relative position based on a calibration gesture of the user.

6. The tracking system of claim 5, wherein the processor is further configured to determine an arm length of the user based on the calibration pose and the body image and / or the sensor data.

7. The tracking system of claim 5, wherein the head mounted display device further comprises a display configured to display a virtual wall in a virtual world to help the user to perform the calibration pose.

8. The tracking system according to claim 5, wherein The calibration posture is an L-shaped posture, The body tracker includes a first body tracker and a second body tracker, and The processor is also configured to determine a shoulder width of the user based on the sensor data of the first body tracker and the sensor data of the second body tracker.

9. The tracking system according to claim 5, wherein The calibration posture is a T-shaped posture, The body tracker includes a first body tracker and a second body tracker, and The processor is further configured to determine a length of an outstretched arm of the user based on the sensor data of the first body tracker and the sensor data of the second body tracker.

10. A tracking method comprising: obtaining a body image including a body part of a user; determining a relative position between a body motion sensor of a body tracker and the body part of the user based on the body image, wherein the body tracker is adapted to be worn on the body part of the user; obtaining sensor data of the body motion sensor of the body tracker, wherein the sensor data is configured to indicate movement and / or rotation of the body part; and Target tracking of the body part is performed based on the relative position and the sensor data.

Images