Tracking method, multi-device system and non-transient computer readable storage medium

Abstract

The invention provides a tracking method and a multi-device system. A multi-device system for tracking a player device in a physical environment includes a host device and a peripheral device. The tracking method includes: acquiring, through a peripheral device, a spatial relationship between a player device and a peripheral device mounted on the player device; transmitting, by the peripheral device, the spatial relationship and a first gesture of the peripheral device to the host device, where the first gesture is generated by the peripheral device mounted on the player device; generating, by the host device, a second posture of the player device according to the spatial relationship and the first posture; and in response to detecting that the spatial relationship is unqualified, updating the spatial relationship through the peripheral device. By retrieving and / or verifying a spatial relationship between a player device and a peripheral device installed on the player device, the host device can accurately display a virtual player device corresponding to the player device in an immersive environment, which provides a user with a visual perception that matches a user's tactile perception of the player device.

Description

Technical Field

[0001] This disclosure relates to a method and system, and in particular to a tracking method and a multi-device system. Background Technology

[0002] In the fields of Virtual Reality (VR), Augmented Reality (AR), and / or Mixed Reality (MR), some related technologies can provide users with various experiential devices that have different physical forms, such as guns, baseball bats, rackets, fire extinguishers, and boxing gloves, allowing users to interact with specific virtual reality content as if they were in the real world. However, if the experiential device is incorrectly rendered into the user's field of vision due to the inability to properly track it, the user may have a poor experience or lack of immersion. Therefore, it is necessary to propose new methods for tracking experiential devices. Summary of the Invention

[0003] One aspect of this disclosure is a tracking method. This tracking method is applicable to a multi-device system for tracking a player device in a physical environment, and includes a host device and a peripheral device. The tracking method includes: obtaining a spatial relationship between the player device and the peripheral device installed on the player device via the peripheral device; transmitting the spatial relationship and a first pose of the peripheral device to the host device via the peripheral device, wherein the first pose is generated by the peripheral device installed on the player device; generating a second pose of the player device based on the spatial relationship and the first pose via the host device, wherein the spatial relationship, the first pose, and the second pose are defined in a spatial coordinate system; and updating the spatial relationship between the player device and the peripheral device installed on the player device via the peripheral device in response to detecting an invalid spatial relationship.

[0004] In some embodiments, obtaining the spatial relationship between the player device and the peripheral device installed on the player device through the peripheral device includes: capturing image data of the player device through the peripheral device; generating a feature point model of the player device based on the image data and a preset three-dimensional model of the player device through the peripheral device; and when the peripheral device is installed on the player device, generating a third pose of the player device based on the feature point model through the peripheral device, and using the third pose and the first pose to generate extrinsic parameter data of the player device relative to the peripheral device as the spatial relationship, wherein the third pose is defined in the spatial coordinate system.

[0005] In some embodiments, at least one trackable object is disposed on the player device, and obtaining the spatial relationship between the player device and the peripheral device installed on the player device through the peripheral device includes: when the peripheral device is installed on the player device, generating a third pose of the at least one trackable object based on an image data of the at least one trackable object through the peripheral device, and using the third pose and the first pose to generate extrinsic data of the player device relative to the peripheral device as the spatial relationship, wherein the third pose is defined in the spatial coordinate system.

[0006] In some embodiments, the tracking method further includes: detecting, via the peripheral device, whether the spatial relationship between the player device and the peripheral device installed on the player device is valid, wherein when the spatial relationship is valid, the spatial relationship is not updated.

[0007] In some embodiments, detecting whether the spatial relationship between the player device and the peripheral device installed on the player device is qualified by means of the peripheral device includes: when the peripheral device is installed on the player device, capturing an image frame containing an image of the player device by means of the peripheral device; generating a projection of the player device on the image frame by means of the peripheral device according to the spatial relationship, so as to obtain a difference between the projection and the image; and comparing the difference with a difference threshold by means of the peripheral device, wherein when the difference is less than the difference threshold, the spatial relationship is qualified, and when the difference is greater than the difference threshold, the spatial relationship is unqualified.

[0008] In some embodiments, at least one trackable object is disposed on the player device, and the detection of whether the spatial relationship between the player device and the peripheral device installed on the player device is qualified by the peripheral device includes: when the peripheral device is installed on the player device, capturing an image frame containing an image of the at least one trackable object by the peripheral device; generating a projection of the at least one trackable object on the image frame by the peripheral device according to the spatial relationship to obtain a difference between the projection and the image; and comparing the difference with a difference threshold by the peripheral device, wherein when the difference is less than the difference threshold, the spatial relationship is qualified, and when the difference is greater than the difference threshold, the spatial relationship is unqualified.

[0009] In some embodiments, the tracking method further includes: determining, via the host device or another peripheral device in the multi-device system, whether the spatial relationship between the player device and the peripheral device installed on the player device is valid; and in response to determining that the spatial relationship is invalid, transmitting, via the host device or the other peripheral device, another spatial relationship between the player device and the peripheral device installed on the player device to the peripheral device, wherein the other spatial relationship is generated based on a first pose of the peripheral device and a third pose of the player device, and the third pose is generated by the host device or the other peripheral device based on image information associated with the player device and is defined in the spatial coordinate system.

[0010] In some embodiments, determining whether the spatial relationship between the player device and the peripheral device installed on the player device is acceptable via the host device or the other peripheral device includes: capturing an image frame related to the player device via the host device or the other peripheral device; generating a projection related to the player device on the image frame via the host device or the other peripheral device based on the spatial relationship and the first pose to obtain a difference related to the player device; and comparing the difference with a difference threshold via the host device or the other peripheral device, wherein the spatial relationship is acceptable when the difference is less than the difference threshold, and the spatial relationship is unacceptable when the difference is greater than the difference threshold.

[0011] In some embodiments, the tracking method further includes: detecting, via the peripheral device, whether another spatial relationship between the player device and the peripheral device installed on the player device is valid; and in response to detecting that the other spatial relationship is valid, replacing the spatial relationship with the other spatial relationship via the peripheral device.

[0012] Another embodiment of this disclosure is a multi-device system. This multi-device system is used to track a player device in a physical environment and includes a host device and a peripheral device. The peripheral device is used to obtain a spatial relationship between the player device and the peripheral device installed on the player device, and to update the spatial relationship between the player device and the peripheral device installed on the player device in response to detecting an invalid spatial relationship. The host device is used to receive the spatial relationship and a first pose of the peripheral device from the peripheral device, and to generate a second pose of the player device based on the spatial relationship and the first pose. The first pose is generated by the peripheral device installed on the player device, and the spatial relationship, the first pose, and the second pose are defined in a spatial coordinate system.

[0013] In some embodiments, the multi-device system further includes a connection device, wherein the peripheral device is mounted on the player device via the connection device, wherein the peripheral device includes a camera, and when the peripheral device is mounted on the player device, a portion of the player device or at least one trackable object on the player device is within the field of view of the camera.

[0014] Another embodiment of this disclosure is a non-transitory computer-readable storage medium having a computer program for executing a tracking method, wherein the tracking method is applicable to a multi-device system for tracking a player device in a physical environment, the multi-device system including a host device and a peripheral device, and the tracking method comprising: obtaining a spatial relationship between the player device and the peripheral device installed on the player device via the peripheral device; transmitting the spatial relationship and a first pose of the peripheral device to the host device via the peripheral device, wherein the first pose is generated by the peripheral device installed on the player device; generating a second pose of the player device via the host device based on the spatial relationship and the first pose, wherein the spatial relationship, the first pose, and the second pose are defined in a spatial coordinate system; and updating the spatial relationship between the player device and the peripheral device installed on the player device via the peripheral device in response to detecting that the spatial relationship is invalid.

[0015] In summary, by obtaining and / or verifying the spatial relationship between the player device and peripheral devices installed on the player device, the host device can display the virtual player device corresponding to the player device in an immersive environment with accurate position and / or orientation. This provides the user with a visual perception that matches the user's tactile perception of the player device. Furthermore, any tracking device that can be installed on the player device can be used to replace peripheral devices, thus eliminating the need to redesign additional tracking modules to track the player device. In conclusion, the multi-device system and tracking method disclosed herein offer advantages such as cost savings, high compatibility, and a superior user experience. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a multi-device system according to some embodiments of the present disclosure.

[0017] Figure 2 This is a flowchart illustrating a tracking method according to some embodiments of the present disclosure.

[0018] Figure 3 This is a flowchart illustrating one operation of a tracking method according to some embodiments of the present disclosure.

[0019] Figure 4AThis is a schematic diagram illustrating the process of obtaining the spatial relationship between a player device and peripheral devices installed on the player device in a multi-device system, according to some embodiments of the present disclosure.

[0020] Figure 4B This is a schematic diagram illustrating the process of obtaining the spatial relationship between a player device and peripheral devices installed on the player device in a multi-device system, according to some embodiments of the present disclosure.

[0021] Figure 5 This is a flowchart illustrating one operation of a tracking method according to some embodiments of the present disclosure.

[0022] Figure 6 This is a schematic diagram illustrating another process for obtaining the spatial relationship between a player device and a peripheral device installed on the player device, according to some embodiments of the present disclosure.

[0023] Figure 7 This is a flowchart illustrating one operation of a tracking method according to some embodiments of the present disclosure.

[0024] Figure 8 This is a schematic diagram of an image frame captured by a peripheral device according to some embodiments of the present disclosure.

[0025] Figure 9 This is a flowchart illustrating one operation of a tracking method according to some embodiments of the present disclosure.

[0026] Figure 10 This is a schematic diagram of an image frame captured by a peripheral device according to some embodiments of the present disclosure.

[0027] Figure 11 This is a schematic diagram illustrating the spatial relationship between a host device or another peripheral device in a multi-device system according to some embodiments of the present disclosure, verifying the relationship between the player device and the peripheral device.

[0028] Figure 12 This is a flowchart illustrating a tracking method according to some embodiments of the present disclosure.

[0029] Figure 13 This is a flowchart illustrating one operation of a tracking method according to some embodiments of the present disclosure.

[0030] Figure 14 This is a schematic diagram of an image frame captured by a host device, according to some embodiments of the present disclosure.

[0031] Figure 15 This is a flowchart illustrating a tracking method according to some embodiments of the present disclosure.

[0032] Symbol explanation: 10: Player Device 11: Main unit 13,15: Peripheral devices 20: Connecting device 30: Trackable objects 100: Multi-device system 110, 130: Processor 112,132: Camera 200: Tracking Methods DEX, DEXU: External parameter data E1: Physical Environment FM: Feature Point Model HM: Map Data IM10, IM13, IM30: Imagery IMG1, IMG2: Image Data IMF1, IMF2, IMF3: Image Frames PJ10, PJ30: Projection PM: Preset 3D Model PPE, PPL[1], PPL[2], PTD: Attitude S201~S204, S1201~S1202, S1501, S1502: Operation S301~S304, S501~S503, S701~S703, S901~S903, S1301~S1303: sub-operations U1: User Detailed Implementation

[0033] The following detailed description of embodiments, in conjunction with the accompanying drawings, is provided. However, the specific embodiments described are only for explaining this case and are not intended to limit this case. The description of the structural operations is not intended to limit the order of their execution. Any structure resulting from the recombination of elements and producing an apparatus with equivalent functionality is within the scope of this disclosure.

[0034] The terms "coupled" or "connected" as used in this article can refer to two or more components making direct physical or electrical contact with each other, or making indirect physical or electrical contact with each other, or to two or more components operating or moving together.

[0035] Please see Figure 1 , Figure 1 This is a schematic diagram of a multi-device system 100 according to some embodiments of the present disclosure. In some embodiments, the multi-device system 100 can be operated by a user U1 in a physical environment E1 such as a gaming venue, workplace, or residence.

[0036] At Figure 1 In one embodiment, the multi-device system 100 includes a host device 11 and a peripheral device 13. In some practical applications, such as... Figure 1 As shown, the host device 11 can be implemented using a wearable display device (e.g., a head-mounted device) and can be worn on the head of the user U1. Similarly, the peripheral device 13 can be implemented using a controller device (e.g., a handheld controller, a wearable controller, etc.) and can be used by the user U1 to control, for example, the wearable display device. The wearable display device and the controller device can be operated together as an immersive system, which provides an immersive experience for the user U1. For example, the user U1's perception of the physical environment E1 is replaced or modified by an immersive environment displayed by the wearable display device, such as Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR), etc. The immersive environment includes at least one virtual object that cannot be directly seen by the user U1 in the physical environment E1. Furthermore, the user U1 can control at least one virtual object in the immersive environment by operating the controller device or by performing hand gestures.

[0037] In some embodiments, such as Figure 1 As shown, the immersive system also provides user U1 with a player device 10 to enhance the user U1's immersion in the immersive environment. Specifically, in Figure 1 In one embodiment, the immersive environment is a shooting scene, such as shooting sports or hunting, and the player device 10 is shaped like a rifle that corresponds to the shooting scene. However, this disclosure is not limited to this. For example, in some embodiments, depending on the type of immersive environment, the player device 10 may be shaped like a baseball bat, badminton racket, tennis racket, boxing glove, etc. Further, a peripheral device 13 may be installed on the player device 10. With this configuration, the user U1 can operate the player device 10, which has an appropriate shape, to interact with at least one virtual object in a specific immersive environment, such as a virtual target, virtual prey, virtual ball, virtual opponent, etc., allowing the user U1 to better immerse themselves in the immersive environment.

[0038] In some further embodiments, the multi-device system 100 also includes a connection device 20. For example... Figure 1 As shown, the connecting device 20 can be disposed on the player device 10 and used to secure the peripheral device 13 by various means such as hook-and-loop fasteners, snap-fits, and clamps. In other words, in some embodiments, the peripheral device 13 is mounted on the player device 10 via the connecting device 20.

[0039] In some embodiments, the host device 11 is used to locate at least one of itself, the peripheral device 13, and the player device 10 in the physical environment E1, and to provide visual feedback to the user U1 based on the location of at least one of itself, the peripheral device 13, and the player device 10. Accordingly, as Figure 1 As shown, the host device 11 includes a processor 110 and a camera 112. Specifically, the processor 110 is electrically and / or communicatively coupled to the camera 112.

[0040] In the above embodiment of the host device 11, the camera 112 is used to capture multiple images in the physical environment E1. It should be understood that these images captured by the camera 112 may include at least one of the following: multiple images of the entire or partial physical environment E1, multiple images of the peripheral device 13, multiple images of the player device 10, and multiple images of the user U1. The processor 110 is used to apply some feature extraction-based positioning techniques (e.g., simultaneous localization and mapping) to the images captured by the camera 112 to establish a map data HM for the physical environment E1, and to calculate the position and / or orientation of at least one of the host device 11, peripheral device 13, and player device 10 in the map data HM. The processor 110 is also used to apply some interaction-based tracking techniques (e.g., optical tracking) to the images captured by the camera 112 to calculate the position and / or orientation of at least one of the peripheral device 13 and player device 10 in the map data HM. Furthermore, the processor 110 generates at least one visual content based on the position and / or orientation of at least one of the host device 11, peripheral device 13 and player device 10, thereby enabling the host device 11 to display an immersive environment (i.e., visual feedback) to the user U1.

[0041] In some embodiments, the peripheral device 13 is used to locate itself and at least one of the player device 10 in the physical environment E1, and to communicate with the host device 11 to operate in the same spatial coordinate system as the host device 11. Accordingly, as Figure 1 As shown, the peripheral device 13 includes a processor 130 and a camera 132. Specifically, the processor 130 is electrically and / or communicatively coupled to the camera 132.

[0042] In the above embodiment of the peripheral device 13, the camera 132 is used to capture multiple images in the physical environment E1. It should be understood that these images captured by the camera 132 may include at least one of the following: multiple images of the entire or part of the physical environment E1, multiple images of the host device 11, multiple images of the player device 10, and multiple images of the user U1. By applying some feature extraction-based positioning technology (e.g., simultaneous localization and mapping technology) and / or some interaction-based tracking technology (e.g., optical tracking technology), the processor 130 is used to calculate the position and / or orientation of at least one of the peripheral device 13 and the player device 10 in the map data HM based on the images captured by the camera 132 and the map data HM shared by the host device 11.

[0043] In the above embodiments, processor 110 and processor 130 can each be implemented by a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a microprocessor, a system-on-a-chip (SoC), or other suitable processing circuitry.

[0044] It should be understood that the main unit 11 and the peripheral unit 13 are not limited to Figure 1 The structure is shown. For example, in some embodiments, the host device 11 and the peripheral device 13 may each further include a motion sensor (e.g., an inertial measurement unit (IMU) including an accelerometer, gyroscope, and / or magnetometer), a memory (e.g., volatile memory, non-volatile memory, etc.), and / or a communicator (e.g., a Wi-Fi module, a Bluetooth Low Energy (BLE) module, a Bluetooth module, etc.). The motion sensor is used to sense the movement of the host device 11 or the peripheral device 13 to generate motion data accordingly. Through some mathematical operations (e.g., integration, orthogonal decomposition, etc.), the motion data can be used to calculate the position and / or orientation of the host device 11 or the peripheral device 13. The memory can be used to store signals, data, and / or information, such as motion data, images captured by camera 112 or camera 132, map data HM, the position and / or orientation of at least one of the host device 11, the peripheral device 13, and the player device 10. The host device 11 and peripheral device 13 can communicate with each other or other devices via a communicator (e.g., transmitting signals, data, and / or information). Furthermore, the host device 11 may also include a display panel, which can be implemented using an active-matrix organic light-emitting diode (AMOLED) display, an organic light-emitting diode (OLED) display, or the like. The display panel can be used to display at least one visual content generated by the processor 110, thereby allowing the host device 11 to provide visual feedback to the user U1 (e.g., an immersive environment).

[0045] In some embodiments, when the peripheral device 13 is installed on the player device 10, the peripheral device 13 is used to generate the position and / or orientation of the player device 10 relative to the peripheral device 13 installed on the player device 10. Based on the position and / or orientation of the player device 10 relative to the peripheral device 13 installed on the player device 10 and the position and / or orientation of the peripheral device 13 in the map data HM, the host device 11 can accurately display the virtual player device (i.e., the virtual object corresponding to the player device 10) in the immersive environment, so that the user U1's visual perception of the virtual player device matches the user U1's tactile perception of the player device 10.

[0046] To further explain, when user U1 operates player device 10 with peripheral device 13 installed to interact with at least one virtual object in the immersive environment, at least one of host device 11 and peripheral device 13 is used to verify the position and / or orientation of player device 10 relative to peripheral device 13 installed on player device 10, and to update the position and / or orientation of player device 10 relative to peripheral device 13 installed on player device 10 in response to verification failure, so as to avoid user U1's visual perception of virtual player device and user U1's tactile perception of player device 10 being inconsistent.

[0047] The operation of the main unit 11 and peripheral unit 13 of the aforementioned multi-device system 100 will be described in the following paragraphs. Figure 2 Please provide a detailed explanation. Figure 2 This is a flowchart illustrating a tracking method 200 applicable to a multi-device system 100 according to some embodiments of this disclosure. In some embodiments, the multi-device system 100 performing the tracking method 200 allows for a good match between the user U1's visual perception of the virtual gaming device and the user U1's tactile perception of the gaming device 10. For example... Figure 2 As shown, the tracking method 200 includes multiple operations S201 to S204. However, this disclosure is not limited thereto.

[0048] In some embodiments, before operating S201, the peripheral device 13 receives map data HM established by the host device 11, so that the peripheral device 13 and the host device 11 operate in the same spatial coordinate system.

[0049] In operation S201, the peripheral device 13 obtains a spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10. In some embodiments, the peripheral device 13 calculates the position and / or orientation of the player device 10 relative to the peripheral device 13 when the peripheral device 13 is installed on the player device 10, using feature extraction-based positioning technology and / or interaction-based tracking technology by the processor 130. Specifically, the position and / or orientation of the player device 10 relative to the peripheral device 13 when the peripheral device 13 is installed on the player device 10 can be directly used as the spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10. These embodiments of operation S201 will be described in detail later. Figure 3 , 4A Detailed explanations for ~4B, 5 and 6.

[0050] In operation S202, peripheral device 13 transmits its spatial relationships and a pose PPE (Personal Physical Context) to host device 11. In some embodiments, when peripheral device 13 is installed on player device 10, using feature extraction-based positioning technology via processor 130, peripheral device 13 calculates its spatial coordinates or position and / or orientation in map data HM as its pose PPE. Peripheral device 13 can send the spatial relationships and pose PPE obtained in operation S201 via its communicator, and host device 11 can receive the spatial relationships and pose PPE obtained in operation S201 via its communicator.

[0051] In operation S203, the host device 11 generates a host-based attitude of the player device 10 based on spatial relationships and the attitude PPE of the peripheral device 13. In some embodiments, the attitude PPE indicates the position and / or orientation of the peripheral device 13 installed on the player device 10 in the map data HM, while the spatial relationships transmitted to the host device 11 in operation S202 (or obtained in operation S201) indicate the position and / or orientation of the player device 10 relative to the peripheral device 13 when the peripheral device 13 is installed on the player device 10. Accordingly, when the peripheral device 13 is installed on the player device 10, the host device 11, through the processor 110, can use the spatial relationships and attitude PPE transmitted by the peripheral device 13 in operation S202 to calculate the spatial coordinates or position and / or orientation of the player device 10 in the map data HM as a host-based attitude. To further explain, through the host-based posture of the player device 10, the host device 11 can display the virtual player device in an immersive environment with accurate position and / or orientation, thereby providing the user U1 with a visual perception that matches their tactile perception of the player device 10.

[0052] In some embodiments, when user U1 operates player device 10 equipped with peripheral device 13 in physical environment E1, the peripheral device 13 installed on player device 10 may become misaligned (or not aligned) with player device 10 due to natural attenuation or physical impact. In this case, the spatial relationship transmitted to host device 11 in operation S202 may differ from the position and / or orientation of player device 10 relative to the misaligned peripheral device 13 (i.e., the current spatial relationship between player device 10 and the misaligned peripheral device 13 installed on player device 10). The significant difference between these two relationships may cause a mismatch between user U1's visual perception of the virtual player device and user U1's tactile perception of player device 10. Accordingly, operation S204 may be performed to verify the spatial relationship currently used by host device 11 and peripheral device 13.

[0053] In operation S204, the peripheral device 13 detects whether the spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10 is acceptable. In some embodiments, the peripheral device 13 verifies the spatial relationship transmitted to the host device 11 in operation S202 by the processor 130 based on at least one image related to the player device 10 according to the spatial relationship analysis transmitted to the host device 11 in operation S202. This will be discussed later. Figures 7-10 Detailed explanation. Specifically, at least one image related to the player device 10 is captured by camera 132 when peripheral device 13 is installed on player device 10.

[0054] In some embodiments, when the peripheral device 13 is misaligned, the spatial relationship verification performed in operation S204 may fail (i.e., the spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10 is not qualified). Therefore, as Figure 2 As shown, operation S201 is re-executed. That is, a new spatial relationship will be generated between the player device 10 and the peripheral device 13 installed on the player device 10. Furthermore, when the peripheral device 13 is not misaligned, the spatial relationship verification performed in operation S204 may succeed (i.e., the spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10 is valid). Therefore, as... Figure 2 As shown, operation S202 is re-executed. That is, the spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10 is not updated and continues to be used.

[0055] Details of operating S201 will be provided in the following paragraphs. Figure 3 and 4A ~4B Detailed Description. Figure 3 This is a flowchart of operation S201 of the tracking method 200 shown in some embodiments of the present disclosure. Figure 4A and 4B These are schematic diagrams illustrating the process of obtaining the spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10, according to some embodiments of this disclosure. In some embodiments, such as Figure 3 As shown, operation S201 includes multiple sub-operations S301~S304.

[0056] In some embodiments, multiple sub-operations S301 and S302 are performed before the peripheral device 13 is installed on the player device 10. Specifically, before sub-operation S301, the user U1 may attempt to install the peripheral device 13 into the connection device 20 on the player device 10 and check whether the field of view of the camera 132 covers at least a portion of the player device 10 when the peripheral device 13 is installed in the connection device 20. If not, the user U1 adjusts the connection device 20 to ensure that the field of view of the camera 132 covers at least a portion of the player device 10 when the peripheral device 13 is installed in the connection device 20. If yes, the user U1 removes the peripheral device 13 from the connection device 20 and places the player device 10 in a stable position (e.g., a table or support device) in the physical environment E1. When placing the player device 10, the user U1 should ensure that the observable area of ​​the player device 10 is as exposed and unobstructed as possible (e.g., placing the gun-shaped device upright on a table). Then, when the peripheral device 13 is held near the player device 10 in a stable position, sub-operation S301 can be executed.

[0057] In operation S301, peripheral device 13 captures image data IMG1 from player device 10. In some embodiments, such as Figure 4A As shown, the peripheral device 13 held by the user U1 uses camera 132 to capture images of the player device 10 as image data IMG1. Therefore, it should be understood that image data IMG1 may contain at least one or a series of images (these images may be taken from different angles, thus forming image data IMG1 with multi-angle information).

[0058] In operation S302, the peripheral device 13 generates a feature point model FM of the player device 10 based on the image data IMG1 and a preset 3D model PM of the player device 10. Specifically, the preset 3D model PM is stored in the memory of the peripheral device 13 and records the actual or physical appearance of the player device 10. Figure 4AIn this embodiment, using feature extraction-based localization technology, the peripheral device 130 extracts multiple feature points from at least one image of the player device 10 in image data IMG1, and matches these feature points with multiple map points in map data HM to obtain some feature points that match the map points as feature point data of the player device 10. It should be understood that each feature point in the feature point data has at least one descriptor that can indicate a spatial coordinate in map data HM, such that the distribution of the feature point data presents an approximate appearance of the player device 10. Accordingly, the processor 130 can map the feature point data to a preset 3D model PM by fitting the approximate appearance of the player device 10 with the actual appearance of the player device 10, thus forming a feature point model FM. That is, each feature point in the feature point model FM of the player device 10 is connected to a corresponding part of the preset 3D model PM; therefore, the various parts of the player device 10 can be accurately located in map data HM (or physical environment E1) based on the feature point model FM of the player device 10.

[0059] After the feature point model FM is generated, the user U1 can install the peripheral device 13 into the connection device 20. That is, as Figure 4B As shown, the peripheral device 13 is connected to the player device 10 via the connection device 20. Accordingly, sub-operations S303 to S304 are executed.

[0060] like Figure 3 As shown, in sub-operation S303, the peripheral device 13 generates a pose of the player device 10 based on the peripheral device according to the feature point model FM. Figure 4B In one embodiment, the processor 130, using feature extraction-based positioning technology, can receive at least one image captured by the camera 132 when the peripheral device 13 is installed on the player device 10, and selectively update the spatial coordinates of each feature point in the feature point model FM by analyzing the at least one image (e.g., performing feature extraction on the at least one image). Specifically, since the feature point model FM is generated when the player device 10 is placed in a stable position, the feature point model FM can be updated in response to the displacement of the player device 10 from the stable position. Thereafter, as... Figure 4B As shown, the processor 130 calculates the position and / or orientation of the player device 10 in the map data HM based on the feature point model FM as a pose PPL of the player device 10 [1] (that is, the pose based on the surrounding end).

[0061] For example Figure 3 As shown, in sub-operation S304, peripheral device 13 uses the attitude of the peripheral end and the attitude PPE of peripheral device 13 to generate extrinsic data DEX of player device 10 relative to peripheral device 13. Figure 4BIn this embodiment, the processor 130, using feature extraction-based positioning technology, can also update the attitude PPE in response to the displacement of the player device 10 from a stable position, so that the attitude PPE can indicate the instantaneous position and / or orientation of the peripheral device 13 in the map data HM when the peripheral device 13 is installed on the player device 10. Accordingly, the peripheral device 13 can use the attitude PPL[1] of the player device 10 and the attitude PPE of the peripheral device 13 to calculate the position and / or orientation of the player device 10 relative to the peripheral device 13 installed on the player device 10 as extrinsic parameter data DEX. Specifically, the extrinsic parameter data DEX can be used as the spatial relationship obtained in operation S201.

[0062] This disclosure does not intend to limit operation S201 to... Figure 3 and 4A ~4B embodiment. For example, in Figure 3 and 4A In a further embodiment of ~4B, after sub-operation S302 or before peripheral device 13 is installed on player device 10, peripheral device 13 further determines whether the number of feature points on a preset portion of preset 3D model PM is greater than a quantity threshold. Specifically, the preset portion of preset 3D model PM may correspond to the handheld portion of player device 10 (e.g., rifle grip). By ensuring that the number of feature points on the preset portion of preset 3D model PM is greater than the quantity threshold through peripheral device 13, the position and / or orientation of the handheld portion of player device 10 in map data HM calculated based on a set of feature points (corresponding to the preset portion) of feature point model FM is reliable, thereby enabling the virtual handheld portion of virtual player device (corresponding to the handheld portion of player device 10) to be accurately displayed in the immersive environment, thus providing user U1 with a visual perception that matches the tactile perception of holding the handheld portion of player device 10.

[0063] Based on the above description, in some embodiments, when the number of feature points on the preset portion of the preset 3D model PM exceeds a threshold, the peripheral device 13 can notify the user U1 that the peripheral device 13 is ready to be installed on the player device 10, thereby executing sub-operations S303-S304. Furthermore, when the number of feature points on the preset portion of the preset 3D model PM does not exceed the threshold, the peripheral device 13 can notify the user U1 to move the peripheral device 13 closer to the handheld portion of the player device 10, or adjust the placement position of the player device 10, thereby re-executing sub-operations S301-S302. In other words, the peripheral device 13 can regenerate the feature point model FM to ensure that the preset portion of the preset 3D model PM has sufficient feature points.

[0064] Details on operating the S201 will be provided in the following paragraphs. Figure 5 and6 Detailed explanation. Figure 5 This is a flowchart of operation S201 of the tracking method 200 shown in some embodiments of the present disclosure. Figure 6 This is a schematic diagram illustrating the process of obtaining the spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10, according to some embodiments of this disclosure. In some embodiments, such as Figure 5 As shown, operation S201 includes multiple sub-operations S501~S503.

[0065] In some embodiments, such as Figure 6 As shown, the player device 10 is equipped with at least one trackable object 30. Specifically, the trackable object 30 is disposed on the outer surface of the player device 10 so that it can be directly photographed by the camera 132 of the peripheral device 13, and can be implemented using one or more light emitters (e.g., light-emitting diodes (LEDs), laser diodes (LDs), etc.), one or more marks with preset shapes or patterns, etc. Figure 3 and 4A Compared to the ~4B embodiment, Figure 6 In this embodiment, the peripheral device 13 is used to interact with the trackable object 30 to obtain the spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10, rather than generating a feature point model FM of the player device 10. Therefore, the peripheral device 13 can directly notify the user U1 to install the peripheral device 13 in the connection device 20, thereby executing sub-operations S501~S503. That is, as Figure 5 As shown, sub-operations S501 to S503 are executed when the peripheral device 13 is installed in the player device 10. It should be understood that before sub-operation S501, the user U1 can further check whether the field of view of the camera 132 covers at least one trackable object 30 when the peripheral device 13 is installed in the connection device 20.

[0066] In operation S501, peripheral device 13 acquires image data IMG2 of at least one trackable object 30. In some embodiments, such as Figure 6 As shown, the peripheral device 13 installed on the player device 10 uses a camera 132 to capture images of the trackable object 30 as image data IMG2. Therefore, it should be understood that the image data IMG2 may contain at least one or a series of images.

[0067] In operation S502, peripheral device 13 generates a pose PTD for at least one trackable object 30 based on image data IMG2. Figure 6In one embodiment, the processor 130, using feature extraction-based positioning technology, can receive image data IMG2 captured by the camera 132 when the peripheral device 13 is installed on the player device 10, and can identify at least one image of the trackable object 30 from the image data IMG2. Subsequently, as... Figure 6 As shown, the processor 130 can calculate the pose PTD of the trackable object 30 (that is, the position and / or orientation of the trackable object 30 in the map data HM) based on at least one image of the trackable object 30.

[0068] In operation S503, peripheral device 13 uses the attitude PTD of at least one trackable object 30 and the attitude PPE of peripheral device 13 to generate extrinsic data DEX of player device 10 relative to peripheral device 13. Figure 6 In this embodiment, the pose PTD of the trackable object 30 can be directly used to represent the pose PPL[1] of the player device 10 (i.e., the position and / or orientation of the player device 10 in the map data HM). Furthermore, the processor 130 can use feature extraction-based localization techniques to update the pose PPE to indicate the real-time position and / or orientation of the peripheral device 13 in the map data HM when the peripheral device 13 is installed on the player device 10. Accordingly, the peripheral device 13 can use the pose PTD of the trackable object 30 and the pose PPE of the peripheral device 13 to calculate the position and / or orientation of the player device 10 relative to the peripheral device 13 installed on the player device 10 as extrinsic parameter data DEX. Specifically, the extrinsic parameter data DEX can be used as the spatial relationship obtained in operation S201.

[0069] In the above embodiment of operation S201 (that is, Figure 3 , 4A In embodiments ~4B, 5, and 6), when extrinsic data DEX is generated, peripheral device 13 can further capture an image (not shown) via camera 132 as a reference image. That is, this reference image has a timestamp corresponding to a point in time when the extrinsic data DEX is generated, and can be stored for later retrieval of the extrinsic data DEX. In short, each time the spatial relationship between player device 10 and peripheral device 13 installed on player device 10 is obtained in operation S201, peripheral device 13 records or stores a corresponding reference image.

[0070] In some embodiments of operation S202, such as Figure 4B As shown in Figure 6, the peripheral device 13 transmits the extrinsic parameter data DEX and the attitude PPE of the peripheral device 13 to the host device 11 via a communicator, thereby executing operation S203. In some embodiments of operation S203, such as Figure 4BAs shown in Figure 6, the host device 11 can use the extrinsic parameter data DEX to transform the pose PPE of the peripheral device 13 into a pose PPL[2] of the player device 10 (that is, the pose of the player device 10 based on the host). Specifically, the pose PPL[2] indicates the spatial coordinates or position and / or orientation of the player device 10 in the map data HM. As can be seen from the description of operations S201 to S203, the pose PPE of the peripheral device 13, the extrinsic parameter data DEX, the pose PTD of the trackable object 30, the pose PPL[1] of the player device 10 and the pose PPL[2] of the player device 10 are defined in the spatial coordinate system of the map data HM established by the host device 11.

[0071] Details of operating S204 will be provided in the following paragraphs. Figure 7 Detailed explanation. Figure 7 This is a flowchart of operation S204 of the tracking method 200 shown in some embodiments of this disclosure. In some embodiments, such as Figure 7 As shown, operation S204 includes multiple sub-operations S701 to S703. Continuing from the previous explanation of operation S204, operation S204 is executed when the user U1 operates the player device 10 on which the peripheral device 13 is installed, to verify the currently used spatial relationships. Therefore, sub-operations S701 to S703 can be executed when the peripheral device 13 is installed on the player device 10.

[0072] In operation S701, peripheral device 13 captures an image frame IMF1 containing an image IM10 of player device 10. Please refer to [link / reference]. Figure 8 , Figure 8 This is a schematic diagram of an image frame IMF1 captured by the peripheral device 13 according to some embodiments of the present disclosure. In some embodiments, because the peripheral device 13 is tightly mounted on the player device 10, the field of view of the camera 132 is fixed relative to the player device 10 and typically only covers a portion of the player device 10. Therefore, in Figure 8 In this context, the image IM10 of the player device 10 corresponds to a portion of the player device 10. It should be understood that when the peripheral device 13 installed on the player device 10 is not misaligned with the player device 10, the image IM10 of the player device 10 will be fixed relative to the image frame IMF1. That is, the position of the image IM10 on the image frame IMF1 will be fixed.

[0073] In operation S702, the peripheral device 13 generates a projection PJ10 of the player device 10 on the image frame IMF1 based on spatial relationships, thereby obtaining a difference between the projection PJ10 and the image IMF10. Following the above description related to the reference image, the extrinsic data DEX can be re-obtained by analyzing the reference image (e.g., performing feature extraction on the reference image). On the other hand, by calculating based on the extrinsic data DEX, the position and / or orientation of the player device 10 relative to the peripheral device 13 when the reference image or the extrinsic data DEX is generated can also be generated. Accordingly, in some embodiments, the peripheral device 13 projects the player device 10, having the position and / or orientation recorded or indicated by the reference image, onto the image frame IMF1 by the processor 130 calculating based on the extrinsic data DEX (i.e., spatial relationships), thereby generating a projection PJ10 of the player device 10 on the image frame IMF1.

[0074] Next, in some embodiments, the peripheral device 13 calculates, via the processor 130, a one-pixel offset from the projection PJ10 of the player device 10 to the image IM10 in the image frame IMF1, as the difference between the projection PJ10 and the image IM10 in the image frame IMF1. For example, the pixel offset can be defined as the translation distance from a pixel in the projection PJ10 to a corresponding pixel in the image IM10 in the image frame IMF1. A pixel in the projection PJ10 and a corresponding pixel in the image IM10 can correspond to the same point on the player device 10.

[0075] In operation S703, peripheral device 13 compares the difference with a difference threshold. Specifically, the difference threshold can be the pixel length (e.g., 5 pixels). In some embodiments, peripheral device 13 compares the pixel offset with the pixel length using processor 130 to determine if the difference is greater than the difference threshold. When the pixel offset exceeds the pixel length, processor 130 determines that the difference is greater than the difference threshold, indicating that the verification of the recorded or stored spatial relationship (i.e., extrinsic data DEX) has failed. Accordingly, as... Figure 7 As shown, operation S201 will be executed to update the extrinsic data DEX. When the pixel offset does not exceed the pixel length, the processor 130 determines that the difference is less than the difference threshold, which means that the verification of the recorded or stored spatial relationship is successful. Accordingly, as Figure 7 As shown, operation S202 will be performed without updating the external parameter data DEX.

[0076] Details on operating S204 will be provided in the following paragraphs. Figure 9 Detailed explanation. Figure 9 This is a flowchart of operation S204 of the tracking method 200 shown in some embodiments of this disclosure. In some embodiments, such as Figure 9As shown, operation S204 includes multiple sub-operations S901~S903. (As in...) Figure 7 As explained, sub-operations S901~S903 can also be executed when the peripheral device 13 is installed on the player device 10 to verify the currently used spatial relationships.

[0077] In operation S901, peripheral device 13 captures an image frame IMF2 containing an image IM30 of at least one trackable object 30. (See also...) Figure 10 , Figure 10 This is a schematic diagram of an image frame IMF2 captured by peripheral device 13 according to some embodiments of this disclosure. As in Figure 8 The explanation is as follows: Figure 10 The image frame IMF2 will contain an image of the player device 10 corresponding to the portion of the player device 10 covered by the field of view of the camera 132. To further explain... Figure 10 The embodiments correspond to Figure 6 The embodiments, therefore in Figure 10 The image frame IMF2 also contains the image IM30 of the trackable object 30 set on the player device 10.

[0078] In operation S902, the peripheral device 13, based on spatial relationships, generates a projection PJ30 of at least one trackable object 30 on the image frame IMF2, thereby obtaining a difference between the projection PJ30 and the image IM30. Figure 6 In some embodiments, the spatial relationship (i.e., extrinsic data DEX) is obtained based on the trackable object 30 disposed on the player device 10, and can therefore be retrieved from a reference image that records the position and / or orientation of the trackable object 30 relative to the peripheral device 13 when the extrinsic data DEX is generated. As described above in sub-operation S702, in some embodiments, the peripheral device 13 projects the trackable object 30, having the position and / or orientation recorded or indicated by the reference image, onto the image frame IMF2 by the processor 130 calculating based on the extrinsic data DEX (i.e., spatial relationship), to generate a projection PJ30 of the trackable object 30 on the image frame IMF2. Furthermore, the peripheral device 13 calculates a one-pixel offset (e.g., the translation distance from a pixel in the projection PJ30 of the trackable object 30 to a corresponding pixel in the image IM30 in the image frame IMF2) through the processor 130, as the difference between the projection PJ30 and the image IM30 in the image frame IMF2.

[0079] In sub-operation S903, peripheral device 13 compares the difference with a difference threshold. As described above in sub-operation S703, in some embodiments, peripheral device 13 compares pixel offset with pixel length via processor 130 to determine whether the difference is greater than the difference threshold. Specifically, as... Figure 9 As shown, when the difference is greater than the difference threshold (that is, when the pixel offset exceeds the pixel length), operation S201 will be executed, while when the difference is less than the difference threshold (that is, when the pixel offset does not exceed the pixel length), operation S202 will be executed.

[0080] This disclosure does not intend to limit the tracking method 200 to... Figure 2 The flowchart and / or limiting the multi-device system 100 to Figure 1 The device assembly shown. Please refer to [link / reference]. Figure 11 , Figure 11 This is a schematic diagram of a multi-device system 100 according to some embodiments of the present disclosure. In some embodiments, such as Figure 11 As shown, the multi-device system 100 also includes another peripheral device 15. The configuration of peripheral device 15 is similar to or the same as that of peripheral device 13, and therefore will not be described again here. In some embodiments, the verification of spatial relationships obtained in operation S201 can be performed by either the host device 11 or the peripheral device 15, which will be discussed in the following paragraphs. Figure 12 Detailed explanation. Figure 12 This is a flowchart illustrating a tracking method 200 according to some embodiments of the present disclosure. In some embodiments, such as Figure 12 As shown, the tracking method 200 further includes operations S1201 and S1202, with operation S1201 being executed after operation S202. That is, operation S1201 is executed after the host device 11 receives the spatial relationship and attitude PPE obtained in operation S201 from the peripheral device 13. As previously described, in some embodiments, the host device 11 may be replaced by another peripheral device 15 in subsequent sub-operations.

[0081] In operation S1201, the host device 11 determines whether the spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10 is acceptable. This will be explained in the following paragraphs. Figure 13 Detailed explanation. Figure 13 This is a flowchart of one operation S1201 of the tracking method 200 shown in some embodiments of the present disclosure. In some embodiments, such as Figure 13 As shown, operation S1201 includes multiple sub-operations S1301 to S1303. As described in operation S204, sub-operations S1301 to S1303 can also be executed when peripheral device 13 is installed on player device 10 to verify the currently used spatial relationships.

[0082] In operation S1301, the host device 11 captures an image frame IMF3 associated with the player device 10. (See also...) Figure 14 , Figure 14 This is a schematic diagram of an image frame IMF3 captured by the host device 11 according to some embodiments of the present disclosure. In some embodiments, since the host device 11, worn on the head of the user U1, is typically farther away from the player device 10 than the peripheral devices 13, the field of view of the camera 112 can cover all or part of the player device 10 and all or part of the peripheral devices 13. Therefore, in Figure 14 In the image frame IMF3 captured by the camera 112 of the host device 11, there is an image IM10 of the player device 10 and an image IM13 of the peripheral device 13 installed on the player device 10.

[0083] In operation S1302, the host device 11, based on spatial relationships and the orientation PPE of the peripheral device 13, generates a projection (not shown) related to the player device 10 on the image frame IMF3 to obtain a difference related to the player device 10. Figure 11 In this embodiment, the host device 11 uses the position and / or orientation of the player device 10 recorded in the reference image (which can be calculated based on the extrinsic data DEX (i.e., spatial relationships)) and the pose PPE to generate reference extrinsic information (not shown) of the player device 10 relative to the host device 11. Specifically, the reference extrinsic information indicates the position and / or orientation of the player device 10 relative to the host device 11 when the reference image or extrinsic data DEX is generated. The host device 11 calculates based on the reference extrinsic information, projecting the player device 10 with the position and / or orientation indicated by the reference extrinsic information onto the image frame IMF3 to generate a projection associated with the player device 10. Furthermore, the host device 11 calculates a pixel offset (e.g., the translation distance from a pixel in the projection associated with the player device 10 to a corresponding pixel in the image IM10 in the image frame IMF3) as a difference associated with the player device 10.

[0084] In sub-operation S1303, the host device 11 compares the difference with a difference threshold. As described above in sub-operations S703 or S903, in some embodiments, the host device 11 compares the pixel offset with the pixel length via the processor 110 to determine whether the difference is greater than the difference threshold. Specifically, as... Figure 13As shown, when the difference is greater than the difference threshold (that is, when the pixel offset exceeds the pixel length), operation S1202 will be executed to allow the peripheral device 13 to update the currently used spatial relationship. Furthermore, when the difference is less than the difference threshold (that is, when the pixel offset does not exceed the pixel length), operation S203 will be executed to display the virtual player device in the immersive environment with an accurate position and / or orientation based on the currently used spatial relationship.

[0085] At Figure 13 In some further embodiments, sub-operations S1301 to S1303 are executed only when the peripheral device 13 is installed on the player device 10, the player device 10 is placed in a stable position in the physical environment E1, and the distance between the host device 11 and the player device 10 does not exceed, for example, 0.5 meters. With this setting, because the spatial relationship (i.e., the extrinsic parameter data DEX) and the posture PPE remain unchanged, the computational burden of the processor 110 in the host device 11 is reduced, making the execution of sub-operations S1301 to S1303 smoother.

[0086] To further explain, Figure 13 The flowchart also applies to embodiments where the player device 10 is equipped with a trackable object 30. For example, the image frame IMF3 associated with the player device 10 captured in sub-operation S1301 may include an image IM30 of the trackable object 30. In sub-operation S1302, a projection associated with the player device 10 is generated by projecting the trackable object 30, having a position and / or orientation calculated based on extrinsic data DEX and attitude PPE, onto the image frame IMF3. The difference associated with the player device 10 is a one-pixel offset from the projection of the trackable object 30 onto the image IM30 of the trackable object 30 in the image frame IMF3.

[0087] like Figure 12As shown, in operation S1202, the host device 11 transmits another spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10 to the peripheral device 13. In some embodiments, prior to operation S1202, the host device 11 calculates the position and / or orientation of the player device 10 relative to the peripheral device 13 installed on the player device 10 as another spatial relationship using feature extraction-based positioning technology and / or interaction-based tracking technology by the processor 110. Specifically, as described above in sub-operations S301-S303, the host device 11 can generate another feature point model (not shown) of the player device 10 based on the image of the player device 10 and a preset 3D model PM, and can generate a pose data (not shown) of the player device 10 based on the other feature point model. Furthermore, as described above in sub-operations S501 and S502, the host device 11 can generate pose data of the player device 10 by tracking the trackable object 30 based on the image of the trackable object 30. Next, the host device 11 can use the attitude data of the player device 10 and the attitude PPE received from the peripheral device 13 to generate another extrinsic data DEXU of the player device 10 relative to the peripheral device 13. As can be seen from the above description, the attitude data of the player device 10 is generated by the host device 11 based on image information related to the player device 10, such as the image of the player device 10, the image of the trackable object 30 set on the player device 10, etc.

[0088] In some embodiments of operation S1202, such as Figure 11 As shown, the host device 11 transmits the extrinsic data DEXU as another spatial relationship to the peripheral device 13, so that the peripheral device 13 can update the currently used spatial relationship through the extrinsic data DEXU.

[0089] In some further embodiments, the peripheral device 13 may verify another spatial relationship transmitted from the host device 11, rather than directly using another spatial relationship, as will be discussed in the following paragraphs. Figure 15 Detailed explanation. Figure 15 This is a flowchart illustrating a tracking method 200 according to some embodiments of the present disclosure.

[0090] Accept Figure 13 The description of the embodiments is as follows: Figure 15 As shown, the tracking method 200 also includes multiple operations S1501 and S1502, and operation S1501 is executed after operation S1202. In operation S1501, the peripheral device 13 detects whether another spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10 is valid. A detailed description of operation S1501 is provided below. Figure 7 The detailed description of operation S204 in or 9 is similar or the same, so it is omitted here.

[0091] At Figure 15 In this embodiment, when the peripheral device 13 detects that another spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10 is valid, operation S1502 will be executed. In operation S1502, the peripheral device 13 replaces the spatial relationship with another spatial relationship. That is, the peripheral device 13 updates the currently used spatial relationship. Specifically, the peripheral device 13 replaces the extrinsic parameter data DEX with the extrinsic parameter data DEXU to update the currently used spatial relationship. Furthermore, when the peripheral device 13 detects that another spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10 is invalid, operation S202 will be executed. That is, the currently used spatial relationship will continue to be used, and the other spatial relationship received from the host device 11 may be discarded or not stored. However, this disclosure is not limited thereto. For example, in some embodiments, when the peripheral device 13 detects that another spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10 is not qualified, operation S204 may also be performed so that the peripheral device 13 can further verify the spatial relationship currently in use.

[0092] As can be seen from the above embodiments of this disclosure, by obtaining and / or verifying the spatial relationship between the player device 10 and the peripheral device 13 installed on the player device 10, the host device 11 can display the virtual player device corresponding to the player device 10 in an immersive environment with accurate position and / or orientation. This provides the user U1 with a visual perception that matches the user U1's tactile perception of the player device 10. Furthermore, any tracking device that can be installed on the player device 10 can be used to replace the peripheral device 13, thus eliminating the need to redesign an additional tracking module to track the player device 10. In summary, the multi-device system 100 and tracking method 200 of this disclosure have advantages such as cost savings, high compatibility, and a good user experience.

[0093] The methods of this disclosure can exist in the form of program code. The program code can be contained in a physical medium, such as a floppy disk, optical disk, hard disk, or any other transient or non-transitory computer-readable storage medium, wherein when the program code is loaded and executed by a computer, the computer becomes an apparatus for implementing the methods. The program code can also be transmitted via some transmission medium, such as wires or cables, via optical fibers, or via any other transmission mode, wherein when the program code is received, loaded, and executed by a computer, the computer becomes an apparatus for implementing the methods. When implemented on a general-purpose processor, the program code, in conjunction with the processor, provides a unique apparatus that operates similarly to an application-specific integrated circuit (ASIC).

[0094] Although the present disclosure has been described above with reference to embodiments, it is not intended to limit the present disclosure. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be determined by the appended claims.

Claims

1. A tracking method, characterized in that, Applicable to a multi-device system, wherein the multi-device system is used to track a player device in a physical environment, and includes a host device and a peripheral device, the tracking method comprising: The spatial relationship between the player device and the peripheral device installed on the player device is obtained through the peripheral device; The spatial relationship and a first posture of the peripheral device are transmitted to the host device through the peripheral device, wherein the first posture is generated by the peripheral device installed on the player device. Through the host device, based on the spatial relationship and the first posture, a second posture of the player device is generated, wherein the spatial relationship, the first posture, and the second posture are defined in a spatial coordinate system; and In response to the detection of an invalid spatial relationship, the spatial relationship between the player device and the peripheral device installed on the player device is updated via the peripheral device.

2. The tracking method as described in claim 1, characterized in that, The spatial relationship between the player device and the peripheral device installed on the player device, obtained through the peripheral device, includes: The peripheral device is used to capture image data from the player's device. Using the peripheral device, based on the image data and a preset 3D model of the player device, a feature point model of the player device is generated; and When the peripheral device is installed on the player device, a third pose of the player device is generated based on the feature point model through the peripheral device, and the third pose and the first pose are used to generate extrinsic data of the player device relative to the peripheral device as the spatial relationship, wherein the third pose is defined in the spatial coordinate system.

3. The tracking method as described in claim 1, characterized in that, At least one trackable object is mounted on the player device, and the spatial relationship between the player device and the peripheral device mounted on the player device is obtained through the peripheral device, including: When the peripheral device is installed on the player device, a third pose of the at least one trackable object is generated by the peripheral device based on the image data of the at least one trackable object, and the third pose and the first pose are used to generate extrinsic data of the player device relative to the peripheral device as the spatial relationship, wherein the third pose is defined in the spatial coordinate system.

4. The tracking method as described in claim 1, characterized in that, Also includes: The peripheral device detects whether the spatial relationship between the player device and the peripheral device installed on the player device is valid, and when the spatial relationship is valid, the spatial relationship is not updated.

5. The tracking method as described in claim 4, characterized in that, Detecting whether the spatial relationship between the player device and the peripheral device installed on the player device is acceptable through the peripheral device includes: When the peripheral device is installed on the player device, an image frame containing an image of the player device is captured through the peripheral device; Using the peripheral device, based on the spatial relationship, a projection of the player device is generated on the image frame to obtain a difference between the projection and the image; and The difference is compared with a difference threshold using the peripheral device. When the difference is less than the difference threshold, the spatial relationship is considered acceptable, and when the difference is greater than the difference threshold, the spatial relationship is considered unacceptable.

6. The tracking method as described in claim 4, characterized in that, At least one trackable object is mounted on the player device, and the detection of whether the spatial relationship between the player device and the peripheral device mounted on the player device is acceptable includes: When the peripheral device is installed on the player device, an image frame containing an image of the at least one trackable object is captured through the peripheral device; Using the peripheral device, based on the spatial relationship, a projection of the at least one trackable object is generated on the image frame to obtain a difference between the projection and the image; and The difference is compared with a difference threshold using the peripheral device. When the difference is less than the difference threshold, the spatial relationship is considered acceptable, and when the difference is greater than the difference threshold, the spatial relationship is considered unacceptable.

7. The tracking method as described in claim 1, characterized in that, Also includes: The spatial relationship between the player device and the peripheral device installed on the player device is determined by the host device or another peripheral device in the multi-device system. as well as In response to the determination that the spatial relationship is invalid, another spatial relationship between the player device and the peripheral device installed on the player device is transmitted to the peripheral device via the host device or the other peripheral device. The other spatial relationship is generated based on the first posture of the peripheral device and a third posture of the player device. The third posture is generated by the host device or the other peripheral device based on image information related to the player device and is defined in the spatial coordinate system.

8. The tracking method as described in claim 7, characterized in that, Determining whether the spatial relationship between the player device and the peripheral device installed on the player device is acceptable, through the host device or the other peripheral device, includes: A video frame related to the player device is captured through the host device or the other peripheral device; Through the host device or the other peripheral device, based on the spatial relationship and the first posture, a projection related to the player device is generated on the image frame to obtain a difference related to the player device; and The difference is compared with a difference threshold by the host device or the other peripheral device, wherein the spatial relationship is qualified when the difference is less than the difference threshold, and the spatial relationship is unqualified when the difference is greater than the difference threshold.

9. The tracking method as described in claim 7, characterized in that, Also includes: The peripheral device is used to detect whether the spatial relationship between the player device and the peripheral device installed on the player device is acceptable; and In response to the detection that the other spatial relationship is valid, the spatial relationship is replaced with the other spatial relationship through the peripheral device.

10. A multi-device system, characterized in that, Used to track a player's device in a physical environment, and includes: A peripheral device is used to obtain a spatial relationship between the player device and the peripheral device installed on the player device, and to update the spatial relationship between the player device and the peripheral device installed on the player device in response to detecting that the spatial relationship is invalid; as well as A host device is configured to receive the spatial relationship and a first posture of the peripheral device from the peripheral device, and to generate a second posture of the player device based on the spatial relationship and the first posture. The first posture is generated by the peripheral device installed on the player's device, and the spatial relationship, the first posture, and the second posture are defined in a spatial coordinate system.

11. The multi-device system as claimed in claim 10, characterized in that, It also includes a connection device, through which the peripheral device is mounted on the player device. The peripheral device includes a camera, and when the peripheral device is installed on the player device, a part of the player device or at least one trackable object on the player device is within the field of view of the camera.

12. A non-transitory computer-readable storage medium, characterized in that, A computer program is provided for executing a tracking method, wherein the tracking method is applicable to a multi-device system for tracking a player device in a physical environment, the multi-device system including a host device and a peripheral device, and the tracking method includes: The spatial relationship between the player device and the peripheral device installed on the player device is obtained through the peripheral device; The spatial relationship and a first posture of the peripheral device are transmitted to the host device through the peripheral device, wherein the first posture is generated by the peripheral device installed on the player device. Through the host device, based on the spatial relationship and the first posture, a second posture of the player device is generated, wherein the spatial relationship, the first posture, and the second posture are defined in a spatial coordinate system; and In response to the detection of an invalid spatial relationship, the spatial relationship between the player device and the peripheral device installed on the player device is updated via the peripheral device.

Images