Information processing method and apparatus, and device, storage medium and program product

Abstract

Provided in the embodiments of the present disclosure are an information processing method and apparatus, and a device, a storage medium and a program product. The method comprises: on the basis of collected image data, determining first pose information of a first locator among a plurality of locators, wherein the plurality of locators are physically connected to a target object; and on the basis of the first pose information and first relative pose information between a second locator and the first locator among the plurality of locators, determining second pose information of the second locator.

Description

Information processing method, apparatus, device, storage medium, and program product

[0001] The present application claims priority to the Chinese patent application No. 202410705474.X, filed on May 31, 2024, entitled “Information processing method, apparatus, device, storage medium, and program product”, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to an information processing method, apparatus, device, computer-readable storage medium, and computer program product. BACKGROUND

[0003] Extended Reality (XR) is widely researched and applied. XR combines hardware devices and various technical means to fuse virtual content and real scenes, and provides users with a unique sensory experience. XR includes, for example, Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

[0004] Thus, a user can be provided with an immersive interactive experience in a virtual environment by wearing an XR device. SUMMARY

[0005] In a first aspect of the present disclosure, an information processing method is provided. The method comprises: determining first pose information of a first locator of a plurality of locators based on collected image data, the plurality of locators being physically connected to a target object; and determining second pose information of a second locator of the plurality of locators based on the first pose information and first relative pose information between the second locator and the first locator.

[0006] In a second aspect of the present disclosure, an information processing apparatus is provided. The apparatus includes a first pose information determination module configured to determine first pose information of a first localizer of a plurality of localizers based on captured image data, the plurality of localizers being physically connected to a target object; and a second pose information determination module configured to determine second pose information of a second localizer of the plurality of localizers based on the first pose information and first relative pose information between the second localizer and the first localizer.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the device to perform the method of the first aspect.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon computer-executable instructions that are executable by a processor to implement the method of the first aspect.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0010] It should be understood that all statements herein made regarding the exemplary embodiments of the present disclosure are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future (e.g., any elements developed that perform the same function, regardless of structure). The scope of the present disclosure, therefore, is not intended to be limited to the exemplary embodiments described herein. BRIEF DESCRIPTION OF DRAWINGS

[0011] The above and other features, aspects, and advantages of various embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals denote like elements, and wherein:

[0012] FIG. 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0013] FIG. 2 shows a flowchart of a process of device control according to some embodiments of the present disclosure;

[0014] FIG. 3 shows a three-dimensional model schematic diagram of playing table tennis in a virtual environment according to some embodiments of the present disclosure;

[0015] FIG. 4 shows a block diagram of an information processing apparatus according to some embodiments of the present disclosure; and

[0016] FIG. 5 illustrates a block diagram of an electronic device that can implement one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the range of use, the scenario of use, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0018] For example, in response to receiving an active request of a user, a prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the electronic device, application program, server or storage medium, etc. software or hardware that performs the operation of the technical solutions of the present disclosure according to the prompt information.

[0019] As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending a prompt information to the user may, for example, be a pop-up window manner, in which the prompt information can be presented in a text manner. In addition, the pop-up window can also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0020] It can be understood that the above notification and obtaining of user authorization process is only illustrative, and does not limit the implementation manner of the present disclosure, and other manners meeting the relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0021] It can be understood that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the relevant laws and regulations and the relevant provisions.

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, rather, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the scope of protection of the present disclosure.

[0023] It should be noted that the titles of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and any type of embodiment can be included under any section / subsection. Furthermore, embodiments described in any section / subsection can be combined with any other embodiment described in the same section / subsection and / or in a different section / subsection in any manner.

[0024] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0025] Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Environment 100 can be a physical scene, i.e., environment 100 can be an example of a real-world scene. In environment 100, user 110 can interact by wearing at least one extended reality (XR) device. In some embodiments, at least one XR device may include a head-mounted XR device 120. In other embodiments, at least one XR device may also include XR manipulators (not shown) held in the left and right hands respectively.

[0026] In environment 100, user 110 can utilize an XR device to access virtual environment 140. Virtual environment 140 can exemplarily be a space. In some embodiments, user 110 can engage in immersive interactive experiences within virtual environment 140, such as sports, entertainment, etc. In some embodiments, user 110 can use a target object equipped with a locator (also referred to as a tracker) to engage in immersive interactive experiences within virtual environment 140. For example, as shown in Figure 1, user 110 can use an object 115 (shown as a ping-pong paddle in Figure 1) equipped with locator 125-1 and locator 125-2 (which can also be referred to individually or collectively as locator 125) to play ping-pong in virtual environment 140. In some embodiments, the locator 125 can be physically connected to the target object. It should be understood that the number of locators 125 physically connected to the target object can be set to one or more depending on the actual scenario requirements.

[0027] In some embodiments, a virtual object 150 for user 110 can be generated in the virtual environment 140, allowing user 110 to have an immersive interactive experience through the virtual object 150 within the virtual environment 140. Furthermore, the virtual object 150 can present an interactive experience using a target object within the virtual environment. In some embodiments, the virtual environment 140 may provide virtual tools and / or other users' virtual objects, enabling user 110 to interact with virtual tools in the virtual environment 140 through the virtual object 150, and also to interact with other users' virtual objects through the virtual object 150.

[0028] In some embodiments, the head-mounted XR device 120 can communicate with the XR manipulators and locators 125 to enable interaction with the user 110 in a coordinated manner.

[0029] In some embodiments, the XR devices can include image collectors, such as cameras, infrared cameras, etc., for collecting images to take images of the physical space. The image collectors can be provided on one or more of the XR devices, respectively. Illustratively, the image collectors can be provided on the head-mounted XR device 120 to collect images of a wider spatial range based on a higher spatial position. If an infrared camera is provided on the head-mounted XR device 120, correspondingly, at least one light source (e.g., infrared light source) can be provided on the locators 125 so that the infrared camera can take images of the optical pattern or multiple spots formed by the light source on the locators 125, thereby identifying and locating the locators 125 based on the obtained images. In some embodiments, a specific locator 125 can be identified based on a specific optical pattern or spot of the locator 125. In some embodiments, the identification of a specific locator 125 can not rely on the light source information, but rather, image recognition of each of the locators 125 can be performed on the collected images, and based on the image recognition result, it can be determined whether a specific locator 125 is captured in the images. In such embodiments, one or more of the locators 125 can not need to be provided with a light source.

[0030] In other scenarios, the image collectors for image collection can be independent of the XR devices (e.g., independent of the head-mounted XR device 120). Such image collectors can be independently deployed for image perception of the locators 125 and other devices and environments, etc., in the environment.

[0031] In some embodiments, the user 110 can utilize multiple XR devices and locators 125 to communicate with the electronic device 130 to utilize the electronic device 130 to process data about the locators 125 obtained by the XR devices and to issue data to the XR devices. It should be understood that in some cases, the XR devices 120 can also not communicate with the electronic device 130 (or in other words, the environment 100 can also illustratively not include the electronic device 130). In this case, the interaction with the locators 125 can be performed through the head-mounted XR device 120. Specifically, the data about the locators 125 can be processed through the information processing device 121 in the head-mounted XR device 120. It should be understood that the information processing device 121 can not be limited to being provided in the head-mounted XR device 120, but can be provided in other XR devices, or can exist as an independent control device.

[0032] The electronic device 130 can be a separate device capable of communicating with the XR device and / or other image capture devices, such as a server, a computing node, etc. for image or data processing, and can also be integrated with the XR device and / or other image capture devices. In some embodiments, the electronic device 130 can be implemented as an XR device, that is, in this case, the XR device can implement all the functions of the electronic device 130. It should be understood that the above description of the electronic device 130 is merely exemplary and not limiting, and the electronic device 130 can be implemented as a device in various forms, structures or categories, and the embodiments of the present disclosure have no limitation in this regard.

[0033] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure.

[0034] As briefly mentioned above, a user can achieve an immersive experience in a virtual environment by wearing an XR device and using an object with a locator (i.e. tracker, also referred to as tracker) installed. Illustratively, the locator can be an optical tracker. The optical tracker can be installed on any object, giving physical tracking capability, and complementary tracking.

[0035] However, when using a locator to assist tracking, due to the poor shooting angle of the camera or the locator being blocked, the camera cannot capture the infrared light on the locator for positioning, resulting in the problem of the locator being lost.

[0036] Embodiments of the present disclosure propose an information processing scheme. According to various embodiments of the present disclosure, first pose information of a first locator of a plurality of locators is determined based on collected image data, the plurality of locators being physically connected to a target object. And, second pose information of a second locator of the plurality of locators is determined based on the first pose information and first relative pose information between the second locator and the first locator. Thereby, this scheme can solve the problem of the target object being lost due to the locator 125 being blocked and thus unable to be captured by the image collector, by using the plurality of locators 125 to complementarily assist tracking of any target object, further ensuring stable tracking and efficient positioning of the target object.

[0037] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0038] Figure 2 illustrates a flowchart of an information processing procedure 200 according to some embodiments of the present disclosure. For ease of discussion, these embodiments will be described with reference to the environment 100 of Figure 1. In some embodiments, the information processing method may be implemented at an extended reality (XR) device. In some examples, these embodiments may be implemented in an information processing device 121 of an XR device. In other examples, these embodiments may also be implemented in a remote controller, which may, for example, be located in the electronic device 130 of Figure 1. The following specific embodiments are exemplified by implementation in information processing device 121, but this is merely exemplary and not intended to be limiting.

[0039] It should also be understood that although the example embodiment is described with reference to the environment 100 involving the XR device shown in FIG1, the device control scheme provided by the embodiments of this disclosure can also be applied to non-XR environments, as long as it is necessary to use an image acquisition device to track and locate target objects with locators installed in the environment.

[0040] As shown in Figure 2, in block 210, the information processing device 121 can determine the first pose information of a first locator among a plurality of locators 125 based on acquired image data. The plurality of locators are physically connected to a target object. In some embodiments, the image data may be an image of a physical environment captured by an image acquisition device, which may include a target object on which the locators 125 are installed (or physically connected). In this document, the target object refers to a physical object or an object in the real world. In some embodiments, the target object may include at least a rigid body portion for connecting the locators 125.

[0041] For example, at a certain shooting angle of the image acquisition device, or at a certain position and / or attitude of the target object, the first locator among multiple locators 125 can be captured by the image acquisition device. Taking Figure 1 as an example, assume that locator 125-1 is the first locator that can be captured. In this case, the information processing device 121 can determine the first pose information of the first locator 125-1 based on the acquired image data. In this document, pose information may include position information, such as coordinate information, or attitude information, such as pitch angle information, yaw angle information, and roll angle information. Pose information may also include both position information and attitude information.

[0042] In some embodiments, in determining the first pose information of the first localizer 125-1, the information processing device 121 can determine whether the image data of the target object captures each localizer 125 in the plurality of localizers 125. And if it is determined that the image data captures the first localizer 125-1 in the plurality of localizers 125, the information processing device 121 can determine the first pose information of the first localizer 125-1 in the plurality of localizers 125 based on the captured image data. In some embodiments, in determining whether the image data of the target object captures each localizer 125 in the plurality of localizers 125, the information processing device 121 can determine whether each localizer 125 in the plurality of localizers 125 is captured based on the optical information in the image data of the target object. In some embodiments, in determining that the image data captures the first localizer 125-1 in the plurality of localizers 125, the information processing device 121 can determine the localizer 125 corresponding to the optical information in the image data. And if it is determined that the optical information corresponds to at least the first localizer 125-1, the information processing device 121 can determine that the first localizer 125-1 in the plurality of localizers 125 is captured.

[0043] Since the localizer 125 can realize optical positioning tracking by deploying a specific light source (such as an optical infrared lamp) and taking a specific light source of the localizer 125 by an image collector (such as an infrared camera), according to the optical information (which can also be referred to as light source information) in the captured image data (which can also be referred to as image feature data), the information processing device 121 can determine whether each localizer 125 in the plurality of localizers 125 is captured. This can specifically involve the following two scenarios.

[0044] In one scenario, there can be many light spots in the captured image, and through these light spots, the structural relationship of several localizers 125 can be mapped. At this time, the information processing device 121 can determine which light spots belong to the localizer 125, and the determined localizer 125 can be the first localizer 125-1.

[0045] In another scenario, the information processing device 121 can calculate all the light spots in the captured image using an IMU (Inertial Measurement Unit), as long as the pose information can be calculated, the corresponding first localizer 125-1 of the light spot can be determined, and the first pose information of the first localizer 125-1 is also obtained.

[0046] In some embodiments, in determining the first pose information of the first locator of the plurality of locators based on the captured image data, the information processing device 121 can determine the first pose information of the first locator based on the captured image data at least by a computer vision algorithm. Exemplarily, the computer vision algorithm can be a PNP (Perspective-n-Point) algorithm. At this time, for the captured image data, the pose of the first locator 125-1 can be solved by the PNP algorithm, and in combination with the 6-axis IMU inertial navigation data, the 6DOF (Degree of Freedom) positioning data of the first locator 125-1 can be estimated in real time.

[0047] In this way, by installing a plurality of locators 125 on the target object, and by determining the first pose information of the first locator 125-1 of the plurality of locators 125 based on the captured image data, the problem of the locators 125 being lost due to the camera being unable to capture the infrared lights for positioning on the locators 125 due to the difference in the shooting angle of the camera or the locators 125 being blocked can be solved.

[0048] With continued reference to FIG. 2, at block 220, based on the first pose information and the first relative pose information between the second locator 125-2 and the first locator 125-1 of the plurality of locators 125, the information processing device 121 can determine second pose information of the second locator 125-2. In some embodiments, in determining the second pose information of the second locator 125-2, if the pose information of the second locator 125-2 cannot be determined from the image data, based on the first pose information and the first relative pose information between the second locator 125-2 and the first locator 125-1 of the plurality of locators, the information processing device 121 can determine the second pose information of the second locator 125-2.

[0049] Still taking the example that the target object 115 in FIG. 1 is a table tennis bat, the table tennis bat can be tracked by pre-installing the first locator 125-1 and the second locator 125-2 on the table tennis bat. In some embodiments, the first locator 125-1 and the second locator 125-2 can be symmetrically installed on the table tennis bat. The installation positions of the first locator 125-1 and the second locator 125-2 can be determined according to actual scenarios, which are not limited herein. FIG. 3 shows a three-dimensional model schematic diagram of playing table tennis in a virtual environment according to some embodiments of the present disclosure. As shown in FIG. 3, the user 110 can play table tennis in the virtual environment 140 when using the physical table tennis bat, which is more immersive, by using the embodiments of the present disclosure. During use, if the second locator 125-2 is rotated to the back of the table tennis bat and is blocked by the table tennis bat, the image collector cannot capture the light source of the second locator 125-2. In such a case, the first relative pose information between the first locator 125-1 and the second locator 125-2 can be used to determine the second pose information of the second locator 125-2.

[0050] In some scenarios, based on the settings of the user 110, the second pose information of the second locator 125-2 can also directly represent the pose information of the target object. For example, if the second locator 125-2 is arranged at the center position of the target object, and the calibration between the real table tennis bat in the real environment and the virtual table tennis bat 145 in the virtual environment 140 is set by the user 110, the second pose information of the second locator 125-2 can be directly used to represent the pose information of the target object. The second pose information of the second locator 125-2 can also be used to obtain the pose information of the target object by adding a calibrated offset. In this way, the pose information of the target object can be more efficiently determined, which facilitates faster tracking of the target object.

[0051] In some embodiments, the first relative pose information between the second locator 125-2 and the first locator 125-1 can be determined in the following manner: the information processing device 121 can obtain reference image data of the target object, the reference image captures the first locator 125-1 and the second locator 125-2, and determines the first reference pose information of the first locator 125-1 and the second reference pose information of the second locator 125-2 based on the reference image data. Then, based on the first reference pose information and the second reference pose information, the information processing device 121 can determine the first relative pose information between the first locator 125-1 and the second locator 125-2.

[0052] In such embodiments, the second locator 125-2 can also be considered as a master locator, and the first locator 125-1 can also be considered as a slave locator. It should be appreciated that the number of the first locators 125-1 and the second locators 125-2 can be set to one or more, respectively, according to actual scenarios. When the locators 125 are installed on the target object, the user 110 can install the plurality of locators 125 at different positions of the target object, respectively, so that the target object can be photographed by the image collector to at least one locator 125 from multiple angles when the target object is used. Alternatively, the target object can be provided with a plurality of slots for installing the locators 125, and the plurality of slots are provided in advance at different positions of the target object, so that the target object can be photographed by the image collector to at least one locator 125 from multiple angles when the target object is used. If the target object 115 is a table tennis bat, at least one locator 125 is visible when the table tennis bat is rotated to any angle.

[0053] If the first relative pose information between the first locator 125-1 and the second locator 125-2 is to be determined, the first locator 125-1 and the second locator 125-2 need to be photographed by the image collector. Specifically, the pose of the target object can be adjusted by rotating or the like, so that the first locator 125-1 and the second locator 125-2 are photographed by the image collector. In this way, the first locator 125-1 and the second locator 125-2 can be captured in the reference image data collected for the target object. In some embodiments, the reference image data can include a set of image data. Then, based on the set of image data, the first reference pose information of the first locator 125-1 and the second reference pose information of the second locator 125-2 can be determined by using a computer vision algorithm in combination with 6-axis IMU inertial navigation data. Thus, the first relative pose information between the first locator 125-1 and the second locator 125-2 can be further determined based on the first reference pose information and the second reference pose information.

[0054] In some embodiments, the reference image data can include a plurality of sets of image data. And when determining the first reference pose information of the first locator 125-1 and the second reference pose information of the second locator 125-2 based on the reference image data, the information processing device 121 can determine a plurality of sets of first reference pose information of the first locator 125-1 and a plurality of sets of second reference pose information of the second locator 125-2 from the plurality of sets of image data, respectively. And when determining the first relative pose information between the first locator 125-1 and the second locator 125-2, based on the plurality of sets of first reference pose information and the plurality of sets of second reference pose information, the information processing device 121 can determine the first relative pose information between the first locator 125-1 and the second locator 125-2.

[0055] In such embodiments, for collecting multiple sets of image data, there can be two scenarios. In one scenario, each frame of image data includes images of the first localizer 125-1 and the second localizer 125-2. At this time, the first relative pose information between the first localizer 125-1 and the second localizer 125-2 can be determined in particular by the following manner.

[0056] 1) For multiple sets of image data, assume that the 6DOF data of the first localizer 125-1 and the second localizer 125-2 are represented as wherein represents the 6DOF pose of the second localizer 125-2 in the i-th frame of the world coordinate system (i.e., the coordinate system of the environment space of the current scene), represents the 6DOF pose of the first localizer 125-1 in the i-th frame of the world coordinate system.

[0057] 2) The first relative pose information between the first localizer 125-1 and the second localizer 125-2 can satisfy the following formula:

[0058] Then the above formula 1 problem similar to AX = B is solved by an optimization method (e.g., a fitting algorithm), and the first relative pose information is obtained as

[0059] Thus, by determining the first relative pose information in this manner, the accuracy of the first relative pose information can be ensured, so that the multi-localizer 125 assisted complementary tracking can be more accurate.

[0060] In another scenario of time-division lighting of the localizer 125, each frame of image data in the multiple sets of image data can only include images of the first localizer 125-1 or images of the second localizer 125-2. That is, the light sources of the first localizer 125-1 and the second localizer 125-2 are not lit at the same time, but only the first localizer 125-1 or the second localizer 125-2 can be lit at a time. Therefore, a series of images are needed to calculate the first relative pose information between the first localizer 125-1 and the second localizer 125-2, and the calculation manner is not described herein.

[0061] In some embodiments, based on the second pose information of the second localizer 125-2 and the second relative pose information between the second localizer 125-2 and the target object, the information processing device 121 can determine the object pose information of the target object.

[0062] As discussed above, in some scenarios, the pose information of the target object can be directly represented by the second pose information of the second localizer 125-2, or the pose information of the target object can be obtained by adding the offset after calibration to the second pose information of the second localizer 125-2. In this embodiment, to determine the object pose information of the target object, the second pose information of the second localizer 125-2 and the second relative pose information between the second localizer 125-2 and the target object can be used for calculation.

[0063] The determination manner of the second pose information of the second localizer 125-2 has been discussed above. As for the second relative pose information between the second localizer 125-2 and the target object, it can be determined based on the selection setting of the user 110, for example. For example, the user 110 can select the relative pose relationship between the second localizer 125-2 and the target object from a plurality of options pre-set in the XR device, and then the second relative pose information is determined by the information processing device 121. For another example, the user 110 can also determine the relative pose between the second localizer 125-2 and the target object through at least one calibration. It should be understood that the second relative pose information between the second localizer 125-2 and the target object can be set according to specific application scenarios, and this is not intended to be limited herein. As for determining the object pose information of the target object based on the second pose information and the second relative pose information, the following manner can be adopted.

[0064] Suppose the relative pose relationship (i.e., the second relative pose information) between the second localizer 125-2 and the target object is When the second localizer 125-2 can be photographed by the image collector, the 6DOF pose of the target object in the world coordinate system is:

[0065] When the target object is rotated to the second localizer 125-2 is blocked and invisible (i.e., cannot be photographed by the image collector), while the first localizer 125-1 is visible (i.e., can be photographed by the image collector), the 6DOF pose of the target object in the world coordinate system is:

[0066] In formula 3, the second pose information of the second localizer 125-2 can be calculated according to the first pose information of the first localizer 125-1 and the first relative pose information between the second localizer 125-2 and the first localizer 125-1. Then, based on the second pose information of the second localizer 125-2 and the second relative pose information between the second localizer 125-2 and the target object, the object pose information of the target object can be calculated.

[0067] Thus, it is possible to achieve complementary tracking of the target object by using multiple locators 125. In addition, the number of locators 125 can be determined according to different target objects. More locators 125 can ensure more stable tracking of the target object.

[0068] In some embodiments, the second locator 125-2 can be determined as the locator having the second relative pose information with the target object in various ways. In other words, the primary locator for the target object can be set in advance. The following will specifically illustrate example embodiments of determining the second locator 125-2.

[0069] In some embodiments, the multiple locators 125 can be wirelessly connected to the XR device, and the second locator 125-2 can be determined as the locator having the second relative pose information with the target object in the following way: the information processing device 121 can detect that the second locator 125-2 is first connected to the XR control device among the multiple locators. That is, among the multiple locators 125, the locator 125 that first establishes a connection with the XR device can be considered as the second locator 125-2 (i.e., the primary locator). The locators 125 that establish a connection with the XR device later can be considered as the first locators 125-1 (i.e., the secondary locators). Thus, the primary locator can be determined quickly.

[0070] In some embodiments, the second locator 125-2 can also be determined as the locator having the second relative pose information with the target object in the following way: receiving a user selection of a physical button connected to the second locator 125-2 among the multiple locators. In such embodiments, each locator 125 can be configured with a physical button, which can be configured to receive a press of the user 110 so that the corresponding locator 125 is selected as the second locator 125-2. Exemplarily, the user 110 can select the physical button based on the guidance information presented through the XR device. Thus, the user 110 can be facilitated to determine the second locator 125-2.

[0071] In some embodiments, the second locator 125-2 can also be determined as the locator with the second relative pose information to the target object by presenting guidance information for the second locator 125-2, and the second relative pose information between the second locator 125-2 and the target object is determined based on the relative pose information between the second locator 125-2 and the XR device obtained after the guidance information is presented. Taking the XR device as the head-mounted XR device 120 for example, after the guidance information is presented by the head-mounted XR device 120, the user 110 can adjust the relative pose information between the second locator 125-2 and the head-mounted XR device 120 based on the guidance information. For example, the guidance information can be a virtual circle, and the user 110 can adjust the pose and / or position of the target object so that the corresponding identifier of the second locator 125-2 is moved into the virtual circle. In this way, the second relative pose information between the second locator 125-2 and the target object can also be determined. Thus, while the second locator 125-2 is determined, the second relative pose information between the second locator 125-2 and the target object can be determined more quickly, which is beneficial to improving the user experience.

[0072] The embodiments of the present disclosure can solve the problem that the target object is lost due to the fact that some locators 125 are blocked and thus cannot be photographed by the image collector, by using multiple locators 125 to complementarily assist in tracking any target object, and further ensure stable tracking of the target object.

[0073] FIG. 4 shows a schematic structural block diagram of an information processing apparatus 400 according to certain embodiments of the present disclosure. The apparatus 400 can be implemented as or included in the information processing device 121. Various modules / components in the apparatus 400 can be implemented by hardware, software, firmware, or any combination thereof.

[0074] As shown, the apparatus 400 includes a first pose information determination module 410 configured to determine first pose information of a first locator of multiple locators based on collected image data, the multiple locators being physically connected to a target object; and a second pose information determination module 420 configured to determine second pose information of a second locator of the multiple locators based on the first pose information and first relative pose information between the second locator and the first locator.

[0075] In some embodiments, the first pose information determination module 410 is further configured to determine whether image data of the target object captures each of the multiple locators connected to the target object, and if it is determined that the image data captures the first locator of the multiple locators, determine the first pose information of the first locator of the multiple locators based on the collected image data.

[0076] In some embodiments, the first pose information determination module 410 comprises a capture determination module configured to determine whether each of the plurality of locators is captured based on the optical information in the image data of the target object.

[0077] In some embodiments, the first pose information determination module 410 comprises a first-locator capture determination module configured to determine the locator to which the optical information in the image data corresponds; and determine that the first locator of the plurality of locators is captured if it is determined that the optical information corresponds to at least the first locator.

[0078] In some embodiments, the apparatus 400 further comprises an object pose information determination module configured to determine object pose information of the target object based on the second pose information of the second locator and the second relative pose information between the second locator and the target object.

[0079] In some embodiments, the second pose information determination module 420 is further configured to determine the second pose information of the second locator based on the first pose information and the first relative pose information between the second locator and the first locator of the plurality of locators if the pose information of the second locator cannot be determined from the image data.

[0080] In some embodiments, the apparatus 400 further comprises a first relative pose information determination module configured to obtain reference image data of the target object, the reference image data being captured by the first locator and the second locator; determine first reference pose information of the first locator and second reference pose information of the second locator based on the reference image data; and determine the first relative pose information between the first locator and the second locator based on the first reference pose information and the second reference pose information.

[0081] In some embodiments, the reference image data comprises a plurality of sets of image data, and the first relative pose information determination module is further configured to determine a plurality of sets of first reference pose information of the first locator and a plurality of sets of second reference pose information of the second locator from the plurality of sets of image data, respectively; and determine the first relative pose information between the first locator and the second locator based on the plurality of sets of first reference pose information and the plurality of sets of second reference pose information.

[0082] In some embodiments, the method is implemented at an extended reality (XR) device, and the plurality of locators are wirelessly connected to the XR device.

[0083] In some embodiments, the apparatus 400 further includes a second locator determining module configured to determine a second locator as the locator having the second relative pose information with the target object by at least one of the following: detecting that the second locator is connected to the XR control device first among the plurality of locators; receiving a user selection of a physical button to which the second locator among the plurality of locators is connected; or presenting guidance information for the second locator, the second relative pose information between the second locator and the target object being determined based on relative pose information between the second locator and the XR device obtained after the guidance information is presented.

[0084] In some embodiments, the first pose information determining module 410 is further configured to determine the first pose information of the first locator based on the captured image data by at least a computer vision algorithm.

[0085] The units and / or modules included in the apparatus 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, e.g., machine executable instructions stored on a storage medium. In addition to or alternatively, some or all of the units and / or modules in the apparatus 400 can be implemented at least partially by one or more hardware logic components. As an example and not by way of limitation, example types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0086] FIG. 5 illustrates a block diagram of an electronic device 500 in which one or more embodiments of the disclosure can be implemented. It should be understood that the electronic device 500 illustrated in FIG. 5 is merely an example and should not be construed to limit the functionality and scope of the embodiments described herein. The electronic device 500 illustrated in FIG. 5 can be used to implement the information processing device 121 of FIG. 1.

[0087] As shown in FIG. 5, the electronic device 500 is in the form of a general computing device. Components of the electronic device 500 can include, but are not limited to, one or more processors 510 or processing units, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processor 510 can be a real or virtual processor and is capable of performing various processes according to programs stored in the memory 520. In a multi-processor system, multiple processors perform computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0088] The electronic device 500 typically includes a plurality of computer storage media. Such media can be any available media that is accessible by the electronic device 500 and includes both volatile and nonvolatile media, removable and non-removable media. The memory 520 can be volatile (such as register, cache, RAM), non-volatile (such as ROM, EEPROM, flash memory), or some combination of the two. The storage device 530 can be a removable or non-removable media, and can include machine-readable media, such as flash drives, magnetic disks, or any other media that can be used to store information and / or data and that can be accessed by the electronic device 500.

[0089] The electronic device 500 can further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 5, a disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk (e.g., a CD-ROM) can be provided. In such instances, each drive can be connected to the bus (not shown) by one or more data media interfaces. The memory 520 can include a computer program product 525 having one or more program modules configured to carry out the various methods or actions of the various embodiments of the present disclosure.

[0090] The communication unit 540 enables communications with other computing devices over communication media. Additionally, the functionality of the components of the electronic device 500 can be implemented in a single computing cluster or a plurality of computer machines that are capable of communicating with one another over a communication connection. As such, the electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network nodes in the networking environment.

[0091] The input device 550 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 560 can be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 500 can also communicate with one or more external devices (not shown) such as a storage device, a display device, etc. through the communication unit 540, with one or more devices that enable a user to interact with the electronic device 500, or with any devices (e.g., a network card, a modem, etc.) that enable the electronic device 500 to communicate with one or more other computing devices. Such communication can be carried out via an input / output (I / O) interface (not shown).

[0092] According to an example implementation of the present disclosure, a computer readable storage medium is provided having computer executable instructions stored thereon, where the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is also provided that is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the method described above.

[0093] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0094] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0095] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0096] The computer program product of the present disclosure can have a signal including said computer program. This signal can be electronic, electromagnetic, optical, or any other suitable type of signal. Such a signal can be provided through a communication connection, such as electrical wiring, optical fiber, wireless interface, etc. Examples of computer program products include computer program implemented on a personal computer, server, or other networked device. A non-transitory computer readable medium, such as a floppy disk, CD-ROM, DVD-ROM, Blu-ray Disc, hard disk drive, or any other suitable non-transitory computer readable medium can store the computer program product.

[0097] Having described several implementations of the present disclosure, it will be clear to those skilled in the art that many modifications, additions, and substitutions are possible without departing from the scope and spirit of the described implementations. Many modifications and variations of the present disclosure are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims, the present disclosure can be practiced otherwise than as specifically described. While the present disclosure has been described with reference to the above examples, it is to be understood that modifications and variations of the examples can be configured and that it is therefore intended to cover any and all modifications and variations.

Claims

1. An information processing method, comprising: The first pose information of the first locator among multiple locators is determined based on the acquired image data, and the multiple locators are physically connected to the target object; as well as Based on the first pose information and the first relative pose information between the second locator and the first locator among the plurality of locators, the second pose information of the second locator is determined.

2. The method according to claim 1, wherein determining the first pose information of the first locator includes: Determine whether the image data of the target object is captured by each of the multiple locators; as well as If it is determined that the image data was captured by the first locator among the plurality of locators, the first pose information of the first locator among the plurality of locators is determined based on the acquired image data.

3. The method of claim 2, wherein determining whether image data of the target object is captured by each of the plurality of locators comprises: Based on the optical information in the image data of the target object, it is determined whether each of the plurality of locators has been captured.

4. The method of claim 3, wherein determining the first locator among the plurality of locators to which the image data was captured comprises: Determine the locator corresponding to the optical information in the image data; as well as If it is determined that the optical information corresponds at least to the first locator, the first locator among the plurality of locators is identified.

5. The method according to claim 1, further comprising: Based on the second pose information of the second locator and the second relative pose information between the second locator and the target object, the object pose information of the target object is determined.

6. The method according to claim 1, wherein determining the second pose information of the second positioner includes: If the pose information of the second locator cannot be determined from the image data, the second pose information of the second locator is determined based on the first pose information and the first relative pose information between the second locator and the first locator among the plurality of locators.

7. The method according to claim 1, wherein the first relative pose information between the second locator and the first locator is determined by the following method: Obtain reference image data of the target object, wherein the reference image is captured by the first locator and the second locator; Based on the reference image data, the first reference pose information of the first locator and the second reference pose information of the second locator are determined; as well as Based on the first reference pose information and the second reference pose information, the first relative pose information between the first locator and the second locator is determined.

8. The method according to claim 7, wherein the reference image data includes multiple sets of image data, and determining the first reference pose information of the first locator and the second reference pose information of the second locator based on the reference image data includes: Multiple sets of first reference pose information for the first locator and multiple sets of second reference pose information for the second locator are determined from the multiple sets of image data, respectively. and The first relative pose information between the first locator and the second locator includes: Based on the multiple sets of first reference pose information and the multiple sets of second reference pose information, the first relative pose information between the first locator and the second locator is determined.

9. The method of claim 1, wherein the method is implemented at an extended reality (XR) device, and the plurality of locators are wirelessly connected to the XR device.

10. The method of claim 9, further comprising: The second locator is determined to be a locator having the second relative pose information with respect to the target object by at least one of the following methods: It was detected that the second locator was first connected to the XR control device among the plurality of locators; Receive user selection for the physical button connected to the second locator among the plurality of locators; or Guidance information is presented for the second locator, and the second relative pose information between the second locator and the target object is determined based on the relative pose information between the second locator and the XR device obtained after the guidance information is presented.

11. The method according to claim 1, wherein determining the first pose information of the first locator among a plurality of locators based on the acquired image data comprises: Based on the acquired image data, the first pose information of the first locator is determined at least through a computer vision algorithm.

12. An information processing apparatus, comprising: The first pose information determination module is configured to determine the first pose information of the first locator among a plurality of locators based on the acquired image data, wherein the plurality of locators are physically connected to the target object; as well as The second pose information determination module is configured to determine the second pose information of the second locator based on the first pose information and the first relative pose information between the second locator and the first locator among the plurality of locators.

13. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processor.

14. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 11.

15. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 11.

Images