Method and apparatus for device control, and device and computer-readable storage medium

Abstract

Provided in the embodiments of the present disclosure are a method and apparatus for device control, and a device and a computer-readable storage medium. The method comprises: in response to completion of a communication connection between a locator and an external device, acquiring identification information of the external device detected by the locator; acquiring pose information of the locator; and on the basis of the pose information, processing related information of the external device, wherein the related information is obtained on the basis of the identification information.

Description

Methods, apparatus, devices, and computer-readable storage media for device control

[0001] This application claims priority to Chinese Patent Application No. 202410693018.8, filed on May 30, 2024, entitled "Method, Apparatus, Device and Computer-Readable Storage Medium for Equipment Control", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, and computer-readable storage media for device control. Background Technology

[0003] Extended Reality (XR) has been widely researched and applied. XR combines hardware devices and various technologies to blend virtual content with real-world scenes, providing users with a unique sensory experience. Examples of XR technologies include Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

[0004] XR devices can be used to provide users with an immersive interactive experience within a virtual environment. How external devices are controlled during this interaction is crucial to the overall experience. Summary of the Invention

[0005] In a first aspect of this disclosure, a method for device control is provided. The method may include: acquiring identification information of the external device detected by the locator in response to the completion of a communication connection between the locator and the external device; acquiring pose information of the locator; and processing relevant information of the external device based on the pose information, the relevant information being acquired based on the identification information.

[0006] In a second aspect of this disclosure, an apparatus for device control is provided. The apparatus may include: an identification information acquisition module configured to acquire identification information of the external device detected by the locator in response to the completion of a communication connection between the locator and the external device; a pose information acquisition module configured to acquire pose information of the locator; and a related information processing module configured to process related information of the external device based on the pose information, wherein the related information is acquired based on the identification information.

[0007] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the electronic device to perform the method of the first aspect.

[0008] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The medium stores computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0009] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0010] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0012] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0013] Figure 2 shows a flowchart of a method for device control according to some embodiments of the present disclosure;

[0014] Figure 3 shows an example three-dimensional model of an external device having the shape of a simulated shooting device according to some embodiments of the present disclosure;

[0015] Figure 4 shows an example three-dimensional model of an external device having the shape of a baseball bat according to some embodiments of the present disclosure;

[0016] Figure 5 shows a schematic structural block diagram of a device for device control according to some embodiments of the present disclosure; and

[0017] Figure 6 shows a block diagram of an electronic device that can implement one or more embodiments of the present disclosure. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0020] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0022] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, relevant users should be informed of the type, scope of use, and usage scenarios of the information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and authorization should be obtained from the relevant users. Among them, relevant users may include any type of rights holder, such as individuals, enterprises, and groups.

[0023] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly inform the user that the requested operation will require obtaining and using the user's information, thereby enabling the relevant user to choose whether to provide information to the software or hardware such as the electronic device, application, server, or storage medium that performs the operation of the technical solution disclosed herein based on the prompt message.

[0024] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide information to the electronic device.

[0025] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0026] Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Environment 100 can be a physical scene, that is, environment 100 can be an example of a real-world scene. In environment 100, user 110 can perform interactive operations by wearing at least one XR device. Exemplarily, at least one XR device may include a head-mounted XR device 120, an external device 122, and a locator 123 detachably mounted in the external device 122. The locator may also be called a tracker. External device 122 may be any suitable type of device, such as a handle, a baseball bat-shaped external device, a racket-shaped external device (tennis, ping-pong, etc.), a bow and arrow-shaped external device, a shooting equipment-shaped external device, etc. External devices may include handheld external devices, or external devices in an environment that do not require handheld use, such as devices worn on the user's body, furniture, autonomously movable toys or appliances, pedals simulating vehicle driving, or external devices fixed in a specific location in the environment. In such an example, the head-mounted XR device 120 can communicate with external devices 122 and locators 123 to enable collaborative interaction with the user. Although a single locator 123 is shown in the figure, external devices 122 can actually be connected to multiple locators.

[0027] In some embodiments, the XR device may include an image acquisition device associated with an external device, such as a camera, infrared camera, depth camera, or other image acquisition device, to capture images of the physical space. The image acquisition device may be disposed on one or more XR devices. For example, the image acquisition device may be disposed on a head-mounted XR device 120 to acquire images over a wider spatial range based on a higher spatial position. If an infrared camera is disposed on the head-mounted XR device 120, correspondingly, at least one infrared light source may be disposed on the locator 123 so that the infrared camera can capture images of the infrared light source on the locator 123, thereby identifying and locating the locator 123 based on the obtained images. Based on the positioning result of the locator 123, the positioning result of the external device 122 can be determined accordingly. Furthermore, the XR device may also include a communication device such as a Bluetooth module for information transmission with the locator and / or the external device.

[0028] In other scenarios, the image acquisition device used for image acquisition can be independent of the XR device (e.g., independent of the head-mounted XR device 120). Such an image acquisition device can be deployed independently for image perception of external devices 122, locators 123, and other entities in the environment. Exemplarily, other entities can be furniture, furnishings, etc. in the environment.

[0029] In some embodiments, user 110 can communicate with electronic device 130 using multiple XR devices to process data acquired through the XR devices and send data to the XR devices. It should be understood that in some cases, head-mounted XR device 120 may not communicate with electronic device 130 (or, in other words, environment 100 may exemplary exclude electronic device 130). In this case, head-mounted XR device 120 can interact with devices such as external device 122 and locator 123. Specifically, it can interact with other XR devices through XR control device 121 in head-mounted XR device 120. It should be understood that XR control device 121 is not limited to being located in head-mounted XR device 120; it can be located in other XR devices or exist as a standalone control device.

[0030] In environment 100, user 110 can utilize an XR device to access virtual environment 140. Virtual environment 140 can exemplarily be a space. In some embodiments, virtual objects 150 for user 110 can be generated within virtual environment 140, allowing user 110 to engage in immersive interactive experiences through virtual objects 150, such as adjusting the actions of virtual objects 150. In some embodiments, virtual tools and / or other users' virtual objects can be provided within virtual environment 140, enabling user 110 to interact with virtual tools within virtual environment 140 through virtual objects 150, and also to interact with virtual objects of other users through virtual objects 150.

[0031] Electronic device 130 may be a standalone device capable of communicating with XR devices and / or other image capture devices, such as a server or computing node for image or data processing, or it may be integrated with XR devices and / or other image capture devices. In some embodiments, electronic device 130 may be implemented as an XR device, that is, in this case, the XR device can perform all the functions of electronic device 130. It should be understood that the above description of electronic device 130 is merely exemplary and not limiting, and electronic device 130 can be implemented as a device of various forms, structures, or categories, and the embodiments of this disclosure are not limited in this regard.

[0032] For example, electronic device 130 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, server devices, etc. Terminal devices may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Server devices may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server devices may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0033] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0034] As briefly mentioned earlier, users can achieve an immersive experience in a virtual environment by wearing XR devices. Therefore, for some XR devices, location tracking is required when users are operating them.

[0035] Currently, in the XR field, positioning and tracking of XR peripherals are achieved using locators, primarily by deploying locators with specific light sources (such as optical infrared lights). Specifically, image acquisition devices capture images of the locator's infrared lights, which are then combined with the attitude tracking of the IMU (Inertial Measurement Unit). With the development of XR technology, the shapes and forms of XR peripherals have become increasingly diverse, evolving from traditional handheld controllers to more varied dedicated peripherals. Examples include peripherals shaped like baseball bats, ping-pong paddles, bows and arrows, and shooting equipment. Developing a separate tracking algorithm for each type of peripheral would be highly wasteful.

[0036] Generally speaking, locators are relatively small in size. However, this leads to problems such as the complex shape of the external device or the locator being frequently obstructed during use. In other words, relying solely on the locator's positioning information to control the operating device can easily result in lost tracking or inaccurate tracking of the external device.

[0037] Furthermore, due to the diverse forms of external devices, it is impossible to provide targeted prompts based on the form of the external device when ensuring user safety. For example, external devices shaped like baseball bats are generally long and swing quickly when in use. If the position is determined solely by the locator, it often cannot meet the safety requirements.

[0038] Finally, a general tracking algorithm is currently used to determine the pose of all types of external devices. However, different external devices often have different characteristics. For example, external devices simulating the shape of a bow and arrow exhibit sudden movements and stops in the bowstring, while external devices simulating the shape of a shooting device exhibit high-frequency jitter after simulating firing. Therefore, the general tracking algorithm cannot provide good pose determination results for external devices of different types or shapes. Thus, how to better determine the pose of external devices is an important task that needs to be addressed.

[0039] This disclosure presents an embodiment of a device control scheme. According to this scheme, an XR control device can acquire the identification information of the external device detected by the locator upon completion of a communication connection between the locator and the external device. Based on the acquired positioning information from the locator, the pose information of the external device is determined. Furthermore, based on the identification information and pose information, relevant information of the external device is processed; this relevant information is obtained based on the identification information. Thus, the XR control device can directly obtain relevant information about the external device through the identification information. In other words, by using a general-purpose locator and connecting different peripherals, tracking of peripherals can be achieved based on the peripheral device's identification. For the external device, there is no need for redundant development of adaptive positioning algorithms, making it more flexible and easier to locate and control the external device.

[0040] Embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. Figure 2 shows a flowchart of a device control process 200 according to some embodiments of the present disclosure. Process 200 may be implemented at an XR control device, such as XR control device 121 of Figure 1. For ease of discussion, process 200 will now be described with reference to Figure 1.

[0041] As shown in Figure 2, in box 201, the XR control device 121 obtains the identification information of the external device detected by the locator in response to the completion of the communication connection between the locator and the external device.

[0042] The user can first establish a communication connection between the locator 123 and the external device 122. For example, the external device 122 may have a pre-installed wiring port, through which the locator 123 connects. For example, the wiring port may be a USB Type-C interface, a Micro-USB interface, a POGO pin interface, etc. Alternatively, the locator 123 can also be secured to the external device 122 using a slot on the device itself or a cable tie or other fastening accessory.

[0043] After the XR device is powered on, the XR control device 121 can obtain the identification information of the external device 122 through the locator 123. The acquisition process may include: after the locator 123 connects to the external device in the aforementioned manner and obtains the identification information of the external device 122, it can forward the identification information of the external device 122 to the XR control device 121. Thus, the XR control device 121 can obtain the external device identification information of the external device 122 detected by the locator 123.

[0044] In box 202, the XR control device 121 acquires the pose information of the locator. This pose information may include position information, such as coordinates, or attitude information, such as pitch angle, yaw angle, and roll angle. The pose information may also include both position and attitude information. For example, the pose information can be determined by combining data from one or more sensors (such as an inertial measurement unit, a global positioning system, an image acquisition device, etc.) for state estimation to obtain the pose information of the locator 123. For instance, the locator 123 may be equipped with sensors such as an inertial measurement unit and a global positioning system. Therefore, based on the measurement data from the inertial measurement unit, global positioning system, etc., the locator 123 can independently determine its pose information and send it to the XR control device 121.

[0045] Furthermore, the XR control device 121 can acquire images containing the light sources arranged in the locator 123 based on image acquisition, thereby determining the pose information. Alternatively, the XR control device 121 can combine data acquired by the inertial measurement unit and the image acquisition device, and determine the pose information of the locator 123 in real time by fusing visual feature points and inertial data.

[0046] In box 203, XR control device 121 processes relevant information of external devices based on identification information and pose information, which is obtained based on identification information.

[0047] Based on the identification information of the external device 122, the XR control device 121 can retrieve relevant information about the external device 122 from a designated database. The designated database can be the storage module of the XR control device 121, the electronic device 130, or other databases, etc.

[0048] Furthermore, relevant information can also be obtained through the locator 123. For example, between the locator 123 and the external device 122, relevant information about the external device 122 can be transmitted based on transmission capability or transmission configuration. For instance, if the transmission capability is poor or the transmission configuration is already set, the locator 123 can only obtain the identification information of the external device 122. Alternatively, if the transmission capability is strong or the transmission configuration is already set, the locator 123 can obtain both the identification information of the external device 122 and other relevant information about the external device 122.

[0049] For example, the relevant information may be the shape information of the external device 122, the parameter information of the tracking algorithm, etc. After obtaining the relevant information of the external device 122, the XR control device 121 can perform processing on the relevant information of the external device based on the pose information. For example, the processing may include generating a 3D model, adjusting the parameters of the tracking algorithm accordingly, etc. Therefore, compared to relying solely on the light source of the locator to control the external device 122, more diverse information can be combined to complete the relevant control of the external device 122.

[0050] In some embodiments, the relevant information is used to indicate the shape information of an external device. In this case, the XR control device 121 processes the relevant information of the external device by determining a three-dimensional model of the external device based on the shape information. The XR control device 121 may also perform processing on the three-dimensional model based on pose information.

[0051] The shape information can be parameters of the external rectangular frame of the external device 122, such as length, width, and depth information. Alternatively, the shape information can also be the outline information, cross-sectional information, or positional information of several key points of the external device 122.

[0052] First, based on the shape information of the external device 122, a corresponding 3D model of the external device 122 can be generated. Second, based on the pose information of the locator 123, it can be combined with the 3D model of the external device 122 to determine the pose of the 3D model of the external device 122. For example, the pose information can be combined with the 3D model based on translation or rotation transformations. Translation transformations can move the 3D model from one position to another. Rotation transformations can change the orientation or posture of the 3D model.

[0053] In some embodiments, the XR control device 121 performing processing on the 3D model may include: determining boundary information corresponding to obstacles; and generating prompts about the 3D model based on the 3D model, pose information, and boundary information corresponding to obstacles.

[0054] Taking a real-world scenario as an example, such a scenario typically includes other entities, such as furniture and furnishings. Furthermore, if the scenario is a relatively enclosed space, it will also include architectural structures such as walls, doors, and windows.

[0055] Obstacles can correspond to other entities such as furniture and furnishings, or to building structures such as walls, doors, and windows. In addition, obstacles can also be the boundaries of a user-defined area.

[0056] For obstacle recognition, the XR control device 121 can use computer vision algorithms and machine learning discrimination models to identify obstacles. For example, computer vision algorithms can be used to determine information such as the shape, size, and / or texture of different objects. Then, the machine learning discrimination model determines the obstacles based on the shape, size, and / or texture information. After the obstacles are identified, the boundary information corresponding to the obstacles can be further determined.

[0057] For example, the XR control device 121 can acquire (dynamic) environmental images through an image acquisition device and identify obstacles in the images. Then, based on the intrinsic parameters (focal length and optical center) and extrinsic parameters (position and orientation of the image acquisition device), the two-dimensional coordinates can be converted into normalized two-dimensional coordinates using the intrinsic parameters of the image acquisition device. Next, the depth information of the obstacles is determined using the extrinsic parameters of the image acquisition device. Finally, using the normalized two-dimensional coordinates and depth information, the three-dimensional position information of the obstacles can be obtained. Based on the three-dimensional position information, the boundary information of the obstacles can be the position information of several feature points (e.g., vertices) of the obstacles, or the position information corresponding to the shape of the obstacles expressed by mathematical equations, etc.

[0058] For obstacles that are user-defined boundaries, the XR control device 121 can set corresponding virtual boundaries based on user input (text, voice, gestures, etc.) and obtain the position information corresponding to the virtual boundaries. The position information corresponding to the virtual boundaries can then be used as the boundary information for the obstacles. Furthermore, the virtual boundaries can be visualized in the virtual environment 140.

[0059] Using the three-dimensional model of the external device 122 determined in the aforementioned embodiments, the three-dimensional model can be combined with the determined pose information to obtain the pose information corresponding to the three-dimensional model. Based on the pose information corresponding to the three-dimensional model and the position information corresponding to the obstacle, the XR control device 121 can determine the distance between the external device 122 and the obstacle. Correspondingly, a distance threshold can be set. If the distance between the external device 122 and the obstacle gradually approaches to no greater than the distance threshold, the XR control device 121 can issue a collision warning. For example, the collision warning can be an audio prompt, a vibration prompt, or a visual prompt in the virtual environment 140. The content of the visual prompt can include text prompts, special effects prompts, highlighting obstacles, or obstacle prompts that are visible in mixed reality.

[0060] Furthermore, the distance threshold can be dynamically adjusted based on the moving speed of the external device 122. For example, based on information collected from multiple sampling points, the moving speed or acceleration of the external device 122 can be determined. Then, a distance threshold adjustment parameter is generated based on the moving speed or acceleration. That is, if the external device 122 moves quickly (has a high acceleration), the corresponding adjustment parameter is larger. Conversely, if the external device 122 moves slowly, the corresponding adjustment parameter is smaller or can be directly set to 1.

[0061] Therefore, since the (collision warning) prompt is based on a three-dimensional model of the operating device 122, it is safer and more suitable for different shapes of operating devices 122, compared to relying solely on the position of the locator to issue prompts.

[0062] In some embodiments, the boundary information corresponding to an obstacle includes at least one of the location information of a physical boundary and the location information of a virtual boundary. A common scenario is that users experience XR devices in an indoor environment. Therefore, furniture, furnishings, etc., in the indoor environment can serve as obstacles. Another scenario is user-defined boundaries. Therefore, a corresponding virtual boundary is set based on the user's input (text, voice, gestures, etc.). For example, if the user's voice is "a radius of one meter centered on me," then a hemispherical virtual boundary with a radius of one meter can be set accordingly.

[0063] In some embodiments, the XR control device 121 performs processing on the three-dimensional model, including displaying the rendered three-dimensional model on a display interface based on pose information.

[0064] The display interface can be a corresponding virtual environment 140, such as a virtual reality display interface, an augmented reality display interface, or a mixed reality display interface. For rendering a 3D model, the pose information can first be used to determine the correct position of the 3D model in the virtual environment 140.

[0065] This is followed by material and / or texture rendering for the 3D model. For example, material rendering can be used to indicate color, gloss, reflectivity, transparency, roughness, etc. Texture rendering can be used to indicate surface features of an object, such as wood grain, metallic texture, fabric texture, etc. For example, for both material and texture rendering, corresponding rendering templates can be obtained from a specified database based on identification information.

[0066] For example, Figure 3 shows a schematic diagram of a three-dimensional model of an external device for a simulated shooting device shape 300 according to some embodiments of the present disclosure, and Figure 4 shows a schematic diagram of a three-dimensional model of an external device for a baseball bat shape 400 according to some embodiments of the present disclosure. As can be seen from the exemplary three-dimensional model diagrams, the XR control device 121 can directly render the three-dimensional model of the baseball bat-shaped external device 122 in the head-mounted XR device 120, achieving the effect of seeing the accurate position and shape of the external device 122 in the global display of the head-mounted XR device 120. Therefore, even if the user is not in a specific application (e.g., a specific game), they can still see the three-dimensional model corresponding to the external device 122 in the virtual environment 140, thus providing a better overall experience.

[0067] In some embodiments, the relevant information is also used to indicate attribute information of the first light source arranged on the external device. Based on this, the XR control device 121 controls the first light source arranged in the external device and the second light source arranged in the locator to light up according to the same timing sequence, based on the attribute information.

[0068] The attribute information of the first light source deployed on the external device may include the control parameters, frequency information, wavelength information, and position information of the first light source on the external device. It is easy to understand that there can be multiple first light sources. The first light source includes, but is not limited to, infrared light sources.

[0069] The XR control device 121 can treat the first light source arranged in the external device 122 and the second light source arranged in the positioner 123 as a whole. That is, the XR control device 121 can generate synchronous control signals for the first light source and the second light source to control the first light source arranged in the external device 122 and the second light source arranged in the positioner 123 to be lit according to the same timing.

[0070] The XR control device 121 can control the light sources in various ways, such as illuminating the first and second light sources at the same time, ensuring these light sources are in a common illumination state within a specific time period, or illuminating the positioners so that the image acquisition device can capture these illuminated light sources during an exposure period of image acquisition. Similarly, the second light source can be multiple light sources. The second light source includes, but is not limited to, infrared light sources.

[0071] XR control device 121 can acquire images containing the first light source arranged in external device 122 and the second light source arranged in locator 123 based on image acquisition, thereby determining the overall pose information of the locator and external device, and using it as the pose information of locator 123. Alternatively, XR control device 121 can combine data acquired by inertial measurement unit and image acquisition device, and determine the overall pose information of locator and external device in real time through the fusion of visual feature points and inertial data, and use it as the pose information of locator 123.

[0072] By controlling the first and second light sources to illuminate in the same timing sequence, the locator 123, which has a limited size, can be extended to a certain extent. Even if the light source on the locator 123 is blocked and fails to collect data, the first light source arranged in the external device 122 will serve as an additional light source, which is more conducive to the positioning of the external device 123.

[0073] Furthermore, there may be a situation where multiple positioners 123 are integrated with a single external device 122. In this case, the XR control device 121 will use the light sources on all the positioners integrated with the same external device 122 as secondary light sources. For example, if three positioners are integrated with the same external device 122, then the light sources of all three positioners will be used as secondary light sources. The secondary light sources of the three positioners will use the same timing sequence as the primary light source on the external device 122.

[0074] It is easy to understand that when a user uses multiple external devices, or when multiple users each use at least one external device, the XR control device 121 can distinguish between multiple external devices and the light sources on the locators corresponding to the external devices by different lighting sequences, thereby distinguishing the light sources of different external devices when acquiring images.

[0075] Through the above process, the external device with a light source and the positioner, which also has a light source, can be integrated as a whole. When controlling the light source to illuminate, control is performed on a whole-unit basis, thus increasing the number of light sources. This effectively reduces the occurrence of light source obstruction and facilitates the capture and identification of the light source. Furthermore, compared to the relatively small size of the positioner, the external device is relatively large. Therefore, the arrangement of light sources on the external device increases the distance between light sources to a certain extent, resulting in more accurate positional determination based on the light sources.

[0076] In some embodiments, the attribute information of the first light source includes control parameters of the first light source. Based on this, the XR control device 121 controls the first light source and the second light source to light up according to the same timing sequence, including: acquiring the control parameters of the second light source, the second light source including a light source arranged in the locator. Based on the control parameters of the first light source and the control parameters of the second light source, the first light source arranged in the external device and the second light source arranged in the locator are controlled to light up according to the same timing sequence.

[0077] Once the locator 123 establishes a communication connection with the external device 122, the locator 123 can obtain control parameters from the attribute information of the first light source. For example, the control parameters may include the light source's identifier, flashing mode, flashing frequency, and timestamp. Similarly, for the second light source arranged in the locator 123, there are also control parameters for the second light source, which may also include the light source's identifier, flashing mode, flashing frequency, and timestamp.

[0078] Positioner 123 can send the received control parameters of the first light source and its own control parameters of the second light source to XR control device 121. XR control device 121 can then generate control commands based on the control parameters of the first and second light sources. These control commands can control the first light source in external device 122 and the second light source in positioner 123 to illuminate according to the same timing sequence.

[0079] In some embodiments, the relevant information is also used to indicate configuration information for the tracking algorithm. The tracking algorithm is configured to determine pose information corresponding to an external device. In this case, the XR control device 121 adjusts the parameter values ​​corresponding to at least one parameter of the tracking algorithm based on the configuration information.

[0080] As previously mentioned, the external device 122 can be a variety of devices, such as a baseball bat-shaped device, a tennis racket-shaped device (like a tennis ball or ping-pong ball), a device shaped like a simulated bow and arrow, or a device shaped like a simulated shooting device. For example, for a device shaped like a simulated bow and arrow, the positioner typically needs to be mounted on the bowstring. The characteristic of a device shaped like a simulated bow and arrow is the sudden movement and stopping of the bowstring. For a device shaped like a simulated shooting device, the characteristic during simulated shooting is high-frequency jitter. This means that due to the different characteristics of different external devices, using a general tracking algorithm when performing pose tracking of the external device 122 may lead to a decrease in tracking accuracy. For example, a general tracking algorithm cannot be optimized for the sudden movement and stopping of the bowstring, resulting in low positioning accuracy for the device shaped like a simulated bow and arrow, potentially leading to positional deviation or overshoot. For example, general-purpose tracking algorithms cannot effectively filter and smooth the high-frequency jitter of external devices that simulate the shape of a shooting apparatus, resulting in large jitter errors in the tracking data and affecting the accuracy of shooting and aiming. Furthermore, general-purpose tracking algorithms cannot respond promptly to the rapid movements and changes of external devices, causing operational delays and impacting user experience. These shortcomings of general-purpose tracking algorithms lead to a poor user experience.

[0081] To address this, for different external devices 122, the relevant information can be extended, meaning the relevant information also includes configuration information specific to that external device. For example, the configuration information may correspond to at least one parameter of the tracking algorithm. For instance, parameters may include acceleration threshold, prediction duration, and jitter smoothing parameters, etc.

[0082] For example, for an external device simulating the shape of a bow and arrow, the acceleration threshold can be set to a1, the prediction duration to t1, and the jitter smoothing parameter to m1. For a racket-type external device, the acceleration threshold can be set to a2, the prediction duration to t2, and the jitter smoothing parameter to m2. For an external device simulating the shape of a shooting device, the acceleration threshold can be set to a3, the prediction duration to t3, and the jitter smoothing parameter to m3. Therefore, for different types of external devices, the parameters in the tracking algorithm should be set as follows: a1 < a2 < a3, t1 > t2 > t3, m1 < m2 < m3. By adjusting these parameters, the tracking algorithm can be better optimized, improving positioning accuracy and user experience for external devices with different shapes and usage characteristics.

[0083] In some embodiments, at least one parameter includes at least one of prediction duration and jitter smoothing parameter.

[0084] Prediction duration indicates the time range within which a tracking algorithm predicts future positions based on the current sampled frame. In cases of rapid movement or violent motion, an appropriate prediction duration can help the algorithm more accurately predict future position changes, thereby allowing it to adjust relevant data for determining pose information in advance and avoid delays and overshoot.

[0085] For external devices simulating the shape of a bow and arrow, the bowstring experiences significant acceleration changes during extension and release. Therefore, accurate capture of its trajectory is crucial for prediction. Consequently, the prediction duration can be set relatively long (longer than the default). Due to the high-frequency jitter of external devices simulating the shape of a shooting device, the prediction duration needs to respond quickly to these high-frequency changes; a shorter duration is typically chosen, so it can be set relatively short (shorter than the default). For racket-type external devices, the prediction duration can be set to the default value.

[0086] Jitter parameters define and handle the behavior of the positioner under high-frequency jitter. These parameters can include jitter frequency, amplitude, and filtering coefficients. For external devices simulating the shape of a shooting apparatus, ideally, high-frequency jitter needs to be effectively filtered out; therefore, jitter parameters can be set relatively high (above the default value) to help filter out high-frequency jitter. For external devices simulating the shape of a bow and arrow, jitter is negligible; therefore, jitter parameters can be set relatively low (below the default value). For racket-type external devices, the jitter parameters can be set to the default value.

[0087] In some embodiments, the method for device control can be implemented at the extended reality device.

[0088] Figure 5 shows a schematic structural block diagram of a device 500 for device control according to some embodiments of the present disclosure. The device 500 may be implemented in or included in an XR device, for example. The various modules / components in the device 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0089] As shown in the figure, the device 500 includes an identification information acquisition module 501, configured to acquire identification information of the external device detected by the locator in response to the completion of a communication connection between the locator and the external device. A pose information acquisition module 502 is configured to acquire the pose information of the locator; and a related information processing module 503 is configured to process related information of the external device based on the pose information, wherein the related information is acquired based on the identification information.

[0090] In some embodiments of this disclosure, relevant information is used to indicate the shape information of an external device. Based on this, the relevant information processing module 503 may include a 3D model determination submodule, configured to determine a 3D model of the external device based on the shape information. A 3D model processing submodule is configured to perform processing on the 3D model based on pose information.

[0091] In some embodiments of this disclosure, the 3D model processing submodule may include an obstacle determination unit configured to determine boundary information corresponding to obstacles. A prompting unit is configured to generate prompts about the 3D model based on the 3D model, pose information, and the boundary information corresponding to the obstacles.

[0092] In some embodiments of this disclosure, the boundary information corresponding to the obstacle includes at least one of the location information of the physical boundary and the location information of the virtual boundary.

[0093] In some embodiments of this disclosure, the 3D model processing submodule is also configured to display the rendered 3D model on the display interface based on pose information.

[0094] In some embodiments of this disclosure, the relevant information is also used to indicate attribute information of a first light source arranged on an external device. Based on this, the device 500 further includes a light source control module configured to control the first light source arranged in the external device and the second light source arranged in the locator to illuminate according to the same timing sequence based on the attribute information.

[0095] In some embodiments of this disclosure, the attribute information of the first light source includes control parameters of the first light source. In this case, the light source control module may include: a control parameter acquisition submodule configured to acquire control parameters of a second light source, the second light source including a light source arranged in the locator; and a control execution submodule configured to control the first light source arranged in the external device and the second light source arranged in the locator to illuminate according to the same timing sequence based on the control parameters of the first light source and the control parameters of the second light source.

[0096] In some embodiments of this disclosure, the relevant information is also used to indicate configuration information for a tracking algorithm configured to determine pose information corresponding to an external device. In this case, the device 500 further includes a parameter adjustment module configured to adjust the parameter values ​​corresponding to at least one parameter of the tracking algorithm based on the configuration information.

[0097] In some embodiments of this disclosure, at least one parameter includes at least one of prediction duration and jitter smoothing parameter.

[0098] In some embodiments of this disclosure, device 500 is implemented at an extended reality device.

[0099] Figure 6 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 600 shown in Figure 6 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 600 shown in Figure 6 may include or be implemented as the electronic device 130 of Figure 1, the XR control device 121, or the device 500 of Figure 5.

[0100] As shown in Figure 6, the electronic device 600 is in the form of a general-purpose electronic device. Components of the electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processing unit 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 600.

[0101] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 600.

[0102] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 6, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0103] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0104] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0105] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0106] According to an exemplary implementation of this disclosure, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative embodiments of FIG2, which will therefore not be described further herein.

[0107] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0108] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0109] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0111] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for device control, comprising: in response to a locator being communicatively connected with an external device, obtaining identification information of the external device detected by the locator; obtaining pose information of the locator; and based on the pose information, processing relevant information of the external device, the relevant information being obtained based on the identification information.

2. The method of claim 1, the relevant information being used to indicate shape information of the external device, and the processing the relevant information of the external device based on the pose information comprises: based on the shape information, determining a three-dimensional model of the external device; and based on the pose information, performing processing on the three-dimensional model.

3. The method of claim 2, the performing processing on the three-dimensional model based on the pose information comprises: determining boundary information corresponding to an obstacle; and based on the three-dimensional model, the pose information, and the boundary information corresponding to the obstacle, generating a hint about the three-dimensional model.

4. The method of claim 3, wherein the boundary information corresponding to the obstacle comprises at least one of position information of a real boundary and position information of a virtual boundary.

5. The method of claim 2, the performing processing on the three-dimensional model based on the pose information comprises: based on the pose information, displaying the rendered three-dimensional model on a display interface.

6. The method of claim 1, the relevant information being further used to indicate attribute information of a first light source arranged on the external device, and the method further comprises: based on the attribute information, controlling the first light source arranged in the external device and a second light source arranged in the locator to be lit according to a same timing.

7. The method of claim 6, the attribute information of the first light source comprising control parameters of the first light source, and wherein the controlling the first light source arranged in the external device and the second light source arranged in the locator to be lit according to the same timing comprises: obtaining control parameters of the second light source, the second light source comprising the light source arranged in the locator; and based on the control parameters of the first light source and the control parameters of the second light source, controlling the first light source arranged in the external device and the second light source arranged in the locator to be lit according to the same timing.

8. The method of claim 1, the relevant information being further used to indicate configuration information for a tracking algorithm configured to determine pose information corresponding to the external device; and the method further comprises: based on the configuration information, performing adjustment on a parameter value corresponding to at least one parameter of the tracking algorithm.

9. The method of claim 8, wherein the at least one parameter comprises at least one of a prediction length and a jitter smoothing parameter.

10. The method of claim 1, wherein the method is implemented at an extended reality device.

11. An apparatus for device control, comprising: ​ ​ ​ ​ An identifier information obtaining module, configured to, in response to the locator being connected with the external device, obtain identifier information of the external device detected by the locator; A pose information obtaining module, configured to obtain pose information of the locator; A related information processing module, configured to process related information of the external device based on the pose information, the related information being obtained based on the identifier information.

12. An electronic device, comprising: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions, when executed by the at least one processor, causing the electronic device to perform the method according to any one of claims 1-10.

13. A computer-readable storage medium having computer-executable instructions stored thereon that are executable by a processor to implement the method according to any one of claims 1-10.

14. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1-10.

Images