Virtual object control method and apparatus, device, storage medium, and program product

Abstract

Embodiments of the present disclosure provide a virtual object control method and apparatus, a device, a storage medium, and a program product. The method comprises: in response to a matching instruction for a first virtual object, obtaining a recognition signal of a first locator; determining a first correspondence between the first locator and the first virtual object on the basis of the recognition signal; and at least on the basis of the first correspondence and the monitoring of pose information of the first locator, controlling the first virtual object.

Description

Methods, apparatus, devices, storage media, and program products for controlling virtual objects

[0001] This application claims priority to Chinese Patent Application No. 202410703172.9, filed on May 31, 2024, entitled “Method, Apparatus, Device, Storage Medium and Program Product for Controlling Virtual Objects”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for controlling virtual objects. Background Technology

[0003] Extended Reality (XR) has been widely researched and applied. XR combines hardware devices and various technologies to merge virtual content with real-world scenes, providing users with a unique sensory experience. Examples of XR technologies include Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). VR uses computers to simulate a three-dimensional virtual world, providing users with an immersive experience encompassing visual, auditory, and tactile senses. AR allows real-world environments and virtual objects to be superimposed on each other in real time, existing simultaneously. MR is a new visual environment that merges the real and virtual worlds, where objects in the physical real-world scene coexist with objects in the virtual world in real time.

[0004] Users can access virtual environments by wearing XR devices to obtain an immersive interactive experience. However, current extended reality technology still has some issues that affect the user's participation experience. Summary of the Invention

[0005] In a first aspect of this disclosure, a method for controlling a virtual object is provided. The method includes: obtaining an identification signal of a first locator in response to a matching instruction for a first virtual object; determining a first correspondence between the first locator and the first virtual object based on the identification signal; and controlling the first virtual object based at least on the first correspondence and monitoring of pose information of the first locator.

[0006] In a second aspect of this disclosure, an apparatus for controlling a virtual object is provided. The apparatus includes: an identification acquisition module configured to acquire an identification signal of a first locator in response to a matching command for a first virtual object; a relationship determination module configured to determine a correspondence between the first locator and the first virtual object based on the identification signal; and an object control module configured to control the first virtual object based at least on the first correspondence and monitoring of the pose information of the first locator.

[0007] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0008] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the method of the first aspect.

[0009] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0010] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0012] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0013] Figure 2 shows a flowchart of a process for controlling a virtual object according to some embodiments of the present disclosure;

[0014] Figure 3 shows a schematic diagram of presenting locator identification information according to some embodiments of the present disclosure;

[0015] Figure 4 shows a schematic structural block diagram of an apparatus for controlling virtual objects according to certain embodiments of the present disclosure; and

[0016] Figure 5 shows a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented. Detailed Implementation

[0017] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0018] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0019] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0020] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0024] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0025] Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Environment 100 can be a physical scene, i.e., environment 100 can be an example of a real-world scene. In environment 100, user 110 can perform interactive operations by wearing at least one XR device 120. At least one XR device 120 may include a head-mounted XR device. In some embodiments, the user can operate manipulation devices 121, 124, which may include one or more detachably mounted locators 122, 123. A locator may also be referred to as a tracker. Handheld manipulation devices 121, 124 may be, for example, a handle, a baseball bat-shaped manipulation device, a racket-shaped manipulation device (tennis, ping-pong, etc.), a bow and arrow-shaped manipulation device, a shooting equipment-shaped manipulation device, or any suitable type of device controlled by the hand. In such an example, head-mounted XR device 120 can communicate with manipulation devices 121, 124 and locators 122, 123 to achieve interaction with the user in a collaborative manner. Although only locators 122 and 123 are shown in the figure, the manipulators 121 and 124 can actually be connected to multiple other locators.

[0026] In some embodiments, the XR device 120 may be connected to other external devices besides the control devices 121 and 124. Other external devices may include handheld external devices, or external devices in environments where hands are not required, such as devices worn on the user's body, furniture, autonomously movable toys or appliances, pedals that simulate vehicle driving, or external devices fixed in a specific location in the environment.

[0027] In some embodiments, locators 122 and 123 can exist independently. User 110 can interact with the virtual environment by operating locators 122 and 123. XR device 120 controls corresponding virtual objects based on the pose information of locators 122 and 123 and control commands sent by locators 122 and 123. For example, locator 122 is matched with a baseball bat in the virtual environment. User 110 holds locator 122 to control the movement of the baseball bat. User 110 can also send control commands to XR device 120 by shaking and / or tapping locator 122.

[0028] In some embodiments, the XR device 120 may include an image acquisition device associated with the manipulation device, such as a camera, infrared camera, depth camera, or other image acquisition device, to capture images of the physical space. The image acquisition device may be disposed on one or more XR devices 120. For example, the image acquisition device may be disposed on a head-mounted XR device 120 to acquire images over a wider spatial range based on a higher spatial position. If an infrared camera is disposed on the head-mounted XR device 120, correspondingly, at least one infrared light source may be disposed on the locators 122 and 123, so that the infrared camera can capture the optical pattern or multiple light spots formed by the infrared light source on the locators 122 and 123, thereby identifying and locating the manipulation device 121 and the locators 122 and 123 based on the obtained images.

[0029] In some embodiments, the XR device 120 may include an image acquisition device, such as a camera, infrared camera, etc., for capturing images of the physical space. The image acquisition device may be disposed on one or more XR devices. Exemplarily, the image acquisition device may be disposed on a head-mounted XR device 120 to acquire images over a wider spatial range based on a higher spatial position. In some embodiments, the locators 122, 123 may not require a light source. The XR device 120 performs image recognition on images including those of the locators 122, 123, and performs localization of the locators 122, 123 based on the image recognition results. In other scenarios, the image acquisition device may be independent of the XR device 120 (e.g., independent of the head-mounted XR device 120). Such an image acquisition device may be independently deployed for image perception of manipulators 121, 124, locators 122, 123, and other entities in the environment. Exemplarily, other entities may be furniture, furnishings, etc., in the environment.

[0030] In some embodiments, user 110 may communicate with server 130 using multiple XR devices 120 to process data acquired through XR devices 120 and send data to XR devices 120. It should be understood that in some cases, the head-mounted XR device 120 may not communicate with server 130 (or, in other words, environment 100 may exemplary exclude server 130). In this case, the head-mounted XR device 120 can be used to interact with devices such as manipulation devices 121, 124 and locators 122, 123. Specifically, the head-mounted XR device 120 can be used to interact with other XR devices.

[0031] In environment 100, user 110 can use XR device 120 to access virtual environment 140. Virtual environment 140 can exemplarily be a space. In some embodiments, virtual objects 150 and 151 can be generated in virtual environment 140, allowing user 110 to interact with virtual objects 150 and 151, adjusting their poses and trigger states to achieve interaction with virtual environment 140. In some embodiments, virtual environment 140 is a game scene, and correspondingly, virtual objects 150 and 151 are game items. Users can participate in the game by manipulating virtual objects 150 and 151. In some embodiments, virtual objects of other users can be provided in virtual environment 140, allowing user 110 to interact with other users' virtual objects through virtual objects 150 and 151 within virtual environment 140.

[0032] Server 130 may be a standalone device capable of communicating with XR devices and / or other image capture devices, such as a server or computing node for image or data processing, or it may be integrated with XR devices and / or other image capture devices. In some embodiments, server 130 may be implemented as an XR device, that is, in this case, the XR device can perform all the functions of server 130. It should be understood that the above description of server 130 is merely exemplary and not restrictive, and server 130 can be implemented as a device of various forms, structures, or categories, and the embodiments of this disclosure are not limited in this regard.

[0033] For example, server 130 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, server devices, etc. Terminal devices may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Server devices may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server devices may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0034] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0035] As briefly mentioned earlier, users can achieve an immersive experience in a virtual environment by wearing XR devices. Therefore, for some XR devices, location tracking is required when users are operating them.

[0036] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0037] As briefly mentioned earlier, users can achieve an immersive experience in a virtual environment by wearing XR devices. Therefore, for some XR devices, location tracking is required when users operate them. Currently, in the XR field, location tracking of locators (i.e., positioners) or handheld controllers (such as gamepads) is mainly achieved by deploying locators with specific light sources (such as optical infrared lights). Specifically, tracking is performed by using a camera to capture the infrared light from the gamepad or other locators, combined with the attitude of the IMU (Inertial Measurement Unit).

[0038] Ideally, a unified motion tracker should be produced and used, facilitating manufacturing, use, and expansion. However, for better control of virtual objects, it's necessary to differentiate the trackers for each virtual object. Currently, users need to specify the tracker's location (e.g., left knee, handheld) via third-party software, which then controls the tracker to emit response signals so users can distinguish between different trackers. On one hand, third-party software has limited customizable locations, failing to meet users' flexible tracker usage needs. On the other hand, since third-party software typically cannot be integrated into XR devices, setting the tracker's location requires users to operate across multiple platforms and confirm the tracker's response signals, causing inconvenience.

[0039] Embodiments of this disclosure propose a scheme for controlling virtual objects. According to various embodiments of this disclosure, in response to a matching command for a first virtual object, an identification signal of a first locator is obtained. Based on the identification signal, a first correspondence between the first locator and the first virtual object is determined. And based at least on the first correspondence and monitoring of the pose information of the first locator, the first virtual object is controlled. Thus, this scheme enables the determination of the correspondence between the first locator and the first virtual object based on the identification signal after obtaining it, eliminating the need to determine the locator's location based on a scheme provided by third-party software, thereby improving the flexibility of using the locator. Furthermore, users do not need to operate on multiple platforms and confirm the locator's response signals, reducing the locator setup process and improving the user experience.

[0040] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0041] Figure 2 illustrates a flowchart of a process 200 for controlling virtual objects according to some embodiments of the present disclosure. Process 200 may be implemented at an XR device 120, such as the XR device 120 of Figure 1. For ease of discussion, process 200 will be described below with reference to Figure 1. As shown in Figure 2, the XR device 120 is used to provide a virtual environment 140, including virtual objects, to a user 110. In some embodiments, a server 130 runs a scene instance and renders it using computing resources to generate the virtual environment, which the user 110 accesses through the XR device 120. Alternatively, the scene instance is run by the XR device 120. Such a virtual environment may include, but is not limited to, games (such as bullet hell games), virtual reality scenes, XR device user interfaces (UI), etc.

[0042] In block 201, in response to a matching instruction for a first virtual object, an identification signal for a first locator is obtained. In the example of FIG1, when XR device 120 detects a matching instruction for a virtual object 150 in virtual environment 140, it performs an operation to match a locator for virtual object 150. In some embodiments, the matching instruction is an instruction generated by XR device 120. XR device 120 determines whether to generate a matching instruction based on the status information of the first matched object (e.g., matching status, active status, etc.). Alternatively, XR device 120 may generate a matching instruction upon receiving an instruction from user 110. In some embodiments, the matching instruction is an instruction sent to XR device 120 by another device or process.

[0043] In some embodiments, virtual objects 150 include, but are not limited to, objects set in the UI of an XR device, game characters, etc. One or more virtual objects that are not matched with a locator may exist in the virtual environment 140. To distinguish different virtual objects and better match locators for each virtual object, guidance information corresponding to each virtual object can be displayed to the user 110 during the matching process to guide the user 110 to match virtual objects 150 and 151.

[0044] Upon detecting a matching command, the XR device 120 obtains an identification signal from the locator 122. The identification signal can be a signal actively sent by the locator 122 to the XR device 120. Alternatively, the identification signal can be a signal obtained by the XR device 120 based on data analysis (e.g., location information) collected from the locator 122.

[0045] In some embodiments, the identification signal may be a signal generated when a physical button on the locator 122 is triggered. If the user 110 needs to match the locator 122 with the virtual object 150, the physical button on the locator 122 is triggered in a predetermined triggering manner to control the locator 122 to generate an identification signal. In some embodiments, existing physical buttons on the locator 122 can be reused to control the locator 122 to generate an identification signal, thereby reducing the hardware requirements of the locator 122 in method 200, eliminating the need to design a special locator, and reducing equipment production costs. For example, the predetermined triggering manner of the locator 122 is that the power button is pressed and held for 3 seconds. After detecting that the user 110 has pressed and held the power button on the locator 122 for 3 seconds, the locator 122 generates a trigger signal. In some embodiments, the physical button triggered by the user 110 may be an additional matching button set on the locator 122.

[0046] User 110 sends a wake-up command to the extended reality (XR) device (e.g., user 110 puts on the wearable device) to activate the XR device 120. After activation, the XR device 120 establishes a communication connection with the locator 122. In some embodiments, the XR device 120 can establish a connection with all locators within range by searching, or it can establish a connection only with the locator 122 based on pre-saved locator 122 information. User 110 secures the locator 122 to the operating device 121 by manipulating the slots or straps on the device body according to the user manual.

[0047] In some embodiments, after user 110 starts XR device 120, the matching task between virtual object 150 and locator 122 is completed in the UI interface of XR device 120. Alternatively, user 110 can also complete the matching task between locator 122 and virtual object 150 within the virtual environment 140 after accessing the virtual environment 140 provided by the application via XR device 120. After XR device 120 detects the establishment of a communication connection with locator 122, it checks whether locator 122 has matched a virtual object. If no virtual object is matched, virtual object 150 is identified, and a matching instruction for virtual object 150 is generated. Virtual object 150 is the virtual object that virtual environment 140 needs to match.

[0048] In some embodiments, the virtual object 150 and the locator 122 have been matched. When the user 110 controls the virtual object 150 through the locator 122, the correspondence between the virtual object 150 and the locator can be updated according to the user's own needs.

[0049] As an example, virtual environment 140 is a game scene, and the process of generating matching instructions is explained. Accordingly, the game scene includes a jump object (used to control the character's jump when triggered) and a recovery object (used to restore the character's state when triggered). Neither the jump object nor the recovery object is matched with a locator. After the XR device 120 starts up and enters the game scene, it detects that locator 122 is not matched with a virtual object. The jump object is identified as virtual object 150, and the XR device 120 generates a matching instruction for the jump object (i.e., virtual object 150). After establishing the correspondence between locator 122 and the jump object, if user 110 needs to match locator 122 with the recovery object according to their own needs, they send an update instruction for the correspondence of virtual object 150 to the XR device 120, generating an update instruction for the correspondence of virtual object 150. The update instruction can instruct the removal of the correspondence between virtual object 150 and locator 122 and the establishment of correspondences between other virtual objects and locator 122. The update command can also instruct the removal of the correspondence between the virtual object 150 and the locator 122, and to establish a correspondence between the locator 122 and other locators.

[0050] There may be multiple locators in the scene that do not match the virtual object. In order to better match the virtual object with the locator, it is necessary to distinguish between multiple locators. The locator corresponding to the identification signal obtained by the XR device 120 is locator 122.

[0051] In some embodiments, the identification signal is a signal actively sent by the locator 122 to the XR device 120. For example, the locator 122 sends an identification signal after being triggered by the user 110 or other objects. The locator 122 may also actively send an identification signal to the XR device 120 when its own state meets the triggering conditions. For example, if the locator 122 detects that its own movement speed is greater than a speed threshold, it sends an identification signal to the XR device 120. In some embodiments, the identification signal may also be a signal obtained based on the analysis of the collected data from the locator 122 (e.g., location information). For example, the signal from the IMU unit in the locator 122 collected when the locator 122 is tapped or shaken can be determined as the identification signal, or the sound signal collected when the locator 122 is tapped can be used as the identification signal.

[0052] In some embodiments, to distinguish different virtual objects, before receiving the identification signal from the locator 122, first guidance information can be presented according to the matching instruction for the virtual object 150. The first guidance information can be in the form of text or an image instructing the matching of the virtual object 150. The first guidance information can display the virtual object 150 corresponding to the matching instruction, or it can display the operation to be performed. For example, the first guidance information includes an image of the virtual object 150 and the text information "Please press the locator power button at the prop corresponding to the virtual object 150". By presenting the first guidance information, the user 110 can be guided to match the virtual object 150, reducing the difficulty of operation and improving the user experience.

[0053] In some embodiments, to complete the matching task of multiple virtual objects, second guidance information corresponding to virtual object 151 may be presented. The second guidance information is used to instruct the user to perform a matching operation on virtual object 151. To prevent confusion in the matching process due to repetitive guidance information, the second guidance information differs from the first guidance information to better guide the user 110 in completing the matching of multiple virtual objects. For example, the difference between the second and first guidance information may be in the timing of appearance, the content of the guidance information, or the display format.

[0054] For example, the first guidance information may present an image of virtual object 150 to instruct the user to match locator 122 with virtual object 150. The second guidance information may present an image of virtual object 151 to instruct the user to match locator 123 with virtual object 151. Alternatively, the first guidance information may instruct the user to press the power button on the locator to perform the matching of locator 122 with virtual object 150. The second guidance information may instruct the user to shake the locator to match locator 123 with virtual object 151. Alternatively, the first guidance information may instruct the user to place the locator matched with virtual object 150 in the left area of ​​their field of vision, and the second guidance information may instruct the user to place the locator matched with virtual object 151 in the right area of ​​their field of vision.

[0055] It should be understood that while examples of two locators are given in some embodiments herein, more locators may exist in practice. In cases with more than two locators, the guidance information for each locator can also differ if it is desired to associate different locators with different virtual objects.

[0056] In some embodiments, the locator 122 can also be determined as a locator corresponding to the virtual object 150 by presenting guidance information for the second locator 122, wherein the correspondence between the locator 122 and the virtual object 150 is determined based on the relative pose information between the locator 122 and the XR device obtained after the guidance information is presented. Taking a head-mounted XR device 120 as an example, after the guidance information is presented through the head-mounted XR device 120, the user 110 can adjust the relative pose information between the locator 122 and the head-mounted XR device 120 based on the guidance information. For example, the guidance information can be a virtual circle, and the user 110 can adjust the posture and / or position of the virtual object 150 so that the identifier corresponding to the locator 122 is moved into the virtual circle. This also determines the correspondence between the locator 122 and the virtual object 150. Thus, while determining the locator 122, the correspondence between the locator 122 and the virtual object 150 can be determined more quickly, which is beneficial to improving the user experience.

[0057] In some embodiments, the XR device 120 may connect to multiple locators, and each locator may be connected to a virtual object to control the virtual object. After the XR device 120 displays the second guidance information to the user 110, it determines a second correspondence between the locator 123 and the virtual object 151 based on the identification signal received from the locator 123, so as to control the virtual object 151.

[0058] In some embodiments, the timing of the appearance of the first and second guidance messages is not limited. The appearance of the second guidance message may be earlier than the appearance of the first guidance message. Accordingly, the operation of presenting the second guidance message is not causally related to establishing the first correspondence.

[0059] In box 202, a first correspondence between the first locator 123 and the first virtual object 150 is determined based on the identification signal. After obtaining the identification signal from the locator 122, the XR device 120 establishes a correspondence between the locator 122 and the virtual object 150 based on the identification signal. The identification signal of the locator 122 includes the identification information of the locator 122. The XR device 120 binds the identification information of the locator 122 with the virtual object 150 to establish the first correspondence. Thus, in the case of multiple locators connected to the XR device 120, the locator 122 corresponding to the virtual object 150 can be conveniently determined according to the correspondence. Compared with the method of matching the locator 122 based on a fixed location provided by the UI interface, the correspondence can be established based on the identification signal of the first locator, eliminating the need for the user to switch between multiple interfaces or platforms (e.g., the XR device 120 and the computer platform used to configure the XR device 120), improving configuration efficiency and user experience. In some embodiments, in order to prevent user 110 from accidentally establishing an incorrect first correspondence (e.g., user 110 mistakenly identifies another virtual object as virtual object 150 and matches it with the first correspondence), XR device 120 may acquire the identification signal only after displaying the first guidance information and determine the correspondence between locator 122 and virtual object 150 based on the acquired identification signal.

[0060] In block 203, the first virtual object is controlled at least based on the first correspondence and the monitoring of the pose information of the first locator. After determining the first correspondence between the virtual object 150 and the locator 122, the user 110 can control the virtual object 150 through the locator 122. In some embodiments, the XR device 120 can control the virtual object 150 corresponding to the locator 122 based on the pose information of the locator 122 and control information from the locator 122. The pose information here may include position information, such as coordinate information, or attitude information, such as pitch angle information, yaw angle information, and roll angle information. The pose information may also include both position information and attitude information. Controlling the first virtual object here includes controlling the movement of the first virtual object in the virtual environment or controlling the display of the first virtual object.

[0061] As an example, the virtual environment 140 is a game scene, and correspondingly, the locator 122 depends on the game interaction device (i.e., the control device 121). The control device 121 can be a variety of control devices, such as a baseball bat-shaped control device, a tennis racket-shaped control device (tennis, ping-pong, etc.), a control device shaped like a simulated bow and arrow, or a control device shaped like a simulated shooting device. For example, for a baseball-shaped control device 121, the locator usually needs to be mounted on the baseball bat. Referring to Figure 1, the user 110 can interact with the game scene by waving the control device 121. In this process, the XR device 120 obtains the pose information of the locator 122 based on communication with the locator 122. In some embodiments, the locator 122 collects its own pose information and sends the pose information to the XR device 120. In some embodiments, the XR device 120 can obtain the pose information of the locator 122 based on the collected image information including the locator 122 and / or the optical pattern or multiple light spots formed by the infrared light source on the locator 122. The XR device can also obtain the IMU information of the locator 122 based on communication with the locator 122, and combine it with the pose information of the locator 122 obtained according to the optical pattern, for use in controlling the virtual object 150. In such an embodiment, there may be no communication connection between the XR device 120 and the locator.

[0062] Alternatively, the XR device 120 obtains its relative position with the locator 122 and acquires the pose information of the locator 122 based on the relative position. After acquiring the pose information of the locator 122, the XR device 120 determines the virtual object 150 corresponding to the locator 122 based on a first correspondence and controls the virtual object 150 based on the pose information. For easier understanding, an example is given here. Assume the virtual scene 140 is an archery game scene, and the locator 122 corresponds to the bow, and the locator 123 corresponds to the arrow. Under this correspondence, the XR device 120 can control the bow and arrow based on the position information of the locators 122 and 123. If the distance between the locators 122 and 123 is detected to be less than a first distance threshold, an image of the arrow being nocked on the bow is displayed. If the distance between the locators 122 and 123 increases but is not greater than a second distance threshold, an image of the bowstring being pulled is displayed. The pulling state of the bowstring is positively correlated with the distance between the locators 122 and 123. If, during the process of drawing the bowstring, it is detected that user 110 taps the locator 123, then a scene of releasing the arrow will be displayed to shoot the arrow out.

[0063] In some embodiments, the XR controller can also acquire control information sent by the locator 122 to control the virtual object 150 based on the first correspondence, the monitoring of the pose information of the locator 122, and the control information. The XR controller can also acquire control information sent by the manipulation device 121 and control the virtual object 150 based on the control information sent by the manipulation device 121.

[0064] In some embodiments, the XR controller may also present identification information of at least one locator in a connected state, the at least one locator including locator 122. And in response to receiving an identification signal from locator 122, the identification information of locator 122 may be highlighted. In some embodiments, the identification information of the locator includes, but is not limited to, the locator's serial number, locator ID information, etc.

[0065] For example, Figure 3 illustrates a schematic diagram 300 of locator identification information presented according to some embodiments of the present disclosure. As shown in Figure 3, the UI of the XR device 120 displays the ID information (e.g., Tracker 1, Tracker 2) and serial number (e.g., xxx001) of the locators in a connected state. Furthermore, the ID of Tracker 1 is highlighted for easy viewing by the user. This allows the user to more intuitively view the connection and matching status of each locator, reducing the difficulty of matching locators with virtual objects and improving the user experience. In some embodiments, the methods of highlighting the locator ID include, but are not limited to, highlighting, flashing, etc.

[0066] In some embodiments, the XR device 120 searches for a first correspondence between the locator 122 and the virtual object 150 from a pre-established correspondence between locators and virtual objects based on an identification signal.

[0067] For some commonly used locators, the XR device 120 can save the correspondence between them and the locators. After the user 110 starts the XR device 120, the XR device 120 establishes a connection with each locator 122. When it receives an identification signal from a locator 122, it first checks whether a virtual object 150 corresponding to the locator 122 exists in the pre-established correspondence between locators and virtual objects. If it exists, the first correspondence between the locator 122 and the virtual object 150 is determined. If it does not exist, the identification signal is ignored, and the corresponding virtual object is matched for the locator 122 according to the matching instruction. In this way, the correspondence between virtual objects and locators can be quickly established based on the user's usage habits and historical records.

[0068] Figure 4 shows a schematic structural block diagram of an apparatus 400 for controlling virtual objects according to certain embodiments of the present disclosure. The apparatus 400 may be implemented as or included in an XR device 120. The various modules / components in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0069] As shown in the figure, the device 400 includes an identification acquisition module 410 configured to acquire an identification signal from the locator 122 in response to a matching command for the virtual object 150. A relationship determination module 420 is configured to determine a correspondence between the locator 122 and the virtual object 150 based on the identification signal. An object control module 430 is configured to control the virtual object 150 based at least on the first correspondence and monitoring of the pose information of the locator 122.

[0070] In some embodiments, the object control module 430 is further configured to control the virtual object 150 based on the first correspondence, the monitoring of the pose information of the locator 122, and the control information received from the locator 122.

[0071] In some embodiments, the device 400 further includes a first guidance information presentation module configured to present first guidance information in response to a matching instruction for the virtual object 150 before receiving an identification signal from the locator 122, the first guidance information indicating that a matching of the virtual object 150 is performed.

[0072] In some embodiments, the relationship determination module 420 is further configured to determine the correspondence between the locator 122 and the virtual object 150 in response to receiving an identification signal from the locator 122 after presenting the first guidance information.

[0073] In some embodiments, the device 400 further includes a second guidance information presentation module, configured to present second guidance information in response to a matching instruction for the virtual object 151, the second guidance information indicating that matching is performed on the virtual object 151, the second guidance information being different from the first guidance information; in response to receiving an identification signal from the locator 123 after presenting the second guidance information, to determine a second correspondence between the locator 123 and the virtual object 151; and to control the virtual object 151 based on the second correspondence and monitoring of the pose information of the locator 123.

[0074] In some embodiments, the second guidance information is presented at a different time than the first guidance information.

[0075] In some embodiments, the apparatus 400 further includes a first matching instruction generation module, configured to generate a matching instruction for the virtual object 150 in response to receiving an update instruction for the correspondence of the virtual object 150.

[0076] In some embodiments, the apparatus 400 further includes a second matching instruction generation module configured to establish a connection between the locator 122 and the XR device 120 in response to receiving a wake-up instruction for an extended reality (XR) device. In response to determining that the locator 122 has not matched a virtual object, a virtual object 150 that is not in a matching state is identified. A matching instruction is generated for the virtual object 150.

[0077] In some embodiments, the device 400 further includes an identification information display module configured to display identification information of at least one locator in a connected state, the at least one locator including locator 122, and to highlight the identification information of locator 122 in response to receiving an identification signal from locator 122.

[0078] In some embodiments, the identification signal is a signal generated when the physical button of the locator 122 is triggered, or the identification signal is a status signal collected from the locator 122.

[0079] In some embodiments, the relationship determination module 420 is further configured to search for a first correspondence between the locator 122 and the virtual object 150 from a pre-established correspondence between the locator and the virtual object based on the identification signal.

[0080] The units and / or modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 600 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0081] Figure 5 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 500 shown in Figure 5 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 500 shown in Figure 5 can be used to implement the XR device 120 of Figure 1.

[0082] As shown in Figure 5, the electronic device 500 is in the form of a general-purpose computing device. Components of the electronic device 500 may include, but are not limited to, one or more processors 510 or processing units, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processor 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0083] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0084] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 5, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0085] The communication unit 540 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the electronic device 500 can be implemented as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0086] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user 110 to interact with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0087] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0088] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0089] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0090] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0092] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for controlling virtual objects, comprising: obtaining an identification signal of a first locator in response to a matching instruction for a first virtual object; determining a first correspondence between the first locator and the first virtual object based on the identification signal; and controlling the first virtual object based on at least the first correspondence and monitoring of pose information of the first locator.

2. The method of claim 1, wherein controlling the first virtual object comprises: controlling the first virtual object based on the first correspondence, monitoring of pose information of the first locator, and control information received from the first locator.

3. The method of claim 1, further comprising: prior to receiving the identification signal from the first locator, presenting first guidance information in response to the matching instruction for the first virtual object, the first guidance information indicating to perform matching for the first virtual object.

4. The method of claim 3, wherein determining the correspondence between the first locator and the first virtual object comprises: determining the correspondence between the first locator and the first virtual object in response to receiving the identification signal from the first locator after presenting the first guidance information.

5. The method of claim 3, further comprising: presenting second guidance information in response to a matching instruction for a second virtual object, the second guidance information indicating to perform matching for the second virtual object, the second guidance information being different from the first guidance information; determining a second correspondence between a second locator and the second virtual object in response to receiving the identification signal from the second locator after presenting the second guidance information; and controlling the second virtual object based on the second correspondence and monitoring of pose information of the second locator.

6. The method of claim 5, wherein the second guidance information is presented at a different time than the first guidance information.

7. The method of claim 1, further comprising: generating the matching instruction for the first virtual object in response to receiving an update instruction for the correspondence of the first virtual object.

8. The method of claim 1, further comprising: establishing a connection between the first locator and an extended reality (XR) device in response to receiving a wake-up instruction for the XR device; determining the first virtual object that is not in a matching state in response to determining that the first locator is not matched to a virtual object; and generating the matching instruction for the first virtual object.

9. The method of claim 1, further comprising: presenting identification information of at least one locator that is in a connected state, the at least one locator including the first locator; and highlighting the identification information of the first locator in response to receiving the identification signal from the first locator.

10. The method of claim 1, wherein the identification signal is a signal generated when a physical button of the first locator is triggered, or the identification signal is a status signal collected from the first locator. ​ ​ 11. The method of claim 1, wherein determining the first correspondence between the first locator and the first virtual object comprises: finding, based on the identification signal, the first correspondence between the first locator and the first virtual object from pre-established correspondences between locators and virtual objects.

12. An apparatus for controlling virtual objects, comprising: an identification obtaining module configured to, in response to a matching instruction for a first virtual object, obtain an identification signal of a first locator; a correspondence determining module configured to determine, based on the identification signal, a correspondence between the first locator and the first virtual object; and an object controlling module configured to control the first virtual object based at least on the first correspondence and monitoring of pose information of the first locator.

13. An electronic device, comprising: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions, when executed by the at least one processor, cause the electronic device to perform the method according to any one of claims 1-11.

14. A computer-readable storage medium having computer-executable instructions stored thereon that are executable by a processor to implement the method according to any one of claims 1-11.

15. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1-11.

Images