Method, apparatus, device and storage medium for manipulating virtual objects

Abstract

According to embodiments of this disclosure, a method, apparatus, device, and storage medium for manipulating virtual objects are provided. The method includes determining a first position of a first control device associated with a virtual environment and a second position of a second control device associated with the virtual environment; determining first orientation information based on the first and second positions; determining a rotation matrix associated with a virtual object drawn in the virtual environment based on the first orientation information and the second orientation information of the first control device; and determining the pose of the virtual object in the virtual environment based on the rotation matrix. Thus, a user can manipulate virtual objects in the virtual environment to perform expected and natural rotations.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and more particularly to methods, apparatus, devices, and computer-readable storage media for manipulating virtual objects. Background Technology

[0002] Extended Reality (XR) technology includes Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). With the advancement of extended reality technology, the construction of virtual scenes is no longer limited to simple viewing, but more about interacting with virtual objects, such as simulating holding virtual objects with both hands. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for manipulating a virtual object is provided. The method includes: determining a first position of a first control device associated with a virtual environment and a second position of a second control device associated with the virtual environment; determining first orientation information based on the first position and the second position; determining a rotation matrix associated with a virtual object drawn in the virtual environment based on the first orientation information and the second orientation information of the first control device; and determining the pose of the virtual object in the virtual environment based on the rotation matrix.

[0004] In a second aspect of this disclosure, an apparatus for manipulating virtual objects is provided. The apparatus includes: a position determination module configured to determine a first position of a first control device associated with a virtual environment and a second position of a second control device associated with the virtual environment; an orientation information determination module configured to determine first orientation information based on the first and second positions; a rotation matrix determination module configured to determine a rotation matrix associated with a virtual object drawn in the virtual environment based on the first orientation information and the second orientation information of the first control device; and a pose determination module configured to determine the pose of the virtual object in the virtual environment based on the rotation matrix.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this summary section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram of the rotation of virtual objects in the prior art is shown;

[0010] Figure 2 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 3 A schematic diagram illustrating an example of determining column vectors in a rotation matrix according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A flowchart illustrating a process of manipulating virtual objects according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A block diagram of an apparatus for manipulating virtual objects according to some embodiments of the present disclosure is shown; and

[0014] Figure 6 A block diagram of an apparatus capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0015] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0016] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0017] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0018] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0019] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0022] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0023] As briefly mentioned earlier, XR technology utilizes hardware devices combined with various software technologies to present virtual scenes or merge virtual content with real-world scenes, transforming people's interaction methods from 2D to 3D. VR technology, for example, uses head-mounted devices to simulate real-world 3D interactive scenes. AR technology, for example, uses electronic devices (such as mobile phones, tablets, glasses, etc.) to overlay various information and images onto the real world. MR technology lies between VR and AR technologies, using digital technology to achieve complex scenarios of real-time interaction between the virtual world, the real world, and the user.

[0024] In virtual environments, users hold virtual objects with their virtual hands. The posture of the virtual objects should match the posture of the user's hands as closely as possible, making the user feel that their movements are interacting naturally with the virtual objects. For example, in VR-based shooting games, when the user's virtual hands hold a virtual weapon, the posture of the virtual weapon should match the posture of the user's hands as closely as possible, and the user should be given the freedom to change the angle of holding the weapon. There should be as few restrictions as possible on how the user holds the weapon, so that the user feels natural and the sense of immersion is enhanced.

[0025] The finite rotation of an object in three-dimensional space can be represented by three relative angles: precession angle, nutation angle, and spin angle. These three angles are collectively called Euler angles. Euler angles can be used to describe the orientation of a rigid body in three-dimensional Euclidean space. However, when using Euler angles to describe the rotation of an object, the problem of "omnidirectional lock" can occur.

[0026] Figure 1 A schematic diagram of gimbal rotation in the prior art is shown. The gimbal's operating mode is completely consistent with the way Euler angles express rotation, therefore the gimbal lock problem of Euler angles can be explained by the gimbal. The gimbal consists of three interconnected ring arms. Each ring arm represents a different axis, for example, the three ring arms represent the X-axis, Y-axis, and Z-axis respectively. The arrow in the middle represents the object. When describing the object's posture using Euler angles, the rotation order of the three axes must be determined, for example, Y-axis → X-axis → Z-axis. The Y-axis is the parent level of the X-axis, and the X-axis is the parent level of the Z-axis. The rotation of the parent level will cause all child levels to rotate, while the rotation of the child levels does not affect the state of the parent level. Specifically, when the Y-axis rotates, it will cause the X-axis, Z-axis, and arrow to rotate together. When the X-axis rotates, it will cause the Z-axis and arrow to rotate together. When the Z-axis rotates, it can only cause the arrow to rotate. When rotating the X-axis ring arm, there is a probability that the Y-axis ring arm and the Z-axis ring arm will be in the same plane (e.g., ...). Figure 1As shown in the right figure, at this point, the Y-axis and Z-axis ring arms are essentially equivalent in function. Only two axis ring arms of the entire universal joint are functioning normally, and the arrow inside can no longer maintain its original balance. This phenomenon is called universal lock. To allow the arrow to rotate to the desired position, all three axis ring arms must be rotated simultaneously. However, this rotation trajectory in actual space is an arc, preventing the middle arrow from rotating in the expected straight line. This improper rotation cannot meet the requirements of practical engineering applications. Adjusting the rotation order of the three axes will change the probability of two ring arms being coplanar, but it cannot completely eliminate the problem of universal lock at a certain angle.

[0027] In specific virtual scenes, when using Euler angles to describe the rotation of virtual objects, for example, following the order of Y-axis → X-axis → Z-axis, a new axis is lost when the virtual object's pitch angle reaches 90°, making the object's rotation from 90° unnatural. This is especially problematic when handling the rotation of two-handed virtual objects, as the object's posture is determined by the postures of both hands. Calculating and merging the postures of each hand results in severe lag and renders the system unusable. Some products use quaternions to provide smooth interpolation and avoid omnidirectional lock-up, but these methods struggle to simultaneously account for the influence of both hand postures on the object's rotation.

[0028] Embodiments of this disclosure provide a scheme for manipulating virtual objects. According to various embodiments of this disclosure, a first position of a first control device associated with a virtual environment and a second position of a second control device associated with the virtual environment are determined. Based on the first and second positions, first orientation information is determined. Based on the first orientation information and the second orientation information of the first control device, a rotation matrix associated with a virtual object drawn in the virtual environment is determined. Based on the rotation matrix, the pose of the virtual object in the virtual environment is determined. Thus, a user can manipulate virtual objects in the virtual environment to rotate smoothly and naturally as expected.

[0029] Example embodiments of this disclosure are described below with reference to the accompanying drawings.

[0030] Figure 2 A schematic diagram of an example environment 200 in which embodiments of the present disclosure can be implemented is shown. In environment 200, a user 210 wears an XR device 220. The XR device 220 communicates with a computing device 230 to reconstruct virtual scenes for the user 210 or to merge virtual content with real scenes. In some embodiments, the XR device 220 and the computing device 230 may also be designed as an integrated unit. In this disclosure, virtual scenes reconstructed based on VR technology, and scenes that merge virtual content with real scenes based on AR or MR technology, are collectively referred to as virtual scene 240 (e.g., Figure 2 (As shown in the dashed box).

[0031] XR device 220 can be a head-mounted or wearable near-eye display device, such as a head-mounted display or smart glasses, supporting technologies such as VR, AR, and MR. XR device 220 may include an image generation component and an optical display component for reconstructing a virtual scene 240 in a monocular or binocular field of view and displaying the user's virtual hands 270 and virtual objects 280. Virtual objects 280 can be game props or other three-dimensional elements; the embodiments disclosed herein are not limited in this regard.

[0032] The first control device 250 and the second control device 260 can be handheld control devices, such as handles, bracelets, or rings. The first control device 250 and the second control device 260 may include motion sensors. The computing device 230 receives sensor data and uses it to control the virtual hands 270 corresponding to the user 210 to perform corresponding actions in the virtual scene 240. In some embodiments, the user 210's hands respectively hold the first control device 250 and the second control device 260. For example, in the virtual scene of a VR shooting game, the virtual hands 270 corresponding to the user 210 are presented as holding virtual firearms.

[0033] The computing device 230 can be a separate device capable of communicating with the XR device 220, the first control device 250, and the second control device 260, such as a server or computing node for image or data processing, or it can be integrated with the XR device 220. In some embodiments, the computing device 230 can be implemented as the XR device 220, that is, in this case, the XR device 220 can perform all the functions of the computing device 230. It should be understood that the above description of the computing device 230 is merely exemplary and not restrictive, and the computing device 230 can be implemented as a device of various forms, structures, or categories, and the embodiments of this disclosure are not limited in this regard.

[0034] It should be understood that the structure and function of environment 200 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0035] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0036] To avoid omnidirectional lock-up and to consider the influence of hand posture on the rotation of the held object, the various embodiments of this disclosure use rotation matrices to describe the rotation of the object. Continue to refer to... Figure 2 In the virtual scene 240, virtual hands 270 hold virtual object 280, and the pose of virtual object 280 can be derived from the position of virtual hands 270 combined with the rotation matrix.

[0037] In some embodiments, computing device 230 determines a first position of a first control device 250 associated with a virtual environment 240 and a second position of a second control device 260 associated with the virtual environment 240.

[0038] It should be understood that, depending on the type of control device used, the computing device 230 may employ appropriate methods to determine the position and / or orientation of the control device. For example, the computing device 230 may acquire pose data through sensors of the control device and determine the position of the control device in physical space through methods such as image localization. This disclosure is not intended to limit the methods for determining the position and / or orientation information of the control device.

[0039] The various embodiments of this disclosure use rotation matrices to describe the relationship between the coordinate systems of the virtual object 280 before and after rotation. The coordinate system in which the virtual object 280 is located before rotation is the coordinate system in which the virtual hands 270 are located. The positions of the virtual hands 270 correspond to the first position of the first control device 250 and the second position of the second control device 260, respectively. The computing device 230 obtains the first and second positions, for example, through an interface of a game engine.

[0040] To ensure calculations are performed within the same coordinate system, it is necessary to obtain the transformations of the first control device 250 and the second control device 260 relative to the same coordinate system. For example, the computing device 230 obtains the transformations of the first control device 250 and the second control device 260 relative to the world coordinate system through the game engine's interface. Deriving the coordinates of the virtual object 280 in its own coordinate system from the coordinates of the virtual hands 270 in the world coordinate system requires constructing a rotation matrix. The world coordinate system is a three-dimensional coordinate system established with the world origin as its origin. The virtual object's own coordinate system is a three-dimensional coordinate system established with the virtual object itself as its origin. The virtual object's own coordinate system is a local coordinate system, which can define the orientation and initial local position of the virtual object 280.

[0041] Figure 3 A schematic diagram of Example 300, illustrating the determination of column vectors in a rotation matrix according to some embodiments of the present disclosure, is shown. Example 300 describes the process of determining column vectors in a rotation matrix. Assume that the coordinate system in which the virtual object 280 resides before rotation is coordinate system A, which may be, for example, the virtual engine coordinate system. The coordinate system in which the virtual object 280 resides after rotation is coordinate system B, which may be, for example, the virtual object 280's own coordinate system. Example 300 can be implemented at computing device 230. References below... Figure 2 Description example 300.

[0042] The first position of the first control device 250 in coordinate system A is P1, represented by three-dimensional vectors F1, U1, and R1. Vectors F1, U1, and R1 are all unit vectors. Vector F1 is in the same direction as the X-axis of the coordinate system of the first control device 250, vector U1 is in the same direction as the Z-axis, and vector R1 is in the same direction as the Y-axis.

[0043] The second position of the second control device 260 in coordinate system A is P2, represented by three-dimensional vectors F2, U2, and R2. Vectors F2, U2, and R2 are all unit vectors. Vector F2 is in the same direction as the X-axis of the coordinate system of the second control device 260, vector U2 is in the same direction as the Z-axis, and vector R2 is in the same direction as the Y-axis.

[0044] Suppose the virtual object 280 is positioned P in coordinate system B. A rotation matrix can convert the coordinates of position P in coordinate system B to coordinates in coordinate system A, and vice versa. The coordinates of position P in coordinate system A are determined based on the first position P1 and the second position P2. Therefore, the rotation matrix can be derived based on the first position P1 and the second position P2.

[0045] A rotation matrix describes the rotation of an object in Cartesian coordinates. For 3D rotation, the 2D matrix can be derived first and then extended to 3D. That is, first obtain the two column vectors of the rotation matrix, and then obtain the third column vector.

[0046] In some embodiments, the computing device 230 determines first orientation information based on a first position P1 and a second position P2. Further, the computing device 230 determines a first column vector in the rotation matrix based on the first orientation information.

[0047] In some embodiments, the computing device 230 determines a direction vector from the first position P1 to the second position P2 based on the first position P1 and the second position P2, and uses the direction vector as first orientation information.

[0048] As an example, in a VR shooting game, virtual object 280 represents a virtual firearm. The virtual firearm includes a grip and barrel. When the user 210 holds the virtual firearm with their virtual hands 270, one control device is the primary device, and the other is the secondary device. The primary and secondary devices can be distinguished by the primary / secondary grip points. For example, the primary control device 250 held by the left hand corresponds to the grip position and can be considered the primary device; the secondary control device 260 held by the right hand corresponds to the barrel position and can be considered the secondary device. Generally, the virtual firearm is oriented from the primary control device 250 towards the secondary control device 260, as shown in the reference diagram. Figure 3 This can be represented by a direction vector F from the first position P1 to the second position P2. The direction vector F can be used to control the direction of the gun muzzle.

[0049] In some embodiments, the computing device 230 determines the forward vector of the first control device 250 and uses it as second orientation information. Further, the computing device 230 determines a second column vector in the rotation matrix based on the second orientation information. It should be understood that the forward vector may, for example, represent the orientation of a predetermined axis (e.g., a horizontal or vertical axis) of the first control device 250, or it may represent the direction formed by two reference portions of the first control device 250 (e.g., the front end and the rear end of the handle).

[0050] Continuing with the VR shooting game example, user 210 twists their wrist at various angles to control the rotation of the virtual gun around the barrel's axis. Generally, the rotation of the virtual gun is controlled by rotating the host device. (Reference) Figure 3 The rotation of the gun around itself is controlled by the forward vector F1 of the first control device 250.

[0051] In some embodiments, to determine the rotation matrix, the computing device 230 performs orthogonalization on the direction vector F and the forward vector F1 to determine the orthogonalized direction vector and the orthogonalized forward vector. Further, based on the orthogonalized direction vector and the orthogonalized forward vector, the computing device determines the first column vector and the second column vector of the rotation matrix.

[0052] Orthogonalizing the direction vector F and the forward vector F1 ensures that adjusting one vector along different orthogonal dimensions has little or no effect on the vectors along other dimensions. Methods for orthogonalizing vectors include Gram-Schimidt orthogonalization, Modified Gram-Schimidt orthogonalization, Householder orthogonalization, and Givens orthogonalization.

[0053] In some embodiments, the computing device 230 performs Gram-Schmidt orthogonalization on the direction vector F and the forward vector F1 to obtain vectors F' and U'. Gram-Schmidt orthogonalization on the direction vector F yields the orthogonalized direction vector F'; Gram-Schmidt orthogonalization on the forward vector F1 yields the orthogonalized forward vector U'. Specifically:

[0054]

[0055] U`=Fl-(F1·F`)F` (2)

[0056] In some embodiments, the computing device 230 determines the vector product based on the orthogonalized direction vector and the orthogonalized forward vector. Further, the computing device 230 may be able to compute the vector product of the orthogonalized direction vector F' and the orthogonalized forward vector U'.

[0057] As an example, the cross product of vectors F' and U' yields a vector product, which is perpendicular to both vectors F' and U'. The cross product is obtained by multiplying and subtracting the components. If we only care about the direction of the vectors and not their magnitudes, we can further standardize the vector product, which can be expressed as:

[0058] R`=U`×F` (3)

[0059] Therefore, the computing device 230 can further determine the three corresponding column vectors in the rotation matrix based on vectors F', U', and R', respectively, as the first column vector, the second column vector, and the third column vector. In some embodiments, the computing device 230 can determine the column vectors corresponding to vectors F', U', and R' based on the coordinate system of the virtual object 280 itself.

[0060] As an example, in game modeling, the coordinate system of the virtual object 280 is a left-handed coordinate system, with the Y-axis parallel to the virtual object 280 and the Z-axis above it. After the virtual object 280 is rotated, in its own coordinate system, vector R` is opposite to the X-axis, vector F` is in the same direction as the Y-axis, and vector U` is in the same direction as the Z-axis. Therefore, the third column vector corresponding to the vector product can be represented as -R`, the first column vector corresponding to the orthogonalization direction vector can be represented as F`, and the second column vector corresponding to the orthogonalization forward vector can be represented as U`.

[0061] In some embodiments, the computing device 230 determines the relative order of the first column vector, the second column vector, and the third column vector in the rotation matrix based on the coordinate system of the virtual object 280 itself. The relative order refers to the order in which the first column vector, the second column vector, and the third column vector are arranged based on the order of the coordinate axes. Accordingly, the computing device 230 can determine that the rotation matrix can be represented as [-R` F` U`]. Further, the computing device 230 can determine the pose of the virtual object 280 in the virtual environment 240 based on this rotation matrix.

[0062] In some embodiments, computing device 230 renders a virtual object 280 in a virtual environment 240 and presents it as being held by virtual hands 270. Computing device 230 derives a rotation matrix describing the rotation of the virtual object 280 based on the position of the virtual hands 270. This rotation matrix can be applied to the mesh components of the virtual object 280, for example, to the mesh components of a firearm provided by a game engine interface, thereby rendering a virtual firearm held by the virtual hands 270. When user 210 controls the virtual firearm through a control device, they can obtain a natural feel of holding the firearm with both hands, consistent with user 210's control expectations.

[0063] In some embodiments, computing device 230 receives a preset interaction with first control device 250. Further, based on this preset interaction, computing device 230 triggers virtual functions associated with virtual object 280. For example, in virtual scene 240, virtual hands 270 hold virtual firearms, and when user 210 instructs shooting via first control device 250, computing device 230 controls the virtual firearms to fire virtual bullets.

[0064] In summary, the various embodiments of this disclosure control the rotation of the manipulated object through a rotation matrix constructed by two control devices, achieving the position and posture of the manipulated object based on the positions of both hands, which conforms to the user's realistic perception. The orientation of the virtual object 280 is controlled by a directional vector pointing from the first control device 250 to the second control device 260, and the rotation of the virtual object 280 around itself is controlled by the forward vector of the first control device 250. In this way, the problem of the user 210 twisting their wrist at various angles can be solved by controlling the virtual object 280 to produce a natural and smooth rotation that meets expectations.

[0065] Example process

[0066] Figure 4 A flowchart of a process 400 for manipulating a virtual object according to some embodiments of the present disclosure is shown. Process 400 can be implemented at a computing device 230. Reference is made below. Figure 2 Describe the process 400.

[0067] In box 410, computing device 230 determines a first position of a first control device associated with a virtual environment and a second position of a second control device associated with a virtual environment.

[0068] In frame 420, computing device 230 determines first orientation information based on a first position and a second position.

[0069] In box 430, computing device 230 determines a rotation matrix associated with a virtual object drawn in the virtual environment based on first orientation information and second orientation information of the first control device.

[0070] In frame 430, computing device 230 determines the pose of virtual objects in the virtual environment based on a rotation matrix.

[0071] In some embodiments, the computing device 230 determines a direction vector from the first position to the second position based on the first position and the second position, as first orientation information.

[0072] In some embodiments, computing device 230 determines the forward vector of the first control device as second orientation information.

[0073] In some embodiments, the computing device 230 performs orthogonalization on the direction vector and the forward vector to determine the orthogonalized direction vector and the orthogonalized forward vector; and determines the rotation matrix based on the orthogonalized direction vector and the orthogonalized forward vector.

[0074] In some embodiments, the computing device 230 determines the vector product based on the orthogonalized direction vector and the orthogonalized forward vector; and determines the column vectors of the rotation matrix based on the orthogonalized direction vector, the orthogonalized forward vector, and the vector product.

[0075] In some embodiments, the relative order of the first column vector corresponding to the orthogonalized direction vector, the second column vector corresponding to the orthogonalized forward vector, and the third column vector corresponding to the vector product in the rotation matrix is ​​determined based on the virtual object's own coordinate system.

[0076] In some embodiments, virtual objects are drawn in a virtual environment to appear as if they are held in both hands.

[0077] In some embodiments, the computing device 230 receives a preset interaction with a first control device; and based on the preset interaction, triggers a virtual function associated with a virtual object in a virtual environment.

[0078] Example devices and equipment

[0079] Figure 5 A schematic structural block diagram of a device 500 for manipulating virtual objects according to certain embodiments of the present disclosure is shown. The device 500 may be implemented as or included in a computing device 230. The various modules / components in the device 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0080] As shown in the figure, the device 500 includes a position determination module 510, configured to determine a first position of a first control device associated with a virtual environment and a second position of a second control device associated with the virtual environment. The device 500 also includes an orientation information determination module 520, configured to determine first orientation information based on the first and second positions. The device 500 further includes a rotation matrix determination module 530, configured to determine a rotation matrix associated with a virtual object drawn in the virtual environment based on the first orientation information and the second orientation information of the first control device. The device 500 also includes a pose determination module 540, configured to determine the pose of the virtual object in the virtual environment based on the rotation matrix.

[0081] In some embodiments, the orientation information determination module 520 is configured to: determine a direction vector from the first position to the second position based on the first position and the second position, as the first orientation information.

[0082] In some embodiments, the orientation information determination module 520 is further configured to: determine the forward vector of the first control device as second orientation information.

[0083] In some embodiments, the rotation matrix determination module 530 is configured to: perform orthogonalization on the direction vector and the forward vector to determine the orthogonalized direction vector and the orthogonalized forward vector; and determine the rotation matrix based on the orthogonalized direction vector and the orthogonalized forward vector.

[0084] In some embodiments, the rotation matrix determination module 530 is configured to: determine the vector product based on the orthogonalized direction vector and the orthogonalized forward vector; and determine the column vectors of the rotation matrix based on the orthogonalized direction vector, the orthogonalized forward vector, and the vector product, respectively.

[0085] In some embodiments, the relative order of the first column vector corresponding to the orthogonalized direction vector, the second column vector corresponding to the orthogonalized forward vector, and the third column vector corresponding to the vector product in the rotation matrix is ​​determined based on the virtual object's own coordinate system.

[0086] In some embodiments, virtual objects are drawn in a virtual environment to appear as if they are held in both hands.

[0087] In some embodiments, the device 500 further includes: a receiving module configured to receive a preset interaction with a first control device; and a triggering module configured to trigger a virtual function associated with a virtual object in a virtual environment based on the preset interaction.

[0088] Figure 6 A block diagram is shown illustrating an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that... Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown can be used to achieve Figure 2 The computing device 230.

[0089] like Figure 6 As shown, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0090] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 600.

[0091] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0092] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0093] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0094] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0095] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0096] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0097] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0099] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for manipulating virtual objects, comprising: Determine the first position of a first control device associated with the virtual environment and the second position of a second control device associated with the virtual environment; Based on the first position, the second position, and the gripping method of the virtual object in the virtual environment, the first orientation information is determined; Based on the first orientation information and the second orientation information of the first control device, a rotation matrix associated with the virtual object drawn in the virtual environment is determined; as well as Based on the rotation matrix, the pose of the virtual object in the virtual environment is determined, wherein the orientation of the virtual object is determined based on the first orientation information, and the rotation of the virtual object is determined based on the second orientation information.

2. The method according to claim 1, wherein determining the first orientation information includes: Based on the first position and the second position, a direction vector from the first position to the second position is determined as the first orientation information.

3. The method according to claim 2, further comprising: The forward vector of the first control device is determined as the second orientation information.

4. The method of claim 3, wherein determining the rotation matrix associated with the virtual object drawn in the virtual environment comprises: Orthogonalization is performed on the direction vector and the forward vector to determine the orthogonalized direction vector and the orthogonalized forward vector; as well as The rotation matrix is ​​determined based on the orthogonalized direction vector and the orthogonalized forward vector.

5. The method of claim 4, wherein determining the rotation matrix comprises: Based on the orthogonalized direction vector and the orthogonalized forward vector, determine the vector product; as well as The column vectors of the rotation matrix are determined based on the orthogonalized direction vector, the orthogonalized forward vector, and the vector product.

6. The method according to claim 5, wherein the relative order of the first column vector corresponding to the orthogonalization direction vector, the second column vector corresponding to the orthogonalization forward vector, and the third column vector corresponding to the vector product in the rotation matrix is ​​determined based on the virtual object's own coordinate system.

7. The method of claim 1, wherein the virtual object is drawn in the virtual environment to appear as if held in both hands.

8. The method according to claim 1, further comprising: Receive preset interactions for the first control device; as well as Based on the preset interaction, virtual functions associated with the virtual object are triggered in the virtual environment.

9. A device for manipulating virtual objects, comprising: The location determination module is configured to determine a first location of a first control device associated with a virtual environment and a second location of a second control device associated with the virtual environment; An orientation information determination module is configured to determine first orientation information based on the first position, the second position, and the gripping method of a virtual object in the virtual environment; The rotation matrix determination module is configured to determine a rotation matrix associated with the virtual object drawn in the virtual environment based on the first orientation information and the second orientation information of the first control device. as well as The pose determination module is configured to determine the pose of the virtual object in the virtual environment based on the rotation matrix, wherein the orientation of the virtual object is determined based on the first orientation information, and the rotation of the virtual object is determined based on the second orientation information.

10. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 8.

Images