Interaction control method and apparatus, and device, storage medium and program product

Abstract

On the basis of the embodiments of the present disclosure, provided are an interaction control method and apparatus, and a device, a storage medium and a program product. The method comprises: on the basis of position information and orientation information which correspond to an extended reality device, determining target orientation information and target position information of a virtual object in a virtual environment, wherein the target orientation information at least indicates a target orientation of the head of the virtual object, and the target position information at least indicates a target position of at least one of limb portions of the virtual object; on the basis of the target orientation of the head of the virtual object, executing a pre-rotation operation on a trunk portion of the virtual object; on the basis of the target position of at least one of the limb portions of the virtual object, determining the orientation of a predetermined part in the trunk portion of the virtual object; and at least on the basis of the orientation of the predetermined part in the trunk portion, determining the posture of the pre-rotated trunk portion of the virtual object.

Description

Method, device, equipment, storage medium and program product for interactive control

[0001] The present application claims priority to the Chinese patent application No. 202410418274.6, filed on April 8, 2024, entitled “Method, device, equipment, storage medium and program product for interactive control”, the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The example embodiments of the present disclosure generally relate to the field of computer, and in particular, to a method, device, equipment, computer readable storage medium and computer program product for interactive control. BACKGROUND

[0003] Extended Reality (XR) is widely researched and applied. XR combines hardware devices and various technical means to fuse virtual content and real scenes, and provides users with unique sensory experiences. XR includes, for example, Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

[0004] Thus, a virtual environment can be used to provide users with an immersive interactive experience. SUMMARY

[0005] In a first aspect of the present disclosure, a method for interactive control is provided. The method comprises: determining target orientation information and target position information of a virtual object in a virtual environment based on position information and orientation information corresponding to an extended reality device, the target orientation information indicating at least a target orientation of a head of the virtual object, and the target position information indicating at least a target position of at least one part of a limb part of the virtual object; performing a pre-rotation operation on a torso part of the virtual object based on the target orientation of the head of the virtual object; determining an orientation of a predetermined site in the torso part of the virtual object based on the target position of the at least one part of the limb part of the virtual object; and determining a pose of the pre-rotated torso part of the virtual object based on at least the orientation of the predetermined site in the torso part.

[0006] In a second aspect of the disclosure, an apparatus for interaction control is provided. The apparatus includes: an information determining module configured to determine target orientation information and target position information of a virtual object in a virtual environment based on position information and orientation information corresponding to an extended reality device, the target orientation information indicating at least a target orientation of a head of the virtual object, and the target position information indicating at least a target position of at least one part of a limb part of the virtual object; a pre-rotation module configured to perform a pre-rotation operation on a torso part of the virtual object based on the target orientation of the head of the virtual object; an orientation determining module configured to determine an orientation of a predetermined site in the torso part of the virtual object based on the target position of the at least one part of the limb part of the virtual object; and a pose determining module configured to determine a pose of the pre-rotated torso part of the virtual object based at least on the orientation of the predetermined site in the torso part.

[0007] In a third aspect of the disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the device to perform the method of the first aspect.

[0008] In a fourth aspect of the disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon a computer program, the computer program being executable by a processor to implement the method of the first aspect.

[0009] In a fifth aspect of the disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer- executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0010] It should be understood that the description in this section is not intended to define key or essential features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] The above and other features, aspects, and advantages of embodiments of the disclosure will become more apparent from the following detailed description in conjunction with the accompanying drawings. In the drawings, the same or similar reference numerals designate the same or similar elements, in which:

[0012] FIG. 1 shows a schematic diagram of an example environment in which embodiments of the disclosure can be implemented;

[0013] FIG. 2 shows a flowchart of a process of interaction control according to some embodiments of the disclosure;

[0014] FIG. 3 shows a schematic diagram of an example of a virtual object and / or a target object in a standard pose, according to some embodiments of the present disclosure;

[0015] FIG. 4 shows a schematic diagram of an example of a relationship between a head target orientation and a torso direction of a virtual object, according to some embodiments of the present disclosure and according to a conventional manner;

[0016] FIG. 5 shows a block diagram of an apparatus for interaction control, according to some embodiments of the present disclosure; and

[0017] FIG. 6 shows a block diagram of an electronic device capable of implementing one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained in a proper manner according to relevant laws and regulations.

[0019] For example, in response to receiving an active request of a user, a prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using personal information of the user. Thus, the user can autonomously choose whether to provide personal information to the software or hardware such as an electronic device, an application program, a server or a storage medium, etc. performing the operation of the technical solutions of the present disclosure according to the prompt information.

[0020] As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending a prompt information to the user may, for example, be a pop-up window manner, in which the prompt information can be presented in a text manner. In addition, the pop-up window can also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0021] It can be understood that the above notification and obtaining of user authorization process is only illustrative and does not limit the implementation manners of the present disclosure, and other manners meeting relevant laws and regulations can also be applied to the implementation manners of the present disclosure.

[0022] It can be understood that the data (including but not limited to the data itself, the acquisition or use of the data) involved in the present technical solutions should comply with the requirements of relevant laws and regulations and relevant provisions.

[0023] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein, but rather, the embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0024] It should be noted that the titles of any sections / subsections provided herein are not limiting. Various embodiments are described throughout this document, and any type of embodiment can be included under any section / subsection. Furthermore, embodiments described in any section / subsection can be combined with any other embodiment described in the same section / subsection and / or a different section / subsection in any manner.

[0025] In the description of embodiments of the present disclosure, the term "includes" and its derivatives, such as "including," should be understood in an open, non- limiting sense, that is, "including, but not limited to." The term "based on" should be understood as "based at least in part on." The term "one embodiment" or "an embodiment" should be understood as "at least one embodiment." The term "some embodiments" should be understood as "at least some embodiments." Other explicit or implicit definitions can also be included below. The terms "first," "second," etc. can refer to different or the same objects. Other explicit and implicit definitions can also be included below.

[0026] As briefly mentioned above, XR technology is a combination of hardware devices and various technical means, which can generate a virtual environment according to the use needs of users for different users to use and interact. Users can generate virtual objects (also referred to as avatars or characters) in a virtual environment by wearing an extended reality device. Through the avatar, users can be immersed in a highly simulated virtual environment and experience, such as sports, entertainment, travel, etc., to achieve a similar sensory experience to the real world.

[0027] Therefore, in order to make the user's experience in the virtual environment better, it is necessary to make the user's avatar achieve a high degree of simulation of the human body. However, this effect has not yet been achieved, for example, there may be cases where the height and arm length of the real person and the avatar model do not match, the avatar's torso posture is stiff, the avatar's motion jumps or distorts, or the avatar's lower body motion is single and unnatural. As a result, the user's interaction effect is poor, the interaction efficiency is low, and the user experience is affected.

[0028] Embodiments of the present disclosure propose a scheme of interaction control. According to various embodiments of the present disclosure, based on position information and orientation information corresponding to an extended reality device, target orientation information and target position information of a virtual object in a virtual environment can be determined, the target orientation information at least indicating a target orientation of a head of the virtual object, and the target position information at least indicating a target position of at least one part of a limb part of the virtual object. And, based on the target orientation of the head of the virtual object, a pre-rotation operation can be performed on a torso part of the virtual object. Then, based on the target position of the at least one part of the limb part of the virtual object, an orientation of a predetermined site in the torso part of the virtual object can be determined. Accordingly, based on at least the orientation of the predetermined site in the torso part, a pose of the pre-rotated torso part of the virtual object can be determined. Thereby, the scheme can enable the virtual object in the virtual environment to realize flexible and natural body movements, and further realize high simulation to the user, so that the user can have a better immersive experience in the virtual environment.

[0029] FIG. 1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. The environment 100 can be a physical scene, i.e., the environment 100 can be an example of a real-world actual scene. In the environment 100, a user 110 can utilize an extended reality device 120 to communicate with an electronic device 130, to utilize the electronic device 130 to process data acquired by the extended reality device 120, and to issue data to the extended reality device 120. In the environment 100, the user 110 can utilize the extended reality device 120 to acquire a virtual environment 140, and to generate a virtual object 150 of the user 110 in the virtual environment 140, so that the user 110 can have an immersive interactive experience in the virtual environment 140 through the virtual object 150. The user 110 is sometimes also referred to as a target object or a physical object manipulated by the extended reality device.

[0030] The extended reality device 120 can support various types of extended reality technologies, including XR technology, VR technology, AR technology, MR technology, etc. The extended reality device 120 can thus include an XR device, a VR device, an AR device, an MR device, or any appropriate combination of the foregoing. In the example shown in FIG. 1, the extended reality device 120 includes at least one of a head-mounted device 120-1 (e.g., a VR glasses), a handheld device 120-2, and a handheld device 120-3. However, it should be understood that the extended reality device capable of implementing embodiments of the present disclosure can be configured to have any appropriate device structure and wearing form according to the specific design of the device.

[0031] Further, virtual environment 140 can exemplarily be a space, exemplarily for user interaction in a sports scenario, such as a room for sports, such that user 110 can perform sports. It should be appreciated that the room for sports is merely an example of virtual environment 140, and is not intended to be limiting thereto.

[0032] In some embodiments, virtual tools and / or virtual objects of other users can be provided in virtual environment 140, such that user 110 can interact with virtual tools in virtual environment 140 through virtual object 150, and can also interact with virtual objects of other users through virtual object 150.

[0033] It should be appreciated that when user 110 interacts in virtual environment 140 through virtual object 150, if it does not need to interact with virtual objects of other users, for example, in a case where virtual environment 140 is acquired based on data stored natively by extended reality device 120 for interaction, extended reality device 120 can also not communicate with electronic device 130 (or in other words, environment 100 can exemplarily not include electronic device 130).

[0034] In some embodiments, extended reality device 120 can include a separate head-mounted device 120-1, and can also include separate handheld devices 120-2 and / or 120-3, and can also include a separate device worn on a foot, etc. In general, extended reality device 120 can be any appropriate type of device for being worn on a user. In other embodiments, extended reality device 120 can include multiple devices of the same or different types worn on a user. As an example, extended reality device 120 can include head-mounted device 120-1, handheld device 120-2, and handheld device 120-3. In such an example, head-mounted device 120-1 can communicate with handheld device 120-2 and handheld device 120-3 to enable user 110 to experience pose adjustment of virtual object 150 and the like in a cooperative manner.

[0035] In some embodiments, head-mounted device 120-1 can be a head-mounted display, for example. Handheld device 120-2 and handheld device 120-3 can be handheld controllers worn on a right hand and a left hand, respectively, for example. The handheld controllers can be appropriate types of devices controlled by hands, such as hand grips, for example. In some embodiments, extended reality device 120 can include a camera for capturing images of a physical space.

[0036] The electronic device 130 can be a separate device capable of communicating with the extended reality device 120 and / or other image capturing devices, such as a server, a computing node, etc. for image or data processing, and can also be integrated with the extended reality device 120 and / or other image capturing devices. In some embodiments, the electronic device 130 can be implemented as or include the extended reality device 120, i.e., in this case, the extended reality device 120 can implement all functions of the electronic device 130. It should be understood that the above description of the electronic device 130 is merely exemplary and not limiting, and the electronic device 130 can be implemented as a device in multiple forms, structures or categories, and embodiments of the present disclosure have no limitation in this regard.

[0037] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure.

[0038] Some example embodiments of the present disclosure will be described below with continuous reference to the accompanying drawings.

[0039] FIG. 2 shows a flowchart of a process 200 of interaction control according to some embodiments of the present disclosure. For ease of discussion, these embodiments will be described with reference to the environment 100 of FIG. 1. These embodiments can be implemented in the extended reality device 120 of FIG. 1.

[0040] As shown in FIG. 2, at block 210, based on the position information and orientation information corresponding to the extended reality device 120, the extended reality device 120 can determine target orientation information and target position information of the virtual object 150 in the virtual environment 140. The target orientation information can at least indicate a target orientation of a head of the virtual object 150, and the target position information can at least indicate a target position of at least one of the limb portions of the virtual object 150.

[0041] To achieve the conversion from the position information and orientation information corresponding to the extended reality device 120 to the target orientation information and target position information of the virtual object, a mapping relationship between the real world and the virtual world is first established. During the manipulation of the virtual object, a VR inverse kinematics (IK) scheme can be used to divide the real object (e.g., a human body) skeleton into five chains (trunk and limbs). For the upper body portion, the device input values can be calibrated with the initial state of the virtual object model according to the position information and orientation information input by the extended reality device, and the modified six degrees of freedom (6Dof) values can be used as the driving target points of the head skeleton and the left and right hand skeletons for IK solving, respectively.

[0042] When driving, the extended reality device 120 can calculate the chest rotation according to the relative position of the hand controller (for example, the device 120-2, 120-3) and the shoulder of the current virtual object, determine the target orientation information and the target position information of the virtual object 150 using the corresponding algorithm, including the degree of whole torso bending, rotation, etc. Next, the neck will need to be rotated according to the head orientation, the skull skeleton, and the whole body will be gradually rotated.

[0043] When controlling the overall posture of the virtual object 150, at block 220, based on the target orientation of the head of the virtual object 150, the extended reality device 120 can perform a pre-rotation operation on the torso part of the virtual object 150. At block 230, based on the target position of at least one of the limb parts of the virtual object 150, the extended reality device 120 can determine the orientation of a predetermined part in the torso part of the virtual object 150. The limb part can include parts such as hands, arms, legs, feet, etc., for example, when moving the single arm, the left and right arms, the single foot, the left and right feet, the torso will also move left and right. At block 240, based at least on the orientation of the predetermined part in the torso part, the extended reality device 120 can determine the posture of the pre-rotated torso part of the virtual object 150. In this way, the entire virtual object can be moved according to the head position, so that the virtual object head is completely coincident with the target position.

[0044] Thus, by using the extended reality device 120, pre-rotating the torso according to the target orientation of the head of the virtual object 150 in the virtual environment 140, the bending trend of the torso as a whole can be kept consistent with the head, avoiding the gap between the head and the torso after subsequent calculation. The rotation angle is too large, which causes the body of the virtual object 150 to be excessively distorted. Then, according to the target position of a part of the limb part, the orientation of a part (such as the chest) in the torso can be determined, so as to continue to rotate the torso, ensuring that the posture of the spine of the virtual object 150 is in a natural and agile state, avoiding the posture of the torso to be in a rigid state. Further, the body movement of the virtual object 150 is natural and coherent, achieving high simulation of the body movement of the user 110, enabling the user 110 to have a better immersive experience in the virtual environment 140, effectively improving the user's experience. The specific implementation steps of the process 200 will be described in detail below.

[0045] In some embodiments, in establishing the real world to virtual world conversion, i.e., determining the target orientation information and target position information of the virtual object 150 in the virtual environment 140 based on the corresponding position information and orientation information of the extended reality device 120, the displacement and rotation of the extended reality device 120 (which can include one or more of the head-mounted device 120-1, the handheld device 120-2, and the handheld device 120-3) can be taken as actual input, with a certain point on the ground of the real world as the origin. Then, the head and / or hand target displacement and rotation of the virtual object 150 in the virtual environment 140 are taken as target input, with the position where the origin of the virtual object 150 (which can also be referred to as an avatar or virtual character) is located as the origin. The driving target of the virtual object 150 is to make the head of the virtual object 150 consistent with the head-mounted device 120-1, and / or make the hands of the virtual object 150 consistent with the right-hand handheld device 120-2 and / or the left-hand handheld device 120-3.

[0046] In some cases, the real pose of the target object (i.e., the user 110 actually manipulating the extended reality device) can not correspond to the target pose of the virtual object 150. For example, the user 110 needs the virtual object 150 to stand when sitting, or the user 110 needs the virtual object 150 to sit when standing, and so on. In this case, the extended reality device 120 needs to perform displacement adjustment for the virtual object 150, including height adaptation adjustment, arm length scaling adjustment, rotation adjustment, and the like. Details will be described below.

[0047] In some embodiments, the target position information further includes the height of the virtual object 150. And in determining the target position information of the virtual object 150 in the virtual environment 140, the extended reality device 120 determines a target pose mode of the virtual object 150, the target pose mode including a standing mode or a sitting mode. The extended reality device 120 can adjust the height indicated by the position information based on a preset offset value corresponding to the target pose mode, to calculate the height of the virtual object 150 in the virtual environment 140.

[0048] Specifically, in the IK driving stage, the actual device input and the preset offset value can be directly read, and then the pose of the virtual object 150 is calculated. IK is a mathematical process of calculating the joint parameters of the end of a moving joint (e.g., the palm of the end of a human skeleton arm) to reach a required position relative to the starting position and direction of the joint. The actual device input may, for example, be height information when the user 110 wears the extended reality device 120. The preset offset value can be set by the user 110 or preconfigured. For standing or sitting, the preset offset value can be different.

[0049] For example, if the user manipulates the virtual object 150 in a sitting posture, but wants the virtual object 150 to remain in a standing posture, the height in the position information input by the extended reality device 120 is lower (lower than the actual height of the virtual object 150), and at this time, the preset height offset value needs to be used to adjust the position information input by the extended reality device 120, so that the virtual object 150 can remain in a standing posture.

[0050] In some embodiments, the extended reality device 120 can determine an offset value between the height of the virtual object 150 in a standard posture and the height of the target object holding the extended reality device 120 in a standard posture as a preset offset value corresponding to the standing posture mode.

[0051] FIG. 3 shows a schematic diagram of an example of a virtual object 150 and / or a target object in a standard posture, according to some embodiments of the present disclosure. The standard posture 300 can be a posture assumed by the virtual object 150 and / or the target object of FIG. 1. The standard posture 300 can be applied in the process 200 of FIG. 2. As shown in FIG. 3, the standard posture 300 can be, for example, a T-pose posture, i.e., a posture in which the arms (including a left arm 310 and a right arm 320) are spread in a standing position.

[0052] Specifically, when the virtual object 150 is loaded in the virtual environment 140, in the initialization stage, the extended reality device 120 can read the standing height of the virtual object 150 in the standard T-pose posture, and calibrate according to the current real height of the target object (which can be the user 110) and the standing height of the virtual object 150.

[0053] In some embodiments, if it is determined that the height difference between the height of the virtual object 150 and the target height corresponding to the target posture mode exceeds a threshold value, the extended reality device 120 can update the preset offset value corresponding to the target posture mode based on the height difference.

[0054] Specifically, for the case where the target posture is standing, the calculated actual height of the virtual object 150 can be compared with the target height, a new preset offset value can be calculated by different strategies and user parameter settings, and a callback can be triggered to inform the user to adjust the extended reality device 120. At this time, the input of the next frame into the IK solver is the target position after the preset offset value is updated.

[0055] In addition, for the case where the target posture is a sitting posture or other postures, the user 110 can also input the target height of the virtual object 150 through the extended reality device 120, and automatically calibrate when the difference between the current actual posture of the virtual object 150 and the target posture is too large.

[0056] In addition, the extended reality device 120 also supports setting the virtual object 150 to be in a standing posture for a long time. For example, the user 110 can set the virtual object 150 to resume standing when crouching for more than a certain distance in the standing mode, or the virtual object 150 to resume standing when crouching or hovering for more than a certain time.

[0057] Thus, through the above embodiments, the user 110 can adjust the target posture of the virtual object 150 through the extended reality device 120, thereby achieving height adaptation with the virtual object 150 based on the needs of the user 110, and improving the user experience.

[0058] In some embodiments, the extended reality device 120 can include at least one of a handheld device held by the target object and a head-mounted device. And in determining the target orientation information and the target position information of the virtual object 150 in the virtual environment 140, the extended reality device 120 can determine at least one of a first scaling relationship between the arm length of the target object and the arm length of the virtual object 150, and a second scaling relationship between the height of the target object and the height of the virtual object 150. According to the determined at least one of the first scaling relationship and the second scaling relationship, the extended reality device 120 can adjust the corresponding position information and orientation information to determine the target orientation information and the target position information of the virtual object 150.

[0059] Exemplarily, in the scaling adjustment process of the present embodiment, the target object and the virtual object 150 can both be in the T-pose posture shown in FIG. 3. Specifically, in the initial stage, when the target object is in the T-pose posture, the extended reality device 120 (specifically, the handheld device 120-2 and the handheld device 120-3) can measure the real arm length of the target object. It should be understood that without this measurement step, the extended reality device 120 can estimate the height and arm length of the user 110 according to the height data of the head-mounted device 120-1 of the user 110 when entering the XR application in the initial stage. Then, the extended reality device 120 can calculate the scaling relationship between the height and arm length of the target object and the height and arm length of the virtual object 150, respectively, and subsequent device inputs can be scaled proportionally as the driving target (including the target orientation information and the target position information) of the virtual object 150. At the same time, the cameras in the extended reality device 120 also need to be scaled proportionally to ensure that the perspective of the user 110 is consistent with the actual sense of presence.

[0060] Thus, through the scaling adjustment process, the inconsistency between the height and arm length of the real user 110 and the height and arm length of the virtual object 150 can be solved, so that the user 110 can have a better immersive experience in the virtual environment 140.

[0061] In some embodiments, the extended reality device 120 can be worn by the target object. And in determining the target orientation information and the target position information of the virtual object 150 in the virtual environment 140, the extended reality device 120 can determine a rotation offset value between an initial rotation value of the virtual object 150 in a standard pose and an initial rotation value of the extended reality device 120 in a standard pose of the target object. Accordingly, the extended reality device 120 adjusts its corresponding position information and orientation information based on the rotation offset value to determine the target orientation information and the target position information of the virtual object 150.

[0062] Exemplarily, in the process of performing the rotation adjustment of the present embodiment, the target object and the virtual object 150 can still be in the T-pose pose shown in FIG. 3. Specifically, when the target object and the virtual object 150 are both in the T-pose pose, the extended reality device 120 can record the initial rotation value of the head-hand skeleton of the virtual object 150 at this time, and the initial rotation value of the data transmitted by the extended reality device 120, and then calculate the rotation offset value between the two. In this way, subsequent device inputs can be used as the driving target of the virtual object 150 after applying the rotation offset value.

[0063] In some embodiments, in addition to the height adaptive adjustment, the arm length scaling adjustment, and the rotation adjustment, a customized position and a customized rotation offset of the virtual object can also be provided. For example, there can still be a small deviation between the position and orientation of the handheld device and the arm skeleton position and orientation of the virtual object, and / or between the head-mounted device (corresponding to the eye position) and the head skeleton position (near the neck) of the virtual object. At this time, the target position and orientation of the corresponding virtual object skeleton can be calculated according to the set offset value.

[0064] In calculating the whole torso pose, the extended reality device 120 can calculate the hip position and correspondingly generate the spine pose. In some embodiments, the extended reality device 120 can determine the target hip position of the virtual object 150 based on the target head position of the virtual object 150. If it is determined that the target head position of the virtual object 150 is lowered relative to the height of the virtual object 150, the extended reality device 120 can determine the lowering height of the hip from the current position based on the lowering height of the head. And based on the proportion of the lowering height of the head target position relative to the height of the virtual object 150 and the height of the virtual object 150, the extended reality device 120 can determine the moving distance of the hip in a two-dimensional plane, which is a plane defined by two directions other than the vertical direction corresponding to the lowering height in the three-dimensional space. Accordingly, while lowering the hip of the virtual object 150 from the current position in the vertical direction by the lowering height, the extended reality device 120 can move the hip in the two-dimensional plane by the moving distance.

[0065] Since the target object's hip moves gradually downward and backward to keep the center of gravity balanced when the target object actually squats with the head moving downward, in the process of calculating the hip position of the target object in the squatting case of the embodiment, a preset curve and a direction vector can be used to simulate the driving process of the virtual object 150. Specifically, a vector v = (x, y, z) can be given, where y represents the height of the hip descending in the vertical direction, and at the same time the hip moves a certain distance along the xz component of the v vector (corresponding to the plane defined by the other two directions in addition to the y direction in the three-dimensional space). The distance of the xz component moving can be represented by the formula: (1-ratio)*(1-ratio)*Height, where ratio is the ratio of the height of the head descending to the height of the virtual object 150 (i.e., the ratio of the height of the virtual object 150 descending), and Height represents the height of the virtual object 150.

[0066] In some embodiments, when determining the hip target position of the virtual object 150 based on the head target position of the virtual object 150, if it is determined that the head target position of the virtual object 150 is elevated relative to the height of the virtual object 150, the extended reality device 120 can determine the elevation of the hip from the current position based on the elevation of the head, and make the hip of the virtual object 150 rise from the current position by the elevation in the vertical direction.

[0067] In the process of calculating the hip position of the target object in the jumping case of the embodiment, since the hip of the target object moves vertically upward, in the driving process of the virtual object 150, when the head target moves upward relative to the initial position, no additional special processing is performed on the xz axis position of the hip. Instead, only the head orientation change slightly moves in the xz plane, and the y axis follows the head height change.

[0068] Thus, by calculating the change of the hip position of the virtual object 150 with the squatting and jumping of the target object, the virtual object 150 can be realistically and naturally active in the virtual environment 140 when driving the virtual object 150, improving the user experience.

[0069] In some embodiments, when performing pre-rotation on the torso part of the virtual object 150 based on the target orientation of the head of the virtual object 150, based on the target orientation and the weight of each of the plurality of torso bones in the torso part, the extended reality device 120 can determine the rotation amplitude of each of the plurality of torso bones, and pre-rotate the plurality of torso bones by the determined rotation amplitudes.

[0070] Specifically, each intermediate bone can be rotated by a certain proportion towards the head target direction according to the weight of each bone in the torso of the virtual object 150, so as to keep the overall bending trend of the torso consistent with the head, and avoid the deviation angle between the head and the torso being too large after subsequent calculation, so that the torso of the virtual object 150 is in an unnatural twisted state.

[0071] Therefore, through the rotation adjustment process, the problem that the orientation of the head of the user 110 (that is, the orientation of the head-mounted device 120-1) is inconsistent with the orientation of the torso bone axis of the virtual object 150 can be solved, the high simulation of the virtual object 150 to the user 110 is realized, and the user experience is improved.

[0072] In addition, similarly, the consistency of the orientations of the handheld devices 120-2 and 120-3 and the orientations of the hand bones of the virtual object 150, and the consistency of the eye positions of the user 110 corresponding to the head-mounted device 120-1 and the head bone positions (such as the positions near the neck) of the virtual object 150 can also be adjusted. And still can use the corresponding rotation offset value to calculate and adjust.

[0073] In some embodiments, in determining the pose of the pre-rotated torso part of the virtual object 150, the extended reality device 120 can rotate the torso bone of the pre-rotated torso part based on the orientation of the predetermined part in the torso part. And based on the head target position of the virtual object 150, the extended reality device 120 can determine the hip target position of the virtual object 150. Accordingly, based on the hip target position and the head target position, the extended reality device 120 can rotate the torso part as a whole to the orientation indicated by the target orientation information.

[0074] Exemplarily, the extended reality device 120 can determine the orientation of the chest in the torso according to the hand target position of the virtual object 150 (because the torso will rotate left and right when the arms move left and right), and continue to rotate the torso bone. Then, adjust the hip position according to the change of the head height, and calculate the torso bone pose by using CCD (Cyclic Coordinate Descent) IK or FABRIK (Forward and Backward Reaching Inverse Kinematics) and the like based on the currently determined hip and head positions.

[0075] Specifically, the vectors from the neck of the virtual object 150 to the left and right hand targets can be calculated, which are represented by leftDirection and rightDirection respectively. Then, the weights of the left and right hands to the chest orientation are calculated respectively, and the calculation method is as follows:

[0076] float leftWeight = 1.f - abs(Dot(directionLeft, upVec) / leftLength);

[0077] float rightWeight = 1.f - abs(Dot(directionRight, upVec) / rightLength);

[0078] leftWeight *= Magnitude(directionLeft) / leftLength;

[0079] rightWeight *= Magnitude(directionRight) / rightLength.

[0080] Wherein, upVec is a direction along the virtual object 150 trunk upward, leftLength and rightLength are the arm length of the virtual object 150. When calculating the weight, the arm target orientation is close to the up axis, and the weight is small, and the hand is close to the shoulder and neck, and the weight is small.

[0081] In addition, it is necessary to avoid the actual hand position not to move greatly, but the xz plane projection direction changes greatly, and the target orientation changes greatly. For example, the right hand is close to the body, and has little influence on the chest orientation, and the left hand is far away from the body, and the proportion of driving the chest to rotate is large. Specifically, the leftDirection and rightDirection can be projected on the xz plane to obtain new leftDirection and rightDirection. Then, the vector leftShoulderDirection and rightShoulderDirection from the neck to the left and right shoulders of the virtual object 150 are calculated as the arm orientation when the virtual object 150 holds the arm horizontally at the current orientation.

[0082] The calculated chest orientation is the direction after the leftDirection, rightDirection, leftShoulderDirection, rightShoulderDirection are mixed according to the weight. At the same time, the chest orientation needs to ensure that the angle with the head orientation is less than 90 degrees and is continuous within a certain time range, and there is no direction mutation.

[0083] In some embodiments, in rotating the torso portion as a whole to the orientation indicated by the target orientation information based on the hip target position and the head target position, the extended reality device 120 can determine a direction corresponding to a line connecting the hip target position and the head target position. Then, based on a difference between the target orientation of the head of the virtual object 150 and the direction corresponding to the line, the extended reality device 120 can determine a degree of rotation of the torso portion. Accordingly, the extended reality device 120 can rotate the torso portion of the virtual object 150 as a whole based at least on the degree of rotation of the torso portion to meet the orientation indicated by the target orientation information.

[0084] Specifically, a pre-rotation of the torso as a whole can be performed according to the head target position to the target direction, and the degree of rotation is determined by a parameter a. If the head target orientation is close to the direction of the hip-head line, a smaller value of a is used to promote the rest of the torso skeleton to continue to rotate to the target direction in subsequent calculations. If the head target orientation is opposite to the direction of the hip-head line, a larger value of a is used to promote the rest of the torso skeleton to continue to rotate in the opposite direction in subsequent calculations.

[0085] FIG. 4 shows a schematic diagram of an example of the relationship between the head target orientation of the virtual object 150 and the torso direction according to some embodiments of the present disclosure and according to a conventional way. The parts of the example 400 related to the embodiments of the present disclosure can be implemented in the environment 100 of FIG. 1. The parts of the example 400 related to the embodiments of the present disclosure can be shown as part of the embodiments of the steps of the process 200. The parts of the example 400 related to the embodiments of the present disclosure can apply the standard pose 300 of FIG. 3.

[0086] As shown in FIG. 4, in the example 410 of the first step, it is assumed that the target position and the target orientation of the head 490 of the virtual object 150 are determined. In the example 420 of the second step, the torso 480 is initially rotated based on the target position of the head 490. The process from the example 420 to the example 430 of the third step is a pre-rotation of the torso 480 as a whole to the target direction. The process from the example 430 to the example 440 of the fourth step can be solved by multiple iterations through the conventional chain IK or other classic ways, so that the skeleton of the head 490 is close to the target position. Finally, in the process from the example 440 to the example 450 of the fifth step, the skeleton of the head 490 is adjusted to the target position. As can be seen from the example 450, the line connecting the head 490 and the torso 480 of the virtual object 150 is in a natural bending state. In the conventional way of the example 460, the line connecting the head 490 and the torso 480 of the virtual object 150 is in a twisted state. When using the conventional chain IK or other classic ways, physical constraints (hinges) can be used to connect the bones for solving.

[0087] Thus, by taking the target orientation into account in the process of solving the target position of the head 490 of the virtual object 150, the overall pose after solving can be smoother, avoiding the phenomenon that although the target condition is met, the angle between the bones is too large and the pose is distorted in the traditional way.

[0088] In some embodiments, the virtual object 150 is composed of a plurality of bones, and the target orientation information and the target position information are target orientation information and target position information of the virtual object 150 in the current frame. For a given bone in the plurality of bones, based on the target orientation information and the target position information of the virtual object 150 in the current frame, the extended reality device 120 can determine a rotation angle of the given bone, and determine a first rotation angle and a second rotation angle corresponding to the given bone after increasing 360 degrees and decreasing 360 degrees on the rotation angle. The extended reality device 120 can apply the rotation angle constraint corresponding to the given bone to the first rotation angle and the second rotation angle respectively to obtain a first corrected rotation angle and a second corrected rotation angle corresponding to the given bone. Accordingly, the extended reality device 120 can select a corrected rotation angle closer to the target rotation angle of the given bone in the previous frame from the first corrected rotation angle and the second corrected rotation angle as the target rotation angle of the given bone in the current frame.

[0089] In some embodiments, the arm driving of the virtual object 150 can refer to the arm IK algorithm. The extended reality device 120 can calculate the shoulder rotation according to the relative position of the controller and the current shoulder, and then use an IK solver (such as Trigonometric IK) to solve the upper arm, elbow, lower arm, wrist rotation, etc. by applying a set of rotation constraints, so that each joint of the character's arm is rotated to the appropriate position. In this process, it is necessary to try to keep the motion continuous in each frame.

[0090] Since when calculating the rotation constraint of the bone of the virtual object 150, angle jump may occur due to the positive and negative changes of the angle, it is necessary to take the pose of the previous frame as a reference to make the bone rotation as continuous as possible. For example, the activity range of a certain bone is (-100, 100) degrees, when the actual rotation is 180 degrees, it will be limited to 100 degrees by the angle constraint. And continue to rotate, from 180 degrees to -179 degrees, the actual angle is continuous, but after applying the angle constraint, it will be limited to -100 degrees, thus the jump phenomenon occurs. To avoid this situation, when applying the rotation constraint, it is necessary to judge, and specifically, the rotation constraint can be applied to the two angles of a (i.e. the rotation angle of the given bone) and a±2p (i.e. the first rotation angle and the second rotation angle corresponding to the given bone), and the value closer to the previous frame is selected.

[0091] Exemplarily, when the first rotation angle and the second rotation angle are corrected, the rotation constraint can be applied to the two angles of α±2π. For example, if the activity angle range of a given bone is (-100, 100), but α+2π exceeds 100 degrees, then the first corrected rotation angle is equal to 100 degrees. If α-2π is lower than -100 degrees, then the second corrected rotation angle is equal to -100 degrees. If the two angles of α±2π do not exceed the highest activity angle or are lower than the lowest activity angle, then the first and second corrected rotation angles are equal to α±2π. Then, when calculating the target rotation angle of the given bone in the current frame, for example, if the target selection angle of the given bone in the last frame is 99 degrees, and the first corrected rotation angle determined in the current frame is 100 degrees, and the second rotation correction angle is -100 degrees, then the first corrected rotation angle (i.e., 100 degrees) should be selected.

[0092] Taking the given bone as the forearm bone, for example, the wrist rotation assigned to the forearm bone will also jump due to positive and negative changes if it is always calculated according to the shortest path. For example, when the wrist rotation is 180 degrees, three bones are assigned to the forearm, and each bone rotates 60 degrees. When the movement continues to -179 degrees, the shortest path will change to -60 degrees for each bone. At this time, similar processing is also required. The bones are continuously rotated in a continuous period of time.

[0093] In some embodiments, the virtual object 150 is composed of a plurality of bones. And for a given bone in the plurality of bones, based on the target orientation information and the target position information of the virtual object 150, the extended reality device 120 can determine a continuous rotation angle range of the given bone in a period of time. If it is determined that the continuous rotation angle range exceeds a predetermined angle range threshold, the extended reality device 120 can reset the given bone to an initial state along the shortest path.

[0094] The embodiment provides a reset mode for bone rotation, which can return to the shortest path calculation mode when the continuous rotation angle is too large (such as the wrist rotating a full circle), and avoids the phenomenon of “twisting a doughnut” when continuously rotating in the same direction.

[0095] Therefore, through the processing of the bone action continuity of the virtual object 150, the virtual object 150 achieves high simulation of the user 110, further achieves a realistic effect, and is beneficial to improve the user experience.

[0096] In some embodiments, based on the target orientation information and the target position information of the virtual object 150, the extended reality device 120 can determine the target position and the target orientation of the two feet of the virtual object 150. Based on the target position and the target orientation of the two feet, the extended reality device 120 can determine that the virtual object 150 is to perform a step. If it is determined that the virtual object 150 is to step from the current position, and the virtual object 150 is in the small stride mode, the foot with a distance between the current position and the target position farther than the other foot is determined as the stepping foot, and the other foot is determined as the supporting foot, to perform the step of the virtual object 150.

[0097] Specifically, the extended reality device 120 can calculate the center of gravity movement according to the head and hand data, and then calculate the target positions and the target orientations of the left and right feet through a preset curve and a set position threshold and a rotation threshold, and use the target positions and the target orientations as the input of the footstep IK. According to the position threshold and the rotation threshold, if the difference between the target position and the target orientation of the foot and the current position and the current orientation respectively exceeds the threshold, and there is no foot currently stepping, a step from the current landing point to the target landing point is triggered, that is, it is determined that the virtual object 150 is to step from the current position. When the step is triggered, the foot with a distance farther than the other foot from the current position to the target position is selected as the stepping foot, and the other foot is selected as the supporting foot, so as to complete the small pacing of the footstep solver in a small range of activities.

[0098] In some embodiments, based on the current position of the supporting foot and the line connecting the current position of the stepping foot and the target position, the extended reality device 120 can perform collision detection. Accordingly, based on the result of the collision detection, the extended reality device 120 can determine the landing point of the two feet of the virtual object 150 after stepping. Specifically, when calculating the landing point, the extended reality device 120 will perform collision detection of a line sphere according to the current position of the supporting foot and the line connecting the current position of the stepping foot and the target position, to avoid the situation that the two feet are inserted or too close during the stepping process.

[0099] In some embodiments, during the stepping process of the virtual object 150, the extended reality device 120 can adjust the positions of the two feet of the virtual object 150 based on the stepping parameters related to the virtual object 150. The stepping parameters can include at least one of the following: a stepping curve (for example, a sine curve), a stepping speed, a stepping height. Specifically, during the stepping process, the extended reality device 120 can adjust the target position of the stepping foot of the virtual object 150 based on these stepping parameters every frame.

[0100] In some embodiments, during the stepping of the virtual object 150, based on the received new target position information and target orientation information of the virtual object 150, the extended reality device 120 can also update the position and orientation of the stepping foot every frame, and update the orientation of the supporting foot every frame. During the stepping, the extended reality device 120 can simultaneously update the target position and target orientation of the stepping and the orientation of the supporting foot according to the newly calculated predicted position every frame. During the stepping of the virtual object 150, the extended reality device 120 can also keep the position of the supporting foot unchanged, which can ensure that the grip does not slip.

[0101] In some embodiments, if it is determined that the virtual object 150 is to step from the current position and the virtual object 150 is in the large stride mode, i.e., the large stride action occurs, the extended reality device 120 can perform the stepping of the virtual object 150 according to a preset animation algorithm based on the target position and target orientation of the two feet. Specifically, according to the moving direction and speed, the extended reality device 120 can trigger the playing and mixing of different animations through an animation state machine and a 2D (two-dimensional) blending tree, and adjust the animation playing speed according to the moving distance of the animation root node and the actual center of gravity moving distance of the virtual object 150 every frame to avoid slipping of the foot bottom. The position and orientation of the foot of the virtual object 150 in the animation, and the knee joint orientation are taken as the input of the IK algorithm, so as to realize the large amplitude action of the virtual object 150.

[0102] In some embodiments, if it is determined that the virtual object 150 is to step from the current position and the virtual object 150 is in the intermediate transition state between the small stride mode and the large stride mode, the extended reality device 120 can perform the stepping of the virtual object 150 in the intermediate transition state by mixing the stepping determined for the virtual object 150 in the small stride mode and the stepping determined for the virtual object 150 in the large stride mode. Specifically, the intermediate transition state can mix the target foot and knee joint orientations obtained in the above two ways according to a certain weight, and the mixed result can be taken as the input of the foot IK algorithm.

[0103] In some embodiments, the extended reality device 120 can also realize diversified walking postures by configuring parameters such as displacement and rotation threshold value for triggering stepping, stepping height, step speed, collision radius, speed threshold value for switching to animation, transition duration, etc.

[0104] Therefore, through the related calculation of the footstep activity of the virtual object 150, the state of the virtual object 150 highly simulating the physical activity of the user 110 is further achieved, which can effectively improve the immersive experience of the user in the virtual environment 140.

[0105] In some embodiments, in a case where the virtual object 150 is in the sitting mode, if it is determined that the movement range of the head of the virtual object 150 exceeds a movement range threshold or the rotation range of the head exceeds a rotation range threshold, the extended reality device 120 can cause the virtual object 150 to exit the sitting mode.

[0106] Specifically, the lower body posture of the virtual object 150 in the sitting mode is mainly controlled by animation. The system of the extended reality device 120 has multiple different heights of sitting modes, such as high, medium, and low, which can be mixed according to the actual height to ensure that the posture is natural. In order to avoid the posture of the virtual object 150 being too distorted due to the arbitrary movement of the user 110, the embodiment provides a function of automatically exiting the sitting mode. For example, when the movement range of the head of the user 110 is too large, i.e., the distance between the head and the hip skeleton exceeds the length of the torso of the virtual object 150 (assuming it is a movement range threshold), the sitting mode (i.e., the sitting mode) is automatically exited. Accordingly, the hip is restored to the default mode of following the head movement, and the height is automatically restored to the standing mode. For another example, when the head of the user 110 rotates too much, i.e., the angle between the head orientation and the chair orientation exceeds a certain angle, the sitting mode can also be automatically exited.

[0107] In some embodiments, when entering the sitting mode, the height adaptation is adjusted to the sitting mode, and the camera position can be adjusted to the height of the seat accordingly. The footstep solver can also be switched to the animation mode, specifically, the small pacing of the footstep can be turned off, and the position of the foot skeleton in the animation and the knee orientation are directly read as the IK target. Moreover, the target position of the foot can be adjusted to the ground height according to the ground height, and the IK calculation is based on the animation to stretch or bend the legs. Accordingly, when exiting the sitting mode, the height adaptation is adjusted to the standing mode, and the camera position is adjusted to the standing height of the virtual object 150. Moreover, the footstep solver can be switched back to the small pacing plus front-back-left-right animation mode.

[0108] Through the above disclosed embodiments, the virtual object 150 in the virtual environment 140 can realize flexible and natural body movement, and further realize high simulation of the user 110, so that the user 110 can have a better immersive experience in the virtual environment.

[0109] FIG. 5 shows a schematic structural block diagram of an apparatus 400 for interaction control according to certain embodiments of the present disclosure. The apparatus 500 can be implemented as or included in the extended reality device 120. Various modules / components in the apparatus 500 can be implemented by hardware, software, firmware, or any combination thereof.

[0110] As shown, the apparatus 500 includes an information determining module 510 configured to determine target orientation information and target position information of a virtual object in a virtual environment based on position information and orientation information corresponding to an extended reality (XR) device, the target orientation information indicating at least a target orientation of a head of the virtual object, and the target position information indicating at least a target position of at least one of limb portions of the virtual object; a pre-rotation module 520 configured to perform a pre-rotation operation on a torso portion of the virtual object based on the target orientation of the head of the virtual object; an orientation determining module 530 configured to determine an orientation of a predetermined site in the torso portion of the virtual object based on the target position of the at least one of the limb portions of the virtual object; and a pose determining module 540 configured to determine a pose of the pre-rotated torso portion of the virtual object based on at least the orientation of the predetermined site in the torso portion.

[0111] In some embodiments, the target position information further includes a height of the virtual object, and the information determining module 510 includes a target position information determining module configured to determine a target pose mode of the virtual object, the target pose mode including a standing mode or a sitting mode; and adjust the height indicated by the position information based on a preset offset value corresponding to the target pose mode to calculate the height of the virtual object in the virtual environment.

[0112] In some embodiments, the apparatus 500 further includes an offset value determining module configured to determine an offset value between a height of the virtual object in a standard pose and a height of the target object holding the extended reality device in the standard pose as the preset offset value corresponding to the standing mode.

[0113] In some embodiments, the apparatus 500 further includes an offset value updating module configured to update the preset offset value corresponding to the target pose mode based on a height difference between the height of the virtual object and a target height corresponding to the target pose mode in response to determining that the height difference exceeds a threshold value.

[0114] In some embodiments, the extended reality device includes at least one of a handheld device and a head-mounted device held by the target object, and the information determining module 510 is further configured to determine at least one of a first scaling relationship between an arm length of the target object and an arm length of the virtual object, and a second scaling relationship between a height of the target object and a height of the virtual object; and adjust the position information and the orientation information corresponding to the extended reality device according to the determined at least one of the first scaling relationship and the second scaling relationship to determine the target orientation information and the target position information of the virtual object.

[0115] In some embodiments, the extended reality device is worn by the target object, and the information determination module 510 is further configured to determine a rotation offset value between an initial rotation value of the virtual object in the standard pose and an initial rotation value of the extended reality device when the target object is in the standard pose; and adjust the position information and the orientation information corresponding to the extended reality device based on the rotation offset value to determine the target orientation information and the target position information of the virtual object.

[0116] In some embodiments, the pre-rotation module 520 is further configured to determine a rotation amplitude of each of the plurality of torso skeletons based on the target orientation and the weight of each of the plurality of torso skeletons; and pre-rotate the plurality of torso skeletons by the determined rotation amplitudes.

[0117] In some embodiments, the pose determination module 540 is further configured to rotate the torso skeletons of the pre-rotated torso part based on the orientation of the predetermined part in the torso part; determine a target hip position of the virtual object based on a target head position of the virtual object; and rotate the torso part as a whole to the orientation indicated by the target orientation information based on the target hip position and the target head position.

[0118] In some embodiments, the pose determination module 540 includes a torso rotation module configured to determine a direction corresponding to a line connecting the target hip position and the target head position; determine a rotation degree of the torso part based on a difference between the target orientation of the head of the virtual object and the direction corresponding to the line; and rotate the torso part of the virtual object as a whole based at least on the rotation degree of the torso part to meet the orientation indicated by the target orientation information.

[0119] In some embodiments, the pose determination module 540 includes a target hip position determination module configured to, in response to determining that the target head position of the virtual object is lowered relative to the height of the virtual object, determine a lowering height of the hip from a current position based on a lowering height of the head; determine a moving distance of the hip in a two-dimensional plane based on a proportion of the lowering of the target head position relative to the height of the virtual object and the height of the virtual object, the two-dimensional plane being a plane defined by two directions other than a vertical direction corresponding to the lowering height in a three-dimensional space; and move the hip in the two-dimensional plane by the moving distance while lowering the hip of the virtual object from the current position along the vertical direction by the lowering height.

[0120] In some embodiments, the target hip position determination module is further configured to, in response to determining that the target head position of the virtual object is raised relative to the height of the virtual object, determine a raising height of the hip from a current position based on a raising height of the head; and raise the hip of the virtual object from the current position along the vertical direction by the determined raising height.

[0121] In some embodiments, the virtual object is composed of a plurality of bones, the target orientation information and the target position information are target orientation information and target position information of the virtual object in the current frame, and the apparatus 500 further includes a rotation angle module configured to, for a given bone of the plurality of bones, determine a rotation angle of the given bone based on the target orientation information and the target position information of the virtual object in the current frame; determine a first rotation angle and a second rotation angle corresponding to the given bone after adding 360 degrees and subtracting 360 degrees from the rotation angle; apply a rotation angle constraint corresponding to the given bone to the first rotation angle and the second rotation angle respectively to obtain a first corrected rotation angle and a second corrected rotation angle corresponding to the given bone; and select a corrected rotation angle closer to a target rotation angle of the given bone in a previous frame from the first corrected rotation angle and the second corrected rotation angle as the target rotation angle of the given bone in the current frame.

[0122] In some embodiments, the virtual object is composed of a plurality of bones, and the apparatus 500 further includes a reset module configured to, for a given bone of the plurality of bones, determine a continuous rotation angle range of the given bone in a period of time based on the target orientation information and the target position information of the virtual object; in response to determining that the continuous rotation angle range exceeds a predetermined angle range threshold, reset the given bone to an initial state along a shortest path.

[0123] In some embodiments, the apparatus 500 further includes a step execution module configured to determine target positions and target orientations of both feet of the virtual object based on the target orientation information and the target position information of the virtual object; determine that the virtual object is to perform a step based on the target positions and the target orientations of the both feet; and in response to determining that the virtual object is to perform a step from a current position and that the virtual object is in a small stride mode, determine a foot farther away between the current position and the target position of the both feet as a step foot and the other foot as a support foot to perform the step of the virtual object.

[0124] In some embodiments, the apparatus 500 further includes a landing point determination module configured to perform collision detection based on a current position of the support foot and a line connecting the current position and the target position of the step foot; and determine a landing point of the both feet of the virtual object after the step based on a result of the collision detection.

[0125] In some embodiments, the apparatus 500 further includes an execution module configured to perform at least one of the following: adjusting the positions of the two feet of the virtual object based on a step parameter related to the virtual object during a step of the virtual object, the step parameter comprising at least one of the following: a step curve, a step speed, a step height; updating the position and orientation of the stepping foot and updating the orientation of the supporting foot based on the received new target position information and target orientation information for the virtual object during the step of the virtual object; keeping the position of the supporting foot unchanged during the step of the virtual object.

[0126] In some embodiments, the apparatus 500 further includes a large-step execution module configured to, in response to determining that the virtual object is to step from a current position and the virtual object is in a large-step mode, perform the step of the virtual object according to a preset animation algorithm based on the target position and target orientation of the two feet.

[0127] In some embodiments, the apparatus 500 further includes a transition state step execution module configured to, in response to determining that the virtual object is to step from a current position and the virtual object is in an intermediate transition state between the small-step mode and the large-step mode, perform the step of the virtual object in the intermediate transition state by mixing a step determined for the virtual object in the small-step mode and a step determined for the virtual object in the large-step mode.

[0128] In some embodiments, the apparatus 500 further includes a sitting mode exit module configured to, in response to determining that a movement range of a head of the virtual object exceeds a movement range threshold or a rotation range of the head exceeds a rotation range threshold, cause the virtual object to exit the sitting mode when the virtual object is in the sitting mode.

[0129] The units and / or modules included in the apparatus 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, e.g., machine executable instructions stored on a storage medium. In addition to or alternatively, some or all of the units and / or modules in the apparatus 500 can be implemented at least partially by one or more hardware logic components. As an example and not by way of limitation, example types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0130] FIG. 6 illustrates a block diagram of an electronic device 600 in which one or more embodiments of the disclosure can be implemented. It should be understood that the electronic device 600 illustrated in FIG. 6 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 600 illustrated in FIG. 6 can be used to implement the extended reality device 120 of FIG. 1.

[0131] As shown in FIG. 6, the electronic device 600 is in the form of a general- purpose computing device. Components of the electronic device 600 can include, but are not limited to, one or more processors or processing units 610, a memory 620, a storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processing unit 610 can be a real or virtual processor and capable of executing various processing in accordance with programs stored in the memory 620. In a multi-processing system, multiple processing units execute computer-executable instructions in parallel to improve the processing power of the electronic device 600.

[0132] The electronic device 600 typically includes a plurality of computer storage media. Such media can be volatile and nonvolatile media, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and other data. The memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 can be a removable or non-removable media, and can include machine readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible by the electronic device 600.

[0133] The electronic device 600 can further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 6, a disk drive and / or a CD drive can be provided for reading from or writing to a removable, non-removable, volatile, or non-volatile media such as a floppy disk, a CD-ROM, and so on. In such instances, each drive can be connected to the bus (not shown) by one or more data media interfaces. The memory 620 can include a computer program product 625 having one or more program modules configured to carry out the various methods or actions of the various embodiments of the present disclosure.

[0134] The communication unit 640 enables communications with other computing devices over a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented in a single computing cluster or multiple computer machines that are capable of communicating over a communication connection. Thus, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0135] The input device 650 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 660 can be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 600 can also communicate with one or more external devices (not shown), such as a storage device, a display device, etc., through the communication unit 640, as necessary, with one or more devices that enable a user to interact with the electronic device 600, or with any device (e.g., a network card, a modem, etc.) that enables the electronic device 600 to communicate with one or more other computing devices. Such communication can be carried out via an input / output (I / O) interface (not shown).

[0136] According to an example implementation of the present disclosure, there is provided a computer-readable storage medium having computer-executable instructions stored thereon, where the computer-executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, there is also provided a computer program product tangibly stored on a non-transitory computer-readable medium and including computer-executable instructions, where the computer-executable instructions are executed by a processor to implement the method described above.

[0137] Various aspects of the disclosure are now described with reference to the drawings. In general, the drawings described below are diagrammatic and schematic representations of actual or conceptual structures and processes, and are not limiting of the scope of the present disclosure. In the drawings, the same reference numerals are used to represent similar or like items.

[0138] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable medium, thereby creating a computer program product, tangible computer-readable media having computer readable program instructions thereon for execution by the computer or other programmable data processing apparatus. The computer readable program instructions can be executed by the computer or other programmable data processing apparatus to cause the computer or other programmable data processing apparatus to carry out acts in the function / act specified in the flowchart and / or block diagram block or blocks.

[0139] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0140] The flow diagrams and the block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various implementations of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions (s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in some cases, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or combinations of hardware and software.

[0141] implementations. Numerous modifications and adaptations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The herein disclosed implementations are meant to be illustrative and not limiting, of the scope or spirit of the implementations. It will be apparent to those reasonably skilled in the art that varying substitutions and modifications can be made to the implementations disclosed here without departing from the scope and spirit of the implementations. And, other implementations that are apparent to those of ordinary skill in the art, are within the scope of the disclosed implementations.

Claims

1. A method of interactive control, comprising: determining, based on the position information and orientation information corresponding to the extended reality device, target orientation information and target position information of a virtual object in the virtual environment, the target orientation information at least indicating a target orientation of a head of the virtual object, and the target position information at least indicating a target position of at least one of a limb of the virtual object; performing a pre-rotation operation on a torso of the virtual object based on the target orientation of the head of the virtual object; determining an orientation of a predetermined portion of the torso of the virtual object based on a target position of at least one of the limbs of the virtual object; as well as The posture of the torso of the virtual object after the pre-rotation is determined based on at least the orientation of the predetermined portion of the torso.

2. The method of claim 1 , wherein the target position information further includes a height of the virtual object, and determining the target position information of the virtual object in the virtual environment comprises: determining a target posture mode of the virtual object, wherein the target posture mode includes a standing mode or a sitting mode; as well as Based on a preset offset value corresponding to the target posture pattern, the height indicated by the position information is adjusted to calculate the height of the virtual object in the virtual environment.

3. The method according to claim 2, further comprising: An offset value between a height of the virtual object when in a standard posture and a height of a target object holding the extended reality device when in the standard posture is determined as a preset offset value corresponding to the standing posture mode.

4. The method according to claim 2, further comprising: In response to determining that a height difference between the height of the virtual object and a target height corresponding to the target posture mode exceeds a threshold, the preset offset value corresponding to the target posture mode is updated based on the height difference.

5. The method of claim 1 , wherein the extended reality device comprises at least one of a handheld device and a head-mounted device held by the target object, and wherein determining target orientation information and target position information of the virtual object in the virtual environment comprises: Determining at least one of the following: a first scaling relationship between an arm length of the target object and an arm length of the virtual object, and a second scaling relationship between a height of the target object and a height of the virtual object; as well as According to the determined at least one of the first scaling relationship and the second scaling relationship, the position information and orientation information corresponding to the extended reality device are adjusted to determine the target orientation information and target position information of the virtual object.

6. The method of claim 1 , wherein the extended reality device is worn by a target object, and wherein determining target orientation information and target position information of the virtual object in the virtual environment comprises: Determining a rotation offset value between an initial rotation value of the virtual object when in a standard posture and an initial rotation value of the extended reality device when the target object is in a standard posture; as well as The position information and orientation information corresponding to the extended reality device are adjusted based on the rotation offset value to determine the target orientation information and target position information of the virtual object.

7. The method according to claim 1 , wherein performing a pre-rotation operation on a torso portion of the virtual object based on the target orientation of the head of the virtual object comprises: Determining a rotation magnitude of each of the plurality of torso bones based on the target orientation and respective weights of the plurality of torso bones in the torso portion; as well as Pre-rotate the multiple torso bones according to the determined rotation amplitude.

8. The method of claim 1 , wherein determining the pose of the torso portion of the virtual object after pre-rotation comprises: rotating the pre-rotated trunk skeleton of the trunk portion based on the orientation of the predetermined portion of the trunk portion; determining a hip target position of the virtual object based on the head target position of the virtual object; as well as The torso is rotated as a whole toward the orientation indicated by the target orientation information based on the hip target position and the head target position.

9. The method according to claim 8, wherein rotating the torso as a whole toward the orientation indicated by the target orientation information based on the hip target position and the head target position comprises: determining a direction corresponding to a line connecting the hip target position and the head target position; determining a degree of rotation of the torso based on a difference between the target orientation of the head of the virtual object and a direction corresponding to the connecting line; as well as The torso portion of the virtual object is rotated as a whole based on at least the degree of rotation of the torso portion to satisfy the orientation indicated by the target orientation information.

10. The method of claim 8, wherein determining a hip target position of the virtual object based on a head target position of the virtual object comprises: In response to determining that the target position of the head of the virtual object is lowered relative to the height of the virtual object, determining a downward displacement height of the hips from a current position based on the downward displacement height of the head; determining, based on a ratio of the head target position to the height reduction of the virtual object and the height of the virtual object, a movement distance of the hips in a two-dimensional plane, the two-dimensional plane being a plane defined by two directions in a three-dimensional space except a vertical direction corresponding to the reduced height; and The buttocks of the virtual object are moved by the movement distance in the two-dimensional plane while lowering the buttocks of the virtual object from the current position in the vertical direction by the downward movement height.

11. The method according to claim 10, wherein determining a target position of a hip of the virtual object based on a target position of a head of the virtual object further comprises: in response to determining that the target position of the head of the virtual object is elevated relative to the height of the virtual object, determining a height at which the hips are elevated from a current position based on the elevated height of the head; as well as The buttocks of the virtual object are raised from a current position by the raised height in a vertical direction.

12. The method according to claim 1, wherein the virtual object is composed of a plurality of skeletons, and the method further comprises: For a given bone in the plurality of bones, determining a continuous rotation angle range of the given skeleton within a period of time based on the target orientation information and target position information of the virtual object; In response to determining that the continuous rotation angle range exceeds a predetermined angle range threshold, the given bone is reset to an initial state along the shortest path.

13. The method according to claim 1, further comprising: determining target positions and target orientations of both feet of the virtual object based on the target orientation information and target position information of the virtual object; Determining, based on the target positions and target orientations of the two feet, that the virtual object is about to take a step; as well as In response to determining that the virtual object is about to take a step from the current position and the virtual object is in a small stride mode, the foot of the two feet with a greater distance between the current position and the target position is determined as the stepping foot, and the other foot is used as the supporting foot to perform the stepping of the virtual object.

14. The method according to claim 13, further comprising: In response to determining that the virtual object is about to take a step from the current position and the virtual object is in a stride mode, the stepping of the virtual object is executed according to a preset animation algorithm based on the target positions and target orientations of the two feet.

15. The method according to claim 14, further comprising: In response to determining that the virtual object is to take a step from the current position and the virtual object is in an intermediate transition state between the small stride mode and the large stride mode, the step of the virtual object in the intermediate transition state is performed by blending the step determined for the virtual object in the small stride mode and the step determined for the virtual object in the large stride mode.

16. The method according to claim 1, further comprising: When the virtual object is in a sitting mode, in response to determining that a movement range of the head of the virtual object exceeds a movement range threshold or a rotation range of the head exceeds a rotation range threshold, the virtual object is caused to exit the sitting mode.

17. A device for interactive control, comprising: an information determination module configured to determine target orientation information and target position information of a virtual object in a virtual environment based on position information and orientation information corresponding to the extended reality device, wherein the target orientation information at least indicates a target orientation of a head of the virtual object, and the target position information at least indicates a target position of at least one of a limb of the virtual object; a pre-rotation module configured to perform a pre-rotation operation on a torso of the virtual object based on the target orientation of the head of the virtual object; an orientation determination module configured to determine an orientation of a predetermined portion of the torso of the virtual object based on a target position of at least one of the limbs of the virtual object; as well as The posture determination module is configured to determine the posture of the torso of the virtual object after pre-rotation based on at least the orientation of the predetermined portion of the torso.

18. An electronic device comprising: at least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 16 when executed by the at least one processing unit.

19. A computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the method according to any one of claims 1 to 16.

20. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 16.

Images