Interaction control method and apparatus, electronic device, and storage medium

Abstract

Provided are an interaction control method and apparatus, an electronic device, and a storage medium. The interaction control method comprises: at the position of an electronic device in communication with a display generation component and one or more input devices: the display generation component generating a three-dimensional environment generated by a computer; displaying a target virtual object in the three-dimensional environment; detecting a user interaction of a user by the one or more input devices; when the position relationship between an interactor corresponding to the user interaction and the target virtual object is a first position relationship, displaying a ray indicator from the interactor to the target virtual object; and when the position relationship between the interactor corresponding to the user interaction and the target virtual object is a second position relationship, adjusting the ray indicator, such that the projection direction of the ray indicator approximates a near-field interaction poke direction for the target virtual object.

Description

Interactive control method, device, electronic device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on October 31, 2023, with application number 202311426609.0 and invention name “Interactive control method, device, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of computer technology, and in particular to an interactive control method, device, electronic device, and storage medium. Background Art

[0003] Extended Reality (XR) technology combines the real and virtual through computers, providing users with a virtual reality space where humans and machines can interact. In XR-based application scenarios, far-field interaction methods are generally different from near-field interaction methods. Far-field interaction is usually based on indirect interaction with virtual objects using rays, while near-field interaction is usually based on user gestures or finger touches (Poke) of virtual objects.

[0004] Typically, these two interaction methods switch based on the user's operating distance, and are often incompatible. When the user is in near-field interaction, although the user can interact with virtual objects through finger touch, the ray-guided interaction function is unavailable, resulting in reduced operational efficiency in some scenarios.

[0005] Summary of the Invention

[0006] This summary is provided to briefly introduce concepts that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0007] In a first aspect, according to one or more embodiments of the present disclosure, an interactive control method is provided, comprising:

[0008] At an electronic device in communication with a display generation component and one or more input devices:

[0009] displaying a computer-generated three-dimensional environment by the display generation component;

[0010] displaying a target virtual object in the three-dimensional environment;

[0011] detecting, by the one or more input devices, a user interaction by a user;

[0012] When the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a first positional relationship, displaying a ray indication from the interactor to the target virtual object;

[0013] When the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship, the ray indication is adjusted so that the projection direction of the ray indication approaches the near-field interactive touch direction for the target virtual object.

[0014] In a second aspect, according to one or more embodiments of the present disclosure, an interactive control device is provided, comprising:

[0015] An environment display unit, configured to display a three-dimensional environment generated by a computer by a display generation component;

[0016] An object display unit, configured to display a target virtual object in the three-dimensional environment;

[0017] an interaction detection unit, configured to detect user interaction of a user via one or more input devices;

[0018] An interaction control unit is used to display a ray indication from the interactor to the target virtual object when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a first positional relationship, and to adjust the ray indication when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship so that the projection direction of the ray indication approaches the near-field interaction touch direction for the target virtual object.

[0019] In a third aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one memory and at least one processor; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory so that the electronic device executes the interactive control method provided according to one or more embodiments of the present disclosure.

[0020] In a fourth aspect, according to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, wherein the non-transitory computer storage medium stores a program code, and when the program code is executed by a computer device, the computer device executes the interactive control method provided according to one or more embodiments of the present disclosure.

[0021] According to one or more embodiments of the present disclosure, the ray indication is adjusted when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship, so that the projection direction of the ray indication approaches the near-field interaction touch direction for the target virtual object, thereby making the ray interaction mode and the touch interaction mode of the interactor non-conflicting in pointing to close-range objects, and thus making the near-field interaction of the interactor compatible with both ray interaction and touch interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0023] 1A-1D are interactive schematic diagrams provided according to related technologies;

[0024] FIG2 is a schematic diagram of an extended reality device according to an embodiment of the present disclosure;

[0025] FIG3 is an optional schematic diagram of a virtual field of view of an extended reality device according to an embodiment of the present disclosure;

[0026] FIG4 is a flow chart of an interactive control method according to an embodiment of the present disclosure;

[0027] 5A-5B are interactive diagrams provided according to an embodiment of the present disclosure;

[0028] FIG6 is a schematic diagram of interaction provided according to an embodiment of the present disclosure;

[0029] FIG7 is an interaction diagram provided according to another embodiment of the present disclosure;

[0030] FIG8 is a schematic structural diagram of an interactive control device according to an embodiment of the present disclosure;

[0031] FIG9 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0033] It should be understood that the steps described in the embodiments of the present disclosure can be performed in a different order and / or in parallel. In addition, the embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0034] As used herein, the term "including" and its variations are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The term "responsive to" and related terms refer to a signal or event being affected to a certain extent by another signal or event, but not necessarily completely or directly. If event x occurs "responsive to" event y, then x may be directly or indirectly responsive to y. For example, the occurrence of y may ultimately lead to the occurrence of x, but there may be other intermediate events and / or conditions. In other cases, y may not necessarily lead to the occurrence of x, and x may occur even if y has not yet occurred. In addition, the term "responsive to" may also mean "at least partially responsive to".

[0035] The term "determine" broadly encompasses a variety of actions, and may include obtaining, calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), ascertaining, and similar actions, and may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and similar actions, as well as parsing, selecting, choosing, establishing, and similar actions. Other terms are defined below. Other terms are defined below.

[0036] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0037] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0038] For the purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B).

[0039] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0040] It should be noted that the steps for obtaining a user's personal data mentioned in this disclosure are performed with the user's authorization. For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, or storage medium, such as software or hardware, that performs the operations of the technical solution of this disclosure based on the prompt message. As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt message can be sent to the user in the form of a pop-up window, in which the prompt message can be presented in text form. In addition, the pop-up window can also contain a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure. It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws and regulations and relevant provisions.

[0041] The interactive control method provided by one or more embodiments of the present disclosure utilizes Extended Reality (XR) technology. XR technology can combine the real and virtual through computers to provide users with a virtual reality space capable of human-computer interaction. In this virtual reality space, users can use XR devices such as head-mounted displays (HMDs) to engage in social interaction, entertainment, learning, work, telecommuting, and user-generated content (UGC).

[0042] Referring to Figure 2, users can enter the virtual reality space through an extended reality device such as a head-mounted display, and control their own virtual characters (Avatars) in the virtual reality space to engage in social interaction, entertainment, learning, remote work, etc. with virtual characters controlled by other users.

[0043] In one embodiment, in a virtual reality space, a user can perform relevant interactive operations using a controller, such as a handheld gamepad. For example, the user can operate buttons on the handheld gamepad to perform relevant operations. Of course, in other embodiments, the controllable objects in the augmented reality device can be controlled using gestures, voice, or multimodal control methods instead of a controller.

[0044] The extended reality devices described in the embodiments of the present disclosure may include, but are not limited to, the following types:

[0045] Computer-based extended reality (PCVR) devices use the PC to perform calculations and output data related to extended reality functions. External computer-based extended reality devices use the data output by the PC to achieve the effect of extended reality.

[0046] Mobile extended reality devices support the configuration of mobile terminals (such as smartphones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations for the extended reality function and outputs data to the mobile extended reality device, such as watching extended reality videos through the mobile terminal's APP.

[0047] The all-in-one extended reality device has a processor for performing related calculations for virtual functions, and thus has independent extended reality input and output functions. It does not need to be connected to a PC or mobile terminal and has a high degree of freedom of use.

[0048] Of course, the form of the extended reality device is not limited to this, and it can be further miniaturized or enlarged as needed.

[0049] The extended reality device is equipped with a posture detection sensor (such as a nine-axis sensor) to detect the posture changes of the extended reality device in real time. If the user wears the extended reality device, then when the user's head posture changes, the real-time posture of the head will be transmitted to the processor to calculate the user's gaze point in the virtual environment. Based on the gaze point, the image within the user's gaze range (i.e., virtual field of view) in the three-dimensional model of the virtual environment is calculated and displayed on the display screen, giving people an immersive experience as if they were watching in the real environment.

[0050] FIG3 shows an optional schematic diagram of the virtual field of view of an extended reality device provided by an embodiment of the present disclosure. The horizontal field of view angle and the vertical field of view angle are used to describe the distribution range of the virtual field of view in the virtual environment. The vertical distribution range is represented by the vertical field of view angle BOC, and the horizontal distribution range is represented by the horizontal field of view angle AOB. The human eye can always perceive the image located in the virtual field of view in the virtual environment through the lens. It can be understood that the larger the field of view angle, the larger the size of the virtual field of view, and the larger the area of ​​the virtual environment that the user can perceive. Among them, the field of view angle represents the distribution range of the viewing angle when the environment is perceived through the lens. For example, the field of view angle of the extended reality device represents the distribution range of the viewing angle of the human eye when perceiving the virtual environment through the lens of the extended reality device; for another example, for a mobile terminal equipped with a camera, the field of view angle of the camera is the distribution range of the viewing angle when the camera perceives the real environment for shooting.

[0051] Extended reality devices, such as HMDs, are integrated with several cameras (e.g., depth cameras, RGB cameras, etc.), and the purpose of the cameras is not limited to providing a direct view. The camera images and the integrated inertial measurement unit (IMU) provide data that can be processed by computer vision methods to automatically analyze and understand the environment. In addition, the HMD is designed to support not only passive computer vision analysis, but also active computer vision analysis. Passive computer vision methods analyze image information captured from the environment. These methods can be monoscopic (images from a single camera) or stereoscopic (images from two cameras). They include but are not limited to feature tracking, object recognition, and depth estimation. Active computer vision methods add information to the environment by projecting patterns that are visible to the camera but not necessarily visible to the human visual system. Such technologies include time-of-flight (ToF) cameras, laser scanning, or structured light to simplify stereo matching problems. Active computer vision is used to achieve scene depth reconstruction.

[0052] In extended reality applications, far-field interaction methods differ from near-field interaction methods. Far-field interaction typically involves indirect interaction with virtual objects using rays, while near-field interaction typically involves user gestures or finger touches (Poke) with virtual objects.

[0053] Taking gesture interaction as an example, see Figure 1A. In far-field interaction mode, rays are typically extended in a direction with the user's palm or wrist as the origin, and intersect with virtual objects to enable the user to interact with them (for example, pick up the virtual object). See Figure 1B. In near-field interaction mode, a collision body (not shown) can be bound to the user's finger, which is used to detect virtual objects that collide with it. This allows the user to interact with the virtual object by colliding (for example, touching or pressing) the virtual object with their finger (i.e., the collision body).

[0054] Taking a controller (e.g., a gamepad) as an example, see Figure 1C . In far-field interaction mode, rays are typically extended in a direction based on the front anchor point of the controller and intersect with virtual objects to interact with them. See Figure 1D . In near-field interaction mode, a collision body 10 can be associated with the handheld controller, so that interaction with the virtual object can be achieved by touching the collision body with the virtual object.

[0055] However, the aforementioned far-field and near-field interaction modes are typically switched based on the user's operating distance, and the two are often incompatible. When the user is in near-field interaction, although they can interact through touch mode, the ray interaction function is unavailable, resulting in reduced operational efficiency in some application scenarios.

[0056] Referring to Figure 4, Figure 4 shows a flowchart of an interactive control method 100 provided by an embodiment of the present disclosure. In some embodiments, method 100 is performed at an electronic device (such as the head-mounted display shown in Figure 2 or the electronic device shown in Figure 9), which can communicate with a display generation component (such as a display screen) and one or more input devices (such as an eye tracking device, a hand tracking device, a camera, or other input device). In some embodiments, the display generation component can be integrated into the electronic device; the input device can be integrated into or external to the electronic device, but the present disclosure is not limited to this. In some embodiments, the input device can be a handheld controller. In other embodiments, the input device can be a camera integrated into the head-mounted display.

[0057] The method 100 includes steps S110 to S150 .

[0058] Step S110: The display generation component displays the computer-generated three-dimensional environment.

[0059] Step S120: Displaying the target virtual object in the three-dimensional environment.

[0060] In some embodiments, a three-dimensional environment (e.g., a virtual reality space) can be a simulation of the real world, or a semi-simulated and semi-fictitious virtual scene, or a purely fictitious virtual scene, and the present disclosure does not limit this. The virtual scene can be any one of a two-dimensional virtual scene, a 2.5-dimensional virtual scene, or a three-dimensional virtual scene. The embodiments of the present application do not limit the dimensions of the virtual scene. For example, a virtual scene can include the sky, land, ocean, etc., and the land can include environmental elements such as deserts and cities, and users can control virtual objects to move in the virtual scene.

[0061] In some embodiments, the target virtual object is the object with which the user intends to interact. In some embodiments, the target virtual object can be determined as the object the user is looking at. For example, a cone projection can be performed based on tracking the user's eye movements and posture, and the target virtual object displayed in the virtual reality space can be determined based on the cone projection. In addition, the target virtual object can also be determined using other related eye tracking technologies, which are not limited by this disclosure. In some embodiments, the target virtual object can also be determined based on voice control or other methods.

[0062] Step S130: Detecting user interaction of a user by one or more input devices.

[0063] In some embodiments, the input device can detect user interaction based on a motion sensing detection method or a computer vision-based detection method. For example, the position of a certain body part of the user can be detected based on a camera (such as a depth camera) through a motion tracking algorithm based on computer vision, or the position of the body part can be detected by holding or wearing a tracking device (such as a handheld controller) on the body part (such as six-degree-of-freedom data), but the present disclosure is not limited to this. Among them, the six degrees of freedom include the degrees of freedom of movement in the directions of the three rectangular coordinate axes of x, y, and z and the degrees of freedom of rotation around these three coordinate axes, namely, front and back, up and down, left and right, pitch, yaw, and roll, a total of 6 degrees of freedom.

[0064] In some embodiments, the HMD is integrated with a hand tracking device, through which the user's hand information, such as the user's gestures, can be obtained. The hand tracking device is part of the HMD (e.g., embedded in or attached to the head-mounted device).

[0065] In some embodiments, the hand tracking device includes an image sensor (e.g., one or more infrared cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information including at least a human user's hand. The image sensor captures hand images with sufficient resolution to enable fingers and their corresponding positions to be distinguished.

[0066] In some embodiments, a hand tracking device captures and processes a time series of depth maps containing the user's hand as the user moves their hand (e.g., the entire hand or one or more fingers). Software running on the image sensor and the HMD's processor processes the 3D mapping data to extract image patch descriptors of the hand from these depth maps. The software can match these descriptors with image patch descriptors stored in a database based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D position of the user's hand joints and fingertips. The software can also analyze the trajectory of the hand and / or fingers across multiple frames in the sequence to recognize gestures. The pose estimation function described herein can be interleaved with the motion tracking function, so that the image patch-based pose estimation is only performed once every two (or more) frames, and tracking is used to find changes in pose that occur in the remaining frames and provide pose, motion, and gesture information to an application running on the HMD. The application can, for example, move and modify the image presented on the HMD display generation component in response to the pose and / or gesture information, or perform other functions, such as executing control instructions corresponding to gestures.

[0067] In some embodiments, the HMD is integrated with a gaze tracking device that can be used to obtain visual information about the user, such as the user's line of sight, gaze point, etc. In one embodiment, the gaze tracking device includes at least one eye-tracking camera (e.g., an infrared (IR) or near-infrared (NIR) camera) and an illumination source (e.g., an IR or NIR light source, such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye-tracking camera can be pointed at the user's eyes to receive IR or NIR light reflected directly from the eyes by the light source, or alternatively, can be pointed at "hot" mirrors located between the user's eyes and the display panel. These hot mirrors reflect IR or NIR light from the eyes toward the eye-tracking camera while allowing visible light to pass through. The gaze tracking device optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes these images to generate gaze tracking information, and transmits the gaze tracking information to the HMD, thereby enabling certain human-computer interaction functions based on the user's gaze information, such as enabling gaze-based content navigation. In some embodiments, the user's two eyes are tracked separately by corresponding eye-tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by a corresponding eye tracking camera and illumination source.

[0068] Step S140: When the positional relationship between the interactor and the target virtual object corresponding to the user interaction is a first positional relationship, a ray indication from the interactor to the target virtual object is displayed.

[0069] Step S150: When the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship, the ray indication is adjusted so that the projection direction of the ray indication approaches the near-field interactive touch direction for the target virtual object.

[0070] The interactor includes interactive control devices (such as handheld controllers) or body parts (such as hands, head, feet, eyes, etc.) used by the user to interact in the three-dimensional environment, or visual representations of these control devices or body parts in the three-dimensional environment, such as virtual models displayed in the three-dimensional environment to reflect the actual posture of the controller or hand.

[0071] For example, in some virtual reality (VR) scenarios, the position of a corresponding virtual model (e.g., a controller model, a hand model, or other animation model) in the virtual reality space can be determined based on the position of an interactive control device or a user's body part in the real scene, and the positional relationship between the virtual model and a target virtual object can be determined. In some augmented reality (AR) scenarios that provide a visual display effect of superimposing virtual objects on real objects, the positional relationship between the real position of the control device or body part and the corresponding real position of the target virtual object in the real space can be determined, but the present disclosure is not limited to this.

[0072] In one specific embodiment, if the distance between the interactor and the target virtual object exceeds a preset threshold, the positional relationship between the interactor and the target virtual object is a first positional relationship; if the distance between the interactor and the target virtual object does not exceed the preset threshold, the positional relationship between the interactor and the target virtual object is a second positional relationship. For example, the preset threshold may be 30 centimeters, or another distance value.

[0073] In one specific embodiment, if the interactor is located within a preset area corresponding to the target virtual object, the positional relationship between the interactor and the target virtual object is the second positional relationship, and otherwise, the positional relationship is the first positional relationship. For example, the area near the target virtual object can be used as the preset area (i.e., the near-field area), and within the near-field area, a near-field interaction method (such as the touch interaction shown in Figures 1B and 1D) can be performed.

[0074] In some embodiments, ray pointers are used to detect virtual objects that collide with the path, allowing the user to select the virtual object using the ray pointer. In ray interaction mode, the direction of the ray pointer (i.e., the projection direction) will be changed based on the position of the interactive controller or hand in real space. In this way, the user can interact with virtual objects in the three-dimensional environment by manipulating the controller or hand to use the ray pointer.

[0075] In some embodiments, when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is the second positional relationship, the projection direction of the ray indication can be made equal to or gradually approach the near-field interactive touch direction for the target virtual object.

[0076] In some embodiments, the direction of movement of the interactor can be determined as the direction of the near-field interactive touch with respect to the target virtual object. For example, the direction of movement of the interactor can be determined based on motion data detected by a motion sensor (e.g., an accelerometer) built into the interactive control device, or the direction of movement of the interactor (e.g., a hand) can be determined using a motion tracking algorithm based on computer vision, but the present disclosure is not limited thereto.

[0077] In some embodiments, the normal direction of the side of the target virtual object facing the interactor (i.e., the contact surface) can be used as the near-field interactive touch direction of the target virtual object. When the user intends to touch an object, the user will eventually apply at least one force perpendicular to the contact surface to the contact surface. Accordingly, as the user's finger approaches the contact surface, the near-field interactive touch direction approaches the normal direction of the contact surface. Therefore, in this embodiment, by directly using the normal direction of the contact surface as the near-field interactive touch direction, the calculation of the near-field interactive touch direction of the interactor for the target virtual object during movement can be simplified.

[0078] In this way, according to one or more embodiments of the present disclosure, the ray indication is adjusted when the position relationship between the interactor corresponding to the user interaction and the target virtual object is the second position relationship, so that the projection direction of the ray indication is close to the near-field interaction touch direction of the target virtual object, so that the ray interaction mode and the touch interaction mode of the interactor do not conflict with the direction of the close-range object, and thus the near-field interaction of the interactor can be compatible with ray interaction and touch interaction at the same time. For example, the user can use touch interaction to control close-range objects, and can also use rays originally suitable for far-field interaction to control close-range objects.

[0079] For example, when a user uses gestures for near-field interaction, they can use either the touch interaction mode shown in Figure 1B or the ray interaction mode shown in Figure 5A. Similarly, when a user uses a controller for near-field interaction, they can use either the touch interaction mode shown in Figure 1D or the ray interaction mode shown in Figure 5B. In the ray interaction mode, the projection direction of the ray matches the user's touch direction, so no matter which interaction mode the user ultimately uses, they can ultimately point to the same virtual object.

[0080] In some embodiments, the positional relationship between a collision body associated with an interactor and a target virtual object may be used as the positional relationship between the interactor and the target virtual object. The collision body is used to detect objects that collide with it in a touch interaction mode, so that the user can interact with the virtual object by touch.

[0081] For example, referring to FIG6 , a virtual object 61 (e.g., a UI control) that the user is looking at can be determined and used as the target virtual object. When a collision body 621 associated with an interactor 62 (e.g., a handle) is located within the near-field region 610 of the virtual object 61, the projection direction indicated by the ray is shifted from the projection direction 63 in the far-field interaction mode to the near-field interaction touch direction 64 of the target virtual object 61. The near-field interaction touch direction 64 is the normal direction of the contact surface 611 of the virtual object 61. As the interactor 62 gradually approaches the target virtual object 61, the angle α between the projection direction 63 and the near-field interaction touch direction 64 can be gradually reduced.

[0082] In some embodiments, the position of the controller may be determined based on multiple sensors built into the controller (eg, an inertial sensor, an accelerometer, a gyroscope, or a time-of-flight sensor, etc.), but the present disclosure is not limited thereto.

[0083] In some embodiments, if the interactor is a hand, the emission starting point corresponding to the ray indication may be determined based on the fingertips of the hand when the positional relationship between the interactor and the target virtual object is the second positional relationship.

[0084] For example, referring to FIG7 , the virtual object 71 (e.g., a UI control) that the user is looking at can be determined and determined as the target virtual object. When the collision body (not shown in the figure) associated with the interactor 72 (e.g., a hand) is located within the near-field area 710 of the virtual object 71, the projection direction indicated by the ray is changed from the projection direction 73 under the far-field interaction mode to the near-field interaction touch direction 74 of the target virtual object 71. The near-field interaction touch direction 74 is the normal direction of the contact surface 711 of the virtual object 71. As the interactor 72 gradually approaches the target virtual object 71, the angle β between the projection direction 73 and the near-field interaction touch direction 74 can be gradually reduced. Furthermore, the starting point of the ray emission can be changed from being determined based on the position of the wrist 721 (e.g., located at the wrist) to being determined based on the fingertip 722 (e.g., located at the fingertip), so as to achieve further convergence of the two interaction modes of finger ray interaction and finger touch interaction in terms of the direction of close-range objects.

[0085] In one specific embodiment, the side of the target virtual object facing the interactor can be determined based on the user's gaze. For example, the surface of the target virtual object that the user is gazing at can be used as the "contact surface" in the aforementioned embodiment, and the normal direction of the contact surface, i.e., the near-field interactive touch direction, can be determined.

[0086] In some embodiments, when the positional relationship between the interactor and the target virtual object is the second positional relationship, a first visual cue is displayed at a preset position of the interactor, where the first visual indication is used to indicate the direction of the near-field interactive touch. In this embodiment, by displaying the first visual cue to indicate the direction of the near-field interactive touch, the user's touch control accuracy in the virtual reality space can be improved.

[0087] In some embodiments, when the positional relationship between the interactor and the target virtual object is the second positional relationship, the ray indicator is retained. In this embodiment, by retaining the display of the ray indicator during near-field interaction, it is convenient for the user to control a close-range object using the ray interaction method originally suitable for far-field interaction.

[0088] Accordingly, referring to FIG8 , an information interaction control device 600 is provided according to an embodiment of the present disclosure, including:

[0089] The environment display unit 601 is used to display a three-dimensional environment generated by a computer by a display generation component;

[0090] An object display unit 602 is configured to display a target virtual object in the three-dimensional environment;

[0091] An interaction detection unit 603 is configured to detect user interactions of a user via one or more input devices;

[0092] The interaction control unit 604 is used to display a ray indication from the interactor to the target virtual object when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a first positional relationship, and to adjust the ray indication when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship so that the projection direction of the ray indication approaches the near-field interaction touch direction for the target virtual object.

[0093] In some embodiments, the near-field interactive touch direction for the target virtual object is a normal direction of a side of the target virtual object facing the interactor or a movement direction of the interactor.

[0094] In some embodiments, the interactor comprises an interactive control device or a body part of the user.

[0095] In some embodiments, the apparatus further comprises:

[0096] A ray starting point determination unit is used to determine the emission starting point corresponding to the ray indication based on the fingertip of the hand if the interactor includes a hand and when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship.

[0097] In some embodiments, the apparatus further comprises:

[0098] The first display unit is used to display a first visual prompt at a preset position of the interactor when the position relationship between the interactor corresponding to the user interaction and the target virtual object is a second position relationship, and the first visual indication is used to indicate the near-field interaction touch direction.

[0099] In some embodiments, the apparatus further comprises:

[0100] The second display unit is configured to keep displaying the ray indication when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship.

[0101] In some embodiments, when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is the second positional relationship, the closer the distance between the interactor and the target virtual object is, the closer the far-field interaction projection direction is to the near-field interaction touch direction.

[0102] In some embodiments, the target object is the object that the user is gazing at.

[0103] In some embodiments, the apparatus further comprises:

[0104] A determining unit is configured to determine a side of the target virtual object facing the interactor based on a user's gaze.

[0105] In some embodiments, the second positional relationship includes that the distance between the interactor and the target virtual object does not exceed a preset threshold.

[0106] For the embodiments of the device, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, and the modules described as separation modules may or may not be separate. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0107] Accordingly, according to one or more embodiments of the present disclosure, there is provided an electronic device, including:

[0108] at least one memory and at least one processor;

[0109] The memory is used to store program codes, and the processor is used to call the program codes stored in the memory to enable the electronic device to execute the interactive control method provided according to one or more embodiments of the present disclosure.

[0110] Accordingly, according to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, which stores program code, and the program code can be executed by a computer device to enable the computer device to perform the interactive control method provided according to one or more embodiments of the present disclosure.

[0111] Reference is now made to FIG9 , which illustrates a schematic diagram of the structure of an electronic device (e.g., a terminal device or server) 800 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device illustrated in FIG9 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0112] As shown in Figure 9, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0113] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although FIG9 shows the electronic device 800 with various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0114] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0115] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0116] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0117] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0118] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method of the present disclosure.

[0119] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0122] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0123] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0124] According to one or more embodiments of the present disclosure, an interaction control method is provided, comprising: at an electronic device that communicates with a display generation component and one or more input devices: displaying a computer-generated three-dimensional environment by the display generation component; displaying a target virtual object in the three-dimensional environment; detecting user interaction by the one or more input devices; when the positional relationship between an interactor corresponding to the user interaction and the target virtual object is a first positional relationship, displaying a ray indication from the interactor to the target virtual object; and when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship, adjusting the ray indication so that the projection direction of the ray indication approaches the near-field interaction touch direction for the target virtual object.

[0125] According to one or more embodiments of the present disclosure, the near-field interactive touch direction for the target virtual object is a normal direction of a side of the target virtual object facing the interactor, or a movement direction of the interactor.

[0126] According to one or more embodiments of the present disclosure, the interactor includes an interactive control device or a body part of the user.

[0127] According to one or more embodiments of the present disclosure, if the interactor includes a hand, the method further includes: when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship, determining the emission starting point corresponding to the ray indication based on the fingertips of the hand.

[0128] The method provided according to one or more embodiments of the present disclosure further includes: when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship, displaying a first visual prompt at a preset position of the interactor, wherein the first visual indication is used to indicate the near-field interaction touch direction.

[0129] According to one or more embodiments of the present disclosure, the method further includes: when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship, retaining the visual representation of the far-field interaction projection direction.

[0130] According to one or more embodiments of the present disclosure, when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is the second positional relationship, the closer the distance between the interactor and the target virtual object is, the closer the far-field interaction projection direction is to the near-field interaction touch direction.

[0131] According to one or more embodiments of the present disclosure, the target object is an object that the user is gazing at.

[0132] According to one or more embodiments of the present disclosure, determining the target virtual object displayed in the virtual reality space includes: determining a side of the target virtual object facing the interactor based on a user's gaze.

[0133] According to one or more embodiments of the present disclosure, the second positional relationship includes that the distance between the interactor and the target virtual object does not exceed a preset threshold.

[0134] According to one or more embodiments of the present disclosure, an interactive control device is provided, comprising: an environment display unit, configured to display a three-dimensional environment generated by a computer by a display generation component; an object display unit, configured to display a target virtual object in the three-dimensional environment; an interaction detection unit, configured to detect user interaction by one or more input devices; and an interaction control unit, configured to display a ray indication from the interactor to the target virtual object when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a first positional relationship, and to adjust the ray indication when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship, so that the projection direction of the ray indication approaches the near-field interactive touch direction for the target virtual object.

[0135] According to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one memory and at least one processor; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory so that the electronic device executes the interactive control method provided according to one or more embodiments of the present disclosure.

[0136] According to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, wherein the non-transitory computer storage medium stores program code, and when the program code is executed by a computer device, the computer device executes the interactive control method provided according to one or more embodiments of the present disclosure.

[0137] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0138] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0139] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. An interactive control method, comprising: At an electronic device in communication with a display generating component and one or more input devices: displaying a computer-generated three-dimensional environment by the display generation component; Displaying a target virtual object in the three-dimensional environment; detecting, by the one or more input devices, a user interaction by a user; When the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a first positional relationship, displaying a ray indication from the interactor to the target virtual object; When the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship, the ray indication is adjusted so that the projection direction of the ray indication approaches the near-field interactive touch direction for the target virtual object.

2. The method according to claim 1, wherein the near-field interactive touch direction for the target virtual object is a normal direction of a side of the target virtual object facing the interactor or a movement direction of the interactor. The method according to claim 1 , wherein the interactor comprises an interactive control device or a body part of a user.

4. The method according to claim 1, wherein if the interactive device comprises a hand, the method further comprises: When the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship, the emission starting point corresponding to the ray indication is determined based on the fingertips of the hand.

5. The method according to claim 1, further comprising: When the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship, a first visual prompt is displayed at a preset position of the interactor, and the first visual indication is used to indicate the near-field interaction touch direction.

6. The method according to claim 1, further comprising: When the position relationship between the interactor corresponding to the user interaction and the target virtual object is the second position relationship, The ray indication remains displayed.

7. The method according to claim 1, wherein when the positional relationship between the interactor corresponding to the user interaction and the target virtual object is a second positional relationship, the closer the distance between the interactor and the target virtual object is, the closer the projection direction is to the near-field interaction touch direction. The method according to claim 1 , wherein the target virtual object is an object that the user is gazing at.

9. The method according to claim 2, further comprising: A side of the target virtual object facing the interactor is determined based on the gaze of the user. 10 . The method according to claim 1 , wherein the second positional relationship includes that a distance between the interactor and the target virtual object does not exceed a preset threshold.

11. An interactive control device, comprising: An environment display unit, configured to display a three-dimensional environment generated by a computer by a display generation component; An object display unit, used for displaying a target virtual object in the three-dimensional environment; an interaction detection unit, configured to detect user interaction of a user via one or more input devices; An interaction control unit is used to display a ray indication from the interactor to the target virtual object when the position relationship between the interactor corresponding to the user interaction and the target virtual object is a first position relationship, and to adjust the ray indication when the position relationship between the interactor corresponding to the user interaction and the target virtual object is a second position relationship so that the projection direction of the ray indication approaches the near-field interaction touch direction for the target virtual object.

12. An electronic device comprising: at least one memory and at least one processor; The memory is used to store program codes, and the processor is used to call the program codes stored in the memory to enable the electronic device to execute the method according to any one of claims 1 to 10.

13. A non-transitory computer storage medium, wherein: The non-transitory computer storage medium stores a program code, and when the program code is executed by a computer device, the computer device executes the method according to any one of claims 1 to 10.

Images