Interaction method and device, electronic equipment and storage medium

Abstract

The invention discloses an interaction method and device, electronic equipment and a storage medium, and belongs to the technical field of computers. The method includes displaying, at a display component, a computer-generated three-dimensional environment; presenting an interactive object in the three-dimensional environment; obtaining interaction information input by a user through an input device; inputting the interaction information to an interaction manager; when the interaction manager judges that the interaction information meets the current interaction scene, a target interaction object corresponding to the interaction information is determined; and when the target interaction object receives the interaction information, executing an operation corresponding to the interaction information on the target interaction object. According to the invention, rich and natural interaction experience can be provided for the user.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and particularly relates to an interaction method, apparatus, electronic device, and storage medium. Background Art

[0002] Extended Reality (XR) technology can combine the real and the virtual through a computer to provide users with an extended reality space for human-computer interaction. In the extended reality space, users can conduct social interactions, entertainment, learning, work, etc. through extended reality devices. How to provide users with a rich and natural interaction experience has become an urgent problem to be solved. Summary of the Invention

[0003] This application aims to solve at least one of the technical problems existing in the prior art. For this purpose, this application provides an interaction method, apparatus, electronic device, and storage medium, which can provide users with a rich and natural interaction experience.

[0004] In a first aspect, this application provides an interaction method, which is applied to an electronic device. The electronic device includes a display component and one or more input devices. The method includes:

[0005] Display a computer-generated three-dimensional environment at the display component;

[0006] Present an interaction object within the three-dimensional environment;

[0007] Obtain interaction information input by the user through the input device;

[0008] Input the interaction information to an interaction manager;

[0009] When the interaction manager determines that the interaction information meets the current interaction scenario, determine the target interaction object corresponding to the interaction information;

[0010] When the target interaction object accepts the interaction information, perform the operation corresponding to the interaction information on the target interaction object.

[0011] According to the interaction method of this application, by presetting interaction information that meets different interaction scenarios and presetting the interaction information accepted by different interaction objects, when the interaction information input by the user meets the current interaction scenario, the target interaction object corresponding to the interaction information is determined, and when the target interaction object accepts the interaction information input by the user, the corresponding operation is performed on the target interaction object, avoiding unnatural interaction operations, thereby providing users with a rich and natural interaction experience.

[0012] According to an embodiment of this application, the method further includes:

[0013] The interaction manager detects whether the interaction information belongs to the preset type of interaction information corresponding to the current interaction scenario;

[0014] If so, the interaction manager determines that the interaction information meets the current interaction scenario;

[0015] If not, the interaction information is discarded.

[0016] According to an embodiment of the present application, the method further includes:

[0017] Detect whether the interaction information belongs to the preset type of interaction information corresponding to the target interaction object;

[0018] If so, it is determined that the target interaction object accepts the interaction information;

[0019] If not, the interaction information is discarded.

[0020] According to an embodiment of the present application, the current interaction scenario includes a far-field interaction scenario, the preset type of interaction information corresponding to the far-field interaction scenario includes ray information and eye movement information, and the preset type of interaction information corresponding to the target interaction object includes ray information and eye movement information;

[0021] When the interaction information belongs to the ray information or the eye movement information, determining the target interaction object corresponding to the interaction information includes:

[0022] Detect whether the interaction information points to an interaction object;

[0023] If so, the interaction object pointed to by the interaction information is used as the target interaction object;

[0024] Performing the operation corresponding to the interaction information on the target interaction object includes:

[0025] Performing a selection operation on the target interaction object.

[0026] According to an embodiment of the present application, the current interaction scenario includes a near-field interaction scenario, the preset type of interaction information corresponding to the near-field interaction scenario includes gesture information, and the preset type of interaction information corresponding to the target interaction object includes gesture information;

[0027] When the interaction information belongs to the gesture information, determining the target interaction object corresponding to the interaction information includes:

[0028] Detect whether there is a currently selected interaction object;

[0029] If so, the currently selected interaction object is used as the target interaction object;

[0030] Performing the operation corresponding to the interaction information on the target interaction object includes:

[0031] Performing a grasping, poking, or pushing / pulling operation corresponding to the interaction information on the target interaction object.

[0032] According to an embodiment of the present application, after performing the operation corresponding to the interaction information on the target interaction object, it further includes:

[0033] Detecting, by the interaction scene manager, whether the interaction information meets the conversion condition from the current interaction scene to the target interaction scene;

[0034] If so, converting the current interaction scene to the target interaction scene.

[0035] According to an embodiment of the present application, the current interaction scene includes a far-field interaction scene, and the target interaction scene includes a near-field interaction scene;

[0036] The detecting, by the interaction scene manager, whether the interaction information meets the conversion condition from the current interaction scene to the target interaction scene includes:

[0037] Detecting, by the interaction scene manager, whether the interaction position corresponding to the interaction information is located in the near-field interaction area of the target interaction object;

[0038] If so, the interaction scene manager determines that the interaction information meets the conversion condition from the far-field interaction scene to the near-field interaction scene.

[0039] According to an embodiment of the present application, the method further includes:

[0040] Setting a new interaction scene in the interaction manager, and setting conversion conditions for the new interaction scene to be converted with other interaction scenes in the interaction scene manager.

[0041] According to an embodiment of the present application, the input device includes at least one of a handle, a mouse, a keyboard, a touchpad, a head-mounted device, an image acquisition device, and a voice acquisition device; the interaction information includes at least one of ray information, key information, and trigger information input through the handle, mouse information input through the mouse, keyboard information input through the keyboard, touch information input through the touchpad, key information input through the head-mounted device, voice information input through the voice acquisition device, and gesture information and eye movement information input through the image acquisition device.

[0042] According to an embodiment of the present application, the current interaction scenario includes any one of a near-field interaction scenario, a far-field interaction scenario, a far-field pure ray interaction scenario, and a pure eye-hand interaction scenario.

[0043] In a second aspect, the present application provides an interaction device applied to an electronic device. The electronic device includes a display component and one or more input devices. The display component is used to display a computer-generated three-dimensional environment in which interaction objects are presented. The device includes:

[0044] An acquisition module, configured to acquire interaction information input by a user through the input device;

[0045] An input module, configured to input the interaction information to an interaction manager;

[0046] A determination module, configured to determine a target interaction object corresponding to the interaction information when the interaction manager determines that the interaction information meets the current interaction scenario;

[0047] An execution module, configured to perform an operation corresponding to the interaction information on the target interaction object when the target interaction object accepts the interaction information.

[0048] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the interaction method described in the first aspect above is implemented.

[0049] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the interaction method described in the first aspect above is implemented.

[0050] In a fifth aspect, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the interaction method described in the first aspect above is implemented.

[0051] One or more of the above technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0052] By presetting interaction information that meets different interaction scenarios and presetting the interaction information accepted by different interaction objects, when the interaction information input by the user meets the current interaction scenario, the target interaction object corresponding to the interaction information is determined, and when the target interaction object accepts the interaction information input by the user, the corresponding operation is performed on the target interaction object, avoiding unnatural interaction operations, thereby providing a rich and natural interaction experience for the user.

[0053] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. Description of the Drawings

[0054] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0055] Figure 1 is a schematic flowchart of the interaction method provided by an embodiment of the present application;

[0056] Figure 2 is a schematic diagram of the principle of the interaction method provided by an embodiment of the present application;

[0057] Figure 3 is a schematic structural diagram of the interaction device provided by an embodiment of the present application;

[0058] Figure 4 is a schematic structural diagram of the electronic device provided by an embodiment of the present application;

[0059] Figure 5 is a schematic hardware diagram of the electronic device provided by an embodiment of the present application. Detailed Embodiments

[0060] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application belong to the scope of protection of the present application.

[0061] The terms "first", "second", etc. in the description and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order different from those illustrated or described herein, and the objects distinguished by "first", "second", etc. generally belong to the same category, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the description and claims means at least one of the connected objects, and the character " / " generally represents an "or" relationship between the associated objects before and after.

[0062] The interaction method, device, electronic device, and storage medium provided by the embodiments of the present application will be described in detail below with reference to the drawings, through specific embodiments and their application scenarios.

[0063] Among them, the interaction method can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0064] Figure 1 It is a schematic flowchart of an interaction method provided by an embodiment of the present application.

[0065] For the interaction method provided by an embodiment of the present application, the execution subject of the interaction method can be an electronic device or a functional module or functional entity in the electronic device that can implement the interaction method. Hereinafter, the interaction method provided by an embodiment of the present application will be described by taking the electronic device as the execution subject as an example. Among them, the electronic device can be a virtual reality (VR) device, an augmented reality (AR) device, an extended reality (XR) device, a mixed reality (MR) device, etc. The electronic device can include a display component and one or more input devices.

[0066] As Figure 1 shown, the interaction method includes: Step 110, Step 120, Step 130, Step 140, Step 150, and Step 160.

[0067] Step 110: Display a computer-generated three-dimensional environment at the display component. Among them, the three-dimensional environment can be a corresponding virtual environment simulated by an electronic device (such as an XR device) for a certain real interaction scenario selected by the user, so as to perform corresponding interactions in the extended reality space. Among them, the three-dimensional environment can be a simulation environment of the real world, or a semi-simulated and semi-virtual virtual scene, or a pure virtual virtual scene.

[0068] Step 120: Present an interaction object in the three-dimensional environment.

[0069] Among them, the interaction object can be any type of object, such as an item, a virtual character, etc.

[0070] The user can enter the three-dimensional environment through an electronic device (such as an XR device) and control the interaction object to perform corresponding operations in the three-dimensional environment.

[0071] The three-dimensional environment can have multiple interaction scenarios, such as a near-field interaction scenario, a far-field interaction scenario, a far-field pure ray interaction scenario, and a pure eye-hand interaction scenario, etc. Multiple interaction scenarios can be converted with each other.

[0072] Step 130: Obtain interaction information input by the user through the input device.

[0073] Among them, the input device includes at least one of a handle, a mouse, a keyboard, a touchpad, a head-mounted device, an image acquisition device, and a voice acquisition device. The image acquisition device can be a camera device on the head-mounted device. The input device can be other types of devices, which are not specifically limited here.

[0074] The interaction information input by the user through the input device may include at least one of ray information, key information, and trigger information input through the handle, mouse information input through the mouse, keyboard information input through the keyboard, touch information input through the touchpad, key information input through the head-mounted device, voice information input through the voice acquisition device, and gesture information and eye movement information input through the image acquisition device. The interaction information may also include other types of information, which are not specifically defined here.

[0075] Among them, the eye movement information can be realized by eye tracking (gaze tracking) technology. Eye tracking is an algorithm technology for calculating the current fixation point of the eyes on the electronic device. The technical principle of eye tracking is to obtain an eye image through the image acquisition device in the electronic device, then perform image processing on the eye image to locate the pupil, and use a certain algorithm to calculate its viewing angle, and then calculate the position (such as coordinates) of the point where the eyes are looking or staring on the screen from this viewing angle. In addition, eye tracking can also identify the current opening and closing degree of the user's eyes.

[0076] The gesture information can be realized by gesture recognition technology. Gesture recognition technology is a technology for recognizing the spatial position and direction posture of the user's hand. The technical principle of gesture recognition technology is to obtain a hand image through the image acquisition device in the electronic device, then perform image processing on the hand image to locate each joint point on the hand, so as to determine the distance that the user's hand moves in space, or to determine the action made by the user's hand (such as pinching, making a fist, etc.).

[0077] The voice information may include the user's voice information input through the voice acquisition device.

[0078] The handle is equipped with controls such as keys and triggers. The ray information, key information, trigger information, etc. input through the handle can be information generated by the electronic device based on the user's operation of the controls on the handle...

[0079] The head-mounted device is equipped with controls such as keys. The key information, etc. input through the head-mounted device can be information generated by the electronic device based on the user's operation of the controls on the head-mounted device.

[0080] The mouse is equipped with controls such as keys and scroll wheels. The mouse information input through the mouse can be information generated by the electronic device based on the user's operation of the controls on the mouse.

[0081] The keyboard is equipped with controls such as keys. The keyboard information input through the keyboard can be information generated by the electronic device based on the user's operation of the controls on the keyboard.

[0082] The touch information input through the touchpad can be information generated by the electronic device based on the user's touch operations (such as click operations, swipe operations, etc.) on the touchpad.

[0083] The user can input at least one type of interaction information through the input device. In this embodiment, each type of interaction information can be abstracted into an interactor. For example, Figure 2 As shown, the user input A1 is the interaction information input through the handle. The interaction information input through the handle can have multiple types, such as a ray (interaction data triggered by a ray control), a poke (interaction data triggered by a button control), and a trigger (interaction data triggered by a trigger control), etc. The interaction information input through the handle can correspond to multiple interactors, such as interactor B1 (ray), interactor B2 (poke), and interactor B3 (trigger), etc.

[0084] The user input A2 is the gesture information input through the image acquisition device. The gesture information can have multiple types, such as a pinch gesture, a poke gesture, and a grab gesture, etc. The gesture information can correspond to multiple interactors, such as interactor B4 (pinch gesture), interactor B5 (grab gesture), and interactor B6 (grab gesture), etc.

[0085] The user input A3 is the eye movement information input through the image acquisition device. The eye movement information can have one type, that is, fixation. The eye movement information can correspond to one interactor, that is, interactor B7 (fixation).

[0086] The user input A4 is the voice information input through the voice acquisition device. The voice information can have one type, that is, voice. The voice information can correspond to one interactor, that is, interactor B8 (voice).

[0087] The user input A5 is the key information input through the head-mounted device. The key information can have multiple types, such as a long press (interaction data triggered by a long press key control) and a short press (interaction data triggered by a short press key control), etc. The key information can correspond to multiple interactors, such as interactor B9 (long press) and interactor B10 (short press).

[0088] Step 140: Input the interaction information to the interaction manager.

[0089] In this embodiment, an Interaction Manager can be used to set various interaction scenarios and the preset type of interaction information corresponding to various interaction scenarios. After the user inputs the interaction information through the input device, the interaction information is input to the interaction manager to determine whether the interaction information meets the current interaction scenario through the interaction manager.

[0090] The current interaction scenario is the interaction scenario in the three - dimensional environment where the user is currently located. The current interaction scenario can include any one of a near - field interaction scenario, a far - field interaction scenario, a far - field pure ray interaction scenario, and a pure eye - hand interaction scenario.

[0091] In some embodiments, the interaction method further includes:

[0092] Detecting, by an interaction manager, whether the interaction information belongs to a preset type of interaction information corresponding to the current interaction scenario;

[0093] If so, the interaction manager determines that the interaction information meets the current interaction scenario;

[0094] If not, the interaction manager determines that the interaction information does not meet the current interaction scenario and discards the interaction information.

[0095] For each interaction scenario, the type of interaction information applicable to it can be preset. The types of interaction information applicable to different interaction scenarios can be the same or different. After obtaining the interaction information input by the user, the interaction manager can detect whether the interaction information input by the user belongs to the preset type of interaction information applicable to the current interaction scenario. If so, it is determined that the interaction information input by the user meets the current interaction scenario. If not, it is determined that the interaction information input by the user does not meet the current interaction scenario.

[0096] For example, the preset type of interaction information corresponding to the far - field interaction scenario can include ray information and eye movement information, etc. The preset type of interaction information corresponding to the near - field interaction scenario can include gesture information (including gesture pinching, gesture grasping, and gesture making a fist), etc. The preset type of interaction information corresponding to the far - field pure ray interaction scenario can include ray information. The preset type of interaction information corresponding to the pure eye - hand interaction scenario can include eye movement information and gesture information (including gesture pinching, gesture grasping, and gesture making a fist), etc.

[0097] Step 150: When the interaction manager determines that the interaction information meets the current interaction scenario, determine the target interaction object corresponding to the interaction information.

[0098] The target interaction object corresponding to the interaction information is an interaction object in the three - dimensional environment. Any operable object in the three - dimensional environment can be used as an interaction object (Interactable). The interaction manager is used to manage the interaction between the interactors and the interaction objects.

[0099] The target interaction object corresponding to the interaction information is the interaction object that the user wants to control.

[0100] For example, in a current interaction scenario that includes a far-field interaction scenario, and the preset type of interaction information corresponding to the far-field interaction scenario includes ray information and eye movement information, when the interaction information meets the far-field interaction scenario, that is, the interaction information belongs to ray information or eye movement information, determining the target interaction object corresponding to the interaction information in step 150 includes: detecting whether the interaction information points to an interaction object; if so, using the interaction object pointed to by the interaction information as the target interaction object.

[0101] In the far-field interaction scenario, both ray information and eye movement information belong to the selected operation type of interaction information. Detecting whether the interaction information points to an interaction object means detecting whether the interaction information selects the interaction object. If so, using the selected interaction object as the target interaction object; if not, discarding the interaction information.

[0102] In some embodiments, the preset type of interaction information corresponding to the far-field interaction scenario may further include trigger information, gesture pinching information, etc. Determining the target interaction object corresponding to the interaction information in step 150 includes: detecting whether there is a currently selected interaction object; if so, using the currently selected interaction object as the target interaction object.

[0103] In the far-field interaction scenario, both trigger information and gesture pinching information belong to the determination operation type of interaction information. If an interaction object has been selected before inputting the interaction information (trigger information or gesture pinching information), using the selected interaction object as the target interaction object; if no interaction object has been selected before inputting the interaction information (trigger information or gesture pinching information), discarding the interaction information.

[0104] It should be noted that the preset type of interaction information corresponding to the far-field interaction scenario may further include voice information, keyboard information, touch information, etc.

[0105] For another example, in a current interaction scenario that includes a near-field interaction scenario, and the preset type of interaction information corresponding to the near-field interaction scenario includes gesture information, when the interaction information meets the near-field interaction scenario, that is, the interaction information belongs to gesture information, determining the target interaction object corresponding to the interaction information in step 150 includes: detecting whether there is a currently selected interaction object; if so, using the currently selected interaction object as the target interaction object.

[0106] In the near-field interaction scenario, gesture information belongs to the action type of information. If an interaction object has been selected before inputting the interaction information (gesture information), using the selected interaction object as the target interaction object; if no interaction object has been selected before inputting the interaction information (gesture information), discarding the interaction information.

[0107] It should be noted that the preset type of interaction information corresponding to the near-field interaction scenario may also include key information, mouse information, keyboard information, touch information, key information and voice information input through a head-mounted device, etc.

[0108] In addition, the preset type of interaction information corresponding to the far-field pure ray interaction scenario may include ray information, etc. The preset type of interaction information corresponding to the pure eye-hand interaction scenario may include eye movement information, gesture information, etc.

[0109] Step 160: When the target interaction object receives the interaction information, perform the operation corresponding to the interaction information on the target interaction object.

[0110] In some embodiments, the interaction method further includes:

[0111] Detect whether the interaction information belongs to the preset type of interaction information corresponding to the target interaction object;

[0112] If so, determine that the target interaction object accepts the interaction information;

[0113] If not, determine that the target interaction object does not accept the interaction information, and discard the interaction information.

[0114] Each interaction object can preset the type of interaction information it can accept. The types of interaction information that different interaction objects can accept can be the same or different. After determining the target interaction object corresponding to the interaction information, it can be detected whether the interaction information belongs to the preset type of interaction information corresponding to the target interaction object. If so, it is determined that the target interaction object accepts the interaction information. If not, it is determined that the target interaction object does not accept the interaction information, and the interaction information is discarded.

[0115] When it is determined that the target interaction object accepts the interaction information, control the target interaction object to perform the interaction operation corresponding to the interaction information. As Figure 2 shown, there are five interaction objects in the three-dimensional environment, namely interaction object C1 to interaction object C5. Distribute the interaction information input by the user to the corresponding target interaction object, that is, interaction object C3, and detect whether interaction object C3 accepts the interaction information input by the user. If so, control interaction object C3 to perform the corresponding interaction operation. When it is determined that the target interaction object does not accept the interaction information, discard the interaction information.

[0116] For example, in a current interaction scenario that includes a far-field interaction scenario, where the preset type of interaction information corresponding to the far-field interaction scenario includes ray information and eye movement information, and the preset type of interaction information corresponding to the target interaction object includes ray information and eye movement information. When the interaction information meets the far-field interaction scenario and the target interaction object accepts the interaction information, that is, when the interaction information belongs to ray information or eye movement information, the operation corresponding to the interaction information performed on the target interaction object in step 160 includes: performing a selection operation on the target interaction object.

[0117] In the far-field interaction scenario, both ray information and eye movement information belong to interaction information of the selection operation type. If the target interaction object accepts the interaction information, a selection operation is performed on the target interaction object. If the target interaction object does not accept the interaction information, the interaction information is discarded.

[0118] In some embodiments, the preset type of interaction information corresponding to the far-field interaction scenario may further include trigger information, gesture pinching information, etc. The operation corresponding to the interaction information performed on the target interaction object in step 160 includes: performing a confirmation operation on the target interaction object.

[0119] In the far-field interaction scenario, both trigger information and gesture pinching information belong to interaction information of the confirmation operation type. If the target interaction object accepts the interaction information, a confirmation operation is performed on the target interaction object. If the target interaction object does not accept the interaction information, the interaction information is discarded.

[0120] For another example, in a current interaction scenario that includes a near-field interaction scenario, and the preset type of interaction information corresponding to the near-field interaction scenario includes gesture information, and the preset type of interaction information corresponding to the target interaction object includes gesture information. When the interaction information meets the near-field interaction scenario and the target interaction object accepts the interaction information, that is, when the interaction information belongs to gesture information, the operation corresponding to the interaction information performed on the target interaction object in step 160 includes: performing a grasping, poking, or pushing / pulling operation corresponding to the interaction information on the target interaction object.

[0121] In the near-field interaction scenario, gesture information belongs to action type information. For example, gesture information may include gesture grasping actions, gesture poking actions, and gesture pushing / pulling actions, etc. If the interaction information includes a gesture grasping action, a grasping operation is performed on the target interaction object; if the interaction information includes a gesture poking action, a poking operation is performed on the target interaction object; if the interaction information includes a gesture pushing / pulling action, a pushing / pulling action is performed on the target interaction object. Gesture information may also include other gesture actions. When the interaction information includes other gesture actions, other operations may be performed on the target interaction object.

[0122] This embodiment can connect multi-modal interaction methods to suit different interaction scenarios in a three-dimensional environment, providing users with a rich and natural interaction experience.

[0123] In some embodiments, after the step of performing an operation corresponding to the interaction information on the target interaction object, the method further includes:

[0124] Detecting, by an interaction scenario manager, whether the interaction information meets the conversion condition from the current interaction scenario to the target interaction scenario;

[0125] If so, converting the current interaction scenario to the target interaction scenario.

[0126] As Figure 2 shown, in this embodiment, an interaction scenario manager can be set up. There are conversion conditions between multiple interaction scenarios in the interaction scenario manager. Each interaction scenario can establish a conversion relationship with at least one other interaction scenario, that is, conversion conditions can be set between any two interaction scenarios. The conversion conditions between different interaction scenarios can be the same or different.

[0127] After controlling the target interaction object to perform the corresponding operation, detecting, by the interaction scenario manager, whether the interaction information meets the conversion condition for converting the current interaction scenario to another interaction scenario. If it is determined that the conversion condition for converting the current interaction scenario to the target interaction scenario is met, then convert the current interaction scenario to the target interaction scenario, that is, the three-dimensional environment is in the target interaction scenario after the conversion; if it is determined that the interaction information does not meet the conversion condition for converting the current interaction scenario to another interaction scenario, then keep the current interaction scenario, that is, the three-dimensional environment remains in the current interaction scenario.

[0128] In some embodiments, the current interaction scenario includes a far-field interaction scenario, and the target interaction scenario includes a near-field interaction scenario; detecting, by the interaction scenario manager, whether the interaction information meets the conversion condition from the current interaction scenario to the target interaction scenario includes: detecting, by the interaction scenario manager, whether the interaction position corresponding to the interaction information is located in the near-field interaction area of the target interaction object; if so, the interaction scenario manager determines that the interaction information meets the conversion condition from the far-field interaction scenario to the near-field interaction scenario.

[0129] The target interaction object can be pre-set with a near-field interaction area. For example, if the target interaction object is a water cup, the near-field interaction area of the target interaction object is the area where the handle of the water cup is located. When it is detected that the interaction position corresponding to the interaction information (such as eye movement information) is located on the handle of the water cup, the current far-field interaction scenario is converted to a near-field interaction scenario.

[0130] In some embodiments, the current interaction scenario includes a near-field interaction scenario, and the target interaction scenario includes a far-field interaction scenario; detecting whether the interaction information meets the conversion condition from the current interaction scenario to the target interaction scenario through the interaction scenario manager includes: detecting whether the interaction position corresponding to the interaction information is located in the near-field interaction area of any interaction object through the interaction scenario manager; if not, the interaction scenario manager determines that the interaction information meets the conversion condition from the near-field interaction scenario to the far-field interaction scenario.

[0131] In this embodiment, by setting the conversion condition for converting the current interaction scenario to the target interaction scenario, when the conversion condition is met, the current interaction scenario is converted to the target interaction scenario, providing users with a rich and natural conversion interaction experience.

[0132] In some embodiments, the interaction method further includes:

[0133] Setting a new interaction scenario in the interaction manager, and setting the conversion condition for the new interaction scenario to be converted with other interaction scenarios in the interaction scenario manager.

[0134] In this embodiment, the interaction manager also provides an interface for creating a new interaction scenario. Users can customize the interaction scenario through this interface and set the preset type of interaction information corresponding to the new interaction scenario, that is, set the interaction information that meets the new interaction scenario. The interaction scenario manager also provides an interface for setting conversion conditions. Users can customize the conversion conditions through this interface, that is, set the conversion conditions for the new interaction scenario to be converted with other interaction scenarios, including the conversion conditions for the new interaction scenario to be converted to other interaction scenarios and / or the conversion conditions for other interaction scenarios to be converted to the new interaction scenario.

[0135] In this embodiment, users can customize the interaction scenario and conversion conditions, which helps to develop more diverse interaction applications, improve the potential of XR applications, and has good scalability.

[0136] According to the interaction method provided by the embodiments of the present application, by presetting the interaction information that meets different interaction scenarios and presetting the interaction information accepted by different interaction objects, when the interaction information input by the user meets the current interaction scenario, the target interaction object corresponding to the interaction information is determined, and when the target interaction object accepts the interaction information input by the user, the corresponding operation is performed on the target interaction object, avoiding unnatural interaction operations, thereby providing users with a rich and natural interaction experience.

[0137] For the interaction method provided by the embodiments of the present application, the execution subject may be an interaction device. In the embodiments of the present application, taking the interaction device executing the interaction method as an example, the interaction device provided by the embodiments of the present application is described.

[0138] An embodiment of the present application further provides an interaction device, which is applied to an electronic device. The electronic device includes a display component and one or more input devices. The display component is used to display a computer-generated three-dimensional environment, and an interaction object is presented in the three-dimensional environment.

[0139] As Figure 3 shown, the interaction device includes: an acquisition module 301, an input module 302, a determination module 303, and an execution module 304.

[0140] The acquisition module 301 is configured to acquire interaction information input by the user through the input device;

[0141] The input module 302 is configured to input the interaction information to an interaction manager;

[0142] The determination module 303 is configured to determine a target interaction object corresponding to the interaction information when the interaction manager determines that the interaction information meets the current interaction scenario;

[0143] The execution module 304 is configured to perform an operation corresponding to the interaction information on the target interaction object when the target interaction object accepts the interaction information.

[0144] According to the interaction device provided by the embodiment of the present application, by presetting interaction information that meets different interaction scenarios and presetting interaction information accepted by different interaction objects, when the interaction information input by the user meets the current interaction scenario, a target interaction object corresponding to the interaction information is determined, and when the target interaction object accepts the interaction information input by the user, a corresponding operation is performed on the target interaction object, avoiding unnatural interaction operations, thereby providing a rich and natural interaction experience for the user.

[0145] In some embodiments, the interaction device further includes a first detection module, and the first detection module is configured to:

[0146] Detect whether the interaction information belongs to preset type interaction information corresponding to the current interaction scenario through the interaction manager;

[0147] If so, the interaction manager determines that the interaction information meets the current interaction scenario;

[0148] If not, discard the interaction information.

[0149] In some embodiments, the interaction device further includes a second detection module, and the second detection module is configured to:

[0150] Detect whether the interaction information belongs to preset type interaction information corresponding to the target interaction object;

[0151] If so, it is determined that the target interaction object accepts the interaction information;

[0152] If not, the interaction information is discarded.

[0153] In some embodiments, the current interaction scenario includes a far-field interaction scenario, and the preset type of interaction information corresponding to the far-field interaction scenario includes ray information and eye movement information. The preset type of interaction information corresponding to the target interaction object includes ray information and eye movement information;

[0154] When the interaction information belongs to the ray information or the eye movement information, the determination module 303 is further configured to:

[0155] Detect whether the interaction information points to an interaction object;

[0156] If so, the interaction object pointed to by the interaction information is used as the target interaction object;

[0157] The execution module 304 is further configured to:

[0158] Perform a selection operation on the target interaction object.

[0159] In some embodiments, the current interaction scenario includes a near-field interaction scenario, and the preset type of interaction information corresponding to the near-field interaction scenario includes gesture information. The preset type of interaction information corresponding to the target interaction object includes gesture information;

[0160] The determination module 303 is further configured to:

[0161] Detect whether there is a currently selected interaction object;

[0162] If so, the currently selected interaction object is used as the target interaction object;

[0163] The execution module 304:

[0164] Perform a grasping, poking, or pushing and pulling operation corresponding to the interaction information on the target interaction object.

[0165] In some embodiments, the interaction device further includes a third detection module, and the third detection module is configured to:

[0166] Detect whether the interaction information meets the conversion condition from the current interaction scenario to the target interaction scenario through the interaction scenario manager;

[0167] If so, the current interaction scenario is converted to the target interaction scenario.

[0168] In some embodiments, the current interaction scenario includes a far-field interaction scenario, and the target interaction scenario includes a near-field interaction scenario;

[0169] The third detection module is further configured to:

[0170] detect, via the interaction scene manager, whether the interaction position corresponding to the interaction information is located in the near-field interaction area of the target interaction object;

[0171] If so, the interaction scene manager determines that the interaction information meets the conversion condition from the far-field interaction scene to the near-field interaction scene.

[0172] In some embodiments, the interaction device further includes a setting module, and the setting module is configured to:

[0173] set a new interaction scene in the interaction manager and set the conversion condition for the new interaction scene to be converted with other interaction scenes in the interaction scene manager.

[0174] In some embodiments, the input device includes at least one of a handle, a mouse, a keyboard, a touchpad, a head-mounted device, an image acquisition device, and a voice acquisition device; the interaction information includes at least one of ray information, key information, and trigger information input through the handle, mouse information input through the mouse, keyboard information input through the keyboard, touch information input through the touchpad, key information input through the head-mounted device, voice information input through the voice acquisition device, and gesture information and eye movement information input through the image acquisition device.

[0175] In some embodiments, the current interaction scene includes any one of a near-field interaction scene, a far-field interaction scene, a far-field pure ray interaction scene, and a pure eye-hand interaction scene.

[0176] The interaction device in the embodiments of the present application may be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than a terminal. Exemplarily, the electronic device may be an augmented reality (AR) / virtual reality (VR) device or an XR device, and the embodiments of the present application do not make specific limitations.

[0177] The interaction device in the embodiments of the present application may be a device with an operating system. The operating system may be a Microsoft (Windows) operating system, an Android operating system, an IOS operating system, or other possible operating systems, and the embodiments of the present application do not make specific limitations.

[0178] The interaction device provided in the embodiments of the present application can implement each process of the method embodiments. To avoid repetition, it will not be elaborated here.

[0179] In some embodiments, such as Figure 4 shown, an embodiment of the present application further provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored on the memory 802 and executable on the processor 801. When the program is executed by the processor 801, it implements each process of the above-described interaction method embodiment and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0180] Figure 5 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application.

[0181] The electronic device 900 includes, but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910, etc.

[0182] Those skilled in the art can understand that the electronic device 900 may further include a power supply (such as a battery) for supplying power to each component. The power supply can be logically connected to the processor 910 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. Figure 5 The structure of the electronic device shown in does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0183] Among them, the processor 910 is used to obtain the interaction information input by the user through the input unit; input the interaction information to the interaction manager; when the interaction manager determines that the interaction information meets the current interaction scenario, determine the target interaction object corresponding to the interaction information; when the target interaction object accepts the interaction information, perform the operation corresponding to the interaction information on the target interaction object.

[0184] According to the electronic device provided by the embodiment of the present application, by presetting the interaction information that meets different interaction scenarios and presetting the interaction information accepted by different interaction objects, when the interaction information input by the user meets the current interaction scenario, the target interaction object corresponding to the interaction information is determined, and when the target interaction object accepts the interaction information input by the user, the corresponding operation is performed on the target interaction object, avoiding unnatural interaction operations, thereby providing a rich and natural interaction experience for the user.

[0185] In some embodiments, the processor 910 is further used for:

[0186] detecting, through the interaction manager, whether the interaction information belongs to the preset type of interaction information corresponding to the current interaction scenario;

[0187] If so, the interaction manager determines that the interaction information meets the current interaction scenario;

[0188] If not, discard the interaction information.

[0189] In some embodiments, the processor 910 is further configured to:

[0190] Detect whether the interaction information belongs to the preset type of interaction information corresponding to the target interaction object;

[0191] If so, determine that the target interaction object accepts the interaction information;

[0192] If not, discard the interaction information.

[0193] In some embodiments, the current interaction scenario includes a far-field interaction scenario, the preset type of interaction information corresponding to the far-field interaction scenario includes ray information and eye movement information, and the preset type of interaction information corresponding to the target interaction object includes ray information and eye movement information;

[0194] When the interaction information belongs to the ray information or the eye movement information, the processor 910 is further configured to:

[0195] Detect whether the interaction information points to an interaction object;

[0196] If so, use the interaction object pointed to by the interaction information as the target interaction object;

[0197] The processor 910 is further configured to:

[0198] Perform a selection operation on the target interaction object.

[0199] In some embodiments, the current interaction scenario includes a near-field interaction scenario, the preset type of interaction information corresponding to the near-field interaction scenario includes gesture information, and the preset type of interaction information corresponding to the target interaction object includes gesture information;

[0200] The processor 910 is further configured to:

[0201] Detect whether there is a currently selected interaction object;

[0202] If so, use the currently selected interaction object as the target interaction object;

[0203] The processor 910 is further configured to:

[0204] Perform a grab, poke, or push-and-pull operation corresponding to the interaction information on the target interaction object.

[0205] In some embodiments, the processor 910 is further configured to:

[0206] Detect, via the interaction scene manager, whether the interaction information satisfies the conversion condition from the current interaction scene to the target interaction scene;

[0207] If so, convert the current interaction scene to the target interaction scene.

[0208] In some embodiments, the current interaction scene includes a far-field interaction scene, and the target interaction scene includes a near-field interaction scene;

[0209] The processor 910 is further configured to:

[0210] Detect, via the interaction scene manager, whether the interaction position corresponding to the interaction information is located in the near-field interaction area of the target interaction object;

[0211] If so, the interaction scene manager determines that the interaction information satisfies the conversion condition from the far-field interaction scene to the near-field interaction scene.

[0212] In some embodiments, the processor 910 is further configured to:

[0213] Set a new interaction scene in the interaction manager, and set the conversion condition for the new interaction scene to be converted with other interaction scenes in the interaction scene manager.

[0214] In some embodiments, the input unit includes at least one of a handle, a mouse, a keyboard, a touchpad, a head-mounted device, an image acquisition device, and a voice acquisition device; the interaction information includes at least one of ray information, key information, and trigger information input through the handle, mouse information input through the mouse, keyboard information input through the keyboard, touch information input through the touchpad, key information input through the head-mounted device, voice information input through the voice acquisition device, and gesture information and eye movement information input through the image acquisition device.

[0215] In some embodiments, the current interaction scene includes any one of a near-field interaction scene, a far-field interaction scene, a far-field pure ray interaction scene, and a pure eye-hand interaction scene.

[0216] It should be understood that in the embodiments of the present application, the input unit 904 may include a Graphics Processing Unit (GPU) 9041 and a microphone 9042. The GPU 9041 processes the image data of static pictures or videos obtained by an image capturing device (such as a camera) in the video capturing mode or the image capturing mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also referred to as a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. The other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here.

[0217] The memory 909 can be used to store software programs and various data. The memory 909 mainly includes a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area can store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory 909 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash memory. The volatile memory may be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 909 in the embodiments of the present application includes, but is not limited to, these and any other suitable types of memories.

[0218] The processor 910 may include one or more processing units; the processor 910 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor may not be integrated into the processor 910 either.

[0219] The embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements each process of the above-mentioned interactive method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0220] Among them, the processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disks, or optical discs, etc.

[0221] The embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned interactive method when executed by a processor.

[0222] Among them, the processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disks, or optical discs, etc.

[0223] The embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned interactive method when executed by a processor.

[0224] Among them, the processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disks, or optical discs, etc.

[0225] The embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above-mentioned interactive method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0226] It should be understood that the chip mentioned in the embodiment of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0227] It should be noted that in this document, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device that includes a series of elements includes not only those elements but also other elements not expressly listed, or elements that are inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "including a..." does not exclude the presence of additional identical elements in the process, method, article or device that includes such element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0228] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present application.

[0229] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Those of ordinary skill in the art, under the inspiration of the present application, without departing from the purpose of the present application and the scope protected by the claims, can also make many forms, all of which fall within the protection scope of the present application.

[0230] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0231] Although embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present application. The scope of the present application is defined by the claims and their equivalents.

Claims

1. An interaction method, characterized in that, Applied to an electronic device, the electronic device including a display component and one or more input devices, the method comprising: Displaying a computer-generated three-dimensional environment at the display component; Presenting an interaction object within the three-dimensional environment; Obtaining interaction information input by a user via the input device; Inputting the interaction information to an interaction manager; When the interaction manager determines that the interaction information meets the current interaction scenario, determining a target interaction object corresponding to the interaction information; When the target interaction object accepts the interaction information, performing an operation corresponding to the interaction information on the target interaction object.

2. The interactive method according to claim 1, wherein The method further comprises: Detecting, by the interaction manager, whether the interaction information belongs to preset type interaction information corresponding to the current interaction scenario; If so, the interaction manager determines that the interaction information meets the current interaction scenario; If not, discarding the interaction information.

3. The interactive method according to claim 2, wherein The method further comprises: Detecting whether the interaction information belongs to preset type interaction information corresponding to the target interaction object; If so, determining that the target interaction object accepts the interaction information; If not, discarding the interaction information.

4. The interactive method according to claim 3, wherein The current interaction scenario includes a far-field interaction scenario, the preset type interaction information corresponding to the far-field interaction scenario including ray information and eye movement information, and the preset type interaction information corresponding to the target interaction object including ray information and eye movement information; When the interaction information belongs to the ray information or the eye movement information, the determining the target interaction object corresponding to the interaction information comprises: Detecting whether the interaction information points to an interaction object; If so, using the interaction object pointed to by the interaction information as the target interaction object; The performing the operation corresponding to the interaction information on the target interaction object comprises: Performing a selection operation on the target interaction object.

5. The interactive method according to claim 3, wherein The current interaction scenario includes a near-field interaction scenario, the preset type interaction information corresponding to the near-field interaction scenario including gesture information, and the preset type interaction information corresponding to the target interaction object including gesture information; When the interaction information belongs to the gesture information, the determining the target interaction object corresponding to the interaction information comprises: Detecting whether there is a currently selected interaction object; If so, using the currently selected interaction object as the target interaction object; The performing the operation corresponding to the interaction information on the target interaction object comprises: Performing a grab, poke, or push-pull operation corresponding to the interaction information on the target interaction object.

6. The interactive method according to claim 1, wherein After performing the operation corresponding to the interaction information on the target interaction object, further comprising: Detecting, by an interaction scenario manager, whether the interaction information meets a conversion condition from the current interaction scenario to a target interaction scenario; If so, converting the current interaction scenario to the target interaction scenario.

7. The interactive method according to claim 6, wherein The current interaction scenario includes a far-field interaction scenario, and the target interaction scenario includes a near-field interaction scenario; The detecting, by the interaction scenario manager, whether the interaction information meets the conversion condition from the current interaction scenario to the target interaction scenario comprises: The interaction scene manager detects whether the interaction position corresponding to the interaction information is located in the near-field interaction area of the target interaction object; If so, the interaction scene manager determines that the interaction information meets the conversion condition from the far-field interaction scene to the near-field interaction scene.

8. The interactive method according to claim 6, wherein The method further includes: Setting a new interaction scene in the interaction manager, and setting conversion conditions for the new interaction scene to be converted with other interaction scenes in the interaction scene manager.

9. The interactive method according to any one of claims 1-8, characterized in that, The input device includes at least one of a handle, a mouse, a keyboard, a touchpad, a head-mounted device, an image acquisition device, and a voice acquisition device; the interaction information includes at least one of ray information, key information, and trigger information input through the handle, mouse information input through the mouse, keyboard information input through the keyboard, touch information input through the touchpad, key information input through the head-mounted device, voice information input through the voice acquisition device, and gesture information and eye movement information input through the image acquisition device.

10. The interactive method according to any one of claims 1-8, characterized in that The current interaction scene includes any one of a near-field interaction scene, a far-field interaction scene, a far-field pure ray interaction scene, and a pure eye-hand interaction scene.

11. An interaction device, characterized in that, Applied to an electronic device, the electronic device includes a display component and one or more input devices, the display component is used to display a computer-generated three-dimensional environment, and an interaction object is presented in the three-dimensional environment. The device includes: An acquisition module, configured to acquire interaction information input by a user through the input device; An input module, configured to input the interaction information to an interaction manager; A determination module, configured to determine a target interaction object corresponding to the interaction information when the interaction manager determines that the interaction information meets the current interaction scene; An execution module, configured to execute an operation corresponding to the interaction information on the target interaction object when the target interaction object accepts the interaction information.

12. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the interaction method according to any one of claims 1-10.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the interaction method according to any one of claims 1-10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the interaction method according to any one of claims 1-10.

Images