Staged interaction method and device in extended reality system, equipment and storage medium

Abstract

The invention relates to the technical field of XR, and provides a staged interaction method and device in an extended reality system, equipment and a storage medium, and the method comprises the steps: obtaining a first specified interaction mode configured for a first interaction stage and a second specified interaction mode configured for a second interaction stage, the first interaction stage is used for updating a target interaction object, and the second interaction stage is used for controlling the target interaction object; acquiring a first action amplitude value of the first specified interaction mode and a second action amplitude value of the second specified interaction mode; according to the relation between the first action amplitude value and the corresponding activation threshold value and the relation between the second action amplitude value and the corresponding activation threshold value, the current interaction stage of the system is recognized; and responding and processing the input of the interaction mode corresponding to the current interaction stage of the system. According to the embodiment of the invention, the flexibility, availability and fault tolerance of interaction in the XR space can be improved, and the system overhead is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of augmented reality systems, and in particular to a method, device, equipment and storage medium for phased interaction in an augmented reality system. Background Art

[0002] At present, devices equipped with XR (Extended Reality) systems can adapt (connect) to multiple input methods, such as keyboards, mice, touchpads, handles, input methods based on eye movement recognition (hereinafter referred to as eye movement), input methods based on hand tracking (hereinafter referred to as hand tracking), input methods based on gesture recognition (hereinafter referred to as gesture recognition), etc. Different input methods generate different input data corresponding to different interaction methods. If a system can recognize the data generated by the eye movement input method, it is usually said that the system supports eye movement interaction. In order to enable people to use a variety of different input methods to interact with the content in the XR system, the XR system generally strictly distinguishes the responsibilities of different interaction methods. For example, in the existing eye-hand bimodal interaction, data based on eye movement recognition is used to determine the user's "interaction target", while data based on gesture recognition is used to determine the user's "interaction action". Specifically, the target seen by the human eye is determined by the system as the "interaction target", and the user can switch the "interaction target" by turning the eyeball and looking at different targets. The user can specify the "interaction action" by changing the hand movement (i.e. gesture). The system executes the interaction instruction for the current "interaction target" according to the current "interaction action" recognized. For example, when the user sees button A (interaction target) and makes a pinching (interaction action) gesture with his hand, button A will be clicked.

[0003] However, this design greatly reduces the usability and flexibility of the system. First, if the device equipped with the XR system has only one input method, it cannot meet the conditions of this dual-modal interaction. Because the completion of its interaction requires that one input method points to the interaction target while another input method completes the interaction action. Secondly, this design often strongly binds a certain input method to the interaction task it undertakes (that is, eye movement is only used for "selecting interaction targets", and gesture recognition is only used for "interaction operations"), which loses the flexibility and scalability of the system. Due to the strong binding of the system, when the eye movement data is lost or cannot be used for some reason, the user cannot select the interaction target and thus cannot complete the interaction. For example, when the device is connected to a mouse, the mouse cannot be used instead of eye movement to select content in the system. Secondly, when the user has determined the "interaction target" and starts to operate the target, he must continue to use the input method of selecting the "interaction target" to select the target, otherwise, the interaction operation will not be successfully executed, or the operation will be performed on a non-original "interaction target". For example, the user moves his eyes to select button A, but when using gestures to interact with button A, his eyes accidentally or unconsciously look at button B. In this case, button B, which was not originally expected to be triggered, will be activated.

[0004] Therefore, how to improve the flexibility, availability, and fault tolerance of interactions in XR space while reducing system overhead has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the embodiments of this specification is to provide a phased interaction method, apparatus, device and storage medium in an augmented reality system to improve the flexibility, availability and fault tolerance of interaction in the XR space, while reducing system overhead.

[0006] To achieve the above objectives, on the one hand, an embodiment of this specification provides a phased interaction method in an augmented reality system, including:

[0007] Acquire a first specified interaction mode configured for a first interaction phase and a second specified interaction mode configured for a second interaction phase, wherein the first interaction phase is used to update a target interaction object, and the second interaction phase is used to control the target interaction object;

[0008] Acquire a first amplitude value of the action of the first specified interaction mode and a second amplitude value of the action of the second specified interaction mode;

[0009] According to the relationship between the first amplitude value of the action and the second amplitude value of the action and the corresponding activation threshold, respectively, identifying the current interaction stage of the system;

[0010] Respond to and process input corresponding to the interaction mode of the current interaction stage of the system.

[0011] In the staged interaction method in the augmented reality system of the embodiment of this specification, identifying the current interaction stage of the system according to the relationship between the first amplitude value of the action and the second amplitude value of the action and the corresponding activation threshold, includes:

[0012] When the first amplitude value of the action exceeds the first activation threshold, and the second amplitude value of the action does not exceed the second activation threshold, confirming that the system is currently in the first interaction stage;

[0013] When the first amplitude value of the action does not exceed the first activation threshold, and the second amplitude value of the action exceeds the second activation threshold, confirming that the system is currently in the second interaction stage;

[0014] The first activation threshold includes that the position change and / or rotation change of the interactive input meets the first feature; the second activation threshold includes that the position change and / or rotation change of the interactive input meets the second feature.

[0015] In the staged interaction method in the augmented reality system of the embodiment of the present specification, according to the relationship between the first amplitude value of the action and the second amplitude value of the action and the corresponding activation threshold, identifying the current interaction stage of the system also includes:

[0016] When the first amplitude value of the action exceeds the first activation threshold, and the second amplitude value of the action exceeds the second activation threshold, switching the interaction stage of the system to an interaction stage different from the current interaction stage;

[0017] The first activation threshold includes that the position change and / or rotation change of the interactive input meets the first feature; the second activation threshold includes that the position change and / or rotation change of the interactive input meets the second feature.

[0018] In the phased interaction method in the augmented reality system of the embodiment of this specification, switching the interaction phase of the system to an interaction phase different from the current interaction phase includes:

[0019] When the current interaction stage of the system is the first interaction stage, switching the interaction stage of the system to the second interaction stage;

[0020] When the current interaction stage of the system is the second interaction stage and the action of the second specified interaction mode has been completed, the interaction stage of the system is switched to the first interaction stage.

[0021] In the phased interaction method in the augmented reality system of the embodiment of this specification, the first feature includes:

[0022] a change in the position and / or rotation of the interactive input relative to the interactive target; and / or,

[0023] The change in the position and / or rotation of the interactive input is relative to the change characteristics of the user's interactive operation habits.

[0024] In the phased interaction method in the augmented reality system of the embodiment of this specification, the second feature includes:

[0025] a change in the position and / or rotation of the interactive input relative to the interactive target; and / or,

[0026] The change in the position and / or rotation of the interactive input is relative to the change characteristics of the user's interactive operation habits.

[0027] In the phased interaction method in the augmented reality system of the embodiment of the present specification, the first designated interaction mode is a first interaction mode dynamically designated from a first interaction mode list, and the second designated interaction mode is a second interaction mode dynamically designated from a second interaction mode list; the first interaction modes in the first interaction mode list all satisfy the first condition; the second interaction modes in the second interaction mode list all satisfy the second condition, wherein the first interaction mode list and the second interaction mode list are saved in a dynamically definable interaction configuration file.

[0028] In the phased interaction method in the augmented reality system of the embodiment of this specification, the first condition includes: the first interaction methods in the first interaction method list can all provide real-time position information and real-time rotation information of the corresponding interaction source; the second condition includes: the interaction sources corresponding to the second interaction methods in the second interaction method list can all provide distinguishable postures whose number reaches a quantity threshold.

[0029] In the phased interaction method in the augmented reality system of the embodiment of this specification, under the system dynamic definition, the first designated interaction mode is the first interaction mode with the lowest fatigue value in the first interaction mode list, and the second designated interaction mode is the second interaction mode with the lowest fatigue value in the second interaction mode list;

[0030] Under user dynamic definition, the first designated interaction mode is any first interaction mode selected by the user from the first interaction mode list, and the second designated interaction mode is any second interaction mode selected by the user from the second interaction mode list.

[0031] In the phased interaction method in the augmented reality system of the embodiment of the present specification, different first designated interaction modes have different first activation thresholds, and different second designated interaction modes have different second activation thresholds.

[0032] In the phased interaction method in the augmented reality system of the embodiment of this specification, the responding and processing of the input corresponding to the interaction mode of the current interaction phase of the system includes:

[0033] When the current interaction stage of the system is the first interaction stage, the target interaction object is dynamically updated according to the real-time position information and real-time rotation information generated by the interaction source of the first specified interaction mode.

[0034] In the phased interaction method in the augmented reality system of the embodiment of the present specification, dynamically updating the target interaction object according to the real-time position information and real-time rotation information generated by the interaction source of the first specified interaction mode includes:

[0035] Acquire real-time position information and real-time rotation information generated by the interaction source of the first specified interaction mode;

[0036] Generate a stereoscopic selection space with the position related to the real-time position information of the interaction source as the starting point, the vector direction related to the real-time rotation information of the interaction source as the longitudinal extension direction, and the length data related to the preset farthest interaction distance as the longitudinal extension length; the coverage range of the stereoscopic selection space is the selection range; and the objects covered or partially covered by the selection range are candidate interaction objects;

[0037] Determining whether there are multiple candidate interaction objects;

[0038] When there are multiple candidate interaction objects, a target interaction object is selected from the multiple candidate interaction objects according to a preset rule.

[0039] In the phased interaction method in the augmented reality system of the embodiment of the present specification, the generation of a stereoscopic selection space with the position related to the real-time position information of the interaction source as the starting point, the vector direction related to the real-time rotation information of the interaction source as the longitudinal extension direction, and the length data related to the preset farthest interaction distance as the longitudinal extension length includes:

[0040] A cone is generated with the real-time position of the interaction source plus the position offset as the vertex position, the real-time rotation direction of the interaction source plus the vector direction of the rotation offset as the mother line direction, the preset farthest interaction distance multiplied by the first coefficient as the height, and the preset anti-shake value multiplied by the second coefficient as the base radius.

[0041] In the phased interaction method in the augmented reality system of the embodiment of this specification, selecting a target interaction object from a plurality of candidate interaction objects according to a preset rule includes:

[0042] Calculate the closest distance between each candidate interaction object and the generatrix of the cone, and obtain multiple closest distances accordingly;

[0043] Updating the candidate interaction object corresponding to the smallest of the multiple closest distances as the target interaction object;

[0044] If there are multiple candidate interaction objects corresponding to the smallest one among the multiple closest distances, the candidate interaction object closest to the user is updated as the target interaction object.

[0045] In the phased interaction method in the augmented reality system of the embodiment of this specification, the responding and processing of the input corresponding to the interaction mode of the current interaction phase of the system includes:

[0046] When the current interaction stage of the system is the second stage, an interaction instruction for controlling the target interaction object is generated according to the input of the second specified interaction mode.

[0047] In the phased interaction method in the augmented reality system of the embodiment of this specification, the method further includes:

[0048] Update the interaction configuration file according to the changes of the interaction source of the system.

[0049] In the phased interaction method in the augmented reality system of the embodiment of this specification, updating the interaction configuration file according to the change of the interaction source of the system includes:

[0050] When the interaction sources of the system are reduced, and the reduced interaction sources are interaction sources corresponding to the first specified interaction mode or the second specified interaction mode, the first interaction mode list or the second interaction mode list is updated accordingly;

[0051] A new designated interaction method is selected from the updated interaction method list and the user is notified.

[0052] In the phased interaction method in the augmented reality system of the embodiment of this specification, updating the interaction configuration file according to the change of the interaction source of the system includes:

[0053] When the interaction sources of the system are reduced, and the reduced interaction sources are not the interaction sources corresponding to the first specified interaction mode and the second specified interaction mode, the first interaction mode list or the second interaction mode list is updated accordingly;

[0054] When an interaction source of the system is increased, the interaction mode of the newly added interaction source is updated in the first interaction mode list or the second interaction mode list accordingly according to the attribute of the newly added interaction source, and the user is notified.

[0055] On the other hand, the embodiment of the present specification further provides a staged interaction device in an augmented reality system, comprising:

[0056] A determination module, used to obtain a first specified interaction mode configured for a first interaction stage and a second specified interaction mode configured for a second interaction stage, wherein the first interaction stage is used to update a target interaction object and the second interaction stage is used to control the target interaction object;

[0057] An acquisition module, configured to acquire a first amplitude value of the action of the first specified interaction mode and a second amplitude value of the action of the second specified interaction mode;

[0058] an identification module, used to identify the current interaction stage of the system according to the relationship between the first amplitude value of the action and the second amplitude value of the action and the corresponding activation threshold respectively;

[0059] The response module is used to respond to and process the input of the interaction mode corresponding to the current interaction stage of the system.

[0060] On the other hand, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, the instructions of the above method are executed.

[0061] On the other hand, an embodiment of the present specification further provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer device, the instructions of the above method are executed.

[0062] On the other hand, an embodiment of the present specification further provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor of a computer device, the computer program executes instructions of the above method.

[0063] It can be seen from the technical solutions provided in the above embodiments of this specification that in the embodiments of this specification, since the interaction mode of selecting the target interaction object and the interaction mode of controlling the target interaction object can be customized by the system, application or user, the mandatory binding of the interaction mode and the interaction function is avoided, and the flexibility and usability of the interaction in the XR space are improved. In addition, in the embodiments of this specification, the interaction is implemented in stages, and the system can identify whether it is currently in the first stage of interaction or the second stage of interaction according to the relationship between the action amplitude of different interaction modes and the corresponding activation threshold, and then only respond to and process the input of the corresponding interaction mode in the interaction stage (for example, when in the first stage of interaction, respond to the user's input to update the target interaction object based on the first specified interaction mode, and when in the second stage of interaction, respond to the user's input to control the target interaction object based on the second specified interaction mode). In this way, the flexibility, usability and fault tolerance of the interaction in the XR space can be improved, and the system overhead can be reduced, thereby improving the system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0065] Figure 1 A schematic diagram of a system for phased interaction in an augmented reality system in some embodiments of this specification is shown;

[0066] Figure 2 A flowchart of a phased interaction method in an augmented reality system in some embodiments of this specification is shown;

[0067] Figure 3 A flowchart showing the current interaction stage of the identification system in some embodiments of this specification;

[0068] Figure 4 A flowchart of updating a target interactive object when the system is in the first interactive stage in some embodiments of this specification is shown;

[0069] Figure 5 A schematic diagram showing an exemplary embodiment of the present specification in which the system is in the first stage of interaction and updates the target interaction object;

[0070] Figure 6 A schematic diagram showing a control target interactive object when the system is in the second stage of interaction in an exemplary embodiment of this specification;

[0071] Figure 7 A schematic diagram showing an exemplary embodiment of the present specification in which the system is in the first stage of interaction and updates the target interaction object according to a preset rule;

[0072] Figure 8 It shows a structural block diagram of a phased interaction device in an augmented reality system in some embodiments of the present specification;

[0073] Fig. 9 The structure block diagram of the computer device in some embodiments of the present specification is shown.

[0074] [Description of Reference Numerals]

[0075] 10. XR devices;

[0076] 20. Interaction source;

[0077] 30. XR space;

[0078] 81. Determine the module;

[0079] 82. Get module;

[0080] 83. Identification module;

[0081] 84. Response module;

[0082] 902. Computer equipment;

[0083] 904, processor;

[0084] 906. Memory;

[0085] 908, driving mechanism;

[0086] 910, input / output interface;

[0087] 912. Input device;

[0088] 914. Output device;

[0089] 916. Presentation equipment;

[0090] 918. Graphical user interface;

[0091] 920, network interface;

[0092] 922, communication link;

[0093] 924. Communication bus. DETAILED DESCRIPTION

[0094] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0095] Figure 1A schematic diagram of a system for phased interaction in an extended reality system of some embodiments of the present specification is shown in FIG; the system includes an XR device 10 and an interaction source 20. Among them, the XR device 10 is configured with an extended reality system (i.e., an XR system). The XR system may include a virtual reality (VR) system, an augmented reality (AR) system, and a mixed reality (MR) system. Therefore, the XR device 10 in the embodiment of the present specification may be any one of a VR device, an AR device, or an MR device. The XR device 10 may display an XR space 30 (i.e., an XR scene), and multiple applications (including three-dimensional applications and / or two-dimensional applications) and / or other visual objects may be presented in the XR space 30. The interaction source 20 is the device corresponding to the input mode or the part of the body tracked by the sensor. The interaction source 20 may be provided with input data by an input device (such as a handle, etc.) and / or a part of the human body (hands, eyes, head, etc.). The system completes the recognition of the input data provided by different interaction sources, and supports different interaction modes accordingly. For example: hand tracking input can provide data such as real-time position information and real-time rotation information of the hand. Systems that support the recognition of these data can use "hand tracking interaction" to complete the interaction. For another example, the eyes can provide data such as real-time position information and real-time rotation information of the eyeballs. Systems that complete the recognition of these data can support "eye movement interaction". For another example, based on the recognition of different gesture data corresponding to the hand recognition input method, different hand gesture information can be obtained, such as pinching, grabbing, scissors gestures, etc., thereby supporting users to use "gesture recognition interaction". The XR device 10 can obtain a first specified interaction mode configured for a first interaction stage and a second specified interaction mode configured for a second interaction stage, wherein the first interaction stage is used to update a target interaction object, and the second interaction stage is used to control a target interaction object; obtain a first amplitude value of an action of the first specified interaction mode and a second amplitude value of an action of the second specified interaction mode; identify the current interaction stage of the system according to the relationship between the first amplitude value of the action and the second amplitude value of the action and the corresponding activation threshold respectively; respond to and process the input of the interaction mode corresponding to the current interaction stage of the system; in this way, the flexibility, availability and fault tolerance of the interaction in the XR space can be improved, while reducing the system overhead, thereby improving the system performance.

[0096] The embodiments of this specification provide a phased interaction method in an augmented reality system, which can be applied to the above-mentioned XR device, refer to Figure 2 As shown, in some embodiments of the present specification, the phased interaction method in the augmented reality system may include the following steps:

[0097] Step 201: Obtain a first specified interaction mode configured for a first interaction stage and a second specified interaction mode configured for a second interaction stage, wherein the first interaction stage is used to update a target interaction object, and the second interaction stage is used to control the target interaction object.

[0098] Step 202: Obtain a first amplitude value of the action of the first specified interaction mode and a second amplitude value of the action of the second specified interaction mode.

[0099] Step 203: Identify the current interaction stage of the system according to the relationship between the first action amplitude value and the second action amplitude value and the corresponding activation threshold.

[0100] Step 204: respond to and process the input of the interaction mode corresponding to the current interaction stage of the system.

[0101] In the embodiments of this specification, since the interaction mode for selecting the target interaction object and the interaction mode for controlling the target interaction object can be customized by the system, application or user, the mandatory binding of the interaction mode and the interaction function is avoided, and the flexibility and usability of the interaction in the XR space are improved. In addition, in the embodiments of this specification, the interaction is implemented in stages, and the system can identify whether it is currently in the first stage of interaction or the second stage of interaction according to the relationship between the action amplitude of different interaction modes and the corresponding activation threshold, and then only needs to respond to and process the input of the corresponding interaction mode in the interaction stage (for example, when in the first stage of interaction, respond to the user's input to update the target interaction object based on the first specified interaction mode, and when in the second stage of interaction, respond to the user's input to control the target interaction object based on the second specified interaction mode). In this way, the flexibility, usability and fault tolerance of the interaction in the XR space can be improved, and the system overhead can be reduced, thereby improving the system performance.

[0102] In the embodiments of this specification, the first stage of interaction is the interaction stage of updating the target interaction object, and the target interaction object can be updated in the first stage of interaction; the second stage of interaction is the interaction stage of controlling the target interaction object, and the target interaction object determined in the previous stage can be controlled in the second stage of interaction. In some embodiments of this specification, updating the target interaction object can be selecting a target application from multiple applications displayed in the XR space, and correspondingly, controlling the target interaction object can be controlling the target application (such as opening, closing, zooming in, zooming out, moving, etc.). In other embodiments of this specification, updating the target interaction object can also be selecting a target object from multiple operable objects of the target application in the XR space (such as characters in game applications, etc.), and correspondingly, controlling the target interaction object can be controlling the target object (such as running, jumping, turning, etc., posture control and other types of control, etc.). It can be understood that the description here using applications or objects in applications as examples is only an exemplary example. In some embodiments of this specification, the target interaction object may also include a part of the XR space that is located outside the application (such as information in the XR environment, physical content in the XR environment, etc.). In addition, in the embodiments of this specification, interactive input generally refers to the user's input to the XR space through an interactive source.

[0103] In some embodiments of the present specification, the first interaction mode list and the second interaction mode list are saved in a dynamically definable interaction configuration file; the first interaction mode list may include multiple first interaction modes, and the second interaction mode list may include multiple second interaction modes. The interaction modes in the first interaction mode list and the second interaction mode list are arranged in order; the first interaction mode ranked first in the first interaction mode list is the first designated interaction mode, and the second interaction mode ranked first in the second interaction mode list is the second designated interaction mode. In addition, the order of the first interaction mode list and the second interaction mode list can be adjusted by the system or the user; for example, the system or the user can adjust the order of the first interaction mode in the first interaction mode list to select a first interaction mode as the designated interaction mode for the first stage of the interaction; the system or the user can adjust the order of the second interaction mode in the second interaction mode list to select a second interaction mode as the designated interaction mode for the second stage of the interaction.

[0104] In some embodiments of this specification, the system may define a first interaction mode and a second interaction mode list, and allow the user to dynamically adjust the order of the interaction modes in the first interaction mode list and / or the second interaction mode list at any appropriate time; in this way, the user's operation convenience can be improved. In the embodiments of this specification, dynamic adjustment means that the first specified interaction mode and / or the second specified interaction mode can be updated at any time according to changes in user needs and / or changes in the interactive input of the system to improve the flexibility and usability of the system.

[0105] In some embodiments of the present specification, the first stage of interaction is characterized by: completing efficient selection of a stable, fault-tolerant, precise, and controllable range. Therefore, an interaction method that can simultaneously provide real-time position (Position) information and real-time rotation (Rotation) information of the corresponding interaction source in the XR space is used as the first interaction method. For example, head movement interaction (interaction based on head posture tracking), eye movement interaction, hand tracking interaction, handle interaction, etc. are suitable as the first interaction method. Since interaction methods such as keyboards cannot provide real-time position information and real-time rotation information in the XR space, they are not suitable as the first interaction method.

[0106] In some embodiments of the present specification, the characteristics of the second stage of interaction are: flexible (there are multiple distinguishable states), easy for users to understand (intuitive, low cognitive load), and easy to operate (low user fatigue operation). Therefore, the interaction method in which the number of distinguishable states reaches a threshold number can be used as the second interaction method. For example, gesture recognition interaction can provide multiple gestures such as pinching, pressing, grabbing, etc. that are easy for users to understand and complete, so gesture recognition interaction is suitable as the second interaction method. Similarly, the handle can provide multiple operations such as pressing, pulling, rotating, etc. that are easy for users to understand and complete, so the handle interaction is also suitable as the second interaction method. However, the system can only obtain its real-time position information and real-time rotation information from eye movement interaction, so as to distinguish between the two states of looking at or not looking at a certain area. Therefore, eye movement interaction is not suitable as the second interaction method.

[0107] In some embodiments of the present specification, in the scenario of a system-defined interaction profile, the first interaction mode list and the second interaction mode list in the interaction profile can be sorted from low to high according to the degree of fatigue (an indicator that reflects the degree of fatigue of the human body caused by using the interaction mode), and the higher the order, the higher the recommendation priority; for example, if an XR device is compatible with both eye movement interaction and head movement interaction, they can both provide data on real-time position information and real-time rotation information in the XR space; however, compared with the rotation of the eyeballs, the muscles of the neck are more easily fatigued when the head rotates; therefore, in the priority sorting of the first interaction mode list, eye movement interaction takes precedence over head movement interaction. For another example, if an XR device is compatible with both gesture recognition interaction and handle interaction, they can both be used as the second interaction mode; however, gesture recognition interaction is easier to operate and more fatigue-resistant than the handle; therefore, in the priority sorting of the second interaction mode list, gesture recognition interaction takes precedence over handle interaction.

[0108] The interaction methods mentioned in the description of the above system-customized interaction profile are only illustrative examples. Since different XR devices are compatible with different interaction methods, in actual implementation, the first interaction method in the first interaction method list and the second interaction method in the second interaction method list may be determined according to the specific device, and this manual does not impose any specific restrictions on this.

[0109] In the scenario where the user dynamically customizes the sorting of the interaction modes in the first interaction mode list and / or the second interaction mode list, the first designated interaction mode can be any first interaction mode selected by the user from the first interaction mode list, and the second designated interaction mode can be any second interaction mode selected by the user from the second interaction mode list, so as to meet the user's interaction preferences or interaction habits. In some embodiments of the present specification, before the user dynamically customizes the sorting of the interaction modes in the first interaction mode list and / or the second interaction mode list, if the first interaction mode list and the second interaction mode list have been pre-sorted by the system in ascending order of fatigue level, then when the user dynamically adjusts the interaction mode on this basis, the user can more quickly select an interaction mode that meets his or her own interaction preferences and has a lower fatigue level.

[0110] In some embodiments of the present specification, after reading the dynamically defined interaction configuration file, the system can determine the first interaction mode desired by the user in the first stage of interaction (i.e., determine the first designated interaction mode for the first stage of interaction) according to the first interaction mode at the first priority in the first interaction mode list of the interaction configuration file, and can determine the second interaction mode desired by the user in the second stage of interaction (i.e., determine the second designated interaction mode for the second stage of interaction) according to the second interaction mode at the first priority in the second interaction mode list of the interaction configuration file.

[0111] If the user adjusts the order of the interaction methods in the first interaction method list and the second interaction method list, the first interaction method in the first interaction method list is the first interaction method desired by the user, and the second interaction method in the second interaction method list is the second interaction method desired by the user. If the user does not adjust the order of the interaction methods in the first interaction method list and the second interaction method list, the order of the interaction methods recommended by the system is used.

[0112] In the embodiments of the present specification, it is possible to identify whether the system is currently in the first interaction stage or the second interaction stage based on the relationship between the action amplitudes of different interaction modes and the corresponding activation thresholds, and then respond to and process the input of the interaction mode corresponding to the current interaction stage; so that in the first interaction stage, the system can ignore the input for the second interaction stage, and in the second interaction stage, the system can ignore the input for the first interaction stage, thereby saving system overhead and improving the stability of the interaction.

[0113] refer to Figure 3 As shown, in some embodiments of the present specification, according to the relationship between the first amplitude value of the action and the second amplitude value of the action and the corresponding activation threshold, identifying the current interaction stage of the system may include the following steps:

[0114] Step 301: Determine whether the first amplitude value of the action exceeds the first activation threshold, and determine whether the second amplitude value of the action exceeds the second activation threshold.

[0115] In the embodiments of the present specification, the first action amplitude value is used to characterize the action amplitude of the first specified interaction mode controlled by the user, and the second action amplitude value is used to characterize the action amplitude of the second specified interaction mode controlled by the user.

[0116] In some embodiments of the present specification, the first activation threshold includes that the position change and / or rotation change of the interactive input meets the first feature; the second activation threshold includes that the position change and / or rotation change of the interactive input meets the second feature. Wherein, both the first feature and the second feature can be the change feature of the position and / or rotation of the interactive input relative to the interactive target; and / or the change feature of the position and / or rotation of the interactive input relative to the user's interactive operation habits.

[0117] In some embodiments of the present specification, the changing characteristics of the position and / or rotation of the interactive input relative to the interactive target may include, but are not limited to: the trajectory of the position change and / or rotation change of the interactive input meets preset characteristics; the time when the data generated by the input source (i.e., the input source of the interactive input) stops updating reaches a first preset duration; the time when the data generated by the input source stops updating reaches a second preset duration, and the rotation data generated by the input source deviates from the user-related angle by a first angle threshold; the input speed of the interactive input reaches a first speed threshold; and / or the user moves to the physical boundary range of the physical space.

[0118] In some embodiments of the present specification, the change characteristics of the position and / or rotation of the interactive input relative to the user's interactive operation habits may include, but are not limited to: the three-dimensional position in the XR space pointed to by the interactive input changes to the boundary range of the UI and / or UI component, the three-dimensional position changes within the boundary range, and the time period that does not exceed the boundary range reaches a third preset time length; and / or the user moves to the physical boundary range of the physical space, the user's position changes within the physical boundary range, and the time period that does not exceed the physical boundary range reaches a fourth preset time length.

[0119] In view of the different characteristics of input data generated by different input modes, different first designated interaction modes may have different first activation thresholds, and different second designated interaction modes may have different second activation thresholds, so as to further improve the fault tolerance performance of the system.

[0120] For example, eye movement interaction, hand tracking interaction, etc. have relatively large deviations during use. That is, even if the user does not want to interact with the system at present, it is difficult for the user's eyes and hands to remain completely still, and a certain degree of deviation will occur. In this case, the activation threshold can be appropriately larger to reduce the risk of false triggering or misoperation. However, head movement interaction, mouse interaction, handle interaction, etc. have relatively low deviations. That is, if the user does not want to interact with the system at present, he can easily keep his head stable and not touch the mouse, handle, etc., so that the data generated by these input devices will not have a large deviation; in this case, the activation threshold can be set smaller to ensure the effectiveness of the input and the efficiency of the system's response to the input. In actual implementation, the activation threshold can be appropriately set according to needs and differences in equipment and individual users.

[0121] In addition, when both the first designated interaction mode and the second designated interaction mode rely on the recognition of input data of the same body part, different thresholds can also be set according to different interaction modes. For example, the first designated interaction mode is hand tracking interaction, and the second designated interaction mode is gesture recognition interaction; then the activation threshold of the hand tracking interaction can be that the displacement distance of the current n-th frame relative to the previous n-1 frame exceeds α meters and or the rotation exceeds θ angle, and the activation threshold of the gesture recognition interaction can be that the distance change between the thumb and the index finger of the current n-th frame relative to the previous n-1 frame exceeds β meters. Since the recognition of the first designated interaction mode and the second designated interaction mode relies on the action of the same body part, the action of the second interaction mode will affect the state of the first interaction mode. For example, when the user makes a pinch operation or presses a button on the handle, the position and rotation of the hand or handle will change accordingly. Therefore, in the embodiments of the present specification, this influence can be eliminated through this differentiated activation threshold design.

[0122] Step 302a: When the first amplitude value of the action exceeds the first activation threshold and the second amplitude value of the action does not exceed the second activation threshold, confirm that the system is currently in the first interaction stage.

[0123] When the first amplitude value of the action exceeds the first activation threshold, it indicates that the user intends to interact with the system through the first specified interaction method; when the second amplitude value of the action does not exceed the second activation threshold, it indicates that the user has no intention to interact with the system through the second specified interaction method. In this case, it can be determined that the system is currently in the first stage of interaction.

[0124] Step 302b: when the first amplitude value of the action does not exceed the first activation threshold, and the second amplitude value of the action exceeds the second activation threshold, confirm that the system is currently in the second interaction stage.

[0125] When the first amplitude value of the action does not exceed the first activation threshold, it indicates that the user has no intention of interacting with the system through the first specified interaction method; when the second amplitude value of the action exceeds the second activation threshold, it indicates that the user intends to interact with the system through the second specified interaction method. In this case, it can be determined that the system is currently in the second stage of interaction.

[0126] Step 302c: when the first amplitude value of the action exceeds the first activation threshold and the second amplitude value of the action exceeds the second activation threshold, switch the interaction phase of the system to an interaction phase different from the current interaction phase.

[0127] When the first amplitude value of the action exceeds the first activation threshold and the second amplitude value of the action exceeds the second activation threshold, the interaction stage of the system can be switched to an interaction stage different from the current interaction stage. Specifically, when the current interaction stage of the system is the first interaction stage, the interaction stage of the system is switched to the second interaction stage; when the current interaction stage of the system is the second interaction stage and the action of the second specified interaction mode has been completed, the interaction stage of the system is switched to the first interaction stage.

[0128] For example, taking the exemplary embodiment corresponding to α meters and β meters as an example, if the displacement distance of the current nth frame relative to the previous n-1th frame exceeds α meters, and the distance change between the thumb and the index finger in the current nth frame and the previous n-1th frame exceeds β meters, if the interaction stage of the system in the nth frame is the first interaction stage, then the interaction stage of the system is switched to the second interaction stage; if the interaction stage of the system in the nth frame is the second interaction stage, and the current second specified interaction action has been completed, then the interaction stage of the system is switched to the first interaction stage. It can be understood that here, one frame is taken as an example only as an exemplary description, and it can also be multiple frames in actual implementation, which can be selected as needed.

[0129] Step 302d: When the first amplitude value of the action does not exceed the first activation threshold, and the second amplitude value of the action does not exceed the second activation threshold, jump to step 301.

[0130] When the first amplitude value of the action does not exceed the first activation threshold, and the second amplitude value of the action does not exceed the second activation threshold, it indicates that the user has no intention to interact with the system at this time (that is, the user does not need to select an interactive object or control an interactive object), so the process can jump to step 301 to continue the judgment. In some embodiments of the present specification, responding to and processing the input corresponding to the interactive mode of the current interactive stage of the system may include: when the current interactive stage of the system is the first interactive stage, dynamically updating the target interactive object according to the real-time position information and real-time rotation information generated by the interactive source of the first specified interactive mode.

[0131] refer to Figure 4 As shown, in some embodiments of the present specification, dynamically updating the target interactive object according to the real-time position information and real-time rotation information generated by the interactive source of the first specified interactive mode may include the following steps:

[0132] Step 401: Acquire real-time position information and real-time rotation information generated by the interaction source of the first specified interaction mode.

[0133] Step 402: generate a stereoscopic selection space (i.e., a stereoscopic selection space) with the position related to the real-time position information of the interaction source as the starting point, the vector direction related to the real-time rotation information of the interaction source as the longitudinal extension direction, and the length data related to the preset farthest interaction distance as the longitudinal extension length; the coverage range of the stereoscopic selection space is the selection range; and the objects covered or partially covered by the selection range are candidate interaction objects.

[0134] Compared with selecting interactive objects in the traditional ray interaction method, in the embodiments of this specification, the coverage formed by the intersection of the stereo selection space and the XR space is larger and the fault tolerance performance is better.

[0135] The position related to the real-time position information of the interactive source refers to: the three-dimensional real-time position of the interactive source + the position offset. The vector direction related to the real-time rotation information of the interactive source refers to: the real-time rotation direction of the interactive source + the rotation offset. The position offset refers to the three-dimensional offset of the stereo selection space relative to the real-time position of the interactive source (which can be zero); the rotation offset refers to the three-dimensional offset of the stereo selection space relative to the real-time rotation direction of the interactive source (which can be zero).

[0136] Among them, in the three-dimensional rectangular coordinate system, the horizontal axis is the x-axis (taking the horizontal right direction as the positive direction), the vertical axis is the y-axis (taking the horizontal forward direction as the positive direction), and the vertical axis is the z-axis (taking the vertical upward direction as the positive direction); then the above-mentioned longitudinal extension direction is the direction corresponding to the positive direction of the y-axis; the above-mentioned longitudinal extension length is the length in the longitudinal extension direction; further, the longitudinal extension length can be set by the user or the system.

[0137] For example, in some embodiments of the present specification, a cone may be generated with the real-time position of the interactive source + the position offset as the vertex position, the real-time rotation direction of the interactive source + the vector direction of the rotation offset as the generatrix direction, the preset farthest interactive distance multiplied by the first coefficient (a preset coefficient) as the height, and the preset anti-shake value (a parameter value preset to meet the anti-shake redundancy requirement) multiplied by the second coefficient (another preset coefficient) as the base radius. The coverage range of the cone is the selection range; the objects covered or partially covered by the selection range are candidate interactive objects.

[0138] Step 403: Determine whether there are multiple candidate interaction objects. If there are multiple candidate interaction objects, execute step 404; otherwise, execute step 405.

[0139] Step 404: Update the candidate interaction object to the target interaction object.

[0140] Step 405: Select a target interaction object from multiple candidate interaction objects according to a preset rule.

[0141] For example, in some embodiments of the present specification, selecting a target interaction object from a plurality of candidate interaction objects according to a preset rule may include:

[0142] (1) Calculate the closest distance between each candidate interaction object (i.e., each of the multiple candidate interaction objects) and the generatrix of the cone, and obtain multiple corresponding closest distances.

[0143] In some embodiments of the present specification, the closest distance between the candidate interactive object and the center line of the cone may refer to: the closest distance between the center point of the candidate interactive object and the center line of the cone; in other embodiments of the present specification, the closest distance between the candidate interactive object and the center line of the cone may refer to: the closest distance between a point on the surface of the candidate interactive object and the center line of the cone; in other embodiments of the present specification, the closest distance between the candidate interactive object and the center line of the cone may also be defined by any other suitable definition, which is not limited to the embodiments of the present specification.

[0144] For example, in Figure 7 In the exemplary embodiment shown, after a cone with the real-time position of the wrist as the vertex position is generated, the coverage range of the cone in the XR space is the selection range; Figure 7 As shown, the selection range includes two candidate interaction objects (application 4 and application 6). In order to further confirm whether the user wants to select application 4 or application 6, the closest distance d1 between application 4 and the busbar can be calculated, and the closest distance d2 between application 6 and the busbar can be calculated.

[0145] (2) Updating the candidate interaction object corresponding to the smallest of the multiple closest distances as the target interaction object.

[0146] For example, in Figure 7 In the exemplary embodiment shown, since d1>d2, that is, d2 is the minimum value of d1 and d2, the application 6 corresponding to d2 can be updated as the target interaction object.

[0147] Furthermore, if there are multiple candidate interaction objects corresponding to the smallest one among the multiple closest distances, the candidate interaction object closest to the user is updated as the target interaction object.

[0148] In such Figure 4 In the embodiment shown, after executing step 404 or step 405, the execution jumps to step 401, that is, when the current interaction stage of the system is the first interaction stage, each time the user changes the position and / or rotation direction of the interaction source of the first specified interaction mode, the above steps will be executed repeatedly, so that the target interaction object can be dynamically updated. Figure 7In the exemplary embodiment shown, when the user changes the position and / or rotation direction of the wrist, the position and / or rotation direction of the cone will change synchronously, so that the selection range of the cone will also change synchronously, so that the target interactive object can be dynamically updated.

[0149] Compared with selecting candidate interactive objects in the traditional ray interaction method, in some embodiments of this specification, when the selection space is a cone, it also has the following advantages:

[0150] (1) The vertices are near the interaction source and the selection range is small near the user, which is conducive to the realization of finer selection of interactive objects. This is in line with the requirement of convenient finer granularity selection of interactive objects because the space near the user is limited and objects are usually small and compactly distributed.

[0151] (2) The bottom surface is on the side far from the interaction source, and the selection range far from the user is larger, which can better prevent jitter and further improve the fault tolerance performance. This is in line with the characteristic that when selecting objects far from the user, the selection deviation is proportional to the distance, thus requiring a stronger anti-shake capability.

[0152] In some embodiments of the present specification, responding to and processing input of an interaction method corresponding to the current interaction stage of the system may also include: when the current interaction stage of the system is the second stage, generating interaction instructions for controlling the target interaction object (for example, interaction instructions for controlling the target interaction object to open, close, move, rotate, etc.) according to the input of the second specified interaction method.

[0153] In the embodiments of this specification, the system may also update the interaction configuration file according to the change of the interaction source to realize the system dynamically configuring the interaction configuration file. For example, in some embodiments of this specification, updating the interaction configuration file according to the change of the interaction source of the system may include:

[0154] (1) When the number of interaction sources of the system is reduced, and the reduced interaction sources are interaction sources corresponding to the first specified interaction mode or the second specified interaction mode (that is, the reduced interaction sources are interaction sources corresponding to the first specified interaction mode, or the interaction sources corresponding to the second specified interaction mode), the first interaction mode list or the second interaction mode list is updated accordingly; a new specified interaction mode is selected from the updated interaction mode list, and the user is notified to remind the user that the specified interaction mode has been changed.

[0155] (2) When the number of interaction sources of the system is reduced, and the reduced interaction sources are not the interaction sources corresponding to the first specified interaction mode and the second specified interaction mode (that is, the reduced interaction sources are neither the interaction sources corresponding to the first specified interaction mode nor the interaction sources corresponding to the second specified interaction mode), the first interaction mode list or the second interaction mode list is updated accordingly. Since the specified interaction mode has not changed at this time, the user may not be notified at this time to reduce disturbance to the user.

[0156] (3) When an interaction source of the system is added, the interaction mode of the new interaction source is updated in the first interaction mode list or the second interaction mode list according to the attributes of the new interaction source, and the user is notified to remind the user that the optional interaction mode has been added.

[0157] For example, in Figure 5 In the exemplary embodiment shown, the first specified interaction mode in the first stage of interaction is eye movement interaction, and the second specified interaction mode in the second stage of interaction is gesture recognition interaction; if the current interaction is in the first stage, the target interaction object corresponding to the eye movement interaction is Figure 5 For 2D applications in XR scenarios, if the motion amplitude of gesture recognition interaction does not reach the corresponding motion amplitude threshold, the system does not respond to the input of gesture recognition interaction. However, when the motion amplitude of gesture recognition interaction reaches the corresponding motion amplitude threshold, for example, the distance change between the index finger and the thumb exceeds the corresponding motion amplitude threshold, the second stage of interaction is entered.

[0158] For example, in Figure 6 In the exemplary embodiment shown, the first specified interaction mode in the first stage of interaction is eye movement interaction, and the second specified interaction mode in the second stage of interaction is gesture recognition interaction; if the second stage of interaction is currently in progress, Figure 5 In the 3D weather application interaction in the XR scene, before the second stage of interaction is completed, or the movement amplitude of the eye movement interaction does not reach the corresponding movement amplitude threshold, even if the eye movement interaction points to Figure 5 The system will not change to control the 2D application in the XR scene. However, when the action amplitude of the eye movement interaction reaches the corresponding action amplitude threshold (for example, the action amplitude of the eye movement interaction pointing to the 2D application reaches the corresponding action amplitude threshold), and the gesture recognition interaction operation is completed (that is, it has ended), the first stage of interaction will be entered.

[0159] Although the process flows described above include multiple operations that occur in a particular order, it should be clearly understood that the processes may include more or fewer operations, which may be performed sequentially or in parallel (eg, using parallel processors or a multi-threaded environment).

[0160] Corresponding to the above-mentioned phased interaction method in the augmented reality system, the embodiment of this specification further provides a phased interaction device in the augmented reality system, which can be configured on the above-mentioned augmented reality system, referring to Figure 8 As shown, in some embodiments of the present specification, the phased interaction device in the augmented reality system may include:

[0161] A determination module 81, configured to obtain a first specified interaction mode configured for a first interaction stage and a second specified interaction mode configured for a second interaction stage, wherein the first interaction stage is used to update a target interaction object and the second interaction stage is used to control the target interaction object;

[0162] An acquisition module 82, configured to acquire a first amplitude value of the action of the first specified interaction mode and a second amplitude value of the action of the second specified interaction mode;

[0163] An identification module 83, configured to identify the current interaction stage of the system according to the relationship between the first amplitude value of the action and the second amplitude value of the action and the corresponding activation threshold respectively;

[0164] The response module 84 is used to respond to and process the input of the interaction mode corresponding to the current interaction stage of the system.

[0165] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0166] It should be noted that in the embodiments of this specification, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user and fully authorized by all parties, that is, the acquisition, transmission, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0167] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0168] The embodiments of this specification also provide a computer device. Fig. 9As shown, in some embodiments of the present specification, the computer device 902 may include one or more processors 904, such as one or more central processing units (CPUs) or graphics processing units (GPUs), and each processing unit may implement one or more hardware threads. The computer device 902 may also include any memory 906, which is used to store any kind of information such as code, settings, data, etc. In a specific embodiment, the computer program on the memory 906 and can be run on the processor 904, when the computer program is run by the processor 904, can execute the instructions of the phased interaction method in the augmented reality system described in any of the above embodiments. Non-limiting, for example, the memory 906 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 902. In one case, when the processor 904 executes an associated instruction stored in any memory or combination of memories, the computer device 902 can perform any operation of the associated instruction. The computer device 902 also includes one or more drive mechanisms 908 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0169] The computer device 902 may also include an input / output interface 910 (I / O) for receiving various inputs (via input devices 912) and for providing various outputs (via output devices 914). A specific output mechanism may include a presentation device 916 and an associated graphical user interface 918 (GUI). In other embodiments, the input / output interface 910 (I / O), input device 912, and output device 914 may not be included, and the computer device 902 may be used as a computer device in a network. The computer device 902 may also include one or more network interfaces 920 for exchanging data with other devices via one or more communication links 922. One or more communication buses 924 couple the components described above together.

[0170] The communication link 922 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 922 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0171] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), computer-readable storage media, and computer program products of some embodiments of the present specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processor to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processor generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0172] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processor to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processor so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0174] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0175] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0176] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computer device. As defined in this specification, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0177] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0178] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processors connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0179] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0180] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0181] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of this specification. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0182] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A phased interaction method in an augmented reality system, characterized in that: include: Acquire a first specified interaction mode configured for a first interaction phase and a second specified interaction mode configured for a second interaction phase, wherein the first interaction phase is used to update a target interaction object, and the second interaction phase is used to control the target interaction object; Acquire a first amplitude value of the action of the first specified interaction mode and a second amplitude value of the action of the second specified interaction mode; According to the relationship between the first amplitude value of the action and the second amplitude value of the action and the corresponding activation threshold, respectively, identifying the current interaction stage of the system; Respond to and process input corresponding to the interaction mode of the current interaction stage of the system.

2. The phased interaction method in an augmented reality system according to claim 1, characterized in that: According to the relationship between the first amplitude value of the action and the second amplitude value of the action and the corresponding activation threshold, identifying the current interaction stage of the system includes: When the first amplitude value of the action exceeds the first activation threshold, and the second amplitude value of the action does not exceed the second activation threshold, confirming that the system is currently in the first interaction stage; When the first amplitude value of the action does not exceed the first activation threshold, and the second amplitude value of the action exceeds the second activation threshold, confirming that the system is currently in the second interaction stage; The first activation threshold includes that the position change and / or rotation change of the interactive input meets the first feature; the second activation threshold includes that the position change and / or rotation change of the interactive input meets the second feature.

3. The phased interaction method in an augmented reality system according to claim 1, characterized in that: According to the relationship between the first amplitude value of the action and the second amplitude value of the action and the corresponding activation threshold, identifying the current interaction stage of the system, further comprising: When the first amplitude value of the action exceeds the first activation threshold, and the second amplitude value of the action exceeds the second activation threshold, switching the interaction stage of the system to an interaction stage different from the current interaction stage; The first activation threshold includes that the position change and / or rotation change of the interactive input meets the first feature; the second activation threshold includes that the position change and / or rotation change of the interactive input meets the second feature.

4. The phased interaction method in an augmented reality system according to claim 3, characterized in that: The switching of the interaction phase of the system to an interaction phase different from the current interaction phase includes: When the current interaction stage of the system is the first interaction stage, switching the interaction stage of the system to the second interaction stage; When the current interaction stage of the system is the second interaction stage and the action of the second specified interaction mode has been completed, the interaction stage of the system is switched to the first interaction stage.

5. The phased interaction method in an augmented reality system according to claim 2 or 3, characterized in that: The first feature includes: a change in the position and / or rotation of the interactive input relative to the interactive target; and / or, The change in the position and / or rotation of the interactive input is relative to the change characteristics of the user's interactive operation habits.

6. The phased interaction method in an augmented reality system according to claim 2 or 3, characterized in that: The second feature includes: a change in the position and / or rotation of the interactive input relative to the interactive target; and / or, The change in the position and / or rotation of the interactive input is relative to the change characteristics of the user's interactive operation habits.

7. The phased interaction method in an augmented reality system according to claim 2, characterized in that: The first specified interaction mode is a first interaction mode dynamically specified from a first interaction mode list, and the second specified interaction mode is a second interaction mode dynamically specified from a second interaction mode list; the first interaction modes in the first interaction mode list all satisfy the first condition; the second interaction modes in the second interaction mode list all satisfy the second condition, wherein the first interaction mode list and the second interaction mode list are saved in a dynamically definable interaction configuration file.

8. The phased interaction method in an augmented reality system according to claim 7, characterized in that: The first condition includes: the first interaction methods in the first interaction method list can all provide real-time position information and real-time rotation information of the corresponding interaction sources; the second condition includes: the interaction sources corresponding to the second interaction methods in the second interaction method list can all provide distinguishable postures whose number reaches a quantity threshold.

9. The phased interaction method in an augmented reality system according to claim 7, characterized in that: Under the system dynamic definition, the first designated interaction mode is the first interaction mode with the lowest fatigue value in the first interaction mode list, and the second designated interaction mode is the second interaction mode with the lowest fatigue value in the second interaction mode list; Under user dynamic definition, the first designated interaction mode is any first interaction mode selected by the user from the first interaction mode list, and the second designated interaction mode is any second interaction mode selected by the user from the second interaction mode list.

10. The phased interaction method in an augmented reality system according to claim 7, characterized in that: Different first designated interaction modes have different first activation thresholds, and different second designated interaction modes have different second activation thresholds.

11. The phased interaction method in an augmented reality system according to claim 1, characterized in that: The responding and processing the input of the interaction mode corresponding to the current interaction stage of the system includes: When the current interaction stage of the system is the first interaction stage, the target interaction object is dynamically updated according to the real-time position information and real-time rotation information generated by the interaction source of the first specified interaction mode.

12. The phased interaction method in an augmented reality system according to claim 11, characterized in that: Dynamically updating the target interactive object according to the real-time position information and real-time rotation information generated by the interactive source of the first specified interactive mode includes: Acquire real-time position information and real-time rotation information generated by the interaction source of the first specified interaction mode; Generate a stereoscopic selection space with the position related to the real-time position information of the interaction source as the starting point, the vector direction related to the real-time rotation information of the interaction source as the longitudinal extension direction, and the length data related to the preset farthest interaction distance as the longitudinal extension length; the coverage range of the stereoscopic selection space is the selection range; and the objects covered or partially covered by the selection range are candidate interaction objects; Determining whether there are multiple candidate interaction objects; When there are multiple candidate interaction objects, a target interaction object is selected from the multiple candidate interaction objects according to a preset rule.

13. The phased interaction method in an augmented reality system according to claim 12, characterized in that: The generating of the stereoscopic selection space with the position related to the real-time position information of the interactive source as the starting point, the vector direction related to the real-time rotation information of the interactive source as the longitudinal extension direction, and the length data related to the preset farthest interactive distance as the longitudinal extension length includes: A cone is generated with the real-time position of the interaction source plus the position offset as the vertex position, the real-time rotation direction of the interaction source plus the vector direction of the rotation offset as the mother line direction, the preset farthest interaction distance multiplied by the first coefficient as the height, and the preset anti-shake value multiplied by the second coefficient as the base radius.

14. The phased interaction method in an augmented reality system according to claim 13, characterized in that: The step of selecting a target interaction object from a plurality of candidate interaction objects according to a preset rule includes: Calculate the closest distance between each candidate interaction object and the generatrix of the cone, and obtain multiple closest distances accordingly; Updating the candidate interaction object corresponding to the smallest of the multiple closest distances as the target interaction object; If there are multiple candidate interaction objects corresponding to the smallest one among the multiple closest distances, the candidate interaction object closest to the user is updated as the target interaction object.

15. The phased interaction method in an augmented reality system according to claim 1, characterized in that: The responding and processing the input of the interaction mode corresponding to the current interaction stage of the system includes: When the current interaction stage of the system is the second stage, an interaction instruction for controlling the target interaction object is generated according to the input of the second specified interaction mode.

16. The phased interaction method in an augmented reality system according to claim 7, characterized in that: Also includes: Update the interaction configuration file according to the changes of the interaction source of the system.

17. The phased interaction method in an augmented reality system according to claim 16, wherein: The updating of the interaction configuration file according to the change of the interaction source of the system includes: When the interaction sources of the system are reduced, and the reduced interaction sources are interaction sources corresponding to the first specified interaction mode or the second specified interaction mode, the first interaction mode list or the second interaction mode list is updated accordingly; A new designated interaction method is selected from the updated interaction method list and the user is notified.

18. The phased interaction method in an augmented reality system according to claim 16, wherein: The updating of the interaction configuration file according to the change of the interaction source of the system includes: When the interaction sources of the system are reduced, and the reduced interaction sources are not the interaction sources corresponding to the first specified interaction mode and the second specified interaction mode, the first interaction mode list or the second interaction mode list is updated accordingly; When an interaction source of the system is increased, the interaction mode of the newly added interaction source is updated in the first interaction mode list or the second interaction mode list accordingly according to the attribute of the newly added interaction source, and the user is notified.

19. A phased interaction device in an augmented reality system, characterized in that: include: A determination module, used to obtain a first specified interaction mode configured for a first interaction stage and a second specified interaction mode configured for a second interaction stage, wherein the first interaction stage is used to update a target interaction object and the second interaction stage is used to control the target interaction object; An acquisition module, configured to acquire a first amplitude value of the action of the first specified interaction mode and a second amplitude value of the action of the second specified interaction mode; an identification module, used to identify the current interaction stage of the system according to the relationship between the first amplitude value of the action and the second amplitude value of the action and the corresponding activation threshold respectively; The response module is used to respond to and process the input of the interaction mode corresponding to the current interaction stage of the system.

20. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: When the computer program is executed by the processor, the computer program executes the instructions of the method according to any one of claims 1 to 18.

21. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor of a computer device, the computer program executes instructions of the method according to any one of claims 1 to 18.

22. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor of a computer device, the computer program executes instructions of the method according to any one of claims 1 to 18.

Images