Interaction method and system for extended reality device

Abstract

The present application relates to the field of virtual reality interactive control technology, and in particular to an interactive method and system for an extended reality device. The method is applicable to an extended reality system; the extended reality system includes an extended reality device and a physical entity that supports touch operations; the method includes: displaying a secondary function menu control required for text editing at a corresponding position of a virtual keyboard in the extended reality device according to the target interactive area and target interactive operation performed by the user on the physical entity; receiving a target secondary function instruction selected by the user from the secondary function menu control, and triggering a corresponding secondary text function operation in the virtual text interface based on the target secondary function instruction. Therefore, the present application can effectively solve the problems of lack of physical feedback, complex operation, and poor user experience when the existing extended reality system processes secondary text functions.

Description

Technical Field

[0001] The present application relates to the field of virtual reality interactive control technology, and in particular to an interactive method and system for an extended reality device. Background Art

[0002] Secondary text functions include copying, pasting, and cutting text, switching to lowercase, switching keyboard types, shortcuts for copying, pasting, cutting, and screenshots, deleting text, undoing and redoing, and positioning the text cursor. Existing extended reality (VR / AR / MR) systems mostly use gesture-based virtual keyboards to handle secondary text functions, but this approach has several drawbacks.

[0003] First, compared to physical keyboards, virtual keyboard keys lack physical feedback, making it difficult for users to tell whether they've pressed the right key. This can easily lead to incorrect operations and prevent users from receiving physical tactile feedback. Second, operations like copying, pasting, and cutting text require multiple steps, increasing operational complexity and reducing text editing efficiency. Finally, prolonged use of hover gestures can easily cause hand fatigue, impacting the user experience. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide an interactive method and system for an extended reality device, which can effectively solve at least one or more of the problems of the existing extended reality system when processing secondary text functions, such as lack of physical feedback, complex operation, and poor user experience.

[0005] In a first aspect, an embodiment of the present application provides an interactive method for an extended reality device, applicable to an extended reality system;

[0006] The extended reality system includes an extended reality device and a physical entity supporting touch operations; the method includes:

[0007] Displaying a secondary function menu control required for text editing at a corresponding position of a virtual keyboard in the extended reality device according to a target interaction area and a target interaction operation performed by the user on the physical entity;

[0008] A target secondary function instruction selected by a user from the secondary function menu control is received, and a corresponding secondary text function operation is triggered in the virtual text interface based on the target secondary function instruction.

[0009] In a second aspect, an embodiment of the present application provides an extended reality system, comprising an extended reality device and a physical entity supporting touch operations, wherein the physical entity is used for a user to perform touch operations;

[0010] The extended reality device is used to implement the interaction method of the extended reality device provided in the first aspect of the present application to process secondary text functions.

[0011] The embodiments of the present application have the following beneficial effects:

[0012] The present application is applicable to an extended reality system; the extended reality system includes an extended reality device and a physical entity that supports touch operations; according to the target interaction area and target interaction operation of the user performing the touch operation on the physical entity, the secondary function menu control required for text editing is displayed at the corresponding position of the virtual keyboard in the extended reality device; the target secondary function instruction of the user from the secondary function menu control is received, and the corresponding secondary text function operation is triggered in the virtual text interface based on the target secondary function instruction. The present application provides physical feedback to the user through the physical entity, which improves the user experience and is less prone to errors; in addition, the secondary function menu control can be displayed by identifying the interaction area and the interaction operation, and the operation steps are simple. Therefore, the present application can effectively solve the problems of lack of physical feedback, complex operation and poor user experience when the existing extended reality system handles secondary text functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0014] Figure 1a A schematic diagram of dividing the rectangular entity interaction area in the secondary text function processing method of an embodiment of the present application is shown;

[0015] Figure 1b A schematic diagram showing a rectangular entity in the secondary text function processing method of an embodiment of the present application is shown;

[0016] Figure 2 A schematic diagram of a detection space obtained by extending a rectangular entity in a secondary text function processing method according to an embodiment of the present application is shown;

[0017] Figure 3 A schematic diagram of a detection subspace obtained by extending a rectangular entity in a secondary text function processing method according to an embodiment of the present application is shown;

[0018] Figure 4 A flow chart showing a method for processing a secondary text function according to an embodiment of the present application is shown;

[0019] Figure 5A schematic diagram of an interactive scenario for displaying a keyboard language menu control in a secondary text function processing method according to an embodiment of the present application is shown;

[0020] Figure 6 A schematic diagram of an interactive scenario for displaying a capitalization menu control in a secondary text function processing method according to an embodiment of the present application is shown;

[0021] Figure 7 A schematic diagram of an interactive scenario for displaying a volume bar control in a secondary text function processing method according to an embodiment of the present application is shown;

[0022] Figure 8 A schematic diagram of an interactive scenario showing a brightness bar control in a secondary text function processing method according to an embodiment of the present application is shown;

[0023] Figure 9 A schematic diagram of an interactive scenario of selecting an option in a capitalization menu control in a method for processing a secondary text function according to an embodiment of the present application is shown;

[0024] Figure 10 A schematic diagram of an interactive scenario of adjusting the volume on a volume bar control in a secondary text function processing method according to an embodiment of the present application is shown;

[0025] Figure 11 A schematic diagram of an interactive scenario for adjusting brightness in a brightness bar control in a secondary text function processing method according to an embodiment of the present application is shown;

[0026] Figure 12 A schematic diagram of an interactive scenario for displaying a text operation shortcut menu control in a secondary text function processing method according to an embodiment of the present application is shown;

[0027] Figure 13 A schematic diagram illustrating an interactive scenario of selecting an option in a text operation shortcut menu control in a secondary text function processing method according to an embodiment of the present application is shown;

[0028] Figure 14 A schematic diagram illustrating another interactive scenario of selecting an option in a text operation shortcut menu control in a secondary text function processing method according to an embodiment of the present application is shown;

[0029] Figure 15 A schematic diagram of an interactive scenario of deleting a character in the secondary text function processing method according to an embodiment of the present application is shown;

[0030] Figure 16 A schematic diagram of an interactive scenario of deleting a segment of characters in the secondary text function processing method according to an embodiment of the present application is shown;

[0031] Figure 17A schematic diagram showing another interactive scenario of deleting a segment of characters in the secondary text function processing method according to an embodiment of the present application is shown;

[0032] Figure 18 A schematic diagram of an interaction scenario of the ESC function in the secondary text function processing method according to an embodiment of the present application is shown;

[0033] Figure 19 A schematic diagram of an interactive scenario showing the positioning of the ESC text cursor in the secondary text function processing method of an embodiment of the present application is shown.

[0034] Description of main component symbols:

[0035] 110 - top surface of the rectangular entity; 120 - side surface of the rectangular entity; 111 - upper edge of the top surface; 112 - left edge of the top surface; 130 - detection space; 131 - detection subspace. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0037] The components of the embodiments of the present application generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but rather merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0038] Hereinafter, the terms "including", "having" and their cognates used in various embodiments of the present application are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the aforementioned items, and should not be understood as excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the aforementioned items or adding the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the aforementioned items. In addition, the terms "first", "second", "third" and the like are only used to distinguish descriptions and should not be understood as indicating or implying relative importance.

[0039] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meaning as in the context of the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning unless clearly defined in the various embodiments of the present application.

[0040] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other.

[0041] In order to effectively solve the problems of lack of physical feedback, complex operation, and poor user experience when existing extended reality systems process secondary text functions, the present application provides an interaction method for an extended reality device.

[0042] This application addresses the issue of secondary text functions through interactive methods such as sliding on the edge of a physical object, simplifying the operation of secondary text functions and improving user comfort. This application aims to provide a new input method that enables rapid switching and precise control of secondary text functions through physical sliding on the edge of a physical object. This interactive method not only provides intuitive physical feedback, but also reduces the complexity of user operations, reduces visual interference and hand fatigue, and improves the overall efficiency of text editing.

[0043] The present application provides an extended reality system. Exemplarily, the system includes: the extended reality system includes an extended reality device and a physical entity that supports touch operations.

[0044] The physical entity is used for users to perform touch operations. In the embodiment of the present application, the interaction area set based on the physical entity includes multiple detection spaces, multiple touch areas, and multiple touch edges surrounding the touch areas. Users can perform touch operations in any interaction area. Each detection space includes detection subspaces.

[0045] Exemplarily, an extended reality device includes, but is not limited to, a head-mounted display (HMD). The HMD system includes a head-mounted display (HMD) with motion capture capabilities, a built-in gyroscope, and a multi-faceted entity with a touchscreen. The HMD includes a built-in camera that captures images of the physical entity. The HMD also includes a built-in processing unit that identifies user interactions on the physical entity and the interaction areas based on the images of the physical entity.

[0046] The extended reality device is used to implement the interaction method of the extended reality device in the embodiment of the present application to process the triggering of the corresponding secondary text function.

[0047] As an optional solution, the physical entity is a multi-faceted entity, and the interaction area of ​​the multi-faceted entity includes multiple detection spaces 130, multiple touch areas, multiple touch tops, and multiple touch edges.

[0048] A touch screen area is provided on the top surface of the multi-faceted entity; the touch area is a part of the touch screen area provided on the top surface of the multi-faceted entity;

[0049] The touch screen area is divided into a left half touch area and a right half touch area by a rectangular center line.

[0050] The plurality of touch zones include a left half touch zone and a right half touch zone of the touch screen area.

[0051] The multiple touch edges include the four edges of the top surface of the multi-faceted entity, namely the left edge, right edge, top edge, and bottom edge. The top edge is further divided into the upper left edge and the upper right edge by the center line of the rectangle. The bottom edge is further divided into the lower left edge and the lower right edge by the center line of the rectangle.

[0052] The plurality of touch tops include four vertex corners of a top surface of the multi-faceted entity.

[0053] The detection space is a parallelepiped space; the parallelepiped space is obtained by extending any target side surface in the polyhedral entity along the perpendicular line of the target side surface in a direction away from the physical entity by a first length.

[0054] The detection subspace is formed by extending the side surface of the target along the perpendicular line of the side surface of the target in the direction away from the physical entity by a second length, corresponding to a parallelepiped space; the first length is greater than the second length. Exemplarily, the first length = 3 cm and the second length = 0.8 cm.

[0055] This application describes a multi-faceted entity as a rounded rectangular parallelepiped (referred to as a rectangular entity). The top surface of the multi-faceted entity is the top surface of the rectangular entity. The multiple side surfaces of the multi-faceted entity are the four side surfaces of the rectangular entity. The multiple touch-sensitive tips are the four rounded edges of the rounded top surface.

[0056] Specifically, the embodiment of the present application divides the edge of the rectangular entity into 8 areas, namely the upper left edge, the upper right edge, the upper left corner (touch top), the upper right corner (touch top), the left edge, the right edge, the lower left edge, and the lower right edge. The touch area of ​​the rectangular entity includes the left half touch area and the right half touch area. Before the operation begins, the rectangular entity is positioned by the camera, and the four corners A, B, C, and D of the rectangle are determined as anchor points, such as Figure 1a Specifically, the rounded rectangle uses the center points of the four arcs at the top as anchor points. Figure 1bAs shown, the side of the rectangular entity facing the user is the top surface 110 of the rectangular entity; the side of the user's right hand is the side surface 120 of the rectangular entity; the side of the top surface far from the user is the upper edge 111 of the top surface; and the side of the top surface close to the user's left hand is the left edge 112 of the top surface.

[0057] like Figure 2 As shown, the plurality of detection spaces 130 include at least three of four parallelepiped spaces; the four parallelepiped spaces are obtained by extending four sides of a rectangular solid along perpendicular lines of the corresponding sides in a direction away from the physical solid by a first length, such that the first length is 3 cm.

[0058] like Figure 3 As shown, detection subspace 131 includes extending each of the four side faces of the rectangular entity along the perpendicular lines of the corresponding side faces in a direction away from the physical entity by a second length, thereby obtaining a corresponding parallelepiped space; the first length is greater than the second length. Exemplarily, the second length = 0.8 cm. Exemplarily, the aforementioned anchor points are used to mark the specific locations of the four corners (A, B, C, D) or the centers of the rounded corners of the rectangular entity. After the camera recognizes these anchor points, the system can clearly determine the orientation and posture of the rectangular entity in space. For example, anchor points such as the upper left corner, upper right corner, lower left corner, and lower right corner define the boundaries of the rectangle. With these anchor points, the system can accurately determine whether the user's target finger is close to a specific interaction area (such as the right edge, upper left corner, etc.) and trigger the corresponding secondary text function.

[0059] The following describes the secondary text function processing method in conjunction with some specific embodiments.

[0060] Figure 4 A flow chart of a method for processing a secondary text function according to an embodiment of the present application is shown. Exemplarily, the method for processing a secondary text function includes the following steps:

[0061] S100 , displaying a secondary function menu control required for text editing at a corresponding position of a virtual keyboard in an extended reality device according to a target interaction area and a target interaction operation performed by a user on a physical entity through a touch operation.

[0062] Furthermore, a physical entity image captured of the physical entity is acquired and recognized to obtain a target interaction area where the user performs a touch operation and a target interaction operation.

[0063] It can be understood that the extended reality device includes a camera, which is used to shoot the physical entity to capture the interactive operations performed by the user in each interactive area of ​​the physical entity.

[0064] The user can perform any interactive operation in any interactive area, and the interactive operation performed by the user in the specific interactive area can be identified through the physical entity image, that is, the target interactive area and the target interactive operation can be obtained.

[0065] The target interaction operation is an interaction operation performed by the user through the target finger in at least one of multiple detection spaces, multiple touch areas, and multiple touch edges; the target interaction area is the interaction area where the target interaction operation is performed.

[0066] S200 , receiving a target secondary function instruction selected by a user from a secondary function menu control, and triggering a corresponding secondary text function operation in a virtual text interface based on the target secondary function instruction.

[0067] Exemplarily, the user selects a target secondary function instruction to be triggered from the secondary function menu control to trigger a corresponding secondary text function operation in the virtual text interface based on the target secondary function instruction, that is, to implement a secondary text function, such as switching between uppercase and lowercase.

[0068] In one embodiment, the types of target interaction operations include a press interaction operation, a lift interaction operation, and a pause interaction operation; and each detection space includes a detection subspace.

[0069] Furthermore, in response to the target finger being within the detection subspace, it is determined that the target finger has performed a press interaction operation. Exemplarily, the target finger is the right thumb, and the detection subspace is the right detection subspace. When the vertical distance between the right thumb and the right side is between 0 cm and 0.8 cm, the right thumb is considered to have performed a press interaction operation; otherwise, the right thumb is considered to have performed a lift interaction operation.

[0070] In response to the target finger staying in the detection space for more than a preset time (for example, the preset time is 1 second), it is confirmed that the target finger performs a stay interaction operation. In response to the target finger leaving the detection subspace, it is confirmed that the target finger performs a lift interaction operation.

[0071] The present application provides a method for controlling the display and hiding of secondary function menu controls. It is understood that in step S100, based on the target interaction area and target interaction operation performed by the user on the physical entity, the secondary function menu controls required for text editing are displayed at the corresponding position of the virtual keyboard in the extended reality device, including:

[0072] S111: If it is recognized that the user performs a press interaction operation or a pause interaction operation in the target detection subspace with the target finger, a secondary function menu control is displayed at a position corresponding to the target detection subspace in the virtual keyboard.

[0073] It can be understood that the embodiment of the present application also includes: hiding the secondary function menu control in response to the target finger leaving the target detection space.

[0074] A secondary function menu control refers to an interactive menu component that provides non-core but necessary auxiliary operation functions during text input or extended reality system operation. The auxiliary functions provided by the secondary function menu control are generally not directly involved in basic character input (such as letters, numbers, etc.), but include a series of shortcuts or enhanced control methods to help users complete the input process, adjust system settings, or execute editing commands more efficiently. Exemplarily, the secondary function menu control includes but is not limited to at least one of the following: a keyboard language menu control, a capitalization menu control, a volume bar control, and a brightness bar control.

[0075] If the secondary function menu control is a keyboard language menu control, the target finger is the right thumb, the target detection subspace is the first detection subspace, the first detection subspace is a parallelepiped space extending from the right side of the rectangular entity, and the corresponding position is the right edge of the virtual keyboard, then steps S111 and S112 include the following steps:

[0076] When the right thumb performs a press interaction operation or the user's thumb is in the parallelepiped space formed by extending 3 cm vertically to the right on the right side for more than 1 second, a keyboard language menu control appears on the right edge of the virtual keyboard in the head-mounted display, such as Figure 5 When the user's thumb leaves the parallelepiped space formed by the right side extending 3 cm vertically to the right, the keyboard language menu control that appears on the right edge of the virtual keyboard disappears.

[0077] If the secondary function menu control is an uppercase and lowercase menu control, the target finger is the left thumb, the target detection subspace is the second detection subspace, the second detection subspace is a parallelepiped space extending from the left side of the rectangular entity, and the corresponding position is the left edge of the virtual keyboard, then steps S111 and S112 include the following steps:

[0078] During the case switching phase, when the user wants to switch to uppercase, lowercase, or caps lock, the user needs to place the left thumb close to the left edge. When the user's thumb is within the parallelepiped space formed by extending 3 cm vertically to the left on the left side for 1 second or the left thumb presses the interactive operation, a case menu control appears on the left edge of the virtual keyboard in the head-mounted display. For example, the case menu control is a one-dimensional case menu list, as shown below. Figure 6 If the user holds down their left thumb for less than 0.1 seconds, the uppercase and lowercase menu list will not appear on the left side of the virtual keyboard in the head-mounted display.

[0079] When the user's thumb leaves the parallelepiped space formed by the left side extending vertically 3 cm to the left, the uppercase and lowercase menu lists on the left edge of the rectangular entity disappear.

[0080] If the secondary function menu control is a volume bar control, the target finger is the right thumb, the target detection subspace is the lower right edge of the rectangular entity, and the corresponding position is the lower edge of the virtual keyboard, then steps S111 and S112 include the following steps:

[0081] By default, the thin volume bar control is abbreviated. When the user wants to adjust the volume, move the right thumb close to the lower right edge. When the user's thumb is in the parallelepiped space extending 3 cm vertically downward from the lower right side (the right half of the lower side) for more than 1 second or the distance between the user's thumb and the lower right edge is 0-0.8 cm, a thin volume bar control will appear below the virtual keyboard, as shown in the following example: Figure 7 When the user's thumb leaves the parallelepiped space formed by the lower right side extending vertically downward by 3 cm, the volume bar on the lower edge of the virtual keyboard disappears.

[0082] If the secondary function menu control is a brightness bar control, the target finger is the left thumb, the target detection subspace is the lower left edge of the rectangular entity, and the corresponding position is the lower edge of the virtual keyboard, then steps S111 and S112 include the following steps:

[0083] By default, it is a thumbnail brightness bar control. When the user wants to adjust the brightness, he or she moves the left thumb close to the lower left edge. Then, when the user's thumb is in the parallelepiped space extending 3 cm vertically downward from the lower left side (the left half of the lower side) for more than 1 second, or when the distance between the user's thumb and the lower left edge is 0-0.8 cm, the thin brightness bar control below the virtual keyboard becomes larger, as shown in the figure below. Figure 8 .

[0084] When the user's thumb leaves the parallelepiped space formed by the lower left side extending vertically downward by 3 cm, the brightness bar control at the lower edge of the virtual keyboard disappears.

[0085] Furthermore, the secondary function menu control includes a selection mark and multiple secondary function options. In order to support the selection of secondary function options and the selection of desired secondary function options in the secondary function menu control, the method of the embodiment of the present application further includes:

[0086] S113: If it is recognized that the target finger is performing a sliding interaction operation or a swinging interaction operation within the target detection subspace, the secondary function option selected in the secondary function menu control is adjusted according to at least one of the movement direction, movement length, and movement speed of the corresponding interaction operation. The movement direction is the direction in which the target finger moves when performing the corresponding interaction operation within the target detection subspace; the movement length is the length the target finger moves when performing the corresponding interaction operation within the target detection subspace; and the movement speed is the speed at which the target finger moves when performing the corresponding interaction operation within the target detection subspace.

[0087] If a slide interaction operation is detected, the secondary function option selected in the secondary function menu control is adjusted based on at least one of the slide direction, slide length, and slide speed. Since a swing interaction operation includes a slide and a lift interaction operation, if a swing interaction operation is detected, the secondary function option selected in the secondary function menu control is adjusted based on at least one of the slide direction, slide length, and slide speed of the slide interaction operation in the swing interaction operation.

[0088] For example, when the user wants to browse from bottom to top and select a secondary function option in the secondary function menu control, the user needs to slide the thumb of the right hand up and down in the detection subspace.

[0089] In response to the target finger's position change in the detection subspace exceeding a preset distance within a preset time period, it is confirmed that the target finger performs a sliding interaction operation.

[0090] In response to the target finger leaving the detection subspace after a position change in the detection subspace exceeds a preset distance within a preset time period, it is confirmed that the target finger performs a swing interaction operation.

[0091] S114: In response to the target finger transitioning from a press interaction operation to a lift interaction operation, a selected target secondary function option is determined based on a selection indicator, and a selected target secondary function instruction is determined based on the target secondary function option. Each secondary function option corresponds to one secondary function instruction. For example, the middle of the right edge of the virtual keyboard represents a selection indicator, indicating the currently selected secondary function option.

[0092] In response to the target finger leaving the detection subspace, it is confirmed that the target finger performs a lift interaction operation.

[0093] As an optional solution, in the embodiment of the present application, the secondary functions include switching keyboard types and switching uppercase and lowercase. It can be understood that the secondary function menu control is a keyboard language menu control or an uppercase and lowercase menu control; the selection mark in the secondary function menu control is a selection mark bit.

[0094] If the secondary function menu control is a keyboard language menu control, the target detection subspace is the first detection subspace; if the secondary function menu control is an uppercase and lowercase menu control, the target detection subspace is the second detection subspace. For example, the first detection subspace is the parallelepiped space extending from the right side of the rectangular entity. The second detection subspace is the parallelepiped space extending from the left side of the rectangular entity.

[0095] If the secondary function menu control is a keyboard language menu control, to facilitate the selection of the keyboard language, in step S113, if it is recognized that the target finger performs a sliding interaction operation or a swing interaction operation in the detection subspace, the secondary function option selected in the first secondary function menu control is adjusted according to at least one of the movement direction, movement length, and movement speed of the corresponding interaction operation, including:

[0096] In the first speed mode, in response to the first finger performing a sliding interaction operation in the target detection subspace, the secondary function options in the secondary function menu control are controlled to scroll through the selected identification position in sequence according to the moving direction and moving speed. When a lift interaction operation is detected, the secondary function option corresponding to the selected identification position is determined to be the target secondary function option.

[0097] Specifically, if the secondary function menu control is a keyboard language menu control, the keyboard language menu control is a keyboard language menu list, and each secondary function option is a keyboard language type. For example, the keyboard language types include simplified Chinese, traditional Chinese, English, Chinese handwriting, Korean, etc. The position of the thumb in the rectangular entity corresponds to the position of the keyboard language menu list in the virtual keyboard. At the same time, the sliding speed of the thumb is positively correlated with the sensitivity of the keyboard type switching in the keyboard language menu list. When the right thumb changes from a press interaction operation to a lift interaction operation, the system determines that the keyboard language type currently staying in the selected identification position in the keyboard language menu list is the selected object, such as Figure 5 Exemplarily, the selected indicator is the middle position in the keyboard language menu list.

[0098] Exemplarily, in response to the target finger being held down for an interactive operation longer than a first duration (e.g., 0.1 seconds), entry into the first speed mode is confirmed. The first speed mode is a slow switching mode for slowly switching between selected secondary function options in the secondary function menu control.

[0099] In the second speed mode, in response to a flick interaction operation performed with a first finger in the target detection subspace, the secondary function options in the secondary function menu control are scrolled one option position in accordance with the direction of the flick interaction operation and the order in which they are arranged in the keyboard language menu control. When a lift interaction operation is detected, the secondary function option corresponding to the selected indicator position is determined to be the target secondary function option. The flick interaction operation includes the swipe and lift interaction operations. Exemplarily, the swipe direction is the same as the switching direction.

[0100] Exemplarily, in response to the target finger being held down for an interactive operation for a duration less than or equal to the first duration (0.1 seconds), entry into the second speed mode is confirmed. The second speed mode is a fast switching mode for quickly switching between selected secondary function options in the secondary function menu control.

[0101] Specifically, if the secondary function menu control is a keyboard language menu control, when the user wants to quickly select the keyboard language type, he needs to use the right thumb to quickly slide up or down and at the same time change the right thumb from a pressing interaction operation to a lifting interaction operation, then it is determined whether to switch the keyboard language type up or down. Specifically, whether it is down or up is determined according to the sliding direction.

[0102] It can be understood that if the secondary function menu control is a capitalized menu control, in step S114, in response to the target finger switching from a press interaction operation to a lift interaction operation, the selected target secondary function option is determined according to the selection indicator, including:

[0103] In the first speed mode (slow switching mode), in response to the first finger performing a sliding interaction operation in the target detection subspace, the selected identifier is controlled to scroll on the various secondary function options according to the moving direction and moving speed. When a lift interaction operation is detected, the secondary function option corresponding to the selected identifier is determined to be the target secondary function option.

[0104] If the secondary function menu control is a capitalization menu control, for example, the capitalization menu control is a capitalization menu list, the secondary function options therein are capitalization options, such as Caps Lock, Uppercase Mode, and Lowercase Mode. Lowercase mode is the default input mode. When switching from Caps Lock to Lowercase mode, all letters entered are lowercase. Uppercase mode means that the current letter is entered in uppercase, and after entering the current letter, it returns to lowercase mode. Caps Lock mode means that the current letter entered and the letters entered thereafter are all uppercase, and they will always be uppercase if the mode is not switched.

[0105] In the second speed mode (fast switching mode), in response to the first finger performing a swing interaction operation in the target detection subspace, the selected identifier is controlled to scroll one option position in the direction of the swing interaction operation between the secondary function options. When a lift interaction operation is detected, the secondary function option corresponding to the selected identifier is confirmed to be the target secondary function option.

[0106] In the case where the secondary function menu control is a case menu control, for example, the case menu control is a case list. In response to the display of the case list, if the left thumb is pressed for less than 0.1 seconds, it is determined that the fast switching mode has been entered. In this case, it is not restricted by the case switch and the system determines that the priority is the same as the case switch.

[0107] If the user swings the thumb and the left edge upward relative to each other within 0.1 seconds, and the position of the thumb relative to the left edge changes by more than a certain value, and the left thumb changes from being pressed to being raised, the system determines that the uppercase mode is selected, such as Figure 9 If no text is entered at this time, and the above action is repeated once on the left edge, it is determined that Caps Lock mode is in effect. In either Uppercase mode or Caps Lock mode, you can swing your left thumb downward on the left edge within 0.1 seconds. When the position of your thumb relative to the left edge changes by more than a certain value, and your left thumb moves from being pressed down to being lifted, the system determines that it has returned to Lowercase mode (located at the bottom of the uppercase and lowercase lists).

[0108] As an optional solution, in the embodiment of the present application, the secondary function includes adjusting the volume and brightness. It can be understood that the secondary function menu control is a volume bar control or a brightness bar control.

[0109] If the secondary function menu control is a volume bar control, the target detection subspace is the third detection subspace. For example, the third detection subspace is the lower left edge. If the secondary function menu control is a brightness bar control, the target detection subspace is the fourth detection subspace; if the secondary function menu control is a volume bar control, the target finger is the right thumb, and the target detection subspace is the fourth detection subspace; the fourth detection subspace is the lower right edge of the rectangular entity. Different secondary function options correspond to different progress ratio values. For step S113, if it is recognized that the target finger performs a sliding interaction operation or a swinging interaction operation in the detection subspace, the secondary function option selected in the secondary function menu control is adjusted according to at least one of the moving direction, moving length and moving speed, and it also includes:

[0110] In response to the first finger performing a sliding interaction operation in the target detection subspace, the size of the progress scale value in the secondary function menu control is adjusted according to the moving direction, and the volume or brightness is set according to the current progress scale value.

[0111] Specifically, if the secondary function menu control is a volume bar control, after turning on the volume adjustment switch, the user can adjust the volume by sliding the thumb of the right hand on the lower right edge. When the user's thumb is placed on the lower right edge, according to a certain ratio, moving it to the left increases the volume, and moving it to the right decreases the volume, such as Figure 10 If the secondary function menu control is a brightness bar control, after turning on the brightness adjustment switch, the user can adjust the brightness by sliding the left thumb on the lower left edge. When the user's thumb is placed on the lower left edge, according to a certain ratio, moving it to the left will reduce the brightness, and moving it to the right will increase the brightness. Figure 11 .

[0112] In step S114, in response to the target finger being converted from a press interaction operation to a lift interaction operation, determining the selected target secondary function option according to the selection indicator includes:

[0113] In response to the first finger performing a lift interaction operation in the target detection subspace, the progress scale value in the secondary function menu control is controlled to stop changing.

[0114] In step S111, if it is recognized that the user performs a press interaction operation or a dwell interaction operation in the target detection subspace with the target finger, a secondary function menu control is displayed at a position corresponding to the target detection subspace in the virtual keyboard, including:

[0115] If it is recognized that the user has performed a press interaction operation or a pause interaction operation in the target detection subspace using a first finger, a secondary function menu control in a second display state is displayed at the corresponding position in the target detection subspace on the virtual keyboard. Specifically, if the secondary function menu control is a volume bar control or a brightness bar control, its initial display state is the first display state, which is a thin, short volume bar control or brightness bar control in an abbreviated state. The secondary function menu control in the second display state is a wide, long volume bar control or brightness bar control.

[0116] As an optional solution, in the embodiment of the present application, the secondary function includes a text operation shortcut function. The text operation shortcut function includes but is not limited to functions such as copy, paste, cut, and screenshot. It can be understood that based on the target interaction area and target interaction operation performed by the user on the physical entity, the secondary function menu control required for text editing is displayed at the corresponding position of the virtual keyboard in the extended reality device, including:

[0117] When the virtual keyboard is in a non-input state, in response to at least one second finger performing a drag operation on the first touch edge, a secondary function menu control is displayed using the third or fourth method; illustratively, the secondary function menu control is a text operation shortcut menu control. The text operation shortcut menu control includes functions such as copy, paste, cut, and screenshot. The third method is to display the text operation shortcut menu control while hiding the virtual keyboard. The fourth method is to display the text operation shortcut menu control next to the virtual keyboard. The drag operation is to move at least one finger along the edge of the rectangular entity in a first direction for a set distance within a set time range. For example, the first direction is toward the endpoint of the rectangular entity.

[0118] Specifically, the second finger can be the index finger. The first touch edge is the upper edge of the rectangular entity. When the user wants to quickly copy, paste, cut or screenshot, he can use at least one index finger to pull outward at least 3cm from the upper edge of the rectangular entity, then the system determines to display the text operation shortcut menu control and hide the virtual keyboard, such as Figure 12 shown.

[0119] It should be noted that when the user is using the keyboard to input letters, the text operation shortcut menu control cannot be opened by dragging the control. If the text operation shortcut menu control is opened and then input is used using the virtual keyboard, the text operation shortcut menu control will automatically close immediately.

[0120] Furthermore, the method of the embodiment of the present application also includes: in response to at least one second finger performing a pull-back operation on the first touch edge, correspondingly hiding the secondary function menu control using the third method or the fourth method. The pull-back operation is to move a set distance in a second direction on the edge of the rectangular entity by at least one finger within a set time range. For example, the second direction is the middle direction of the currently touched edge of the rectangular entity. If the secondary function menu control is a text operation shortcut menu control, specifically, if the index finger is pulled back to the middle of the touch edge by at least 3 cm, the system determines that the text operation shortcut menu control is closed and the keyboard is restored.

[0121] Furthermore, the embodiment of the present application also includes a method for selecting a shortcut function in a text operation shortcut menu control. It can be understood that in one embodiment, when the secondary function menu control is in a displayed state, in response to the target finger performing a sliding operation on the first touch area, the selection mark in the secondary function menu control is controlled to switch to the next secondary function option according to the sliding direction; in response to the target finger being lifted from the first touch area, the secondary function option corresponding to the selection mark is confirmed to be the target secondary function option, and the virtual keyboard is displayed. The target finger can be a thumb or other finger, and this application does not impose any restrictions on this.

[0122] The first touch area is a rectangular solid touch screen area. The sliding operation is to slide the finger on the touch screen area to a set distance within a preset time. Specifically, if the secondary function menu control is a text operation shortcut menu control, the selected mark is highlighted. After the text operation shortcut menu control is displayed, the user can use the thumb (or other finger) to slide on the touch screen, such as Figure 13 When the touchscreen detects the entry of a finger, the first option is highlighted by default. Each time the finger swipes left or right by 0.5 cm, a new option in the corresponding direction is highlighted. When the thumb (or other finger) is lifted, the highlighted option becomes the target object and the corresponding function, such as taking a screenshot, is triggered, and the virtual keyboard is restored.

[0123] In another embodiment, when the secondary function menu control is in a displayed state, the first touch edge is divided into hidden touch segments and function option touch segments corresponding to each secondary function option according to the number and order of the secondary function options in the secondary function menu control; for example, the first touch edge is the upper edge of the rectangular entity.

[0124] In response to the target finger sliding on the first touch edge to any function option touch segment, confirming the selection of a secondary function option corresponding to any function option touch segment;

[0125] In response to the target finger sliding on the first touch edge to the hidden touch segment, the secondary function menu control is hidden.

[0126] Specifically, if the secondary function menu control is a text operation shortcut menu control, after opening the text operation shortcut menu control, the finger can leave the upper edge. At this time, the system will hover the text operation shortcut menu control above the virtual keyboard and it will not disappear. During specific operations, the user can slide one finger on the upper edge of the rectangular entity to select an option in the text operation shortcut menu control. At this time, the upper edge is divided into several areas according to the number of options except for a certain area on the leftmost side, and the secondary function options are obtained. Different absolute positions of the finger on the upper edge correspond to different secondary function options. When the finger is released, it is determined that the secondary function option corresponding to the position is selected, such as Figure 14 When you slide to the leftmost hidden touch segment on the top edge, the system determines it as a hidden text operation shortcut menu control.

[0127] As an optional solution, the secondary function in the embodiment of the present application also includes a text deletion function. It can be understood that the method of the embodiment of the present application also includes:

[0128] (1) In response to the third finger pressing on the touch area and the fourth finger clicking on the top of the first touch, a preset number of characters before the cursor position are deleted; for example, the top of the first touch is the top of the upper right corner of the rectangular entity. The third finger and the fourth finger are either fingers of the left hand or the right hand. Exemplarily, if the user wants to delete the previous character, he needs to keep one finger pressed on the touch screen (it can be the left hand or the right hand), and then use another finger to press the top of the upper right corner and then lift it up (one click), then one click deletes one character, such as Figure 15 shown.

[0129] (2) In response to at least one third finger performing a sliding operation in the touch area, and a fourth finger pressing the top of the first touch, the selected text range is determined according to the sliding direction and sliding distance of the sliding operation and the position of the cursor; for example, the fourth finger is a right-hand finger, and the top of the first touch is still the top of the upper right corner of the rectangular entity.

[0130] (3) When it is detected that at least one third finger has left the touch area, the text corresponding to the text range is deleted.

[0131] For example, after deleting a character using the above step (1), if the user wants to delete continuously, he needs to keep the right index finger at the top of the upper right corner without lifting it, and at the same time, move the other finger to the left or right on the touch screen to select a range, so as to delete the entire section, such as Figure 16 The user's other finger moves left, right, up, or down on the touch screen, corresponding to the left, right, up, or down position of the deletion cursor. After determining the selected text range, the user can also move the finger on the touch screen back to the original cursor position to cancel the selected text range.

[0132] In one way of confirming deletion, the operation at this time is only related to whether the other finger on the touch screen is lifted, and has nothing to do with whether the user's right index finger at the top of the upper right corner is lifted. Only after detecting that the user's finger on the touch screen is lifted, the deletion of the selected text range content will be confirmed. In another way of confirming deletion, if the user turns on range deletion and then places a second finger on the touch screen, the second finger that enters the touch screen can also control the position of the deletion cursor. At this time, the user can execute the deletion command only after lifting both fingers to delete the content corresponding to the selected text range, such as Figure 17 .

[0133] As an optional solution, the secondary function in the embodiment of the present application also includes an exit function, for example, the function corresponding to the ESC key (hereinafter referred to as the ESC function). It can be understood that the method of the embodiment of the present application also includes:

[0134] In response to a third finger pressing on the touch area and a fourth finger clicking the top of the second touch control, an exit function is triggered to exit the currently used application. Exemplarily, the third finger can be any finger on the left hand, and the fourth finger can be any finger on the right hand, but this application does not impose any limitation. The touch area is the left half of the touch area of ​​the touch screen. The top of the second touch control can be the top of the upper left corner of the rectangular entity.

[0135] If the user wants to exit the currently opened web page or exit the full-screen software, the ESC function can be used. The ESC function is triggered when the user presses one finger on the left half of the touch screen. At the same time, the left finger presses down on the top of the upper left corner and then lifts up the upper left corner (a single click). If the left finger moves away from the upper left corner after pressing, it is not considered a single click. A single click is considered to send an ESC command to the system. Figure 18 .

[0136] As an optional solution, the secondary function in the embodiment of the present application includes a cursor positioning function. It can be understood that the method of the embodiment of the present application also includes:

[0137] In response to the third finger and the second target hand performing a double-click operation simultaneously in two interaction areas belonging to the same type, the user's gaze point is captured and the text cursor is moved to the gaze point.

[0138] For example, when a user wants to position a text cursor in a text editing area, the user first looks at the desired location. The head mounted display device captures the location of the eye gaze through eye movements, such as Figure 19 At this point, the user performs any of the following double-click operations to position the text cursor at the location where the eye is currently looking. Double-click operations include any of the following:

[0139] a. Use your left and right index fingers to double-click the top left and top right corners of the rectangle simultaneously.

[0140] b. Use your left and right index fingers to double-click on the upper left and right edges of the rectangle entity simultaneously;

[0141] c. Use your left and right index fingers to double-click on the left and right edges of the rectangle entity simultaneously;

[0142] d. Use your left and right thumbs to double-click simultaneously on the top of the lower left and right corners of the rectangle.

[0143] e. Use your left and right thumbs to double-click on the lower left and right edges of the rectangle entity simultaneously;

[0144] f. Use your left and right thumbs to double-click on the left and right edges of the rectangle entity at the same time.

[0145] This application has the following advantages:

[0146] This application applies physical object edge sliding technology to the secondary text function of the keyboard in VR / AR / MR, which can greatly enhance the user's interactive experience. For functions such as switching between uppercase and lowercase letters and switching between keyboard languages, operations are performed by sliding the physical edge to provide a more intuitive and smooth interaction method. Compared with traditional virtual button clicks or gesture operations, this method combines tactile feedback with the naturalness of physical sliding, which helps reduce misoperations, especially in complex virtual scenes.

[0147] 1. Enhanced tactile feedback: Existing virtual keyboards lack physical feedback, making it easy for users to make mistakes when performing secondary text functions, such as switching between uppercase and lowercase, switching keyboard languages, selecting text, moving the text cursor, deleting text, undoing and redoing. This application uses physical edge sliding to give users a physical sense of touch when performing these operations, allowing them to clearly perceive the start and end of the operation, improve control accuracy, and reduce accidental touches.

[0148] 2. Improved Operational Efficiency: This application uses edge swipes as shortcuts to quickly complete secondary text functions such as switching between uppercase and lowercase, switching keyboard languages, selecting text, moving the text cursor, deleting text, and undoing and redoing. Users no longer need to navigate through submenus as before. This application has fewer steps and is more intuitive, reducing operation time and improving work efficiency, which is particularly advantageous in high-frequency usage scenarios.

[0149] 3. Reduced fatigue: In traditional VR / AR / MR operations, user gestures or controllers hovering in the air can easily cause fatigue. Solid Edge Slide relies on physical support to reduce hand-in-the-air time and strain, keeping users comfortable during extended use and suitable for long-term input tasks.

[0150] 4. Improved precision: The physical path of sliding in this application is fixed, and users can precisely control cursor movement and text selection.

[0151] Compared with virtual button clicks, the sliding path is clear and predictable, which greatly reduces the randomness of gesture input and improves the accuracy of input and the stability of operation.

[0152] 5. Operational portability: Since physical edge sliding does not require complex peripherals or gesture recognition, users can directly use their mobile phones as devices, which is easy to carry and can be flexibly used in various scenarios. It is especially suitable for mobile scenarios that require high-frequency text input, such as mobile office or remote work.

[0153] 6. Adaptability to multiple platforms: This sliding interaction method can be easily adapted to different devices, including VR, AR, and MR devices, and may even be expanded to new terminals such as smart watches and AR glasses in the future. It has a wide range of applications and can be implemented without major changes to device hardware, thereby reducing development and application costs and having the advantage of being cross-platform.

[0154] 7. Reduced visual interference: Virtual keyboards usually occupy the user's field of view, especially in dense interaction scenarios, affecting the user's overall experience. When users need to complete the input of secondary text functions, they need to find the location of the relevant function keys. However, this application proposes that by sliding the physical edge, keyboard function operations can be implemented without taking up additional screen space, ensuring that the user's field of view is clear and the operation experience is more focused.

[0155] 8. Strong adaptability: This application can adapt to a variety of text input tasks, not only for general text editing, but also for complex scenarios such as symbol input and program code editing. Its flexibility allows users to obtain a smooth interactive experience in various input scenarios of VR / AR / MR, improving the wide adaptability of text input.

[0156] An embodiment of the present application provides a schematic diagram of a structure of an interactive device for an extended reality device. Exemplarily, the interactive device for the extended reality device includes:

[0157] The menu trigger module displays the secondary function menu controls required for text editing at the corresponding position of the virtual keyboard in the extended reality device based on the target interaction area and target interaction operation performed by the user on the physical entity;

[0158] The function execution module is used to receive a target secondary function instruction selected by a user from the secondary function menu control, and trigger a corresponding secondary text function operation in the virtual text interface based on the target secondary function instruction.

[0159] It can be understood that the apparatus of this embodiment corresponds to the interaction method of the extended reality device of the above embodiment, and the optional items in the above embodiment are also applicable to this embodiment, so they will not be repeated here.

[0160] The present application also provides a terminal device. Exemplarily, the terminal device includes a processor and a memory, wherein the memory stores a computer program, and the processor runs the computer program to enable the terminal device to execute the functions of each module in the above-mentioned interaction method of the extended reality device or the interaction device of the above-mentioned extended reality device.

[0161] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, etc., and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0162] The memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM). The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving an execution instruction.

[0163] This application also provides a computer-readable storage medium for storing the computer program used in the terminal device. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or flowchart, and the combination of boxes in the structure diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0165] In addition, the functional modules or units in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0166] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a smart phone, personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0167] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. An interactive method for an extended reality device, characterized in that: Applicable to an extended reality system; the extended reality system includes an extended reality device and a physical entity supporting touch operations; the method includes: Displaying a secondary function menu control required for text editing at a corresponding position of a virtual keyboard in the extended reality device according to a target interaction area and a target interaction operation performed by the user on the physical entity; receiving a target secondary function instruction selected by a user from the secondary function menu control, and triggering a corresponding secondary text function operation in the virtual text interface based on the target secondary function instruction; The interaction area set based on the physical entity includes multiple detection spaces; the types of the target interaction operations include a press interaction operation and a stay interaction operation; each of the detection spaces includes a detection subspace; the secondary function menu controls required for text editing displayed at the corresponding position of the virtual keyboard in the extended reality device based on the target interaction area and the target interaction operation performed by the user on the physical entity include: If it is recognized that the user performs the press interaction operation or the dwell interaction operation in the target detection subspace with the target finger, displaying the secondary function menu control at a position corresponding to the target detection subspace in the virtual keyboard; The method further includes hiding the secondary function menu control in response to the target finger leaving the target detection space.

2. The interactive method of an extended reality device according to claim 1, characterized in that: The secondary function menu control includes a selection mark and a plurality of secondary function options; the method further includes: If it is recognized that the target finger performs a sliding interaction operation or a swinging interaction operation in the target detection subspace, adjusting the secondary function option selected in the secondary function menu control according to at least one of a movement direction, a movement length, and a movement speed of the corresponding interaction operation; In response to the target finger being converted from the press interaction operation to the lift interaction operation, a selected target secondary function option is determined according to the selection mark, and the selected target secondary function instruction is determined based on the target secondary function option.

3. The interactive method of an extended reality device according to claim 2, characterized in that: If it is recognized that the target finger performs a sliding interaction operation or a swinging interaction operation in the target detection subspace, adjusting the secondary function option selected in the secondary function menu control according to at least one of a movement direction, a movement length, and a movement speed of the corresponding interaction operation, includes at least one of the following two items: Item 1: The secondary function menu control is a keyboard language menu control; the selected mark in the secondary function menu control is a selected mark bit; If the secondary function menu control is a keyboard language menu control, the target detection subspace is the first detection subspace; otherwise, the target detection subspace is the second detection subspace; In a first speed mode, in response to a first finger performing the sliding interaction operation in the target detection subspace, the secondary function options in the secondary function menu control are controlled to scroll sequentially through the selected identification position according to the movement direction and the movement speed, and when the lift interaction operation is detected, the secondary function option corresponding to the selected identification position is determined to be the target secondary function option; In the second speed mode, in response to a flick interaction operation being performed with the first finger in the target detection subspace, the secondary function options in the secondary function menu control are scrolled one option position according to the direction of the flick interaction operation and the order of arrangement in the keyboard language menu control, and when the lift interaction operation is detected, the secondary function option corresponding to the selected indicator is determined to be the target secondary function option; Item 2: The secondary function menu control is a volume bar control or a brightness bar control; if the secondary function menu control is a volume bar control, the target detection subspace is the third detection subspace, otherwise the target detection subspace is the fourth detection subspace; different secondary function options correspond to different progress ratio values; In response to the first finger performing the sliding interaction operation in the target detection subspace, adjusting the size of the progress scale value in the secondary function menu control according to the movement direction, and setting the volume or brightness according to the current progress scale value; Furthermore, in response to the target finger being converted from the pressing interaction operation to the lifting interaction operation, determining the selected target secondary function option according to the selection mark includes: In response to the first finger performing the lift interaction operation in the target detection subspace, controlling the size of the progress ratio value in the secondary function menu control to stop changing.

4. The interactive method of an extended reality device according to claim 3, characterized in that: The secondary function menu control is an uppercase and lowercase menu control; the selected mark in the secondary function menu control is a selected mark bit; If the secondary function menu control is the uppercase and lowercase menu control, the target detection subspace is the second detection subspace; In response to the target finger being converted from the pressing interaction operation to the lifting interaction operation, determining the selected target secondary function option according to the selection mark includes: In the first speed mode, in response to a first finger performing the sliding interaction operation in the target detection subspace, the selected marker is controlled to scroll on each secondary function option according to the movement direction and the movement speed, and when the lift interaction operation is detected, the secondary function option corresponding to the selected marker is determined to be the target secondary function option; In the second speed mode, in response to the first finger performing the swing interaction operation in the target detection subspace, the selected identifier is controlled to scroll one option position in the moving direction of the swing interaction operation between the secondary function options, and when the lift interaction operation is detected, it is confirmed that the secondary function option corresponding to the selected identifier is the target secondary function option.

5. The interactive method of an extended reality device according to claim 2, characterized in that: The secondary function menu control is a volume bar control or a brightness bar control, and the initial display state is the first display state; if the secondary function menu control is a volume bar control, the target detection subspace is the third detection subspace, otherwise the target detection subspace is the fourth detection subspace; If it is recognized that the user performs the press interaction operation or the dwell interaction operation in the target detection subspace with the target finger, displaying the secondary function menu control at a position corresponding to the target detection subspace in the virtual keyboard includes: If it is recognized that the user performs the press interaction operation or the stay interaction operation in the target detection subspace with a first finger, a secondary function menu control in a second display state is displayed at a position corresponding to the target detection subspace in the virtual keyboard.

6. The interactive method of an extended reality device according to claim 1, characterized in that: The interaction area set based on the physical entity includes a touch area and a plurality of touch edges surrounding the touch area; The method of displaying a secondary function menu control required for text editing at a corresponding position of a virtual keyboard in the extended reality device according to a target interaction area and a target interaction operation performed by the user on the physical entity through a touch operation includes: When the virtual keyboard is in a non-input state, in response to at least one second finger performing a drag operation on the first touch edge, displaying the secondary function menu control in a third manner or a fourth manner; The method also includes: in response to at least one second finger performing a pull-back operation on the first touch edge, correspondingly hiding the secondary function menu control using a third method or a fourth method; wherein the secondary function menu control includes a text operation shortcut menu control; the third method is to display the text operation shortcut menu control while hiding the virtual keyboard; the fourth method is to display the text operation shortcut menu control next to the virtual keyboard.

7. The interactive method of an extended reality device according to claim 1, characterized in that: The method further comprises at least one of the following three items: Item 1: The interaction area set based on the physical entity includes a touch area; the interaction area of ​​the physical entity also includes a plurality of touch tips; In response to a third finger pressing on the touch area and a fourth finger clicking on the top of the first touch, deleting a preset number of characters before the cursor position; In response to at least one of the third fingers performing a sliding operation on the touch area and the fourth finger holding down the top of the first touch, determining a selected text range according to the sliding direction and sliding distance of the sliding operation and the position of the cursor; When it is detected that the at least one third finger leaves the touch area, deleting the text corresponding to the text range; Item 2: The interaction area set based on the physical entity includes a touch area and a plurality of touch tips; the interaction area of ​​the physical entity also includes a plurality of touch tips; In response to a third finger pressing on the touch area and a fourth finger clicking on the top of the second touch area, triggering an exit function to exit the currently used application; Item 3: In response to the third finger and the second target hand performing a double-click operation simultaneously in two interaction areas of the same type, the user's gaze point is captured and the text cursor is moved to the gaze point.

8. The interactive method of an extended reality device according to claim 1 or 5, characterized in that: The physical entity is a multifaceted entity; The detection space is a parallelepiped space; the parallelepiped space is obtained by extending any target side surface of the polyhedral entity along a perpendicular line of the target side surface in a direction away from the physical entity by a first length, thereby obtaining a corresponding parallelepiped space; The detection subspace is formed by extending the side surface of the target along a perpendicular line of the side surface of the target in a direction away from the physical entity by a second length, thereby obtaining a corresponding parallelepiped space; the first length is greater than the second length; The touch area is a part of the touch area in the touch screen area provided on the top surface of the multi-faceted entity; The plurality of touch edges include four sides of the top surface; The plurality of touch tops include four corners of the top surface.

9. An extended reality system, characterized in that: The extended reality system includes an extended reality device and a physical entity supporting touch operations, wherein the physical entity is used for a user to perform touch operations; The extended reality device is used to implement the interaction method of the extended reality device as described in any one of claims 1 to 8 to process a secondary text function.

Images