Interface display method of XR space, XR equipment and storage medium

Abstract

The invention provides an XR space interface display method, XR equipment and a storage medium. The method is applied to an XR device and comprises the steps that a first application window and a second application window are displayed in an XR space, the first application window is used for displaying a program interface of a first application, and the second application window is used for displaying a program interface of a second application; a user can drag the first application window to the second application window, so that the first application window and the second application window are interlocked to form an application window group. The application window group comprises two windows which are respectively used for displaying program interfaces of the first application and the second application. Therefore, the user can complete the creation of the application window group through simple operation, and each window in the application window group is displayed around the user, so that the user can conveniently use the plurality of application programs at the same time.

Description

XR space interface display methods, XR devices and storage media

[0001] This application relates to the field of smart terminal technology, and in particular to a display method, XR device and storage medium for XR space.

[0002] XR (Extended Reality) technology is an emerging graphics and image-related technology that has appeared in recent years. XR technology is a collective term for various technologies such as AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality). It can create a virtual scene that supports human-computer interaction, providing users with visual, auditory, and other sensory experiences that simulate real scenes, making users feel as if they are actually there.

[0003]

[0004] This application provides a display method, XR device, and storage medium for XR space. In this method, users can quickly and efficiently perform multi-window operations in XR space.

[0005] Firstly, this application provides a method for displaying an interface in an XR space. The method includes:

[0006] The XR device displays a first application window and a second application window in the XR space; wherein, the first application window is used to display the program interface of the first application, and the second application window is used to display the program interface of the second application.

[0007] In response to an operation that drags the first application window toward the second application window, the XR device displays the first control;

[0008] In response to a first operation on a first control, the XR device displays a first application window group in XR space; wherein the first application window group includes a first window and a second window, the first window is displayed at a first position in the first application window group and is used to display the program interface of the first application; the second window is displayed at a second position in the first application window group and is used to display the program interface of the second application.

[0009] The first application window and the second application window are both independently displayed XR objects, as shown in application window 903 and application window 902 in Figure 9b (1); the first application window group is an XR object generated based on the first application window and the second application window, as shown in application window group 905 in Figure 9b (3).

[0010] For example, the first control can be an interlock confirmation control, displayed below the first application window and the second application window, as shown in Figure 9b (2), to prompt the user whether to interlock the first application window and the second application window to generate an application window group.

[0011] For example, when a user interacts with an XR interface in a virtual space using hand-eye control, the first operation could be to look at the screen and pinch with two fingers.

[0012] In this embodiment, the first window (e.g., window 9052) and the second window (e.g., window 9051) can be understood as two sub-windows included in the first application window group, displayed around the user. The distance between each sub-window and the user can be a fixed value of 2, as mentioned below, for example, 1.5m.

[0013] In this way, users can create application window groups simply by dragging and dropping application windows and confirming. The operation is simple and convenient, which not only makes it easier for users to use multiple applications at the same time, but also eliminates the need for users to adjust the display position of each window individually.

[0014] According to the first aspect, the first column of the first application window includes: a first control and a second control; the first control is used to minimize the first application window, and the second control is used to close the first application window;

[0015] The second column of the first application window group includes a third control and a fourth control; the third control is used to minimize the first application window group, and the fourth control is used to close the first application window group.

[0016] The third column of the first window includes a fifth control; the third control is used to minimize the first window within the first application window group.

[0017] The first column of the first application window refers to the application bar of the first application window, which can be seen in the application bar 8031 ​​shown in Figure 8b (1). The first control can refer to the window minimize control 80312 in the application bar 8031, and the second control can refer to the window close control 80314 in the application bar 8031.

[0018] Similarly, the application bar of the second application window includes a window minimize control and a window close control.

[0019] The second column of the first application window group refers to the application bar of the first application window group, which can be seen in the window group application bar 9015 shown in Figure 9a. The third control can be referred to as the window group minimize control in the window group application bar 9015, and the fourth control can be referred to as the window group close control in the window group application bar 9015.

[0020] The third column of the first window refers to the application bar of the first window, which can be seen in the window minimize control 90111 of window 9011 shown in Figure 9a. Similarly, the application bar of the second window also includes a window minimize control.

[0021] Before the first application window and the second application window are interlocked to form the first application window group, each application window's application bar includes a minimize control and a close control. At this time, the user can minimize or close any individual application window. After the first application window and the second application window are interlocked to form the first application window group, the application bar of the first application window group includes a group minimize control and a group close control. The user can minimize or close the entire first application window group. At this point, each window in the first application window group only includes a minimize control in its window bar; it no longer includes a close control. Therefore, the user cannot close any individual window in the application window group; they can only minimize it, and this minimization occurs within the application window group, unlike minimizing a single application window (such as the first application window).

[0022] According to the first aspect, or any implementation of the first aspect above, in the XR space, the windows included in the first application window group are displayed around the user, and each window is equidistant from the user; in the first application window group, the first window faces directly in front of the user.

[0023] In this embodiment, when a user drags and drops an application window to create an application window group, the program interface of the application window (i.e., the first application window) dragged by the user is displayed in the window directly in front of the user in the application window group, so as to provide the user with a better visual experience and meet the user's usage needs; each window is displayed around the user, and each window is equidistant from the user, which makes it easy for the user to use each window and simplifies the interaction.

[0024] According to the first aspect, or any implementation of the first aspect above, the method further includes:

[0025] In response to a second operation on the second window, the XR device cancels the display of the second window in the first application window group and displays the icon of the second application in the second column of the first application window group;

[0026] In response to a third operation on the icon of the second application displayed in the second column, the XR device redisplays the second window in the first application window group and removes the icon of the second application from the second column.

[0027] For example, the second operation on the second window can be the second operation on the window minimize control in the application bar of the second window. Taking the interaction between the user and the XR interface in the virtual space through hand and eye control as an example, the second operation and the third operation can be, for example, the operation of looking and pinching with two fingers.

[0028] Referring to Figure 12a, users can minimize an application window within an application window group to avoid negatively impacting their visual experience due to an excessive number of application windows displayed in the virtual space. After minimizing a window within the application window group, users can also re-enable that window, displaying it again within the application window group. This allows users to flexibly adjust the windows displayed within the application window group according to their actual needs.

[0029] It's important to note that minimizing a window within an application window group is different from minimizing a standalone application window. When the user performs the latter operation, the corresponding application icon is added to the minimized application area, while when the user performs the former operation, the corresponding application icon is added to the application bar of the application window group.

[0030] According to the first aspect, or any implementation of the first aspect above, when the XR device redisplays the second window in the first application window group, the method further includes:

[0031] In the first application window group, the XR device displays a second window at a first position and moves the first window to the second position for display.

[0032] In this embodiment, the window displayed in the first position within the first application window is positioned directly in front of the user. Considering that the window the user reawakens in the application bar of the window group is the one the user needs at the moment, the restored window is displayed in the center of the application window bar, that is, directly in front of the user, which is beneficial for user use. Since the restored application window takes up the display space of other application windows, the display positions of the other application windows can be adjusted adaptively, for example, moved to any available placement position.

[0033] According to the first aspect, or any implementation of the first aspect above, the method further includes:

[0034] In response to a fourth operation on the first window, the XR device cancels the display of the first window in the first application window group, displays the icon of the first application in the second column of the first application window group, and moves the second window to the first position for display.

[0035] For example, the fourth operation on the first window can be the fourth operation on the window minimize control in the application bar of the first window. Taking the interaction between the user and the XR interface in the virtual space through hand and eye control as an example, the fourth operation can be, for example, the operation of looking at and pinching with two fingers.

[0036] In this embodiment, the first window is the window located directly in front of the user in the first application window group. After the user minimizes this window, the other windows in the application window group can be automatically moved to be displayed directly in front of the user, thereby ensuring that there is always a window displayed in the best viewing position of the user in the application window group, improving the user's visual experience and avoiding the waste of XR display space.

[0037] According to the first aspect, or any implementation of the first aspect above, the method further includes:

[0038] In response to the fifth operation, the XR device rotates the first application window group by a preset angle centered on the user, so that the window facing directly in front of the user changes from the first window to the second window.

[0039] Referring to the examples in Figures 10a and 10b, users can adjust the orientation of application window groups.

[0040] For example, the application window group can rotate clockwise or counterclockwise around the user. For instance, the application window group can rotate around the user by a preset angle to adjust the application window facing directly in front of the user. This allows users to flexibly adjust the orientation of the application window group according to their actual needs, improving the user's visual experience.

[0041] According to the first aspect, or any implementation of the first aspect above, the method further includes:

[0042] In response to a sixth operation on the second window, the XR device moves the second window to a first position within the first application window group and displays it, and moves the first window to a second position.

[0043] Alternatively, in response to a seventh operation on the second window, the XR device moves the second window to a third position within the first application window group for display.

[0044] For example, the sixth and seventh operations on the second window can be gaze and pinch-and-drag operations, as shown in the examples in Figures 10c and 10d.

[0045] In this way, users can flexibly adjust the order of windows in the application window group according to their actual needs, improving the convenience of using multiple windows at the same time, and also allowing the multi-window order in the application window group to meet the user's personal habits of using multiple windows at the same time.

[0046] According to the first aspect, or any implementation of the first aspect above, the method further includes:

[0047] In response to the eighth operation on the first application window group, the XR device cancels the display of the first application window group in the XR space and displays the third application window and the fourth application window in the XR space respectively; wherein, the third application window is used to display the program interface of the first application, and the fourth application window is used to display the program interface of the second application.

[0048] For example, the eighth operation can be the operation of ungrouping the application window as shown in FIG11a.

[0049] In this way, when users no longer need to use the application window group, they can disassemble it into multiple independent application windows, which can meet the user's personalized needs.

[0050] According to the first aspect, or any implementation of the first aspect above, the fourth column of the third application window includes: a sixth control and a seventh control; the sixth control is used to minimize the third application window, and the seventh control is used to close the third application window.

[0051] The fourth column of the third application window refers to the application bar of the third application window, the sixth control can be referred to as the window minimize control in the application bar, and the seventh control can be referred to as the window close control in the application bar.

[0052] Similarly, the application bar of the fourth application window includes a window minimize control and a window close control.

[0053] Thus, before the first window group is disbanded (refer to (1) in Figure 11a), each window in the first application window group only has a minimize control in its window bar, not a close control. Therefore, the user cannot close any individual window in the application window group; they can only minimize it, and this minimization occurs within the application window group, unlike minimizing a single application window (such as the first application window). After the first window group is disbanded (refer to (3) in Figure 11a), each independently existing application window has both a minimize control and a close control in its application bar. At this point, the user can individually minimize or close any application window.

[0054] According to the first aspect, or any implementation of the first aspect above, the method further includes:

[0055] The XR device displays a fifth application window and a group of first application windows in XR space; the fifth application window is used to display the program interface of the third application.

[0056] The XR device displays the second control in response to an operation that drags the fifth application window toward the second application window group;

[0057] In response to a ninth operation on the second control, the XR device displays a second application window group in XR space; wherein the second application window group includes a third window, a fourth window and a fifth window, the third window is used to display the program interface of the third application, the fourth window is used to display the program interface of the first application and the fifth window is used to display the program interface of the second application.

[0058] Among them, the fifth application window is an independently displayed XR object, which can be referred to as application window 906 in Figure 9e (1); the first application window group is also an independently displayed XR object, which can be referred to as application window group 905 in Figure 9e (1).

[0059] For example, the second control can be an interlock confirmation control, displayed below the fifth application window and the first application window group, as shown in Figure 9e (2), to prompt the user whether to interlock the fifth application window and the first application window group to generate a new application window group.

[0060] For example, when a user interacts with an XR interface in a virtual space using hand-eye control, the ninth operation could be a gaze and a pinching motion with two fingers.

[0061] In this embodiment, the third window (e.g., window 9071), the fourth window (e.g., window 9072), and the fifth window (e.g., window 9073) can be understood as three sub-windows included in the second application window group, displayed around the user. The distance between each sub-window and the user can be a fixed value of 2, as mentioned below, for example, 1.5m.

[0062] In this way, users can aggregate an application window with an application window group to create a new application window group, thus meeting their personalized needs.

[0063] According to the first aspect, or any implementation of the first aspect above, the method further includes:

[0064] In response to the tenth operation on the first application window group, the XR device displays a first prompt message in the XR space, which indicates that the current application group has been saved to the desktop.

[0065] In response to the eleventh operation, the XR device displays a desktop window in XR space, which includes icons for the first application window group.

[0066] In response to the twelfth operation on the icon of the first application window group, the XR device displays the first application window group in XR space.

[0067] Referring to the scenario shown in Figure 11b, users can save application window groups according to their actual needs for later use. After saving the application window group, users can quickly open the application window group through its icon, making the operation simple and convenient.

[0068] According to the first aspect, or any implementation of the first aspect above, the method further includes:

[0069] In response to the thirteenth operation performed by the user at the fourth position, the XR device displays the sixth application window at the fifth position in the XR space at the first moment; the sixth application window is used to display the program interface of the fourth application.

[0070] At the second moment, in response to the fourteenth operation performed by the user at the sixth position, the XR device displays the seventh application window at the seventh position in the XR space; the seventh application window is used to display the program interface of the fifth application.

[0071] At the third moment, in response to the fifteenth operation performed by the user at the eighth position, the XR device displays the sixth window at the ninth position in the XR space, and displays the first XR interface in the sixth window. The first XR interface includes thumbnails of the sixth application window and the seventh application window; wherein the third moment is later than the first moment and the second moment.

[0072] For example, taking the interaction between the user and the XR interface in the virtual space through hand-eye control as an example, the fifteenth operation can be the operation of the user looking at the multi-tasking background icon 13011 and pinching it with two fingers, as shown in the example in Figure 14a (1).

[0073] The sixth window can be the multitasking background management window 1303 shown in Figure 14a (2), and the first XR interface is the interface displayed in the multitasking background management window 1303.

[0074] In this way, users can view all the application windows currently running in the background in the multitasking background management window. These application windows can be opened by the user from anywhere. Users can then manage these application windows through the multitasking background management window.

[0075] According to the first aspect, or any implementation of the first aspect above, when the XR device displays the first XR interface in the sixth window, the method further includes:

[0076] In response to the sixteenth operation on the thumbnail of the sixth application window, the XR device displays the sixth application window at the ninth position in XR space and cancels the display of the sixth application window at the fifth position in XR space.

[0077] For example, taking the interaction between a user and an XR interface in a virtual space through hand-eye control as an example, the sixteenth operation could be to look at the screen and pinch with two fingers.

[0078] For example, the thumbnail of the sixth application window can refer to the application M window thumbnail 13701 shown in Figure 14d (1). For the sixth application window that the user previously opened at the fourth position in the real space, the user can drag it to the current position from the eighth position in the real space through the multitasking background management window. At this time, the sixth application window disappears at the virtual space position (i.e., the fifth position) corresponding to the fourth position in the real space, and is re-displayed at the virtual space position (i.e., the ninth position) corresponding to the eighth position in the real space, so that the user can use the program interface of the fourth application again.

[0079] In this way, users can open application windows that they previously opened in any other location in the multitasking background management window and display them in front of them, improving the convenience of user operation.

[0080] According to the first aspect, or any implementation of the first aspect above, when the XR device displays the first XR interface in the sixth window, the method further includes:

[0081] In response to the seventeenth operation on the thumbnail of the sixth application window, the XR device closes the sixth application window displayed at the fifth position in the XR space and displays a second XR interface in the sixth window, which includes a thumbnail of the seventh application window.

[0082] For example, taking the interaction between a user and an XR interface in a virtual space through hand-eye control as an example, the seventeenth operation could be a gaze and a pinch-and-swipe operation.

[0083] In this implementation, users can close some application windows running in the background through the multitasking background management window.

[0084] According to the first aspect, or any implementation of the first aspect above, the sixth window is displayed around the user, and the thumbnail of each application window displayed in the sixth window is equidistant from the user.

[0085] For example, the sixth window is displayed in a ring around the user.

[0086] For example, the spatial distance between the thumbnail of each application window displayed in the sixth window and the user can be a fixed value of 2 (e.g., 1.5m).

[0087] In this way, the distance between the user and the thumbnail of each application window is equal, making it easier to view and operate each thumbnail, resulting in a better user experience.

[0088] According to the first aspect, or any implementation of the first aspect above, the method further includes:

[0089] XR devices display the first Dock bar in XR space;

[0090] In response to the eighteenth operation on the first Dock bar, the XR device displays a second Dock bar in the XR space; wherein the second Dock bar includes a first area and / or a second area, the first area including a first application icon, or the second area including a first application icon;

[0091] In response to the nineteenth action on the first application icon, the XR device displays the eighth application window in XR space.

[0092] For example, taking the interaction between the user and the XR interface in the virtual space through hand-eye control as an example, the eighteenth operation can be to stare at the large desktop icon 7011 for a preset duration, and the nineteenth operation can be to stare and pinch with two fingers.

[0093] Referring to Figure 7a (2), the second Dock bar can be a transient frequently used application bar 703, the first area can be a minimized application area 7031, and the second area can be a recommended application area 7032.

[0094] In this way, users can quickly open application windows through the application icons displayed in the temporary frequently used application bar, enhancing the user's immersive experience.

[0095] According to the first aspect, or any implementation of the first aspect above, the XR device displays a ninth application window at the tenth position in the XR space; wherein the distance between the tenth position and the user is less than or equal to the first length;

[0096] XR devices display an eighth application window in XR space, including:

[0097] The XR device displays the eighth application window at the eleventh position in the XR space, and moves the ninth application window from the tenth position to the twelfth position in the XR space; wherein, the distance between the eleventh position and the user is equal to the first length, the distance between the twelfth position and the user is equal to the second length, and the second length is greater than the first length.

[0098] For example, the first length can be a fixed value of 2, such as 1.5m; the second length can be a fixed value of 6, such as 1.7m.

[0099] In this way, the user's most recently opened application window, such as the eighth application window, is positioned at a distance of the first length from the user, ensuring the best visual experience. If other application windows are displayed at a distance less than or equal to the first length from the user, these other windows need to be moved away from the user, for example, to a position at a distance of the second length from the user, to avoid obstructing the user's most recently opened application window.

[0100] According to the first aspect, or any implementation of the first aspect above, when the XR device displays the first application window and the second application window in the XR space, it further includes:

[0101] In response to a twentieth operation on the first application window, the XR device cancels the display of the second application window in the XR space;

[0102] In response to the twenty-first operation on the first application window, the XR device redisplays the second application window in XR space.

[0103] For example, the application bar of the first application window also includes a window isolation control. The twentieth operation on the first application window may be the twentieth operation on the window isolation control in the application bar of the first application window; correspondingly, the twenty-first operation on the first application window may be the twenty-first operation on the window isolation control in the application bar of the first application window.

[0104] For example, taking the interaction between a user and an XR interface in a virtual space through hand-eye control as an example, the twentieth and twenty-first operations can be the actions of looking at the screen and pinching with two fingers.

[0105] In this way, users can hide other application windows (such as the second application window) displayed in the XR space through the window isolation control of the first application window, and can also control these application windows to be displayed again through the window isolation control of the first application window.

[0106] In this way, users can isolate (i.e., hide) application windows that are not currently in use, based on actual needs. This avoids an excessive number of application windows displayed in the XR space, which could negatively impact the user's visual experience and prevent application windows from obstructing the view. When users need to use the isolated application windows again, they can use the window isolation control to make these application windows re-display.

[0107] According to the first aspect, or any implementation of the first aspect above, the method further includes:

[0108] The XR device displays the tenth application window in XR space; the tenth application window is used to display the program interface of the sixth application.

[0109] In response to the twenty-second operation on the tenth application window, the XR device minimizes the tenth application window, displays the second Dock, and adds the icon of the sixth application to the first section of the second Dock.

[0110] For example, referring to Figure 8f, the application bar of the tenth application window also includes a window minimize control. The twenty-second operation on the tenth application window can be the twenty-second operation on the window minimize control in the application bar of the tenth application window. For example, taking the user's interaction with the XR interface in the virtual space through hand-eye control as an example, the twenty-second operation can be the operation of looking up and pinching with two fingers.

[0111] In this way, the application icon corresponding to the application window minimized by the user is placed in the first section of the second Dock. When the user needs to use the application window again, they can quickly open it through the application icon displayed in the first section of the second Dock. The operation is simple and convenient, which can enhance the user's immersive experience.

[0112] Secondly, this application provides an XR device. The XR device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the XR device performs the XR space interface display method of the first aspect and any one of the first aspects.

[0113] The second aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the second aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0114] Thirdly, this application provides a computer-readable storage medium. The computer-readable storage medium includes a computer program that, when run on an electronic device, causes the electronic device to execute the XR space interface display method of the first aspect and any one thereof.

[0115] The third aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the third aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0116] Fourthly, this application provides a computer program product, including a computer program that, when run, causes the computer to execute an XR space interface display method as described in the first aspect or any one of the first aspects.

[0117] The fourth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fourth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0118] Fifthly, this application provides a chip including a processing circuit and transceiver pins. The transceiver pins and the processing circuit communicate with each other via an internal connection path. The processing circuit executes an XR space interface display method as described in the first aspect or any one thereof, to control the receiving pin to receive signals and to control the transmitting pin to transmit signals.

[0119] The fifth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fifth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0120] Figure 1a illustrates an example of an application scenario related to XR space;

[0121] Figure 1b illustrates an example of an application scenario related to XR space;

[0122] Figure 2a shows an exemplary structural schematic diagram of an XR device;

[0123] Figure 2b illustrates an exemplary structural diagram of an electronic device in an XR system;

[0124] Figure 3a illustrates an exemplary XR spatial interaction method based on rays;

[0125] Figure 3a illustrates an exemplary XR spatial interaction method based on hand-eye control;

[0126] Figure 4a illustrates an exemplary desktop diagram in XR space;

[0127] Figure 4b illustrates, for example, the positions of the desktop window and the Dock bar in XR space;

[0128] Figure 5a illustrates an exemplary diagram of desktop window operation in XR space;

[0129] Figures 5b-5d illustrate exemplary diagrams of adjusting the position of desktop icons in XR space;

[0130] Figures 5e-5f illustrate exemplary diagrams of adjusting the order of desktop icons in XR space;

[0131] Figure 6a illustrates an example of bringing up the Dock in XR space;

[0132] Figure 6b illustrates an example of collapsing the Dock in XR space;

[0133] Figure 6c illustrates an example of moving the Dock bar in XR space;

[0134] Figure 6d illustrates an exemplary diagram of the Dock bar display in XR space;

[0135] Figures 7a and 7b illustrate, for example, common Dock bars in XR space;

[0136] Figure 7c illustrates, for example, the location of the Dock bar in XR space;

[0137] Figure 7d illustrates, for example, the location of the Dock bar in XR space;

[0138] Figure 7e illustrates an example of collapsing the commonly used Dock in XR space;

[0139] Figures 8a-8b illustrate, for example, schematic diagrams of opening an application window in XR space;

[0140] Figure 8c illustrates, for example, a tab bar for applying a window in XR space;

[0141] Figures 8d-8e illustrate exemplary diagrams of applying windows in XR space;

[0142] Figure 8f illustrates an exemplary scenario of minimizing an application window in XR space;

[0143] Figures 8g-8i illustrate exemplary scenarios of opening application windows based on icons in the commonly used Dock bar in XR space;

[0144] Figure 8j illustrates, for example, the orientation of an application window opened by a user in XR space;

[0145] Figure 8k illustrates an exemplary scenario of adjusting the size of an application window in XR space;

[0146] Figure 81 illustrates an exemplary scenario of dragging an application window in XR space;

[0147] Figure 8m illustrates an exemplary scenario of isolating application windows in XR space;

[0148] Figure 9a illustrates an exemplary schematic diagram of an application window group in XR space;

[0149] Figures 9b-9d illustrate exemplary scenarios of creating application window groups in XR space;

[0150] Figures 9e-9f illustrate exemplary scenarios of creating application window groups in XR space;

[0151] Figures 10a-10b illustrate exemplary scenarios of adjusting the orientation of an application window group in XR space;

[0152] Figures 10c and 10d illustrate exemplary scenarios of adjusting the order of application windows in an application window group in XR space;

[0153] Figures 10e-10f illustrate exemplary scenarios of adjusting the order of application windows in an application window group in XR space;

[0154] Figure 11a illustrates an exemplary scenario of ungrouping application windows in XR space;

[0155] Figure 11b illustrates an exemplary scenario of saving an application window group in XR space;

[0156] Figure 11c illustrates an exemplary scenario of opening an application window group in XR space;

[0157] Figure 12a illustrates an exemplary scenario of minimizing application windows in an application window group in XR space;

[0158] Figure 12b illustrates an exemplary scenario of minimizing application windows in an application window group in XR space;

[0159] Figure 12c illustrates an exemplary scenario of restoring the minimized application window in an application window group in XR space;

[0160] Figure 13 illustrates an exemplary problem scenario of opening multiple application scenarios in XR space;

[0161] Figure 14a illustrates an exemplary scenario of opening a multitasking background management window in XR space;

[0162] Figure 14b illustrates an exemplary scenario of closing an application window in XR space based on a multi-task background management window;

[0163] Figures 14c-14e illustrate exemplary scenarios of adjusting the position of application windows based on a multitasking background management window in XR space.

[0164] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0165] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0166] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0167] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0168] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0169] The user operations mentioned in the embodiments of this application, such as the "pinch with two fingers" operation and the drag operation, can be referred to as the first operation, the second operation, etc., and the specific form of the user operation is not limited to the operation method mentioned in the embodiments of this application. There may be more or fewer operation steps to complete the corresponding function.

[0170] This application provides a method for displaying XR space. XR space can include three-dimensional images, three-dimensional videos, and stereoscopic audio, etc. XR space can be a multi-dimensional space combining real and virtual spaces, or it can be a multi-dimensional space consisting solely of virtual space.

[0171] XR devices refer to electronic devices that employ XR technologies such as AR, VR, and MR, and can display XR spaces to users. When an XR device displays an XR interface within an XR space to a user, the user can experience XR scenes that transcend reality, as if they were actually inside the XR space. For example, XR devices can include head-mounted XR devices and handheld XR devices. For instance, a head-mounted XR device can be XR glasses, an XR helmet, or other XR devices that can be worn on the user's head. This embodiment does not limit the implementation form of head-mounted XR devices.

[0172] In some implementations, the XR space display method provided in this application can be applied to an XR device. In this case, the XR device can not only serve as a display device for the XR space, but also as a master control device for the XR space. The display data of the XR device can be provided by the XR device itself. For example, the XR device can run a specific application to generate the display content of the XR device; this application can be, for example, a video application, a game application, a music application, a desktop application, etc.

[0173] Taking XR glasses 100 as an example, Figure 1a illustrates an exemplary XR application scenario. As shown in Figure 1a, XR glasses 100 can display images within a virtual space 001. Users wearing XR glasses 100 can view the displayed images within the virtual space 001 and experience the XR experience. The virtual space 001 shown in Figure 1a is a planar space within a real space 1. It is understood that in practical applications, the virtual space 001 can be a three-dimensional space comprising multiple planes within the real space 1, or it can be a curved surface with curvature within the real space 1. This embodiment does not limit the representation of the virtual space.

[0174] In the example shown in Figure 1a, the XR space is a multi-dimensional space combining real space 1 and virtual space 001. Real space 1 can be understood as the actual environment in which virtual space 001 resides. Understandably, in practical applications, virtual space 001 can also be displayed within a virtual space of a preset theme; in this case, the XR space is simply a multi-dimensional space of virtual spaces (including the virtual space of the preset theme and virtual space 001).

[0175] In other implementations, the XR space display method provided in this application can be applied to an XR system including interconnected XR devices and electronic devices. In the XR system, the electronic device is the master control device for the XR space, providing display data to the XR devices; the XR device is the display device for the XR space, equivalent to an extension display device of the electronic device. The electronic device can be a portable terminal device such as a mobile phone or tablet computer, or a non-portable terminal device such as a laptop computer or desktop computer with a touch-sensitive or touch-enabled surface. The electronic device can run specific applications to provide content transmitted to the XR device for display; these applications can be, for example, video applications, game applications, music applications, desktop applications, screen mirroring applications, etc.

[0176] In an XR system, XR devices and electronic devices can be connected via wired or wireless connections. Wired connections can include, but are not limited to, wired communication via interfaces such as USB (Universal Serial Bus) and HDMI (High Definition Multimedia Interface). Wireless connections can include, but are not limited to, wireless communication via technologies such as Bluetooth, Wi-Fi (Wireless Fidelity) Direct, Wi-Fi softAP, Wi-Fi LAN, and radio frequency. This embodiment does not limit the connection method between the two.

[0177] Taking XR glasses 100 as the XR device and mobile phone 200 as the electronic device, Figure 1b illustrates an exemplary XR application scenario. As shown in Figure 1b, the XR glasses 100 and mobile phone 200 can be connected via wired or wireless connection. Wearing the XR glasses 100, the user can view the displayed images within the virtual space 002 and experience the XR experience. The virtual space 002 shown in Figure 1b is a plane within a real space 2. It is understood that in practical applications, the virtual space 002 can be a three-dimensional space including multiple planes within the real space 2, or it can be a curved surface with curvature within the real space 2. This embodiment does not limit the form of the virtual space. Furthermore, it should be noted that the virtual spaces indicated in the accompanying drawings of this application embodiment are merely illustrative and are not intended to limit the size, shape, or other characteristics of the virtual space.

[0178] In the example shown in Figure 1b, the XR space is a multi-dimensional space combining real space 2 and virtual space 002. Real space 2 can be understood as the actual environment in which virtual space 002 resides. Understandably, in practical applications, virtual space 002 can also be displayed within a virtual space of a preset theme; in this case, the XR space is simply a multi-dimensional space of virtual spaces (including the virtual space of the preset theme and virtual space 002).

[0179] Figure 2a illustrates an exemplary structural diagram of an XR device. As shown in Figure 2a, the XR device includes a processor 110, a memory 120, a sensor system 130, a communication module 140, a camera 150, a display device 160, and an audio device 170. These components can be coupled together and communicate with each other.

[0180] It is understood that the structure shown in Figure 2a does not constitute a specific limitation on the XR device. In some embodiments of this application, the XR device may include more or fewer components than shown, or combine some components, or split some components, or have different component arrangements. For example, the XR device may also include physical buttons, such as power buttons, volume buttons, screen brightness adjustment buttons, and various interfaces, such as USB interfaces, HDMI, etc. The components shown can be implemented in hardware, software, or a combination of hardware and software. The XR device structure shown in Figure 2a can be applied to head-mounted XR devices, such as the XR glasses 100 shown in Figures 1a and 1b.

[0181] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). Different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution, enabling each component to perform corresponding functions, such as human-computer interaction, motion tracking / prediction, rendering and display, and audio processing.

[0182] Memory 120 can store executable instructions. Memory 120 may include a stored program area and a stored data area. The stored program area may store the operating system, application programs required for at least one function (such as sound playback, image playback, etc.), etc. The stored data area may store data created during the use of the XR device (such as audio data, etc.). In addition, memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of the XR device by running instructions stored in memory 120 and / or instructions stored in memory located in the processor.

[0183] The communication module 140 may include a mobile communication module and a wireless communication module. The mobile communication module can provide solutions for wireless communication applications in XR devices, including 2G / 3G / 4G / 5G. The wireless communication module can provide solutions for wireless communication applications in XR devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module may be one or more devices integrating at least one communication processing module.

[0184] The sensor system 130 may include an accelerometer, compass, gyroscope, magnetometer, or other sensors for detecting motion. The sensor system 130 is used to collect corresponding data, such as an accelerometer collecting the acceleration of the XR device, and a gyroscope collecting the speed of the XR device. The data collected by the sensor system 130 can reflect the movement of the user's head while wearing the XR head-mounted device. In some embodiments, the sensor system 130 may be an inertial measurement unit (IMU) installed within the XR device. In some embodiments, the XR device can send the data acquired by the sensor system to the processor 110 for analysis. The processor 110 can determine the user's head movement based on the data collected by each sensor and execute corresponding functions based on the user's head movement. That is, the user can trigger the XR head-mounted device to execute corresponding functions by inputting head movement operations on the XR head-mounted device. The user's head movement may include whether it rotates, the direction of rotation, etc.

[0185] The sensor system 130 may also include optical sensors for use in conjunction with the camera 150 to track the user's eye position and capture eye movement data. This eye movement data can be used, for example, to determine the user's interpupillary distance, the 3D position of each eye relative to the head-mounted XR device, the magnitude of torsion and rotation (i.e., turning, pitching, and swaying) of each eye, and the direction of gaze, etc. In one example, infrared light is emitted within the head-mounted XR device and reflected from each eye. The reflected light is detected by the camera 150 or the optical sensors, and the detected data is transmitted to the processor 110 so that the processor 110 can analyze the changes in the infrared light reflected from each eye to determine the user's eye position, pupil diameter, movement state, etc.

[0186] Camera 150 can be used to capture still images or videos. These still images or videos can be externally focused images or videos of the user's surroundings, or internally focused images or videos. Camera 150 can track the movement of one or both eyes of the user. Camera 150 includes, but is not limited to, traditional color cameras (RGB cameras), depth cameras (RGB depth cameras), and dynamic vision sensor (DVS) cameras. Depth cameras can acquire depth information of the object being photographed. In some embodiments, camera 150 can be used to capture images of the user's eyes and send these images to processor 110 for analysis. Processor 110 can determine the state of the user's eyes based on the images captured by camera 150 and execute corresponding functions based on the state of the user's eyes, such as determining the user's viewpoint on the XR interface. That is, the user can trigger the head-mounted XR device to execute corresponding functions by inputting eye movement operations on the head-mounted XR device. The state of the user's eyes may include: whether they are turning, the direction of the turn, whether they have not turned for a long time, the duration of the inactivity, the angle at which they are looking outwards, etc.

[0187] XR devices render or display images using a GPU, a display device 160, and an application processor. The GPU is a microprocessor for image processing, connecting the display device 160 and the application processor. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. The GPU performs mathematical and geometric calculations based on data obtained from the processor 110, and renders images using computer graphics technology, computer simulation technology, etc., to provide content for display on the display device 160. The GPU also adds correction or pre-distortion to the image rendering process to compensate for or correct distortion caused by optical components in the display device 160. The GPU can also adjust the content provided to the display device 160 based on data from the sensor system 130. For example, the GPU can add depth information to the content provided to the display device 160 based on the user's 3D eye position, interpupillary distance, etc.

[0188] Display device 160 may include one or more displays and one or more optical components. The displays may include a display panel for displaying images and presenting a stereoscopic virtual scene to the user. The display panel may be LCD, OLED, AMOLED, FLED, MiniLED, MicroLED, Micro-OLED, QLED, etc. The optical components may be used to guide light from the displays to an exit pupil for user perception. In some embodiments, one or more optical elements (e.g., convex lenses, Fresnel lenses, or other types of lenses) in the optical components may have one or more coatings, such as anti-reflective coatings. The amplification of image light by the optical components allows the displays to be physically smaller, lighter, and consume less power. Additionally, the amplification of image light can increase the field of view of the content displayed on the displays. For example, the optical components may make the field of view of the content displayed on the displays the user's entire field of view. The one or more displays may include a first display and a second display. The first and second displays may be different display areas of the same display, or they may be two different displays. The first display is used to display a first image, and the second display is used to display a second image; the first and second images have parallax. Thus, the user can experience an immersive XR space.

[0189] In this embodiment of the application, the display screen in the XR device can be used to display display data transmitted from connected electronic devices (such as mobile phones, tablets, etc.) (such as XR interface data in XR space), or it can be used to display display data on the XR device side (such as locally stored display data, display data downloaded from the Internet, etc.), providing users with an XR experience.

[0190] Audio device 170 is used to acquire and output audio. Audio device 170 may include, but is not limited to, a microphone, speaker, headphones, etc. Audio device 170 can provide stereo audio, allowing users to experience stereo sound effects.

[0191] Understandably, XR devices may also include a charging management module, a power management module, a battery, etc. The charging management module receives charging input from the charger. While charging the battery, the charging management module can also supply power to the electronic device via the power management module. The power management module connects to the battery, and the charging management module connects to the processor 110. The power management module receives input from the battery and / or the charging management module, supplying power to the processor 110, memory 120, sensor system 130, communication module 140, camera 150, display device 160, audio device 170, etc. In some embodiments, the power management module may also be located within the processor 110. In some embodiments, the power management module and the charging management module may also be located in the same device.

[0192] It is understood that, in order to implement the XR space display method in the embodiments of this application, an XR device includes hardware and / or software modules that perform various functions. Based on the algorithm steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0193] Referring to Figure 1b, an XR system may include interconnected XR devices and electronic devices. The electronic devices can generate XR interfaces in a virtual space and transmit these interfaces to the XR devices. The XR devices can then display the XR space to the user through the XR interface. The structure of the XR devices is shown in Figure 2a and will not be described further here.

[0194] Figure 2b illustrates an exemplary structural diagram of an electronic device in an XR system. It should be understood that Figure 2b is merely an example of an electronic device, and the electronic device may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in Figure 2b can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits (ASICs).

[0195] The electronic device may include: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus interface 230, a charging management module 240, a power management module 241, a battery 242, antenna 1, antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0196] Processor 210 may include one or more processing units, such as application processors, modem processors, graphics processors, image signal processors, controllers, memory, video codecs, digital signal processors, baseband processors, and / or neural network processors. These different processing units may be independent devices or integrated into one or more processors.

[0197] The controller can serve as the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0198] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0199] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0200] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor, etc.

[0201] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0202] Mobile communication module 250 can provide wireless communication solutions for electronic devices, including 2G / 3G / 4G / 5G. Wireless communication module 260 can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0203] In some embodiments, the antenna 1 of the electronic device is coupled to the mobile communication module 250, and the antenna 2 is coupled to the wireless communication module 260, enabling the electronic device to communicate with networks and other devices via wireless communication technology.

[0204] Electronic devices implement display functions through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0205] The external storage interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 210 through the external storage interface 220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0206] Internal memory 221 can be used to store computer executable program code, which includes instructions. Processor 210 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 221. Internal memory 221 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, phonebook, etc.). Furthermore, internal memory 221 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0207] Electronic devices can implement audio functions through audio modules 270, speakers 270A, receivers 270B, microphones 270C, headphone jacks 270D, and application processors. Examples include music playback and recording.

[0208] It is understood that, in order to implement the XR space display method in the embodiments of this application, XR devices and electronic devices (including XR devices and electronic devices) may include hardware and / or software modules that perform various functions. Based on the algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0209] Next, several interaction methods between users and XR interfaces in virtual space are briefly described. In practical applications, the embodiments of this application do not limit the interaction methods between users and XR interfaces in virtual space.

[0210] In some implementations, as shown in Figure 3a, users can interact with the XR interface in the virtual space through rays displayed in the virtual space.

[0211] When a controller paired with (or connected to) an XR device is used as an input device for the XR device, the controller can receive user operations such as clicking and swiping, and can project rays into the virtual space to simulate the action of a mouse, making it easier to control the objects displayed on the XR interface in the virtual space.

[0212] In the scenario shown in 1b, electronic devices can also serve as input devices for XR devices. When an electronic device receives user actions such as clicks or swipes, it projects rays into the virtual space to control the objects displayed on the XR interface within that virtual space.

[0213] XR devices can be equipped with various sensors in their input devices, such as touch sensors, accelerometers, gyroscopes, magnetometers, and pressure sensors, to receive user input. Accelerometers and gyroscopes can detect user movement of the input device, which can change the direction of the input signal. Touch sensors and pressure sensors can detect user touch operations on touch panels such as touchscreens, including swipes, taps, short presses, and long presses.

[0214] XR devices can also be equipped with some physical buttons, such as buttons for turning the device on and off, buttons for adjusting the brightness of the XR interface, buttons for bringing up certain XR interfaces or XR windows, etc., to receive some user input.

[0215] In other implementations, as shown in Figure 3b, users can interact with the XR interface in the virtual space through hand-eye control. In this hand-eye control method, user input can include gaze and gesture operations.

[0216] As shown in Figure 3b, the XR device tracks the user's eyes and uses an intent prediction algorithm to determine the user's viewpoint on the XR interface in virtual space. The viewpoint is the gaze projection point, used to indicate the target object the user wants to interact with on the XR interface. For example, the viewpoint can be represented by a dot, as shown by point 301 in Figure 3b. For instance, the viewpoint could be a red dot with some transparency; this embodiment does not limit the representation of the viewpoint. The XR device also detects the user's gestures to determine the user's actions on the target object, such as clicks, double-clicks, and swipes. For example, when the user gesture is a pinch of two fingers (as shown by gesture 302 in Figure 3b), it indicates that the user input is a single click operation on the target object. A pinch of two fingers includes the process of the user's two fingers touching together and then separating, for example, the user's thumb and index finger touching and then separating. Similarly, a pinch of two fingers twice can indicate a double click operation. The meaning of a pinch of two fingers twice can be the same as a single pinch of two fingers, that is, quickly repeating the pinch of two fingers twice. When the user gesture is a pinch and drag gesture, it indicates that the user input is a drag operation, swipe operation, or move operation on the target object. The pinch and drag gesture can refer to the process of the user's thumb and index finger touching and then moving in space.

[0217] Understandably, in hand-eye control, the eye-tracking result determined by the intent prediction algorithm serves as the interaction aimer, and the gesture detection result serves as the interaction trigger. Users can interact with the XR interface in the virtual space through coordinated hand-eye input. This method of interacting with the XR interface via hand-eye control allows users to achieve human-computer interaction more naturally and intuitively, quickly and efficiently, resulting in a better user experience.

[0218] In some implementations, users can directly interact with certain displayed objects in the virtual space. For example, some elements in the virtual space support direct interaction with the user's hands. For instance, with a virtual keyboard, users can use their hands to simulate typing to input information; with certain windows, users can use their hands to perform close-range operations, such as dragging; and with certain virtual objects, users can use their hands to grasp them.

[0219] The following uses the "hand-eye control method" as an example to explain in detail the XR space display method provided in this application embodiment. In the XR space display method provided in this application embodiment, users can quickly and efficiently perform multi-window operations in the XR space, resulting in a better user experience.

[0220] (I) On the Large Desktop in XR Space

[0221] (1) About desktop windows and Dock

[0222] Figure 4a illustrates an exemplary display diagram of an XR space. Exemplarily, this XR space is a multi-dimensional space combining real and virtual spaces. Figure 4a shows this XR space from the perspective of a user wearing XR glasses. Continuing to refer to Figure 4a, a large desktop 40 is displayed in this XR space. The large desktop 40 is an XR object displayed in the XR space. The large desktop 40 can refer to the XR object seen through the XR glasses after the user opens the XR glasses and successfully logs into the system. The large desktop 40 can be equivalently understood as the desktop of an electronic device (such as a mobile phone, tablet, PC, etc.).

[0223] As shown in Figure 4a, the large desktop 40 may include a desktop window (or desktop application display area, etc.) 401 and a Dock bar (or taskbar) 402. In this XR space, the desktop window 401 and the Dock bar 402 may be on the same plane or on different planes (for example, the plane where the desktop window 401 is located is parallel to or intersects the plane where the taskbar 402 is located). This embodiment does not limit this.

[0224] Figure 4b illustrates the exemplary positions of desktop window 401 and Dock 402 in XR space. As shown in Figure 4b, in XR space, the plane containing desktop window 401 is perpendicular to the horizontal plane and also perpendicular to the eye level of the user wearing XR glasses. Dock 402 and desktop window 401 are not on the same plane; the angle between the plane containing Dock 402 and the horizontal plane is a fixed value of 1 (the fixed value of 1 is, for example, 60°). To enhance the user's visual experience: the spatial distance between the desktop window 401 and the user wearing XR glasses can be a fixed value 2 (fixed value 2 is, for example, 1.5m); the center point 4012 of the desktop window 401 can be lower than the eye level, for example, the angle between the user's line of sight 4013 when looking at the center point 4012 and the eye level is a fixed value 3 (fixed value 3 is, for example, 6°); the center point 4023 of the Dock bar 402 can be lower than the eye level, for example, the angle between the user's line of sight 4024 when looking at the center point 4023 and the eye level is a fixed value 4 (fixed value 4 is, for example, 30°).

[0225] It should be noted that the desktop window 401 and the Dock bar 402 can be understood as two independently existing XR objects in the XR space. The desktop window 401 and the Dock bar 402 can be displayed simultaneously in the XR space, or they can be displayed separately in the XR space. In some embodiments, the large desktop only includes the desktop window 401, while the Dock bar 402 is an independently existing XR object in the XR space. This embodiment does not limit this.

[0226] Referring again to Figure 4a, the desktop window 401 may include a desktop background and one or more application icons, such as icon 4011. Each application icon can be used to open an application. The application icons are displayed on the desktop background, which can be a transparent background or a non-transparent colored background; this embodiment does not limit this. Below each application icon, the corresponding application name (or application label, etc., not shown in Figure 4a) may also be displayed; this embodiment does not limit this. In this embodiment, the size of the desktop window 401 can be a fixed size, or it can dynamically change according to the number of rows and / or columns of application icons displayed in the desktop window 401. For example, the more rows of application icons displayed on the desktop window 401, the greater the height of the desktop window 401; the more columns of application icons displayed on the desktop window 401, the greater the width of the desktop window 401.

[0227] In this embodiment, the Dock 402 carries device system-level functions. The Dock 402 may include a content display area 4021 and Dock operation controls (or operation bar, operation bar movement controls, etc.) 4022. The content display area 4021 may include, but is not limited to, system information (such as time, battery level, wireless signal strength, etc.) and at least one function icon, such as a notification center function icon 40211, a control center function icon 40212, a shared space function icon 40213, an environment switching function icon 40214, a large desktop icon 40215, a multitasking background icon 40216, etc.

[0228] The notification center icon 40211 is used to display the notification center XR interface. For example, in response to the user looking at the notification center icon 40211 and pinching it with two fingers, a new XR window is created in the XR space, and the notification center interface is displayed in that XR window. Alternatively, in response to the user looking at the notification center icon 40211 and pinching it with two fingers, the notification center interface is displayed in the desktop window 401.

[0229] The control center function icon 40212 is used to display the control center XR interface. For example, in response to the user looking at the control center function icon 40212 and pinching it with two fingers, a new XR window is created in the XR space, and the control center interface is displayed in that XR window. Alternatively, in response to the user looking at the control center function icon 40212 and pinching it with two fingers, the control center interface is displayed in the desktop window 401.

[0230] The shared space function icon 40213 is used to display the shared space settings interface. For example, in response to the user looking at the shared space function icon 40213 and pinching it with two fingers, a new XR window is created in the XR space, and the shared space settings interface is displayed in that XR window. Alternatively, in response to the user looking at the shared space function icon 40213 and pinching it with two fingers, the shared space settings interface is displayed in the desktop window 401.

[0231] The environment switching icon 40214 is used to display the XR space background settings interface. For example, in response to the user looking at the environment switching icon 40214 and pinching it with two fingers, a new XR window is created in the XR space, and the XR space background settings interface is displayed in that XR window. Alternatively, in response to the user looking at the environment switching icon 40214 and pinching it with two fingers, the XR space background settings interface is displayed in the desktop window 401.

[0232] The multitasking background icon 40216 is used to display the multitasking background management XR interface. Detailed explanations of the multitasking background can be found below and will not be repeated here. The large desktop icon 40215 is used to display the desktop window 401. In the example shown in Figure 4a, the large desktop icon 40215 is selected, indicating that desktop window 401 is currently displayed in the XR space.

[0233] (2) About application icons in the desktop window

[0234] For example, desktop window 401 can display M rows × N columns of application icons. For instance, M is 2 and N is 6. When P (P>M*N) applications are installed on the XR glasses (or an electronic device connected to the XR glasses), the icons of these P applications can be displayed in a split-screen manner. The user can switch between the application icons displayed in desktop window 401 by pinching and swiping with their finger.

[0235] As shown in Figure 5a(1), the desktop window 501 displays the first screen 5011 of desktop application icons, which includes application icons for applications such as application A and application B. Continuing to refer to Figure 5a(1), the user wearing XR glasses pinches their thumb and forefinger together and slides to the left. In response to the user's operation, the first screen 5011 of desktop application icons slides to the left in the desktop window 501, and the second screen 5014 of desktop application icons slides to the left into the desktop window 501. The second screen 5014 of desktop application icons includes application icons for applications such as application M and application N. For example, during the process of the user pinching and sliding to the left, as shown in Figure 5a(2), a portion 5012 of the first screen 5011 of desktop application icons and a portion 5013 of the second screen 5014 of desktop application icons are simultaneously displayed in the desktop window 501. The other areas of the first screen 5011 and the second screen 5014 of desktop application icons are not visible to the user because they are covered by a mask. That is, during the process of switching the display interface of desktop window 501 from the first screen 5011 of desktop application icons to the second screen 5014 of desktop application icons, only those screens appearing in the area of ​​desktop window 501 are visible.

[0236] In response to the user pinching and sliding two fingers to the left, the content of the first screen 5011 of desktop application icons displayed in the desktop window 501 decreases, while the content of the second screen 5014 of desktop application icons displayed in the desktop window 501 increases; correspondingly, in response to the user pinching and sliding two fingers to the right, the content of the first screen 5011 of desktop application icons displayed in the desktop window 501 increases, while the content of the second screen 5014 of desktop application icons displayed in the desktop window 501 decreases. That is, the switching from the first screen 5011 of desktop application icons to the second screen 5014 of desktop application icons is responsive. Referring to Figure 5a (2), when the width of the second screen 5014 of desktop application icons slides into the desktop window 501 and exceeds a preset ratio (e.g., 1 / 3) of the width of the second screen 5014 of desktop application icons, the user releases two fingers. In response to user operation, the display interface of desktop window 501 automatically switches from the first screen 5011 of desktop application icons to the second screen 5014 of desktop application icons, as shown in Figure 5a (3).

[0237] If the width of the second screen 5014 of the desktop application icon slides into the desktop window 501 does not exceed the preset ratio (e.g., 1 / 3) of the width of the second screen 5014 of the desktop application icon, and the user releases two fingers, then in response to this user operation, the first screen 5011 of the desktop application icon will be restored and displayed in the desktop window 501.

[0238] Understandably, when the desktop application icons are displayed in the second screen 5014 of the desktop window 501, the user can also switch the display interface of the desktop window 501 from the second screen 5014 of the desktop application icons to the first screen 5011 of the desktop application icons by pinching and sliding two fingers to the right. This will not be elaborated on here.

[0239] In this embodiment, users can adjust the position, sorting, and other settings of application icons on any screen of the desktop application icons displayed in the desktop window using a "hand-eye control method".

[0240] Figures 5b and 5c exemplify an application scenario. As shown in Figure 5b (1), the desktop window 502 displays application icons for multiple applications, such as application B and application C. When a user wearing XR glasses wants to adjust the position of application J's icon 5021, the user can hover over the icon 5021. In response to this user action, the user's viewpoint 5022 is displayed on the icon 5021 of application J, indicating that the user has selected the icon 5021 of application J. In this embodiment, the user's viewpoint can be represented by a dot marker, specifically by a red dot marker.

[0241] After the user selects the icon 5021 of application J, referring to Figure 5b (1), the user pinches the icon with two fingers for a preset duration (e.g., 1.5 seconds). In response to this user operation, the icon 5021 of application J floats up or falls down on the desktop window display interface (not shown in the figure), as shown in Figure 5b (2). For example, when the icon 5021 of application J floats up on the desktop window 502 display interface, the size of the icon 5021 of application J increases, and the application name (or application label) "Application J" corresponding to the icon 5021 of application J disappears. At the same time, in response to this user operation, a visual style indication is displayed at the placement position on the desktop window 502 display interface, for example, dashed frames are displayed at the placement positions of the icon of application J, as shown in dashed frames 5023, 5024 and 5025. That is, dashed boxes 5023, 5024 and 5025 respectively represent a placeable position for the icon 5021 of application J.

[0242] It should be noted that in this embodiment, when a user looks at any object, as long as the user's viewpoint displayed in the virtual space is within the hot zone of that object, it indicates that the user has successfully selected the object.

[0243] After the user pinches the button with two fingers for a preset duration, the user continues to pinch the button and drags it to the placement position indicated by the dashed frame 5023, as shown in Figure 5b (2). In response to this user operation, the icon 5021 of application J moves to the placement position indicated by the dashed frame 5023, as shown in Figure 5c (1). When the icon 5021 of application J is dragged into the hot zone (or recognition range) of a certain placement position, the placement position is highlighted, for example, the color of the dashed frame corresponding to the placement position changes or the line becomes thicker, thus prompting the user that "the icon 5021 of application J has been dragged to the position indicated by the dashed frame, and you can release your finger to complete the dragging of the application icon." For example, as shown in Figure 5c (1), when the icon 5021 of application J is dragged into the hot zone of the placement position indicated by the dashed frame 5023, the line of the dashed frame 5023 becomes thicker. If the user releases their two fingers at this time, in response to the user's operation, the icon 5021 of application J moves to the placement position indicated by the dashed box 5023, and the icon 5021 of application J is restored to its original size. The application name "Application J" is displayed below the icon 5021 of application J, as shown in Figure 5c (2). At this point, the position of the icon 5021 of application J is adjusted.

[0244] Figures 5b and 5d illustrate an application scenario. An explanation of Figure 5b can be found above and will not be repeated here. As shown in Figure 5d (1), when the icon 5021 of application J is not dragged to any of the available placement locations, the visual style of each placement location remains unchanged, such as the style of each dashed box. If the user releases their two fingers, in response to the user's action, the icon 5021 of application J is restored to its original position, and the icon 5021 of application J is restored to its original size. The application name "Application J" is re-displayed, as shown in Figure 5d (2). In this scenario, the display position of the icon 5021 of application J in the desktop window display interface is not adjusted.

[0245] Figures 5e and 5f illustrate an application scenario. As shown in Figure 5e (1), the desktop window 503 displays application icons for multiple applications, such as application A, application B, etc. When a user wearing XR glasses wants to adjust the position of application K's icon 5031, the user can look at application K's icon 5031. In response to the user's operation, the user's viewpoint 5032 is displayed on application K's icon 5031, indicating that the object selected by the user is application K's icon 5031.

[0246] After the user selects the icon 5031 of application K, referring to Figure 5e (1), the user pinches the icon with two fingers for a preset duration (e.g., 1.5 seconds). In response to this user operation, the icon 5031 of application K floats up or down on the desktop window display interface (not shown in the figure). As shown in Figure 5e (2), for example, when the icon 5031 of application K floats up on the desktop window 503 display interface, the size of the icon 5031 of application K increases, and the application name (or application label) "Application K" corresponding to the icon 5031 of application K disappears. In this scenario, the desktop window 503 display interface shows two rows and six columns of application icons, with no free space to place them, but the user wants to adjust the order of the application icons. At this time, the user can drag the icon 5031 of application K to any position in the application icon sorting to adjust the order of the application icons.

[0247] After the user pinches the screen with two fingers for a preset duration, the user continues to pinch and drags the screen towards the middle of the C and D icons, as shown in Figure 5e (2). In response to this user action, the K icon 5031 is dragged to the hot zone between the C and D icons, as shown in Figure 5f (1). For example, during the dragging of the K icon 5031, a visual style indication is displayed at the original position of the K icon 5031, such as a dashed box 5033. If the user releases their fingers at this time, in response to this user action, the K icon 5031 is dragged between the C and D icons, the positions of other application icons are adjusted accordingly, the K icon 5031 is restored to its original size, and the application name "Application K" is displayed below the K icon 5031, as shown in Figure 5f (2). At this point, the position of the K icon 5031 is adjusted.

[0248] Understandably, if a user drags the icon 5031 of application K within the hotspot area of ​​any application (e.g., application A) and releases two fingers, in response to the user's action, the icon 5031 of application K moves to the original icon position of that application (e.g., application A), and the icons of other applications are adjusted accordingly in sequence. If the user drags the icon 5031 of application K to the hotspot area corresponding to the dashed box 5033, or to any non-placeable location (such as the hotspot area not corresponding to any application icon position) and releases two fingers, in response to the user's action, the icon 5031 of application K is restored to its original position, and the icon 5031 of application K is restored to its original size, and the application name "Application K" is displayed again.

[0249] This allows users to flexibly adjust the position or order of application icons displayed in the desktop window according to their actual needs, resulting in a better user experience.

[0250] (3) Regarding the way the Dock bar is brought up and collapsed.

[0251] In this embodiment, the user can bring up the Dock bar by using a global gesture or by operating the relevant physical buttons on the XR glasses, regardless of the state of the XR space.

[0252] For example, as shown in Figure 6a(1), when no XR interface is displayed in the virtual space, the user performs a global gesture and holds it for a preset duration (e.g., 2 seconds). For example, a global gesture could be a gesture of making a fist with the thumb pointing towards the user. In response to this user action, a Dock bar 601 is displayed in the virtual space, as shown in Figure 6a(2).

[0253] For example, in response to a user action, a gesture progress bar may be displayed before the Dock bar is displayed in the virtual space, and the Dock bar 601 may be displayed in the virtual space when the gesture progress bar has finished loading. If the user releases the gesture or changes the gesture before the gesture progress bar has finished loading, the Dock bar 601 may not be displayed in the virtual space in response to the user action.

[0254] Understandably, when any XR interface (excluding the Dock) is displayed in the virtual space, the user can also display the Dock in the virtual space by performing a global gesture and holding it for a preset duration.

[0255] In this embodiment, when a Dock is displayed in the XR space, the user can also collapse the Dock by using relevant gestures or operating the relevant physical buttons on the XR glasses.

[0256] For example, as shown in Figure 6b(1), a Dock bar 602 is displayed in the virtual space. The user looks at the Dock bar operation control 6021 of the Dock bar 602. In response to the user operation, the user's viewpoint 603 is displayed on the Dock bar operation control 6021, indicating that the object selected by the user is the Dock bar operation control 6021. Continuing to refer to Figure 6b(1), after the user selects the Dock bar operation control 6021, the user pinches two fingers together. In response to the user operation, the Dock bar 602 is de-displayed in the virtual space, as shown in Figure 6b(2).

[0257] In this way, users can flexibly bring up or hide the Dock bar according to their actual needs, avoiding displaying too many XR objects in the XR space and affecting the user's visual experience.

[0258] (4) Adjusting the display position of the Dock bar

[0259] In this embodiment, when a Dock bar is displayed in the XR space, the user can adjust the display position of the Dock bar in the XR space through "hand and eye control".

[0260] For example, as shown in Figure 6c (1), a Dock bar 604 is displayed in a virtual space. The user looks at the Dock bar operation control 6041 of the Dock bar 604. In response to the user operation, the user viewpoint 605 is displayed on the Dock bar operation control 6041, indicating that the object selected by the user is the Dock bar operation control 6041.

[0261] Referring again to Figure 6c(1), after the user selects the Dock bar operation control 6041, the user pinches it with two fingers and drags it to the upper right. In response to this user operation, the Dock bar 604 moves to the upper right in virtual space. During the dragging of the Dock bar 604, the state of the Dock bar operation control 6041 is switched from the default state to the moving state, and / or, the state of the user viewpoint 605 is switched from the default state to the moving state. The default state of the Dock bar operation control 6041 can be seen in Figure 6c(1), and the moving state of the Dock bar operation control 6041 can be seen in Figure 6c(2); the default state of the user viewpoint 605 can be seen in Figure 6c(1), and the moving state of the user viewpoint 605 can be seen in Figure 6c(2).

[0262] When the Dock bar 604 is dragged to the target position, the user releases two fingers. In response to this user action, the Dock bar 604 is displayed at the target position, and the user's viewpoint 605 on the Dock bar operation control 6041 disappears, restoring the form of the Dock bar operation control 6041 to its default state. At this point, the display position of the Dock bar 604 in virtual space is adjusted. Optionally, when the Dock bar 604 is not dragged, it is displayed as completely opaque; while the Dock bar 604 is being dragged, it is displayed as semi-transparent, for example, with a preset transparency value (such as 50%), to enhance visual perception; when the dragging of the Dock bar 604 is complete, it reverts to being completely opaque.

[0263] As an optional implementation, the display state of the Dock bar can include a default state and a non-default state. The default state of the Dock bar can be seen as the Dock bar 606 shown in Figure 6d (1). That is, when the Dock bar 606 is in the default state, only the content display area 6061 is displayed, and the Dock bar operation control 6062 is not displayed. In response to the user's operation of looking at the area below the content display area 6061, the user's viewpoint 607 is displayed below the content display area 6061. When the display of the user's viewpoint 607 reaches a preset duration (such as 0.5s or 1s), the Dock bar operation control 6062 is displayed below the content display area 6061, as shown in Figure 6d (2). At this time, the Dock bar 606 is in a non-default state, that is, the Dock bar 606 displays both the content display area 6061 and the Dock bar operation control 6062. When the Dock 606 is in a non-default state, the user can collapse or move the Dock 606 in virtual space using the Dock operation control 6062. Optionally, when the Dock 606 is in a non-default state, i.e., when the Dock operation control 6062 is not displayed, if the user does not perform any operation on the Dock 606 for a preset threshold period of time, the Dock operation control 6062 will be de-displayed, that is, the Dock 606 will be restored from the non-default state to the default state.

[0264] In this way, users can flexibly adjust the display position of the Dock bar in XR space according to their actual needs, avoiding the problem of the Dock bar obscuring other XR windows.

[0265] (II) Regarding the commonly used Dock bar in XR space

[0266] (1) Functions and activation methods of the transient Dock bar

[0267] Building upon the aforementioned Dock, this embodiment also provides a transient frequently used application bar (or frequently used Dock, transient Dock, etc.). This transient frequently used application bar can display the application icons of minimized applications and / or recommended applications. This allows users to open applications using the icons displayed in the transient frequently used application bar, eliminating the need to search for application icons on the desktop and enhancing the user's immersive experience.

[0268] For example, as shown in Figure 7a(1), a Dock bar 701 and a desktop window 704 are displayed in the virtual space. The Dock bar 701 displays a large desktop icon 7011. Continuing to refer to Figure 7a(1), in response to the user's gaze at the large desktop icon 7011, a user viewpoint 702 is displayed on the large desktop icon 7011. In response to the user's gaze at the large desktop icon 7011 reaching a preset duration (e.g., 2 seconds), a temporary frequently used application bar 703 is displayed in the virtual space, as shown in Figure 7a(2). Continuing to refer to Figure 7a(2), the temporary frequently used application bar 703 includes a minimized application area 7031 and a recommended application area 7032.

[0269] The minimized application area 7031 is used to store the application icon of at least one application that has been minimized and collapsed by the user. For example, when the application window of application 1 is displayed in XR space, the user minimizes the application window of application 1. At this time, application 1 is the application that has been minimized and collapsed by the user, and then the icon of application 1 will be added to the minimized application area 7031. For example, if the user does not minimize any application window, that is, there is no application that has been minimized and collapsed by the user, the minimized application area 7031 disappears. At this time, the transient frequently used application bar 703 only includes the recommended application area 7032, as shown in Figure 7b (1).

[0270] The recommended application area 7032 is used to store a set number (e.g., 3) of application icons recommended to the user. For example, the applications recommended to the user can change dynamically. For instance, the recommended applications might be those recently used by the user, such as the recommended application area 7032 storing the application icons of the three most recently used applications.

[0271] Optionally, to avoid duplicate inclusion of the same applications in the minimized application area 7031 and the recommended application area 7032, it is necessary to deduplicate the applications included in both areas. Taking the recommended application area 7032 containing the icons of application B, application X, and application Y as an example, if the windows of application B, application X, and application Y are all minimized by the user, then the icons of application B, application X, and application Y are added to the minimized application area 7031, and the recommended application area 7032 does not need to duplicate the icons of application B, application X, and application Y. At this time, the recommended application area 7032 disappears, that is, the temporary frequently used application bar 703 only includes the minimized application area 7031, as shown in Figure 7b (2).

[0272] For example, the minimized application area 7031 can display up to N (e.g., 5) application icons simultaneously. When a user minimizes more than N applications, the user can view the icons of other minimized applications by performing a gaze and pinch-to-swipe operation within the minimized application area 7031. For example, if the minimized applications include application A, application C, application D, application E, application F, application G, and application H, and the minimized application area 7031 simultaneously displays icons for application A, application C, application D, application E, and application F, then in response to the user's gaze at the minimized application area 7031, the user's viewpoint is displayed on the minimized application area 7031; in response to the user's pinch-to-swipe operation, the application A icon is slid out of the minimized application area 7031, and the application G icon is slid into the minimized application area 7031. At this time, the application C icon, application D icon, application E icon, application F icon, and application G icon are displayed sequentially in the minimized application area 7031. In response to the user's pinch-and-swipe gesture to the left, app icon C slides out of the minimized app area 7031, and app icon F slides into the minimized app area 7031. At this point, app icons D, E, F, G, and H are displayed sequentially in the minimized app area 7031. It's understandable that a user's pinch-and-swipe gesture to the right can trigger the app icons displayed in the minimized app area 7031 to adjust in the opposite direction; this will not be elaborated further.

[0273] Figure 7c illustrates the position of the transient frequently used application bar 703 in XR space. As shown in Figure 7c (1), in the virtual space, the plane where the desktop window 704 is located and the plane where the transient frequently used application bar 703 is located are both perpendicular to the horizontal plane and perpendicular to the user's eye level. The spatial distance between the transient frequently used application bar 703 and the user can be less than the spatial distance between the desktop window 704 and the user. Optionally, the spatial distance between the plane where the transient frequently used application bar 703 is located and the plane where the desktop window 704 is located is a fixed value of 5 (e.g., 0.2m). Optionally, the upper boundary of the Dock bar 701 can be located on the plane where the transient frequently used application bar 703 is located, or the horizontal distance between the upper boundary of the Dock bar 701 and the user is equal to or the spatial distance between the plane where the transient frequently used application bar 703 is located and the user, or the difference between the two is less than a preset threshold.

[0274] As an optional implementation, if the spatial distance between the desktop window 704 and the user is less than or equal to a fixed value of 2 (e.g., 1.5m) before the user invokes the transient frequently used applications bar 703, then after the user invokes the transient frequently used applications bar 703, the desktop window 704 moves backward, increasing the spatial distance between the desktop window 704 and the user by a fixed value of 5 (e.g., 0.2m), and the transient frequently used applications bar 703 is created and displayed at a position where the spatial distance between the desktop window 704 and the user is a fixed value of 2. At this time, the spatial distance between the transient frequently used applications bar 703 and the user is a fixed value of 2, and the spatial distance between the desktop window 704 and the user is a fixed value of 6 (the value of the fixed value of 6 is the sum of the fixed values ​​of 2 and 5), as shown in Figure 7d.

[0275] If the spatial distance between the desktop window 704 and the user is greater than a fixed value of 2 before the user invokes the transient frequently used application bar 703, then after the user invokes the transient frequently used application bar 703, the position of the desktop window 704 remains unchanged, and the transient frequently used application bar 703 is created and displayed at a position where the spatial distance between the user and the user is a fixed value of 2.

[0276] In this embodiment, desktop window 704 is used as an example for explanation; the same applies to other application windows, and will not be repeated here. If, before the user invokes the transient frequently used application bar 703, the virtual space within the user's current visible range does not include any XR window, then in response to the user's operation of invoking the transient frequently used application bar 703, the transient frequently used application bar 703 is created and displayed at a position with a fixed distance of 2 from the user's space.

[0277] In this way, users can flexibly bring up the temporary frequently used application bar according to their actual needs, and can quickly open application windows through the application icons displayed in the temporary frequently used application bar, thus enhancing the user's immersive experience.

[0278] (2) Regarding common ways to collapse the Dock bar in transient situations

[0279] After the user invokes the temporary frequently used applications bar, the temporary frequently used applications bar can be collapsed. For example, when the temporary frequently used applications bar 707 is displayed in the virtual space, the user looks at any blank area outside the temporary frequently used applications bar 707 (that is, the area does not include any XR objects). As shown in Figure 7e (1), in response to the user operation, the user's viewpoint 708 is displayed at the position the user is looking at in the virtual space. At this time, the user pinches two fingers together. In response to the user operation, the temporary frequently used applications bar 707 is de-displayed in the virtual space, as shown in Figure 7e (2).

[0280] For another example, when the transient frequently used applications bar 707 is displayed in the virtual space, the user gazes at any blank area outside the transient frequently used applications bar 707. In response to this user operation, the user's viewpoint 708 is displayed at the location the user is gazing at in the virtual space. If the user's gaze duration reaches a preset value (e.g., 2 seconds), the transient frequently used applications bar 707 is dedisplayed in the virtual space in response to this user operation.

[0281] It should be noted that this scenario uses the display of desktop windows and Dock in virtual space as an example. When other XR interfaces (such as application windows) and Dock are displayed in virtual space, or when only Dock is displayed in virtual space, the way users bring up and collapse the temporarily frequently used application bar in virtual space is the same, and this embodiment will not elaborate on this.

[0282] In this way, users can flexibly collapse the transient frequently used application bar according to actual needs, avoiding the display of too many XR objects in the XR space and affecting the user's visual experience.

[0283] (III) Application Windows in XR Space

[0284] The following explains the operations related to application windows. Application windows can be opened either through the application icon displayed on the desktop window or through the application icon displayed in the temporary frequently used applications bar.

[0285] (1) Regarding the method of opening application windows through application icons in the desktop window, and the structure of application windows.

[0286] As shown in Figure 8a(1), a desktop window 801 is displayed in the virtual space, and multiple application icons are displayed in the desktop window 801, such as application B icon 8011. The position of the desktop window 801 in the XR space can be referred to as shown in Figure 8a(2). Continuing to refer to Figure 8a(1), in response to the user's operation of looking at application B icon 8011, the user's viewpoint 802 is displayed on application B icon 8011. In response to the user pinching two fingers together to start application B, the window 803 of application B is displayed in the virtual space, and the desktop window 801 is canceled, as shown in Figure 8b(1). The window 803 of application B may include, but is not limited to, the application bar 8031, the window body area 8032, the window movement control 8033, and the tab bar 8034. The position of the window 803 of application B in the XR space can be referred to as shown in Figure 8b(2). Comparing Figure 8a (2) and Figure 8b (2), it can be seen that the position of application B window 803 is the same as the position of the desktop window 801.

[0287] (2) Explanation of the application bar in the application window

[0288] Referring again to Figure 8b(1), the application bar 8031 ​​may include, but is not limited to, the application icon and name 80311, the window minimize control 80312, the window isolation control 80313 (for a detailed explanation of this control, please refer to the example corresponding to Figure 8m), and the window close control 80314. The main window area 8032 is used to display the program interface of application B. The window movement control 8033 is used to move the window 803 of application B in virtual space.

[0289] (3) Explanation of the tab bar in the application window

[0290] The tab bar 8034 may include at least one function icon. This function icon refers to the function icon displayed at the bottom of the page when the application is running on an electronic device (such as a mobile phone). For example, taking a gallery application as an example, when the gallery application is running on a mobile phone, the function icons displayed at the bottom of the page are the four function icons 8041 shown in Figure 8c (1). When the window of the gallery application is displayed in virtual space, these function icons are displayed in the tab bar on the left side of the main area of ​​the window, as shown by the four function icons 8042 in Figure 8c (2). That is, the four function icons 8041 and the four function icons 8042 are in a one-to-one mapping relationship. It is understandable that when the application is running on an electronic device (such as a mobile phone), if there are no function icons displayed at the bottom of the page, the application window may not display a tab bar when displayed in virtual space.

[0291] (4) About the names of the function icons in the tab bar of the application window

[0292] In one alternative implementation, the tab bar includes at least one function icon and a name for each function icon.

[0293] In another alternative implementation, the tab bar displays only function icons, not their names. For example, in response to a user gazing at a function icon for a preset duration (e.g., 1.5 seconds), the name of the function icon is displayed in the area near the icon to provide an operation prompt to the user. As shown in Figure 8d (1), in response to a user gazing at function icon 80341 in the tab bar 8034, the user's viewpoint is displayed on function icon 80341. When the user's gazing duration reaches the preset duration (e.g., 1.5 seconds), the name 80342 of function icon 80341 is displayed near function icon 80341, as shown in Figure 8d (2). As another example, in response to a user gazing at any function icon in the tab bar for a preset duration (e.g., 1.5 seconds), the corresponding name is displayed in the area near each function icon in the tab bar to provide an operation prompt to the user. As shown in Figure 8d(1), in response to the user's gaze at the function icon 80341 in the tab bar 8034, the user's viewpoint is displayed on the function icon 80341. When the user's gaze duration reaches a preset duration (e.g., 1.5s), the corresponding name is displayed near each function icon in the tab bar 8034, as shown in Figure 8d(3) for the function icon name 80343.

[0294] In this way, displaying the corresponding function name near each function icon in the tab bar allows users to understand the meaning of each function icon and avoids the problem of not knowing how to operate due to not understanding the meaning of the icon.

[0295] (5) Hide and unhide the application bar in the application window.

[0296] In this embodiment, when a user uses an application window, the application bar of that application window can be hidden (or undisplayed) in virtual space. When the user needs to use the relevant functions in the application bar, the application bar is displayed again. For example, in response to an operation performed by the user in the main area 8032 of the window, the application bar 8031 ​​is undisplayed in virtual space. As another example, after undisplaying the application bar 8031, in response to an operation performed by the user outside the main area 8032 of the window, such as the user looking at the area above the main area 8032 of the window, or the user looking at the area above the main area 8032 of the window for a preset duration (e.g., 1.5 seconds), the application bar 8031 ​​is redisplayed in virtual space.

[0297] In this way, users can control whether the application bar of an application window is hidden or not, according to their actual needs. When the application bar of an application window is hidden, users can immerse themselves in using the corresponding application.

[0298] (6) Regarding the names of the functional controls in the application bar of the application window.

[0299] To provide operation prompts to users, when a user gazes at an operation control in the application bar for a preset duration (e.g., 1.5s), the name of the corresponding operation control can be displayed in the vicinity of the operation control. For example, in response to a user gazing at an operation control for a preset duration (e.g., 1.5s), the name of the operation control is displayed in the vicinity of the operation control to provide operation prompts to the user. As shown in Figure 8e (1), in response to a user gazing at a window isolation control 80313 in the application bar, the user's viewpoint is displayed on the window isolation control 80313. When the user's gazing time reaches the preset duration (e.g., 1.5s), the name 80315 of the window isolation control 80313 is displayed near the window isolation control 80313, as shown in Figure 8e (2).

[0300] For example, in response to a user's gaze at any operation control in the application bar for a preset duration (e.g., 1.5s), the corresponding name is displayed in the area near each operation control in the application bar to provide operation prompts to the user. As shown in Figure 8e (1), in response to a user's gaze at the window isolation control 80313 in the application bar, the user's viewpoint is displayed on the window isolation control 80313. When the user's gaze duration reaches the preset duration (e.g., 1.5s), the corresponding name is displayed in the area near each operation control in the application bar, such as the operation control name 80316 shown in Figure 8e (3).

[0301] Optionally, in response to a user's gaze lingering on any element in the application window for a preset duration (e.g., 1.5 seconds), the corresponding name is displayed near each element in the application window. At this time, the function icon name 80343 shown in Figure 8d (3) and the operation control name 80316 shown in Figure 8e (3) can be simultaneously referenced to provide operation prompts to the user. The elements include at least the function icons in the tab bar and the operation controls in the application bar.

[0302] In this way, displaying the corresponding function names on the attachments of each element in the application window allows users to understand the meaning of each icon and control, thus avoiding the problem of not knowing how to operate due to a lack of understanding of the meaning of the icons and controls.

[0303] (7) Regarding the use of controls in the application bar of the application window

[0304] Users can minimize application windows using the window minimize control displayed in the application window. Referring to Figure 8f (1), application window 805 is displayed in virtual space, and the user looks at the window minimize control 8051 of application window 805. In response to the user's operation, the user's viewpoint is displayed on the window minimize control 8051. At this time, the user pinches two fingers together. In response to the user's operation, application window 805 is minimized, and the icon of application B is added to the minimized application area 8061 in the temporary frequently used applications bar 806, as shown in Figure 8f (2). For example, the process of minimizing application window 805 can be displayed by an animation of a preset duration (such as 1 second), for example, application window 805 gradually moves away from the user until it disappears. For example, as application window 805 gradually disappears, the transparency of application window 805 gradually increases (i.e., becomes more and more transparent). For example, the process of adding the icon 8062 of application B to the minimized application area 8061 can also be displayed through an animation of a preset duration (e.g., 1 second). For instance, the icon 8062 of application B is added to the rightmost side of the minimized application area 8061, and the other application icons are adaptively shifted to the left (or hidden). Optionally, after the icon of application B is added to the minimized application area 8061, its size is enlarged, as shown by the icon 8062 of application B. The icon 8062 of application B can also be highlighted in the minimized application area 8061 by means of glowing or bright flashing. In this scenario, after the icon of application B is added to the minimized application area 8061, the transient frequently used application bar 806 automatically disappears after the preset duration (e.g., 1 second).

[0305] Users can also close the application window using the window close control displayed in the application window. For example, referring to Figure 8b (1), if the user looks at the window close control 80314, the user's viewpoint is displayed on the window close control 80314 in response to the user's action. If the user then pinches their fingers together, the application window 803 is closed in response to the user's action. For example, the process of closing the application window 803 can be shown through an animation of a preset duration (e.g., 1 second), such as the application window 803 gradually moving away from the user until it disappears. For example, as the application window 803 gradually disappears, the transparency of the application window 803 gradually increases.

[0306] Users can also isolate other application windows and application window groups (see below) using the window isolation controls displayed in the application window. Isolating other application windows and application window groups can be understood as hiding or minimizing them. For an illustration of how a user isolates other application windows using the window isolation controls within an application window, please refer to the example below corresponding to Figure 8m.

[0307] (8) Regarding the method of opening application windows through application icons in the transient frequently used application bar

[0308] Figures 8g-8i illustrate exemplary scenarios of users opening application windows based on application icons displayed in the transient frequently used applications bar.

[0309] As shown in Figure 8g(1), application window 807 and Dock bar 808 are displayed in the virtual space. The positions of application window 807 and Dock bar 808 in the XR space can be illustrated in Figure 8g(2). Application window 807 is displayed directly in front of the user. Dock bar 808 can be activated by the user through a global gesture or through a relevant physical button.

[0310] Referring to Figure 8g(1), in response to the user's gaze at the large desktop icon 8081, the user's viewpoint is displayed on the large desktop icon 8081. In response to the user gazing at the large desktop icon 8081 for a preset duration (e.g., 2 seconds), a temporary frequently used application bar 809 is displayed in the virtual space, as shown in Figure 8h(1). The positions of the temporary frequently used application bar 809 and application window 807 in the XR space can be illustrated in Figure 8h(2).

[0311] In this scenario, if the spatial distance between application window B 807 and the user is less than or equal to a fixed value of 2 (e.g., 1.5m) before the user invokes the temporary frequently used applications bar 809, then application window B 807 moves backward, making the spatial distance between application window B 807 and the user a fixed value of 6 (the fixed value of 6 is the sum of fixed values ​​2 and 5 (e.g., 0.2m)). The temporary frequently used applications bar 809 is then created and displayed at a fixed distance of 2 from the user. If the spatial distance between application window B 807 and the user is greater than the fixed value of 2 (e.g., 1.5m) before the user invokes the temporary frequently used applications bar 809, then when the user invokes the temporary frequently used applications bar 809, the position of application window B 807 remains unchanged, and the temporary frequently used applications bar 809 is created and displayed at a fixed distance of 2 from the user.

[0312] In this way, the transient frequently used applications bar will be displayed in a better viewing position in the virtual space, and the original windows displayed in the virtual space will not obstruct the transient frequently used applications bar, thus ensuring the user's visual experience.

[0313] Optionally, as shown in Figure 8h (2), the Dock bar 808 can be adaptively adjusted to follow the spatial position of the transient frequently used applications bar 809, so that the upper boundary of the Dock bar 808 can be located on the plane where the transient frequently used applications bar 809 is located.

[0314] Referring to Figure 8h (1), the minimized application area of ​​the transient frequently used application bar 809 includes the icon of application D. In response to the user's gaze on the icon of application D, the user's viewpoint is displayed on the icon of application D, and the size of the icon of application D is enlarged, as shown in the application D icon 8091. At this time, in response to the user's pinch operation, the application D window 810 is maximized, and the transient frequently used application bar 809 disappears, as shown in Figure 8i (1). The positions of the application D window 810 and the application B window 807 in XR space can be shown in Figure 8i (2). As shown in Figure 8i (2), the transient frequently used application bar 809 disappears, and the application D window 810 is created and displayed in the original position of the transient frequently used application bar 809, while the position of the Dock bar 808 can remain unchanged.

[0315] In this way, the user's most recently opened application window, such as application window D 810, is kept at a fixed spatial distance of 2 from the user, ensuring the best visual experience. It should be noted that in this embodiment, if other application windows are displayed at a position where the spatial distance from the user is less than or equal to the fixed value 2, these other windows need to be moved away from the user, for example, to a position where the spatial distance from the user is a fixed value 6, to avoid obstructing the user's most recently opened application window.

[0316] It should be noted that newly opened application windows are displayed directly in front of the user, perpendicular to the user's eye level. As the user's head turns, the display orientation (or direction) of each application window opened by the user also changes. For example, as shown in Figure 8j (1), the user opens application window M in XR space, and the orientation of application window M is shown as direction 1 in the figure. Optionally, the spatial distance between application window M and the user is a fixed value of 2. Continuing to refer to Figure 8j (2), the user turns to the left, that is, the user's head turns to the left. At this time, the user opens application window N in XR space, and the orientation of application window N is shown as direction 2 in the figure. Optionally, the spatial distance between application window N and the user is a fixed value of 2. It can be seen that newly opened application windows are always displayed directly in front of the user, perpendicular to the user's eye level. As the user's head turns, the user's eye level will change, and thus the orientation of the application windows opened before and after the user's head turns will be different. If the user's head position is exactly the same when opening application window D 810 and application window B 807, that is, the user's eye level does not change, then the application window D 810, which is opened later, will completely block application window B 807, and the spatial distance between the two will be a fixed value of 5 (e.g., 0.2m).

[0317] The above explanation uses the example of a user opening an application window by minimizing the application icon in the application area. The same applies to users opening application windows by minimizing the application icon in the recommended application area, so it will not be repeated here.

[0318] (9) Adjusting the size of the application window

[0319] For any application window displayed in the virtual space, the user can adjust the size of the application window using "hand and eye control".

[0320] As shown in Figure 8k(1), application window 811 is displayed in virtual space. In response to the user's gaze at the lower right corner of application window 811, the user's viewpoint 812 is displayed at the lower right corner of application window 811. In response to the user's gaze at the lower right corner of application window 811 reaching a preset duration (e.g., 1 second), a window scaling control 813 is displayed at the lower right corner of application window 811, as shown in Figure 8k(2). At this time, in response to the user's pinch and move operation, the size of application window 811 is scaled. For example, in response to the user's pinch and move operation away from the user, the size of application window 811 is shrunk, as shown in Figure 8k(3). Correspondingly, in response to the user's pinch and move operation closer to the user, the size of application window 811 is enlarged. When scaling the size of application window 811, in response to the user releasing their two fingers, the size of application window 811 no longer changes, and the window scaling control 813 is de-displayed. At this point, the size of window 811 in application B has been adjusted.

[0321] It should be noted that the scaling operation is not limited to the above methods. For example, after displaying the window scaling control 813, the user can pinch with two fingers to shrink the window, and open the pinch with two fingers to enlarge the window.

[0322] As mentioned earlier, application window B includes an application bar, a main window area, window movement controls, and a tab bar. During the scaling of application window B, the application bar, main window area, window movement controls, and tab bar are all scaled proportionally. For example, the maximum size (width * height) of the application window is 576cm * 324cm, the default size is 192cm * 108cm, and the minimum size is 32cm * 18cm. That is, when a user opens an application window, its size is the default size of 192cm * 108cm, and the user can adjust the size of the application window according to their needs.

[0323] As an optional implementation, when the application window is shrunk to a preset size (or critical size), in response to the user's further shrinking operation, the main area of ​​the window continues to shrink, while the sizes of the application bar, window movement controls, tab bar, and window resizing controls remain unchanged. For example, when the application window is shrunk to the preset size, the radius of the controls in the application bar (such as the window minimize control, window isolation control, and window close control) is 9cm.

[0324] In this way, the size of controls in the application bar and tab bar, as well as window movement and window scaling controls, will not continue to shrink. This avoids the problem that users may not be able to focus on the relevant hot areas due to the small size of the controls, and will not affect the user's "hand-eye control" experience.

[0325] (10) Regarding the adjustment of the application window position

[0326] For any application window displayed in the virtual space, the user can adjust its position using the window movement controls. Understandably, when using XR glasses, newly opened application windows in the virtual space are always displayed directly in front of the user, and their positions do not change automatically. Thus, as the user turns their head or moves, application windows may appear in different locations within the virtual space, and these windows may obstruct each other, affecting the user's visual experience. Therefore, the user can adjust the display position of application windows in the virtual space using the window movement controls to meet their usage needs.

[0327] As shown in Figure 8l(1), application window B 814 and application window A 815 are displayed in the virtual space, with application window A 815 partially obscuring application window B 814. At this time, the user can adjust the display position of application window A 815 in the virtual space. Continuing to refer to Figure 8l(1), in response to the user's operation of the window movement control 8151 of application window A 815, the user's viewpoint 816 is displayed on the window movement control 8151. At this time, in response to the user's operation of pinching and dragging with two fingers, application window A 815 moves in the virtual space accordingly, and the movement of application window A 815 is responsive, as shown in Figure 8l(2). During the dragging process of application window A 815, application window A 815 is always facing the user, that is, application window A 815 moves with the user as the center. When application window A 815 is dragged to the target position, the user releases two fingers. In response to the user's operation, application window A 815 is displayed at the target position, and the user's viewpoint 816 on the window movement control 8151 disappears, as shown in Figure 8l (3). At this point, the display position of application window A 815 in the virtual space is adjusted. Optionally, when application window A 815 is not dragged, it is displayed in a completely opaque manner; during the dragging process, it is displayed in a semi-transparent manner, for example, with a preset transparency value (e.g., 50%), to enhance the user's visual perception and make the user clearly understand the object being moved; when the dragging of application window A 815 is complete, it is restored to completely opaque. Optionally, during the movement of application window A 815 in the virtual space, the user's viewpoint 816 is in a moving state, as shown in Figure 8l (2).

[0328] In this way, users can flexibly adjust the display position of the application window in XR space according to their actual needs, thereby improving the user's visual experience.

[0329] (11) Regarding the isolation of application windows

[0330] As mentioned earlier, users can open multiple application windows in the virtual space, and can drag any application window to adjust its display position. This allows multiple application windows to be displayed simultaneously in the virtual space. These application windows may or may not overlap.

[0331] In some scenarios, a user may only need to use one application window at a time, and displaying the other application windows might lead to a poor visual experience. In this case, the user can keep only the application window they need and hide the rest that they don't need for the time being. When the user needs to use the other application windows that were isolated, they can also control these application windows to reappear.

[0332] As shown in Figure 8m(1), application window B 817, application window C 818, and application window A 819 are simultaneously displayed in the virtual space. Assuming that the user only needs to use application window B 817 at the current stage and does not need to use application window C 818 and application window A 819 for the time being, the other application windows can be isolated by the window isolation control 8171 of application window B 817, that is, the other application windows are hidden. Continuing to refer to Figure 8m(1), in response to the user's operation of looking at the window isolation control 8171 of application window B 817, the user's viewpoint 8172 is displayed on the window isolation control 8171. At this time, in response to the user's operation of pinching with two fingers, application window C 818 and application window A 819 are hidden (or canceled), as shown in Figure 8m(2). Continuing to refer to Figure 8m(1) and (2), in response to the user's operation of clicking the window isolation control 8171, the state of the window isolation control 8171 is updated from the unselected state to the selected state. When the window isolation control 8171 is selected, it indicates that an application window is currently isolated. When the user needs to use the isolated application window again, they can control the window isolation control 8171 to make it re-display.

[0333] Referring to Figure 8m (2), when the window isolation control 8171 is selected, in response to the user's gaze on the window isolation control 8171, the user's viewpoint 8173 is displayed on the window isolation control 8171. At this time, in response to the user's pinching action, the currently isolated application windows are redisplayed, that is, application C window 818 and application A window 819 are redisplayed, as shown in Figure 8m (3).

[0334] In this way, users can isolate (i.e. hide) application windows that are not currently in use according to their actual needs, avoiding the problem of too many application windows being displayed in the XR space, which would affect the user's visual experience, and also avoiding the problem of application windows being obscured in the XR space.

[0335] (iv) Application window groups in XR space

[0336] (1) About the structure of application window groups

[0337] In some scenarios, users may use multiple applications simultaneously. To provide users with operational convenience, this embodiment can also display application window groups in the virtual space.

[0338] Each application window group may include a window group application bar, a window group move control, and at least two application windows. The window group application bar may include, but is not limited to, a window group minimize control, a window group isolate control, and a window group close control.

[0339] It should be noted that, unlike standalone application windows, each application window in an application window group only includes a minimize control in its application bar, excluding close and isolation controls. Furthermore, unlike standalone application windows, each application window in an application window group does not display window movement controls; the window movement controls for a specific application window within the application window group can be invoked when needed.

[0340] Figure 9a illustrates an example of an application window group displayed in virtual space. As shown in Figure 9a (1), the application window group 901 includes a window group application bar 9015, a window group movement control 9014, and application window F 9011, application window D 9012, and application window E 9013. Application window F 9011 is used to display the program interface of application F, application window D 9012 is used to display the program interface of application D, and application window E 9013 is used to display the program interface of application E.

[0341] The application group application bar 9015 may include, but is not limited to, a window group minimize control, a window group isolate control, and a window group close control. The window group minimize control is used to minimize the application window group 901, the window group isolate control is used to isolate other application windows and application window groups in the virtual space except for application window group 901, and the window group close control is used to close application window group 901.

[0342] Optionally, when a user minimizes an application window group using the window group minimize control, the icon of that application window group is added to the minimized application area of ​​the temporary frequently used applications bar. The icon of the application window group can refer to the application window group icon shown in Figure 11c (1), but this embodiment does not limit it.

[0343] The window group movement control 9014 is used to adjust the position of the application window group 901 in virtual space. An example of how a user adjusts the position of the application window group 901 in virtual space using the window group movement control 9014 can be found in the example of adjusting the position of an application window in virtual space using the window movement control; this will not be repeated here.

[0344] Similarly, users can also adjust the size of the application window group 901 using "hand-eye control". For example, in response to the user looking at the lower right corner of the application window group 901, the user's viewpoint is displayed in the lower right corner of the application window group 901. In response to the user looking at the lower right corner of the application window group 901 for a preset duration (e.g., 1 second), a window group scaling control is displayed in the lower right corner of the application window group 901. At this time, in response to the user's pinch-and-move gesture, the size of the application window group 901 is scaled.

[0345] Referring to Figure 9a(1), the application bars of application windows F 9011, D 9012, and E 9013 only include a minimize control, as shown by the minimize control 90111 displayed in the application bar of application window F 9011. The positional relationship of application windows F 9011, D 9012, and E 9013 in the virtual space can be seen in Figure 9a(2).

[0346] In this way, application window F 9011, application window D 9012, and application window E 9013 are aggregated in application window group 901 and displayed around the user. This not only makes it convenient for the user to use these applications at the same time, but also eliminates the need for the user to adjust the display position of each application window individually, thus improving the user experience.

[0347] (2) About creating application window groups

[0348] In this embodiment, an application window group can be formed by interlocking (or aggregating) two application windows.

[0349] Figures 9b and 9c exemplify schematic diagrams of a user creating an application window group. As shown in (1) of Figure 9b and (1) of Figure 9c, application window F 902 and application window D 903 are displayed in virtual space. The user drags application window D 903 towards application window F 902 using the window movement control 9031 of application window D 903, so that application window D 903 is closer to application window F 902. When the positional relationship between application window D 903 and application window F 902 meets the interlocking condition, the window movement control 9021 of application window F 902 and the window movement control 9031 of application window D 903 interlock to generate an interlocking confirmation control 904, as shown in (2) of Figure 9b. The display of the interlocking confirmation control 904 indicates that application window D 903 and application window F 902 are in an interlocking pending confirmation state. If the user confirms, the two interlock to form an application window group. Referring again to Figure 9b(2), in response to the user's pinch-to-zoom operation, application window D 903 and application window F 902 are interlocked to generate application window group 905, as shown in Figure 9b(3) and Figure 9c(2). Application window group 905 includes application window F 9051, application window D 9052, window group movement control 9053, and window group application bar 9054.

[0350] In this embodiment, the window group movement control 9053, also known as the interlock control or interlock bar, can be derived from the interlock confirmation control 904.

[0351] Referring to Figure 9c (2), in order to ensure the user experience, the spatial distance between each application window in the application window group 905 and the user is a fixed value of 2 (e.g., 1.5m).

[0352] Understandably, when the interlock confirmation control 904 is displayed, if the user does not confirm (e.g., does not perform the pinch-to-open operation), but continues to drag application window D 903, then application window D 903 and application window F 902 will not be interlocked and will remain independent application windows.

[0353] For example, if the angle between application window D 903 and application window F 902 is within a preset range, then the positional relationship between application window D 903 and application window F 902 is determined to satisfy the interlocking condition.

[0354] For example, if the window movement control of application window D 903 and the window movement control of application window F 902 meet preset conditions, then it is determined that the positional relationship between application window D 903 and application window F 902 meets the interlocking condition. For instance, if the overlapping volume of the window movement control of application window D 903 and the window movement control of application window F 902 reaches a preset proportion (e.g., 47%) of the volume of the window movement control itself, then it is determined that the positional relationship between application window D 903 and application window F 902 meets the interlocking condition.

[0355] Figure 9d illustrates an exemplary schematic diagram of application window interlocking based on window movement controls. As shown in Figure 9d (1), when the user drags application window D 903 toward application window F 902, the three-dimensional area 9032 occupied by the window movement control 9031 of application window D 903 overlaps with the three-dimensional area 9022 occupied by the window movement control 9021 of application window F 902. If the overlapping volume of the two reaches a preset proportion (e.g., 47%) of the volume of the window movement control itself (i.e., the volume of the three-dimensional area 9032 or the volume of the three-dimensional area 9022), then it is determined that the positional relationship between application window D 903 and application window F 902 meets the interlocking condition, and the window movement control 9031 of application window D 903 and the window movement control 9032 of application window F 902 interlock to generate an interlocking confirmation control 904, as shown in Figure 9d (2). At this point, if the user confirms, application window D 903 and application window F 902 will interlock to generate application window group 905, and the interlock confirmation control 904 will change to the window movement control 9053 of application window group 905, as shown in Figure 9d (3). It can be understood that when the interlock confirmation control 904 is displayed, if the user does not confirm but continues to drag application window D 903, the interlock confirmation control 904 will revert to the window movement control 9031 of application window D 903 and the window movement control 9032 of application window F 902.

[0356] Optionally, as shown in Figure 9c (2), the application window (i.e., application D window) dragged by the user is the application window located directly in front of the user in the application window group.

[0357] In this embodiment, the application window group can also be formed by combining application windows and interlocking application windows.

[0358] Figures 9e and 9f exemplify schematic diagrams of user-created application window groups. As shown in (1) of Figure 9e and (1) of Figure 9f, application window group 905 and application window E 906 are displayed in virtual space. Application window group 905 includes application window D 9052 and application window F 9051. The user drags application window E 906 towards application window group 905 using the window movement control 9061 of application window E 906 to bring application window E 906 closer to application window group 905. When the positional relationship between application window E 906 and application window group 905 meets the interlocking condition, the interlocking confirmation control 908 is displayed. Specifically, when the window movement control 9061 of application window E 906 and the window group movement control 9053 of application window group 905 meet the interlocking condition, the interlocking confirmation control 908 is generated and displayed, as shown in (2) of Figure 9e.

[0359] The display interlock confirmation control 908 indicates that application window E 906 and application window group 905 are in an interlock pending confirmation state. If the user confirms, the two interlock to form a new application window group. Referring to Figure 9e (2), in response to the user's pinch operation, application window E 906 and application window group 905 interlock to generate application window group 907, as shown in Figure 9e (3) and Figure 9f (2).

[0360] The application window group 907 includes application window F 9071, application window D 9072, application window E 9073, window group movement control 9074, and window group application bar 9075. In this embodiment, the window group movement control 9075 may be derived from the interlock confirmation control 908.

[0361] To ensure a good user experience, the spatial distance between each application window in application window group 907 and the user can be a fixed value of 2 (e.g., 1.5m).

[0362] Understandably, when the interlock confirmation control 908 is displayed, if the user does not confirm (e.g., does not perform the pinch-to-open operation), but continues to drag the application window 906, the application window 906 and the application window group 905 will not be interlocked, and the application window 906 will remain an independent application window.

[0363] For example, if the window movement control 9061 of application window 906 and the window group movement control 9053 of application window group 905 meet preset conditions, then the positional relationship between application window 906 and application window group 905 is determined to meet the interlocking condition. For instance, if the overlapping volume of the window movement control 9061 of application window 906 and the window group movement control 9053 of application window group 905 reaches a preset proportion (e.g., 47%) of the volume of the window movement control (or window group movement control) itself, then the positional relationship between application window 906 and application window group 905 is determined to meet the interlocking condition. The determination of the interlocking between application window 906 and application window group 905 based on the window movement control 9061 of application window 906 and the window group movement control 9053 of application window group 905 is similar to the scenario shown in Figure d, and will not be repeated here.

[0364] In the examples shown in Figures 9e and 9f, the application window dragged by the user (i.e., application window E) is not the application window displayed directly in front of the user in the newly created application window group. Optionally, when the user drags application window E and interlocks it with application window group 905 to form a new application window group, application window E is the application window displayed directly in front of the user in the newly created application window group. For example, in the newly created application window group, the three application windows displayed in a ring are arranged left and right as application window F, application window E, and application window D, respectively. In this way, when the user uses the newly created application window group, application window E can be displayed directly in front of the user.

[0365] The above explanation uses an application window group consisting of two or three application windows as an example. Understandably, an application window group can include more (e.g., four, five, etc.) application windows to suit the user's actual needs. The methods for creating application window groups are similar and will not be repeated here. It should be noted that each application window in a user-created application window group is displayed in a ring around the user, with each application window having a fixed spatial distance of 2 (e.g., 1.5m) from the user, and at least one application window is perpendicular to the user's eye level to improve the user experience.

[0366] (3) Regarding the orientation adjustment of application window groups

[0367] In this embodiment, the user can also adjust the orientation of the application window group. For example, the application window group can rotate clockwise or counterclockwise around the user. For instance, the application window group can rotate clockwise or counterclockwise by a preset angle to adjust the orientation of the application window directly in front of the user.

[0368] Taking an application window group containing two application windows as an example, Figures 10a and 10b exemplarily illustrate a schematic diagram of adjusting the orientation of the application window group. Referring to (1) in Figure 10a and (1) in Figure 10b, the application window group 1001 includes two application windows, namely application F window 10011 and application D window 10012. Application F window 10011 and application D window 10012 are displayed around the user, with application D window 10012 displayed directly in front of the user.

[0369] Since each application window in the application window group does not display window movement controls, the window movement controls of a specific application window in the application window group can be brought up when needed by the user. For example, referring to Figure 10a (1), the user is looking at the lower area of ​​application window F 10011. In response to this user operation, the user's viewpoint 10013 is displayed in the lower area of ​​application window F 10011. When the user's viewing time reaches a preset threshold (e.g., 1 second), the window movement control 10014 of application window F 10011 is displayed in the lower area of ​​application window F 10011, as shown in Figure 10a (2). At this time, in response to the user's pinching operation, application window group 1001 rotates clockwise around the user by a preset angle, so that the application window facing the user directly changes from application window D 10012 to application window F 10011, as shown in Figure 10a (3) and Figure 10b (2).

[0370] Similarly, the same applies to an application window group comprising three application windows. For example, consider an application window group consisting of application A, application B, and application C. Application A, application B, and application C are displayed around the user, with application B positioned directly in front of the user. For instance, the user is looking at the lower area of ​​application A. In response to this user action, the user's viewpoint is displayed in the lower area of ​​application A. When the user's gaze duration reaches a preset threshold (e.g., 1 second), the window movement controls for application A are displayed in the lower area of ​​application A. Then, in response to the user's pinch-to-zoom action, the application window group rotates clockwise around the user, positioning application A directly in front of the user.

[0371] This allows users to flexibly adjust the orientation of application window groups according to their actual needs, enhancing their visual experience.

[0372] (3) Adjustment of the order of application windows in the application window group

[0373] In this embodiment, the user can also adjust the order of application windows in the application window group.

[0374] Taking an application window group containing two application windows as an example, Figures 10c and 10d illustrate a schematic diagram of adjusting the order of application windows within an application window group.

[0375] Referring to Figure 10c (1), the application window group 1002 includes two application windows, namely application F window 10021 and application D window 10022. Application F window 10021 and application D window 10022 are displayed around the user, with application D window 10022 displayed directly in front of the user. Since each application window in the application window group does not display window movement controls, the user can bring up the window movement controls of a specific application window in the application window group when needed. Continuing to refer to Figure 10c (1), the user is looking at the lower area of ​​application F window 10021. In response to this user operation, the user's viewpoint 10023 is displayed in the lower area of ​​application F window 10021. At this time, in response to the user's pinch-and-drag operation with two fingers, application F window 10021 moves, as shown in Figure 10c (2). For example, continuing to refer to Figure 10c (2), when the application F window 10021 is moved, a visual style indication is displayed at the placeable position of the application window group 1002, such as displaying a dashed box at the placeable position for prompting. You can refer to dashed boxes 10024 and 10025 to identify a placeable position of the application F window 10021 respectively.

[0376] In response to the user's pinch-and-drag operation with two fingers to the right a first distance (e.g., 5cm), application window F 10021 moves to the position indicated by dashed frame 10024, while application window D 10022 moves to the position indicated by dashed frame 10026 (i.e., the original position of application window F 10021), as shown in Figure 10c (3). That is, in response to the user's pinch-and-drag operation with two fingers to the right a first distance (e.g., 5cm), the display positions of application window F 10021 and application window D 10022 are swapped. At this time, the positions indicated by dashed frames 10025 and 10026 are the available placement positions for application window F 10021.

[0377] Continuing with reference to Figure 10c (3) and Figure 10d (1), the user pinches and drags to the right with two fingers. When the user pinches and drags to the right a second distance (e.g., 10cm), the application window 10021 moves to the position indicated by the dashed frame 10025, and simultaneously the display position of the application window 10022 is restored, that is, the application window 10022 moves to the position indicated by the dashed frame 10024. At this time, the positions shown by dashed frames 10024 and 10026 are the possible placement positions for the application window 10021.

[0378] If the current display position of each application window in the application window group is what the user needs, the user can release two fingers. In response to the user's two-finger release operation, the user's viewpoint displayed on application window 10021 disappears, dashed boxes 10024 and 10026 disappear, and application window 10021 is displayed at the position indicated by dashed box 10025, as shown in Figure 10d (2). At this point, the sorting of the application window positions in application window group 1002 is completed.

[0379] The example of a user adjusting the order of application windows in application window group 1002 by using the window movement control of application window 10022 is similar and will not be repeated here.

[0380] Taking an application window group containing three application windows as an example, Figures 10e and 10f illustrate a schematic diagram of adjusting the order of application windows within an application window group.

[0381] Referring to Figure 10e (1), the application window group 1003 includes three application windows: application window F 10031, application window D 10032, and application window E 10033. Application window F 10031, application window D 10032, and application window E 10033 are displayed around the user, with application window D 10032 displayed directly in front of the user. Since each application window in the application window group does not display window movement controls, the user can bring up the window movement controls of a specific application window in the application window group when needed. Continuing to refer to Figure 10e (1), the user is looking at the lower area of ​​application window F 10031. In response to this user operation, the user's viewpoint 10034 is displayed in the lower area of ​​application window F 10031. At this time, in response to the user's pinch-and-drag operation with two fingers, application window F 10031 moves, as shown in Figure 10e (2). For example, continuing to refer to Figure 10e (2), when the application F window 10031 is moved, a visual style indication is displayed at the placeable position of the application window group 1003, such as displaying a dashed box at the placeable position for prompting. You can refer to dashed boxes 10035 and 10036 to identify a placeable position of the application F window 10031 respectively.

[0382] In response to the user's pinch-and-drag operation to the right with two fingers, a first distance (e.g., 5cm) is reached, the application window F 10031 moves to the position indicated by the dashed box 10035, while the application window D 10032 moves to the position indicated by the dashed box 10037 (i.e., the original position of application window F 10031), as shown in Figure 10e (3). In other words, in response to the user's pinch-and-drag operation to the right with two fingers, the display positions of application window F 10031 and application window D 10032 are swapped. At this time, the positions indicated by dashed boxes 10036 and 10037 are the available placement positions for application window F 10031.

[0383] Continuing with reference to Figure 10e (3) and Figure 10f (1), the user pinches and drags to the right with two fingers. In response to the user's pinching and dragging to the right a second distance (the second distance is greater than the first distance, for example, 10cm), the application window 10031 moves to the position indicated by the dashed box 10036, while the display position of the application window 10032 is restored (that is, the application window 10022 moves to the position indicated by the dashed box 10035), and the application window 10033 moves to the position indicated by the dashed box 10037. At this time, the positions indicated by the dashed boxes 10035 and 10037 are the placement positions of the application window 10031.

[0384] If the current display position of each application window in the application window group is what the user needs, the user can release two fingers. In response to the user's two-finger release operation, the user's viewpoint displayed on application window F 10031 disappears, dashed boxes 10035 and 10037 disappear, and application window F 10031 is displayed at the position indicated by dashed box 10036, as shown in (2) of Figure 10f. At this point, the sorting of the application windows in application window group 1003 is completed, and the application windows in application window group 1003 are sorted left and right as follows: application window E 10033, application window D 10032, and application window F 10031.

[0385] As an alternative implementation, in the case shown in Figure 10e (3), the user pinches with two fingers and continues to drag to the right. In response to the user's pinching and dragging to the right a second distance (the second distance is greater than the first distance, for example, 10cm), application window F 10031 moves to the position indicated by dashed box 10036, while the display position of application window D 10032 remains unchanged, and application window E 10033 moves to the position indicated by dashed box 10035. That is, in this implementation, after the user completes the adjustment, the application windows in application window group 1003 are arranged left and right as follows: application window D 10032, application window E 10033, and application window F 10031.

[0386] The examples of users adjusting the order of application windows in application window group 1003 using the window movement controls of application window D 10032 or application window E 10033 are similar and will not be repeated here.

[0387] In this way, users can flexibly adjust the order of application windows in the application window group according to actual needs, improving the convenience of using multiple application windows at the same time, and also satisfying users' personal habits of using multiple application windows at the same time.

[0388] (4) Saving and releasing application window groups

[0389] The display of application window groups provides users with the convenience of using multiple application windows simultaneously. Users can not only save their created application window groups for later use, but also unlink the application windows when they are no longer needed, allowing each application window to display independently.

[0390] Figure 11a illustrates an exemplary scenario of a user ungrouping an application window. Referring to (1) of Figure 11a, the application window group 1101 includes two application windows, namely application window F 11012 and application window A 11013. In response to the user's operation of the window group movement control 11011 of the application window group 1101, the user's viewpoint is displayed on the window group movement control 11011. In response to the user's operation of pinching with two fingers for a preset duration (e.g., 2 seconds), a prompt window 1102 pops up in virtual space, as shown in (2) of Figure 11a. The prompt window 1102 may include a "Save Group" option 11021 and a "Ungroup" option 11022.

[0391] In response to the user's gaze on the "Ungroup" option 11022, the user's viewpoint 1103 is displayed on the "Ungroup" option 11022. At this time, in response to the user's pinch-to-zoom operation, the application window group 1101 becomes application window F 1104 and application window A 1105, as shown in Figure 11a (3). That is, the application windows in the application window group 1101 are unlinked and restored to individual application windows. Referring to Figure 11a (3), as independently displayed application windows, application window F 1104 displays window movement control 11041, and application window A 1105 displays window movement control 11051.

[0392] Understandably, when an application window group contains N (integer) application windows, in response to the user's action of ungrouping the application window group, each of the N application windows is displayed individually. At this time, the user can perform individual operations on each application window, including but not limited to moving the application window, closing the application window, minimizing the application window, and isolating the application window.

[0393] In this way, when users no longer need to use the application window group, they can disassemble it into multiple independent application windows, which can meet the user's personalized needs.

[0394] Figure 11b illustrates an exemplary scenario of a user saving an application window group. As shown in Figure 11b (1), in response to the user's gaze on the "Save Group" option 11021, the user's viewpoint is displayed on the "Save Group" option 11021. At this time, in response to the user's pinch-to-zoom operation, a pop-up notification window 1106 appears in the virtual space, as shown in Figure 11b (2). The content displayed in the notification window 1106 indicates that the current application window group has been saved to the desktop. For example, when the display duration of the notification window 1106 reaches a preset threshold (e.g., 1 second), the notification window 1106 is canceled.

[0395] After an application window group is saved to the desktop, when the user views the desktop window again, the desktop window displays the icon corresponding to that application window group. For example, as shown in Figure 11c (1), the desktop window 1107 displays the icon 11071 of application window group 1, the icon 11072 of application window group 2, and the icon 11073 of application window group 3. Referring to icon 11071, it can be seen that icon 11071 displays the icons of application A and application F at the same time, and application window group 1 is formed by the interlocking of application A window and application F window; referring to icon 11072, it can be seen that icon 11072 displays the icons of application D, application F, and application E at the same time, and application window group 2 is formed by the interlocking of application D window, application F window, and application E window; referring to icon 11073, it can be seen that icon 11073 displays the icons of application A, application B, application C, and application D at the same time, and application window group 3 is formed by the interlocking of application A window, application B window, application C window, and application D window.

[0396] It should be noted that the icons of the application window group are only illustrative examples, and the icons of the application window group can also be displayed in other forms, which are not limited in this embodiment.

[0397] Understandably, users can open the corresponding application window group through the icons displayed on the desktop window. For example, in response to the user's pinch-to-open operation on icon 11071, application window group 1 is displayed in virtual space, as shown in application window group 1101 in Figure 11a (1). Similarly, in response to the user's pinch-to-open operation on icon 11072, application window group 2 is displayed in virtual space, as shown in application window group 1103 in Figure 10e (1). Similarly, in response to the user's pinch-to-open operation on icon 11073, application window group 3 is displayed in virtual space, as shown in application window group 1108 in Figure 11c (2). It should be noted that when the user opens the application window group through the application window group icon displayed on the desktop window, the order of the application windows in the application window group can be the same as the order of the application windows in the application window group when the user saves the application window group.

[0398] In this way, after a user saves an application window group, they can still quickly open the application window group using its icon later, making the operation simple and convenient.

[0399] It's important to note that ungrouping an application window group does not affect the application window group icon displayed on the desktop. For example, in response to a user saving application window group 1 to the desktop, the icon for application window group 1 is displayed on the desktop. In response to a user pinching the icon with two fingers, application window group 1 is displayed in virtual space. At this point, in response to ungrouping application window group 1, each application window within application window group 1 is displayed individually. When the user reopens the desktop window, the icon for application window group 1 will still be displayed on the desktop.

[0400] If a user needs to delete a saved application window group, they can delete the corresponding application window group icon in the desktop window. This embodiment does not limit the specific implementation of the deletion operation.

[0401] (5) Regarding minimizing and restoring application windows in the application window group

[0402] When a user is using an application window group, there may be times when they no longer need to use one of the included application windows. In this case, the user can minimize the included application window to avoid the problem of too many application windows displayed in the virtual space affecting the user's visual experience.

[0403] Figure 12a illustrates an exemplary application scenario of minimizing an application window within an application window group. As shown in Figure 12a (1), the application window group 1201 includes three application windows: application window F 12012, application window D 12013, and application window E 12014. The application bar 12011 of the application window group 1201 displays a window group minimize control, a window group isolation control, and a window group close control. Each application window within the application window group 1201 includes a window minimize control in its application bar. For example, application window F 12012 displays a window minimize control 120121 in its application bar.

[0404] Suppose that in the current scenario, the user does not need to use application F temporarily, then the user can minimize application F window 12012. Application F window 12012 is not an application window displayed directly in front of the user in the application window group. Continuing to refer to Figure 12a(1), the user looks at the window minimize control 120121 of application F window 12012, and the user's viewpoint is displayed on the window minimize control 120121. In response to the user's pinch-to-zoom operation, application F window 12012 is minimized. Continuing to refer to Figure 12a(2), of the three application windows included in the window group application bar 12011, only application D window 12013 and application E window 12014 are displayed, while application F window 12012 is minimized, and the icon 12015 of application F is displayed in the window group application bar 12011. That is, in addition to displaying the window group minimize control, window group isolate control and window group close control, the window group application bar 12011 also displays the icon 12015 of application F, thereby indicating that the application F window included in the application window group 1201 has been minimized and included in the window group application bar.

[0405] Figure 12b illustrates an exemplary application scenario of minimizing an application window within an application window group. As shown in Figure 12b (1), the application window group 1202 includes three application windows: application window F 12022, application window D 12023, and application window E 12024. The application bar 12021 of the application window group 1202 displays a window group minimize control, a window group isolation control, and a window group close control. Each application window within the application window group 1202 includes a window minimize control in its application bar. For example, application window D 12022 displays a window minimize control 120231 in its application bar.

[0406] Assuming that the user does not need to use application D in the current scenario, the user can minimize application D window 12023. Application D window 12023 is an application window displayed directly in front of the user in the application window group. Referring to Figure 12b(1), the user looks at the window minimize control 120231 of application D window 12023, and the user's viewpoint is displayed on the window minimize control 120231. In response to the user's pinch-to-zoom operation, application D window 12023 is minimized. Referring to Figure 12b(2), of the three application windows included in application window group 1202, only application F window 12022 and application E window 12024 are displayed, while application D window 12023 is minimized, and the icon 12025 of application D is displayed in the application bar 12021 of the window group. That is, in addition to displaying the window group minimize control, window group isolation control, and window group close control, the window group application bar 12021 also displays the icon 12025 of application D, indicating that the application D window included in application window group 1202 has been minimized and added to the window group application bar. Continuing to compare with (1) and (2) in Figure 12b, since the application D window located directly in front of the user is minimized, the position of application F window 12022 located to its left moves to the right and is displayed in the original position of application D window, that is, in front of the user. As another optional implementation, when application D window 12023 is minimized, the display position of application E window 12024 located to its right can also be moved to the left and displayed in the original position of application D window.

[0407] When the application window in the application window group that is directly in front of the user is minimized, any other application window in the group can automatically move to be displayed directly in front of the user; this embodiment does not limit this. In this way, when the user is using the application window group, there is always one application window displayed in the user's comfortable viewing area, improving the user experience.

[0408] It's important to note that minimizing an application window within an application window group is different from minimizing a standalone application window. When the user performs the latter operation, the corresponding application icon is added to the minimized application area of ​​the temporary frequently used applications bar, while when the user performs the former operation, the corresponding application icon is added to the application bar of the application window group.

[0409] After a user minimizes at least one application window in an application window group, the user can also bring those application windows back up and display them. This allows the user to flexibly adjust the displayed application windows within the application window group according to their actual needs.

[0410] Figure 12c exemplarily illustrates an application scenario where application windows are restored within an application window group. As shown in (1) of Figure 12c, the application window group 1203 includes three application windows, but only application window D 12032 (located in the center of the application window group, i.e., directly in front of the user) is displayed. The icons 12033 of application F and 12034 of application E are displayed in the application bar 12031 of the window group, indicating that application windows F and E are minimized and stored in the application bar 12031 of the window group. Suppose that the user needs to reuse application E, the application window E can be restored.

[0411] Referring again to Figure 12c (1), in response to the user's gaze on the icon 12034 of application E, the user's viewpoint is displayed on the icon 12034 of application E. At this time, in response to the user's pinch-to-zoom action, the icon 12034 of application E displayed in the application bar 12031 of the window group disappears, and the application window 12035 of application E is re-displayed in the application window group 1203. Optionally, the restored application window 12035 of application E is displayed in the center of the application window group 1203 (that is, directly in front of the user), and the display position of the application window 12032 of application D is shifted to the right (or left), as shown in Figure 12c (2).

[0412] Suppose that the user needs to reuse application F, the application F window can be restored. Referring to Figure 12c (2), in response to the user's gaze on the icon 12033 of application F, the user's viewpoint is displayed on the icon 12033 of application F. At this time, in response to the user's pinch operation, the icon 12033 of application F displayed in the application bar 12031 of the window group disappears, and the application F window 12036 is re-displayed in the application window group 1203. Optionally, the restored application F window 12036 is displayed in the center of the application window group 1203 (that is, directly in front of the user), and the display position of application E window 12035 is shifted to the left (or right), as shown in Figure 12c (3).

[0413] Considering that the application window re-activated by the user in the application bar of the window group is the one the user needs at the moment, the restored application window is displayed in the center of the application window bar, which is convenient for the user. Since the restored application window takes up display space from other application windows, the display positions of the other application windows can be adjusted accordingly, for example, moved to any available placement position. Of course, the application window re-activated by the user in the application bar of the window group can also be directly displayed in any available placement position in the application window group; this embodiment does not limit this.

[0414] In this way, users can minimize any application window in the application window group according to their actual needs, thus avoiding the problem of the application window group occupying too much display space. Moreover, when the user needs to use the application, the minimized application window in the application window group can be restored at any time, which is convenient and can meet various user needs.

[0415] (V) Regarding the multi-task background management function in XR space

[0416] Figure 13 illustrates an exemplary application scenario. As shown in Figure 13(1), at time T1, the user opens application B at location 1 in the real space. In response to this user action, the window of application B is displayed at a fixed distance 2 (e.g., 1.5m) from the user in the virtual space. In this scenario, the window of application B is not closed when the user leaves location 1 in the real space.

[0417] As shown in Figure 13(2), at time T2 (which is later than time T1), the user opens application A at location 2 in the real space. In response to this user operation, the window of application A is displayed at a fixed distance of 2 (e.g., 1.5m) from the user in the virtual space. In this scenario, the window of application A is not closed when the user leaves location 2 in the real space.

[0418] Referring again to Figure 13(3), at time T3 (which is later than time T1 and later than time T2), the user is located at position 3 in the real space. If the user needs to use application B again, they must return to position 1 in the real space or find the icon for application B on the desktop to reopen it. Similarly, at time T3, the user is located at position 3 in the real space. If the user needs to use application A again, they must return to position 2 in the real space or find the icon for application A on the desktop to reopen it. This inconvenience, requiring the user to either return to the location where the application window was opened or search for the corresponding application icon on the desktop, negatively impacts the user experience.

[0419] (1) Opening and closing the multi-tasking background management window

[0420] To address this issue, this embodiment provides a multi-task background management function, enabling users to conveniently manage background tasks. Users can quickly reopen application windows previously opened in any other location, displaying them directly in front of them, thus improving user convenience.

[0421] Figure 14a illustrates an exemplary scenario of a user opening a multitasking background management window. As shown in Figure 14a (1), a Dock bar 1301 and a desktop window 1302 are displayed in the virtual space. The Dock bar 1301 includes a multitasking background icon 13011. In response to the user's gaze at the multitasking background icon 13011, the user's viewpoint is displayed on the multitasking background icon 13011. At this time, in response to the user pinching two fingers together, a multitasking background management window 1303 is displayed in the virtual space, as shown in Figure 14a (2). The multitasking background management window 1303 may include one or more application window thumbnails, such as application window thumbnail 13031. Each application window thumbnail represents an application currently running in the system background.

[0422] Referring again to Figure 14a (3), the multitasking background management window 1303 is displayed in a ring around the user. For example, the spatial distance between each application window thumbnail displayed in the multitasking background management window 1303 and the user can be a fixed value of 2 (e.g., 1.5m). For example, the desktop window 1302 is displayed at a distance of 2 from the user. In response to the user pinching two fingers together, the desktop window 1302 is de-displayed, and the multitasking background management window 1303 is created and displayed in the original position of the desktop window 1302.

[0423] It should be noted that, similar to other application windows, if there is another application window within a fixed distance of 2 (e.g., 1.5m) before the user invokes the multitasking background management window, then after the user invokes the multitasking background management window, that application window moves back, making the spatial distance between it and the user a fixed value of 6 (the fixed value of 6 is the sum of fixed values ​​2 and 5 (e.g., 0.2m)). The multitasking background management window is then created and displayed at this fixed distance of 2. Conversely, if there is another application window within a greater distance of 2 (e.g., 1.5m) before the user invokes the multitasking background management window, then after the user invokes the multitasking background management window, the position of that application window remains unchanged, and the multitasking background management window is created and displayed at this fixed distance of 2. This avoids the problem of existing application windows in the virtual space obscuring the multitasking background management window, ensuring a better visual experience for the user.

[0424] The multitasking background management window can display m rows and n columns of application window thumbnails, for example, m=2 and n=4, as shown in Figure 14a (2). If the number of applications running in the background exceeds m*n, the application window thumbnails can be displayed in a split-screen manner in the multitasking background management window. In this case, the user can switch the application window thumbnails displayed in the multitasking background management window by pinching and sliding their fingers.

[0425] It should be noted that when a user invokes the multitasking background management window, if no applications are currently running in the background, a prompt message will pop up in the virtual space in response to the user's pinch-to-pinch action on the multitasking background icon. This prompt message informs the user that no applications are currently running in the background. After a certain period of time, the prompt message disappears, and the multitasking background icon is displayed as unselected.

[0426] When the multitasking background management window is displayed in the virtual space, the user can also close the multitasking background management window.

[0427] For example, when the multitasking background management window and the Dock are displayed in the virtual space, the user looks at the multitasking background icon in the Dock. In response to this user action, the user's viewpoint is displayed on the multitasking background icon. At this moment, the user pinches two fingers together. In response to this user action, the multitasking background management window is de-displayed in the virtual space.

[0428] For another example, when the multitasking background management window is displayed in virtual space, the user looks at any blank area outside the multitasking background management window (i.e., the area does not include any XR objects). At this time, the user pinches two fingers together. In response to this user action, the multitasking background management window is de-displayed in virtual space.

[0429] (2) Regarding closing application windows in the multitasking background management window

[0430] Figure 14b illustrates an exemplary scenario of a user managing background running tasks. As shown in Figure 14b (1), a multi-task background management window 1304 is displayed in a virtual space. The multi-task background management window 1304 displays thumbnails of application A (13041), application B (13042), application C (13043), application D (13044), application E (13045), application F (13046), application G (13047), and application H (13048). If the user wants to close application C, they can look at application C's thumbnail 13043. In response to the user looking at application C's thumbnail 13043, the user's viewpoint 1305 is displayed on application C's thumbnail 13043. At this time, in response to the user's pinch-and-swipe operation, the application C window thumbnail 13043 moves upward out of the multitasking background management window 1304, as shown in Figure 14b (2). After the application C window thumbnail 13043 moves upward out of the multitasking background management window 1304, the display positions of other application window thumbnails in the multitasking background management window 1304 are adaptively adjusted so that the application window thumbnails are arranged in rows and columns in the multitasking background management window 1304, as shown in Figure 14b (3). The diagram of the adaptive adjustment of the display positions of other application window thumbnails in the multitasking background management window 1304 can be seen with reference to the dashed arrow shown in Figure 14b (2). It can be understood that after the application C window thumbnail 13043 moves upward out of the multitasking background management window 1304, it indicates that the application C window running in the background has been closed.

[0431] In this way, users can close some application windows running in the background through the multitasking background management window.

[0432] (3) Regarding dragging and dropping the application window to the current position via the multitasking background management window

[0433] In this embodiment, for application windows that the user previously opened in any other location, the user can also open them in the multitasking background management window and display them in front of the user, improving the convenience of user operation.

[0434] Figures 14c-14e illustrate an exemplary application scenario. As shown in Figure 14c (1), at time T4, the user opens application M at location 4 in the real space. In response to this user action, application M window 1306 is displayed at a fixed distance of 2 (e.g., 1.5m) from the user in the virtual space. In this scenario, the user does not close application M window when leaving location 4 in the real space. Continuing to refer to Figure 14c (2), at time T5 (time T5 is later than time T4), the user opens multitasking background management window 1307 at location 5 in the real space. That is, when the user is at location 5 in the real space, task background management window 1307 is displayed at a fixed distance of 2 (e.g., 1.5m) from the user in the virtual space. Moreover, at this time (i.e., time T5), application M window 1306 is still displayed in its original position in the virtual space. Since application M is still running in the background of the system, the thumbnail of application M window 13701 is displayed in the multitasking background management window 1307, as shown in Figure 14c (2) and Figure 14d (1).

[0435] In this embodiment, if the user needs to use application M again, there is no need to return to the real space location 4. Referring to Figure 14d (1), the user is looking at the thumbnail of application M window 13701 at the real space location 2. In response to the user's operation, the user's viewpoint 1308 is displayed on the thumbnail of application M window 13701. At this time, in response to the user's pinch operation, the multitasking background management window 1307 is canceled in the virtual space, and application M window 1306 is displayed at the original display position of the multitasking background management window 1307. As shown in Figure 14d (2) and Figure 14e, when the user is at the real space location 5 at time T6 (time T6 is later than time T5), application M window 1306 is displayed at a fixed distance of 2 (e.g., 1.5m) from the user in the virtual space, and at the same time, application M window 1306 disappears at its original display position.

[0436] Thus, for the application M window that the user previously opened at location 4 in the real space, the user can drag and drop it from location 5 in the real space to the current location via the multitasking background management window. At this time, the application M window disappears at the virtual space location corresponding to location 4 in the real space and reappears at the virtual space location corresponding to location 5 in the real space (see Figure 14e), so that the user can use application M again.

[0437] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the XR space display method in the above embodiment.

[0438] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the XR space display method in the above embodiment.

[0439] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the XR space display method in the above method embodiments.

[0440] In this embodiment, the XR device (such as XR glasses), electronic device (such as a mobile phone), computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0441] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0442] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0443] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

A method for displaying an interface in an XR space, characterized in that, The method, applied in extended reality (XR) devices, includes: displaying a first application window and a second application window in XR space; wherein the first application window displays the program interface of a first application, and the second application window displays the program interface of a second application; displaying a first control in response to an operation of dragging the first application window toward the second application window; and displaying a first application window group in XR space in response to a first operation on the first control; wherein the first application window group includes a first window and a second window, the first window being displayed at a first position in the first application window group for displaying the program interface of the first application; and the second window being displayed at a second position in the first application window group for displaying the program interface of the second application. The method according to claim 1, characterized in that, The first column of the first application window includes: a first control and a second control; the first control is used to minimize the first application window, and the second control is used to close the first application window; the second column of the first application window group includes: a third control and a fourth control; the third control is used to minimize the first application window group, and the fourth control is used to close the first application window group; the third column of the first window includes a fifth control; the fifth control is used to minimize the first window within the first application window group. The method according to claim 1 or 2, characterized in that, In XR space, the windows in the first application window group are displayed around the user, and each window is equidistant from the user; in the first application window group, the first window faces directly in front of the user. The method according to any one of claims 1-3, characterized in that, Also includes: In response to a second operation on the second window, the second window is de-displayed in the first application window group, and the icon of the second application is displayed in the second column of the first application window group; In response to a third operation on the icon of the second application displayed in the second column, the second window is redisplayed in the first application window group, and the icon of the second application is dedisplayed in the second column. The method according to claim 4, characterized in that, When redisplaying the second window in the first application window group, the method further includes: displaying the second window at the first position in the first application window group, and moving the first window to the second position for display. The method according to claim 3, characterized in that, Also includes: In response to the fourth operation on the first window, the first window is de-displayed in the first application window group, the icon of the first application is displayed in the second column of the first application window group, and the second window is moved to the first position for display. The method according to claim 1, characterized in that, Also includes: In response to the fifth operation, the first application window group rotates by a preset angle centered on the user, so that the window facing directly in front of the user changes from the first window to the second window. The method according to claim 1, characterized in that, Also includes: In response to a sixth operation on the second window, the second window is moved to the first position and displayed in the first application window group, and the first window is moved to the second position and displayed; or, in response to a seventh operation on the second window, the second window is moved to the third position and displayed in the first application window group. The method according to claim 1 or 2, characterized in that, Also includes: In response to the eighth operation on the first application window group, the first application window group is de-displayed in the XR space, and a third application window and a fourth application window are displayed in the XR space respectively; wherein, the third application window is used to display the program interface of the first application, and the fourth application window is used to display the program interface of the second application. The method according to claim 9, characterized in that, The fourth column of the third application window includes a sixth control and a seventh control; the sixth control is used to minimize the third application window, and the seventh control is used to close the third application window. The method according to claim 1, characterized in that, Also includes: A fifth application window and the first application window group are displayed in XR space; wherein the fifth application window is used to display the program interface of the third application; in response to an operation of dragging the fifth application window toward the second application window group, a second control is displayed; in response to a ninth operation on the second control, a second application window group is displayed in XR space; wherein the second application window group includes a third window, a fourth window, and a fifth window, the third window is used to display the program interface of the third application, the fourth window is used to display the program interface of the first application, and the fifth window is used to display the program interface of the second application. The method according to claim 1, characterized in that, Also includes: In response to a tenth operation on the first application window group, a first prompt message is displayed in the XR space, the first prompt message indicating that the current application group has been saved to the desktop; in response to an eleventh operation, a desktop window is displayed in the XR space, the desktop window including the icon of the first application window group; in response to a twelfth operation on the icon of the first application window group, the first application window group is displayed in the XR space. The method according to claim 1, characterized in that, Also includes: At the first moment, in response to the thirteenth operation performed by the user at the fourth position, the sixth application window is displayed at the fifth position in XR space; The sixth application window is used to display the program interface of the fourth application; at the second moment, in response to the fourteenth operation performed by the user at the sixth position, the seventh application window is displayed at the seventh position in the XR space; the seventh application window is used to display the program interface of the fifth application; at the third moment, in response to the fifteenth operation performed by the user at the eighth position, the sixth window is displayed at the ninth position in the XR space, and the first XR interface is displayed in the sixth window, the first XR interface including a thumbnail of the sixth application window and a thumbnail of the seventh application window; wherein, the third moment is later than the first moment and the second moment. The method according to claim 13, characterized in that, When the first XR interface is displayed in the sixth window, the method further includes: in response to a sixteenth operation on the thumbnail of the sixth application window, displaying the sixth application window at the ninth position in the XR space, and de-displaying the sixth application window at the fifth position in the XR space. The method according to claim 13, characterized in that, When the first XR interface is displayed in the sixth window, the method further includes: in response to a seventeenth operation on the thumbnail of the sixth application window, closing the sixth application window displayed at the fifth position in the XR space, and displaying a second XR interface in the sixth window, the second XR interface including the thumbnail of the seventh application window. The method according to claim 13, characterized in that, The sixth window is displayed around the user, and the thumbnail of each application window displayed in the sixth window is equidistant from the user. The method according to claim 1, characterized in that, Also includes: Display a first Dock bar in XR space; in response to an eighteenth operation on the first Dock bar, display a second Dock bar in XR space; wherein the second Dock bar includes a first area and / or a second area, the first area including a first application icon, or the second area including the first application icon; in response to a nineteenth operation on the first application icon, display an eighth application window in XR space. The method according to claim 17, characterized in that, A ninth application window is displayed at the tenth position in XR space; wherein the distance between the tenth position and the user is less than or equal to a first length; displaying an eighth application window in XR space includes: displaying the eighth application window at the eleventh position in XR space, and moving the ninth application window from the tenth position to the twelfth position in XR space; wherein the distance between the eleventh position and the user is equal to the first length, and the distance between the twelfth position and the user is equal to a second length, wherein the second length is greater than the first length. The method according to claim 1, characterized in that, When displaying the first application window and the second application window in XR space, the method further includes: in response to a twentieth operation on the first application window, canceling the display of the second application window in XR space; and in response to a twentieth-first operation on the first application window, redisplaying the second application window in XR space. The method according to claim 17, characterized in that, Also includes: Display the tenth application window in XR space; The tenth application window is used to display the program interface of the sixth application; In response to the twenty-second operation on the tenth application window, the tenth application window is minimized and the second Dock is displayed, and the icon of the sixth application is added to the first section of the second Dock. An XR device, characterized in that, include: One or more processors; Display screen; memory; And one or more computer programs; wherein the one or more computer programs are stored on the memory; when the computer programs are executed by the one or more processors, the XR device performs the interface display method of the XR space as described in any one of claims 1-20. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the interface display method of the XR space as described in any one of claims 1-20.

Images