A window control method, electronic device and computer readable storage medium

Abstract

The present application relates to the field of computer technology, and discloses a window control method, an electronic device, and a computer-readable storage medium. A user can use the movement of a pointing component to select some windows from multiple windows as target windows to be controlled. The selected target windows can be multiple windows at the same level or at different levels. After selecting the target window, the electronic device performs the operation indicated by the pointing component on each target window. The operation indicated by the pointing component includes but is not limited to retracting and releasing operations to reduce or enlarge the target window, replacing operations to replace the selected target windows with each other, deleting operations to delete the selected target window, etc. In this way, the user uses the pointing component to select multiple target windows, thereby facilitating the electronic device to perform batch operations on each target window, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a window control method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Augmented Reality (AR) is a new human-computer interaction technology. Through AR technology, participants can interact with virtual objects in real time, thus gaining a wonderful visual experience, and can break through space, time and other objective limitations, and experience something that cannot be experienced in the real world. Virtual Reality (VR) is a virtual reality technology that uses computer technology to generate a simulated environment and immerse users in the created three-dimensional dynamic real scene. It can be understood as a simulation system of the real world. VR technology was first applied in the military field, and the most common product is the head-mounted display.

[0003] The interaction technology with AR / VR devices mainly uses the motion sensors integrated into the head display. When the user rotates his head, the scene in the field of view can be changed. With the continuous development of AR / VR technology, a series of pointing components have gradually emerged, such as gloves, watches, mobile phones, and handles. The pointing components present both hands in the virtual scene to achieve the purpose of the user moving around. By tracking the rotation and movement of these hands, the motion state of these pointing components is mapped to the movement, selection, rotation, and scaling of virtual objects. At present, the interaction technology using AR / VR devices is still relatively simple, mainly focusing on the interaction between gestures, handles and interfaces. When using gestures and handles to interact with the interface, only a single window on a single plane in the interface can be operated independently. For example, splitting, closing, and moving a single window. In this way, when there are multiple windows in the interface, this method is not applicable, resulting in a poor user experience. Summary of the invention

[0004] The purpose of the present invention is to provide a window control method, an electronic device and a computer-readable storage medium, which are convenient for operating multiple windows at the same time and improve the user experience.

[0005] In a first aspect, an embodiment of the present application discloses a window control method, which is applied to an electronic device based on virtual reality technology or augmented reality, wherein the electronic device is associated with a pointing component, and the window control method is used to control multiple window layers displayed in a visible area of ​​the electronic device, wherein each window layer is arranged in sequence in a direction away from a user side of the electronic device, and each window layer includes multiple windows, wherein the window control method includes:

[0006] Detecting movement of a pointing component;

[0007] Determine a selection area based on the movement trajectory of the pointing component;

[0008] determining at least one target window from a plurality of window layers according to the selected area;

[0009] The operation indicated by the pointing component is simultaneously performed on each of the at least one target windows.

[0010] Through a window control method disclosed in an embodiment of the present application, an electronic device and a pointing component are associated with each other, and the electronic device can be used to control multiple windows selected by the pointing component. When a user is using an electronic device, multiple windows at the same level or at different levels are presented in the visible area of ​​the electronic device. The user can use the pointing component to select some windows from multiple windows as target windows to be controlled. The selected target windows may be multiple windows at the same level or at different levels. After selecting the target window, the device simultaneously performs the operation indicated by the pointing component on each target window. The operation indicated by the pointing device includes but is not limited to a retracting operation to reduce or enlarge the target window, a replacement operation to replace the selected target windows with each other, a deletion operation to delete the selected target window, etc. In this way, the user selects multiple target windows using the pointing component, thereby facilitating the electronic device to perform batch operations on each target window, thereby improving the user experience.

[0011] According to some embodiments provided in the first aspect of the present application, the movement of the pointing component is a click on the window, and the selection area is determined based on the click position of the pointing component.

[0012] According to some embodiments provided by the first aspect of the present application, the pointing component moves along a predetermined trajectory.

[0013] The selection area is determined based on the trajectory formed when the pointing member moves.

[0014] According to some embodiments provided in the first aspect of the present application, the trajectory formed when the pointing component moves forms a closed figure, and the electronic device determines the selection area based on the area covered by the closed figure, and the window within the area of ​​the closed figure serving as the selection area is the selected target window.

[0015] According to some embodiments provided by the first aspect of the present application, the retracting and releasing operations based on the indication of the pointing component include reducing, enlarging, and moving the window position of each target window.

[0016] According to some embodiments provided by the first aspect of the present application, determining at least one target window from multiple window layers according to the selected area includes:

[0017] Based on the selection area, at least one target window in the same window layer is determined.

[0018] According to some embodiments provided by the first aspect of the present application, determining at least one target window from multiple window layers according to the selected area includes:

[0019] Based on the selection area, multiple target windows in multiple window layers are determined, and at least one window in each window layer is selected.

[0020] According to some embodiments provided in the first aspect of the present application, the range of the closed figure determined by the selection area covers multiple windows in multiple window layers, and the multiple windows are determined as target windows.

[0021] According to some embodiments provided by the first aspect of the present application, a window selected by a selection area on one window layer among multiple window layers, and the selection area is mapped to at least one window determined on other window layers as a target window.

[0022] According to some embodiments provided by the first aspect of the present application, in the case where at least one target window includes at least two windows, performing an operation indicated by a pointing component on multiple target windows includes performing an operation indicated by a pointing component on multiple target windows.

[0023] Swap the positions of at least two windows.

[0024] According to some embodiments provided by the first aspect of the present application, at least two windows are divided into two window groups, and an operation indicated by a pointing component is performed on multiple target windows. The operation indicated by the pointing component includes:

[0025] Exchange the positions of two window groups.

[0026] According to some embodiments provided in the first aspect of the present application, the area of ​​the first target window among at least two windows is adjusted to adapt to the display area at the position of the second target window, or the area of ​​the second target window among at least two windows is adjusted to adapt to the display area at the position of the first target window.

[0027] In a second aspect, an embodiment of the present application discloses an electronic device, the electronic device is associated with a pointing component, and the electronic device is an electronic device based on virtual reality technology or augmented reality, and the electronic device includes:

[0028] A memory, the memory is used to store window control instructions;

[0029] Processor, the processor implements the following steps when executing the window control instruction:

[0030] Detecting movement of a pointing component;

[0031] Determine a selection area based on the movement trajectory of the pointing component;

[0032] determining at least one target window from a plurality of window layers according to the selected area;

[0033] The operation indicated by the pointing component is simultaneously performed on each of the at least one target windows.

[0034] An electronic device disclosed in an embodiment of the present application and a window control method disclosed in an embodiment of the present application are associated with each other, and the electronic device can be used to control multiple windows selected by the pointing component. When a user is using an electronic device, multiple windows at the same level or at different levels are presented in the visible area of ​​the electronic device. The user can use the pointing component to select some windows from multiple windows as target windows to be controlled. The selected target windows may be multiple windows at the same level or at different levels. After selecting the target window, the device simultaneously performs the operation indicated by the pointing component on each target window. The operation indicated by the pointing device includes but is not limited to the retracting operation to reduce or enlarge the target window, the replacing operation to replace the selected target windows with each other, the deleting operation to delete the selected target window, etc. In this way, the user selects multiple target windows using the pointing component, so that the electronic device can perform batch operations on each target window, thereby improving the user experience.

[0035] According to some embodiments provided in the second aspect of the present application, when executing the window control instruction, the processor is further used to determine the selection area based on the click position of the pointing component.

[0036] According to some embodiments provided in the second aspect of the present application, when executing the window control instruction, the processor is further used to determine the selection area based on the trajectory formed when the pointing component moves.

[0037] According to some embodiments provided in the second aspect of the present application, when executing the window control instruction, the processor is also used to perform the retraction and expansion operations indicated by the pointing component, including reducing, enlarging, and moving the window position of each target window.

[0038] According to some embodiments provided in the second aspect of the present application, when the processor executes a window control instruction, it is also used to determine at least one target window in the same window layer based on the selection area, or to determine multiple target windows in multiple window layers based on the selection area, and at least one window in each window layer is selected.

[0039] According to some embodiments provided by the second aspect of the present application, the memory is further used to store the selected area so that the processor calls the selected area.

[0040] According to some embodiments provided in the second aspect of the present application, when executing the window control instruction, the processor is further configured to swap positions of at least two windows or swap positions of two window groups.

[0041] In a third aspect, an embodiment of the present application discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a storage window control instruction, and when the storage window control instruction is executed by a processor, the window control method mentioned in any one of the above is implemented.

[0042] Other features and corresponding beneficial effects of the present invention are described in the latter part of the specification, and it should be understood that at least part of the beneficial effects become obvious from the description in the specification of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1A and Figure 1B A scene graph consisting of an electronic device and a pointing component provided in an embodiment of the present application;

[0044] Figure 1C and Figure 1D A schematic diagram of the structure of a pointing component provided in an embodiment of the present application;

[0045] Figure 2 a to Figure 2 c is a schematic diagram of a target window selection method according to an example of an embodiment of the present application;

[0046] FIG. 3A to FIG. 3C A schematic diagram of a selection result corresponding to a target window selection method exemplified in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of the distribution of windows according to an example of an embodiment of the present application;

[0048] Figure 5A The structure diagram of a VR device disclosed in an embodiment of the present application is exemplarily shown;

[0049] Figure 5B The structure diagram of a memory of a VR device disclosed in an embodiment of the present application is exemplarily shown;

[0050] FIG. 6A to FIG. 6B A schematic diagram of retracting and expanding windows at the same level according to an example of an embodiment of the present application;

[0051] 7A to 7C A schematic diagram of retracting and extending windows at different levels according to an example of an embodiment of the present application;

[0052] FIG. 8A to FIG. 8C A schematic diagram of retracting and extending windows at different levels but in the same vertical space according to an embodiment of the present application;

[0053] 9A to 9C This is another schematic diagram of retracting and extending windows at different levels according to an example of an embodiment of the present application;

[0054] FIG. 10A to FIG. 10C A schematic diagram of replacing two windows at the same level according to an example of an embodiment of the present application;

[0055] FIG. 11A to FIG. 11B A schematic diagram of replacing multiple windows at different levels according to an example of an embodiment of the present application;

[0056] FIG. 12A to FIG. 12B A schematic diagram of replacing multiple windows at different levels but in the same vertical space according to an embodiment of the present application;

[0057] Fig.13 A flowchart of a window control method provided in an embodiment of the present application for controlling the retraction and extension of windows at different levels or the same level;

[0058] FIG. 14A to FIG. 14B A flowchart of a window control method provided in an embodiment of the present application for controlling the replacement of windows at different levels or the same level;

[0059] Fig.15 A schematic diagram of the structure of a window control device disclosed in an embodiment of the present application;

[0060] Fig.16 A schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application;

[0061] Fig.17 A schematic diagram of the structure of a SOC disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The window control method provided in the embodiment of the present application is applied to an electronic device, which includes but is not limited to an AR device or a VR device. Among them, the AR device adopts augmented reality technology, which is a new technology that "seamlessly" integrates real-world information and virtual-world information. The AR device adopts virtual reality technology, which mainly includes aspects such as simulated environment, perception, natural technology and sensing equipment. The simulated environment is a dynamic three-dimensional realistic image generated by a computer. Perception means that the ideal VR should have all the perceptions that people have. In addition to the visual perception generated by computer graphics technology, there are also perceptions such as hearing, touch, distance and movement. Natural skills refer to the movement of a person's head, eyes, gestures or other human body behaviors. The computer processes the data that is adapted to the participant's actions, responds to the user's input in real time, and feeds back to the user's five senses respectively.

[0063] A window control method provided by an embodiment of the present application is described below in conjunction with the accompanying drawings. First, see Figure 1A and Figure 1B , Figure 1A and Figure 1B A scene graph consisting of an electronic device and a pointing component provided in an embodiment of the present application.

[0064] Figure 1A and Figure 1B The scene diagram shown includes but is not limited to an electronic device 10 using a VR device as an example, a pointing component 11 using a remote control handle as an example, and a visible area 100 (such as Figure 1A The VR device and the remote controller are associated with each other so that the VR device controls the multiple windows selected by the remote controller. After the user wears the VR device, multiple windows 1000 are presented in the visible area 100 of the VR device. The user can use the pointing component 11 to select some windows from the multiple windows as target windows to be controlled. After the target window is selected, the VR device simultaneously performs the operation indicated by the pointing component 11 on each target window.

[0065] It is worth noting that although the example of the pointing component 11 explained above can be a remote control handle, the present application is not limited thereto, and the pointing component 11 can also be Figure 1B Gloves as shown, Figure 1C The watch shown, Figure 1D The terminal such as the wristband shown, or the user's hand, etc. Although the pointing component 11 explained above is used to control the window displayed by the VR device, the present application is not limited thereto, and the pointing component can also be used for other functional operations, such as controlling the movement of objects in the game, performing actions, etc.

[0066] According to the embodiments of the present application, methods for selecting the target window include but are not limited to point selection (select by clicking, the clicked window is selected as the target window), surface selection (select by drawing a closed figure, the window covered by the closed figure is the target window) and line selection (select by drawing a line, the window covered by the line is selected as the target window).

[0067] Figure 2 This is a schematic diagram of a target window selection method for an example embodiment of the present application. Figure 2 The leftmost Figure 2 As shown in the example of a, the user can use the buttons provided on the remote controller, or perform actions similar to tapping or clicking with the remote controller, to click one or more windows presented in the visible area of ​​the VR device. The clicked window can be used as the selected target window. Figure 2 Middle Figure 2As shown in example b, the user can use tapping or clicking actions on one or more windows, or waving the hand to select the windows in the area where the hand is waving, and select them as the selected target windows. The hand waving can move within the visible area of ​​the VR device. When the movement trajectory forms a closed figure, the window covered by the closed figure is the target window. When the hand moves within the visible area of ​​the VR device, the movement trajectory forms a curve or a straight line, and the window covered by the curve or the straight line can be selected as the target window. It is understandable that the user can also use the remote control handle to move within the visible area of ​​the VR device to establish a closed figure to select the target window. The embodiment of the present application is not limited here. Figure 2 The middle Figure 2 As shown in example c, the user can grab part of the window with his hand as the target window. When the user grabs with his hand, the window covered by the hand is the selected target window.

[0068] FIG. 3A to FIG. 3C Schematic diagram of the selection result corresponding to the target window selection method of the embodiment of the present application. Figure 3A In the example shown, when the user clicks on window A, window B, window C, and window D using the remote control handle or a finger, window A, window C, and window D as the selected target windows can display a selection mark 1001 in the upper right corner. The selection mark 1001 can be as follows: Figure 3A The black dots shown in the figure may also be marks of other shapes and types, such as triangles, diamonds, squares, etc. The embodiments of the present application are not limited thereto.

[0069] like Figure 3B In the example shown, the user uses his hand or remote control handle to move in the visible area of ​​the VR device. The closed figure formed by the movement trajectory is as follows Figure 3B The regular rectangle 1002 or triangle 1003 or irregular shape 1004 etc. are shown. Figure 3B The windows A, B and C covered by the regular rectangle 1002, triangle 1003 and irregular shape 1004 shown in the closed figure are selected target windows. Of course, the shape of the closed figure is not limited to the shape of the closed figure in this embodiment. Figure 3B As shown in the example, it can also be other types of shapes, and the embodiments of the present application are not limited here. It can be understood that the closed figure formed by the moving trajectory determines the selected target window can be: the window is completely located inside the closed figure and is selected as the target window, and the window is partially located inside the closed figure and is regarded as the target window (for example, 50% of the area of ​​the window is located in the closed figure and is regarded as the target window).

[0070] like Figure 3CIn the example shown, the user uses his hand or remote control handle to move in the visible area of ​​the VR device. The trajectory formed by the movement trajectory is as follows Figure 3C The straight line 1005 or the curved line 1006 shown, etc. Figure 3C The window A, window B and window C passed by the straight line 1005 or the curve 1006 are selected target windows.

[0071] For the multiple windows 1000 in the visible area 100 of the VR device, they are located in a three-dimensional space. The three-dimensional space is a space composed of three dimensions: length, width, and height, which correspond to the x-axis (horizontal axis), y-axis (vertical axis), and z-axis (vertical axis) of the three-dimensional space. For the multiple windows 1000, they can be located in the same plane area (same level) of the three-dimensional space, and the multiple windows can also be in different plane areas (different levels). For example Figure 4 As shown, window A, window B, window C, and window D are all in the same plane area (as the first plane area), that is, window A, window B, window C, and window D are windows at the same level (this embodiment of the present application records it as Layer-1). Window E and window F are in the second plane area after the first plane area, that is, window E and window F are at the second level (this embodiment of the present application records it as Layer-2), and window A and window B, window C, and window D are at the first level. That is, window E and window F are at different levels from window A and window B and window C and window D. Window G, window H, and window I are in the third plane area after the second plane area, that is, window G, window H, and window I are at the third level (this embodiment of the present application records it as Layer-3). The windows in the three plane areas are at different levels respectively.

[0072] Of course, the multiple windows 1000 within the visible area 100 of the VR device may also have other types of display modes, which are not limited in the embodiments of the present application.

[0073] According to some embodiments of the present application, the VR device and the remote controller are associated with each other, and the VR device can be used to control multiple windows selected by the remote controller. When the user is using the VR device, the following is presented in the visible area 100 of the VR device: Figure 1A and Figure 1BThe multiple windows 1000 shown in the figure are at the same level or at different levels. The user can use the pointing component 11 to select some windows from the multiple windows as the target windows to be controlled. The selected target windows can be multiple windows at the same level or at different levels. After selecting the target window, the VR device simultaneously performs the operation indicated by the pointing component 11 on each target window. The operations indicated by the pointing device 11 include but are not limited to retracting operations to reduce or enlarge the target window, replacing operations to replace the selected target windows with each other, deleting operations to delete the selected target window, etc. In this way, the user selects multiple target windows using the pointing component, which facilitates the VR device to perform batch operations on each target window, thereby improving the user experience.

[0074] The structure of the VR device that implements the window control method of the above embodiment of the present application is described below:

[0075] like Figure 5A As shown, Figure 5A The VR device 50 shown includes but is not limited to a processor 11, which is used to generate corresponding operation control signals, send them to corresponding components in the device, read and process data in the software, especially read and process data and programs in the memory, so that each functional module in the device performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions. For example, it is used for various media processing algorithms, including human-computer interaction, motion tracking / prediction (such as tracking the user's hand movement, the movement and rotation of the remote control handle in the embodiment of the present application), rendering display, audio processing, window reduction or enlargement, window replacement, and window deletion.

[0076] Sensor system 12: used to collect, acquire or send information, including image information and distance information, such as hand information in the embodiment of the present application and information on ray clicks of the remote control handle. The sensor system in the embodiment of the present application may include a 3-axis or 6-axis sensor for acquiring motion information of the VR device, such as angular velocity and linear acceleration; at the same time, it can locate, track and identify hand movements; the sensor system also acquires static and dynamic features of the hand. Static feature information, such as fingertip fixed point, palm center of mass, hand joints, etc. Such features are generally acquired using single-frame data. Dynamic feature information, such as displacement vector, motion speed, etc. Such feature information is usually acquired through multiple frames of data. As a sensor system, some specific program instructions can also be stored therein.

[0077] The memory 13 is used to store programs and various data, mainly storing software units such as operating systems, applications and functional instructions, or their subsets, or their extensions. It can also include non-volatile random access memory to provide the processor 11 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for users to store the selection range of multiple windows, and to store running programs and applications. Figure 5B As shown, at least one storage unit can be set in the memory 13 of the VR device 50, and each storage unit can have its own storage function. For example, the first storage unit is used to store software units such as operating systems, applications, and functional instructions; the second storage unit is used to store applications and running programs, etc.; the third storage unit is used to store the selection range of the user for selecting multiple windows.

[0078] Display element 14: usually includes a display screen and supporting optical devices, used for content display; usually the display interface is presented on the display screen for human-computer interaction and window browsing.

[0079] Acoustic element 15: such as microphone, speaker, earphone, etc., used to output sound.

[0080] Physical hardware 16: physical function keys such as power button, volume button, mechanical control button, etc.

[0081] The device may also include some other components 17 in addition to the above 11-16, so as to make the function and appearance of the device more abundant and beautiful.

[0082] The above hardware 11-16 and part of the hardware 17 can be electrically connected via a bus to achieve coupled communication.

[0083] A window control method provided by an embodiment of the present application is described below in conjunction with the accompanying drawings:

[0084] According to some embodiments of the present application, the multiple windows in the visible area of ​​the VR device are windows at the same level. The window control method of the embodiment of the present application is applied to control the expansion and contraction of multiple windows at the same level.

[0085] like Fig. 6A As shown, windows at the same layer (Layer-1) include but are not limited to window A, window B, window C, and window D. The user uses Figure 2The selection area is established by any of the methods shown in . For example, the user uses the movement of the hand to establish a rectangular movement track, and the rectangle formed by the movement track covers window A, window B, window C and window D. The selected windows A, window B, window C and window D are used as target windows. In addition, the selected windows A, window B, window C and window D present a selection mark 1001 to prompt the user which pages are currently in the selected state and wait for subsequent operations.

[0086] After window A, window B, window C, and window D are selected, the user can simultaneously reduce or enlarge window A, window B, window C, and window D by moving the hand. Figure 6B As shown, when the fingers of the user's hand are closed, window A, window B, window C and window D are reduced together. The reduction ratio of window A, window B, window C and window D can be determined by the degree of closing of the user's fingers relative to the original position of the fingers. For example, the VR device tracks the movement of the user's fingers in real time. When the user's fingers are completely closed, window A, window B, window C and window D are scaled to the minimum. When the user's fingers are closed to half of the original state, window A, window B, window C and window D are scaled to half of the original size. The reduction ratio of window A, window B, window C and window D can also be determined by other methods, which are not limited in the embodiments of the present application.

[0087] In addition, it can be understood that in the scenario of using a remote control handle, the reduction or enlargement of the control window can be achieved by the case of the remote control handle or by a specific way of waving the remote control handle.

[0088] It is worth noting that the reduced windows A, B, C and D can still be displayed in the plane area of ​​the windows A, B, C and D before they were reduced, and the levels of the reduced windows A, B, C and D can be the same as the levels of the windows A, B, C and D before they were reduced. In addition, the reduced windows can be displayed in the form of icons in a non-center position (such as the lower right corner or the lower left corner) in the visible area of ​​the VR device so that the user can enlarge them again.

[0089] In this way, the window control method provided in the embodiment of the present application is applied to control multiple windows at the same level, and multiple windows can be operated in batches, which improves the user experience compared to the way users operate a single window.

[0090] Although the above embodiment explains that all windows A to D in the first level display area are selected and reduced, the present application is not limited thereto, and some of the multiple windows, such as windows A and D, or windows A, B, and C, may also be selected.

[0091] According to some embodiments of the present application, multiple windows in the visible area of ​​the VR device are windows at different levels. The window control method of the embodiment of the present application is applied to control the retraction of multiple windows at different levels. The window control method provided in the embodiment of the present application is applied to control multiple windows at different levels by taking the windows at the first level (recorded as Layer-1 in the embodiment of the present application), the second level (recorded as Layer-2 in the embodiment of the present application) and the third level (recorded as Layer-3 in the embodiment of the present application) as examples to illustrate that the window control method provided in the embodiment of the present application is applied to control multiple windows at different levels. In the embodiment of the present application, multiple windows at different levels refer to windows at the first level, windows at the second level and windows at the third level, which may be at the same position on the z-axis of the three-dimensional space coordinates. It can be understood that the level of the remaining windows in the VR device may also be different from the first to third levels mentioned in the embodiment of the present application, and the level in the visible area of ​​the VR device may also have more levels, which are not limited in the embodiment of the present application.

[0092] In three-dimensional space, Fig. 7A As shown, the windows of the first level, the windows of the second level, and the windows of the third level may be in the same position on the z-axis of the three-dimensional space coordinates. That is, the vertex Layer-10 of the plane area formed by the windows of the first level, the vertex Layer-20 of the plane area formed by the windows of the second level, and the vertex Layer-30 of the plane area formed by the windows of the third level have the same coordinates on the z-axis, that is, the coordinates on the z-axis are all z1. Among them, the windows at the first level include but are not limited to window A, window B, window C, and window D. The windows at the second level include but are not limited to window E and window F. The windows at the third level include but are not limited to window G, window H, and window I.

[0093] like Figure 7B As shown, the user adopts Figure 2The selection area is established by any of the methods shown in . For example, the user uses the movement of the finger to select window A, window B and window D in the first level as the selected target windows in the first level (first selection area) by clicking or establishing a selection area. Window F in the second level is selected as the selected target window (second selection area) by clicking or establishing a selection area, and window I in the third level is selected as the selected target window (third selection area) by clicking or establishing a selection area. In addition, a selection mark 1001 is presented in the selected windows A, B, D, F and I to prompt the user which pages are now in the selected state and wait for subsequent operations. It is worth noting that when the user clicks on windows of different levels with his fingers to select the target window, when the user clicks on windows A, B and D in the first level as the selected target windows in the first level. Through the first level, when the overlapping area of ​​the window of the second level located at the next level of the first level and the selected window A, window B and window D of the first level exceeds a threshold (the overlapping area may be 80%), the window of the overlapping area of ​​the second level can be directly selected as the target window.

[0094] After window A, window B, window D, window F, and window I are selected, the user can simultaneously reduce or enlarge window A, window B, window D, window F, and window I by moving the hand. Figure 7C As shown, when the fingers of the user's hand are closed, window A, window B, window D, window F and window I are reduced together. The reduction ratio of window A, window B, window D, window F and window I can be determined by the degree of closing of the user's hand fingers relative to the original position of the fingers. For example, the VR device tracks the movement of the user's fingers in real time. When the user's fingers are completely closed, window A, window B, window D, window F and window I are scaled to the minimum. When the user's fingers are closed to half of the original state, window A, window B, window D, window F and window I are scaled to half of the original size. The reduction ratio of window A, window B, window D, window F and window I can also be determined by other methods, which are not limited in the embodiments of the present application.

[0095] It is worth noting that the reduced and displayed windows A, B and D can still be displayed in the plane area where the first level of windows A, B and D are located before they are reduced. That is, the levels of windows A, B and D after reduction can be the same as those of windows A, B and D before they are reduced. The reduced and displayed window F can still be displayed in the plane area where the second level of window F is located before it is reduced. The reduced and displayed window I can still be displayed in the plane area where the third level of window I is located before it is reduced. In this way, since the windows of the first level are reduced, it is convenient to display more windows of the next level. In addition, the reduced windows can be displayed in the form of icons in a non-central position in the visible area of ​​the VR device (such as the lower right corner or lower left corner of the visible area) so that the user can enlarge them again.

[0096] In addition, it is understandable that, in the scenario of using a remote control handle, the reduction or enlargement of the control window can be achieved by buttons on the remote control handle or by a specific way of waving the remote control handle.

[0097] In this way, the window control method provided in the embodiment of the present application is applied to control multiple windows at different levels, and multiple windows can be operated in batches, which improves the user experience compared to the way users operate a single window.

[0098] Although the above embodiment explains that windows A, B, D, F, and I in the first to third level display areas are all selected and reduced, the present application is not limited thereto, and some of the multiple windows may also be selected, for example, selecting windows A and B in the first level, selecting windows E and F in the second level, and selecting windows H and I in the third level.

[0099] According to some embodiments of the present application, multiple windows in the visible area of ​​the VR device are windows at different levels but in the same vertical space. The window control method of the embodiment of the present application is applied to control the retraction and expansion of multiple windows at different levels but in the same vertical space. Taking the windows at the first level (recorded as Layer-1 in the embodiment of the present application), the second level (recorded as Layer-2 in the embodiment of the present application) and the third level (recorded as Layer-3 in the embodiment of the present application) as examples, the window control method provided in the embodiment of the present application is applied to control multiple windows at different levels. In the embodiment of the present application, multiple windows at different levels refer to windows at the first level, windows at the second level, and windows at the third level, which can be in the same position on the y-axis of the three-dimensional space coordinates (being in the same position means that the overlapping area of ​​the windows at the three levels reaches the threshold value, and the threshold value can be any value such as 80%, 85%, etc., which is not limited in the embodiment of the present application). It is understandable that the levels of the remaining windows in the VR device may be different from the first to third levels mentioned in the embodiment of the present application, and there may be more levels in the visible area of ​​the VR device, which is not limited in the embodiment of the present application.

[0100] In three-dimensional space, Fig. 8A As shown, the windows of the first level, the windows of the second level, and the windows of the third level may be in the same position on the y-axis of the three-dimensional space coordinates. That is, the vertex Layer-40 of the plane area formed by the windows of the first level, the vertex Layer-50 of the plane area formed by the windows of the second level, and the vertex Layer-60 of the plane area formed by the windows of the third level have the same coordinates on the y-axis, that is, the coordinates on the y-axis are all y1. Among them, the windows at the first level include but are not limited to window A, window B, window C, and window D. The windows at the second level include but are not limited to window E, window F, window G, and window H. The windows at the third level include but are not limited to window I, window J, and window K.

[0101] like Figure 8B As shown, the user adopts Figure 2The selection area is established by any of the methods shown in . For example, the user uses the movement of the finger to select the window A and window B in the first level by clicking or establishing a selection area, selects the window E in the second level by clicking or establishing a selection area, and selects the window I in the third level by clicking or establishing a selection area as the target window of the first selection area (in the embodiment of the present application, group 1 is used to represent it). The user uses the movement of the finger to select the window D in the first level by clicking or establishing a selection area, selects the window M and window G in the second level by clicking or establishing a selection area, and selects the window K in the third level by clicking or establishing a selection area as the target window of the second selection area (in the embodiment of the present application, group 2 is used to represent it). In addition, the selected windows A, B, E, I, D, M, G, and K present a selection mark 1001 to prompt the user which pages are currently in the selected state and wait for subsequent operations.

[0102] After window A, window B, window E, window I, window D, window M, window G, and window K are selected, the user can reduce or enlarge window A, window B, window E, window I, window D, window M, window G, and window K by moving the hand. Figure 8C As shown, when the fingers of the user's hand are closed, windows A, B, E, I, D, M, G, and K are reduced together. Among them, windows A, B, E, and I are reduced as the first group of windows. Window D, M, G, and K are reduced as the second group of windows. The reduction ratio of windows A, B, E, I, D, M, G, and K can be determined by the degree of closing of the user's fingers relative to the original position of the fingers. For example, the VR device tracks the movement of the user's fingers in real time. When the user's fingers are completely closed, windows A, B, E, I, D, M, G, and K are scaled to the minimum. When the user's fingers are closed to half of the original state, windows A, B, E, I, D, M, G, and K are scaled to half of the original size. The reduction ratios of window A, window B, window E, window I, window D, window M, window G, and window K may also be determined by other methods, which are not limited in the embodiments of the present application.

[0103] In addition, it can be understood that in the scenario of using a remote control handle, the reduction or enlargement of the control window can be achieved by the case of the remote control handle or by a specific way of waving the remote control handle.

[0104] It is worth noting that the reduced displayed windows A, B, E, I, D, M, G and K can still be displayed in the plane area of ​​the level where windows A, B and D are located before they are reduced. That is, the level of reduced windows A, B, E, I, D, M, G and K can be the same as the level of windows A, B, E, I, D, M, G and K before they are reduced. In this way, because the windows of the first level are reduced, it is easier to display more windows of the next level. In addition, the reduced windows can be displayed in the form of icons in a non-central position in the visible area of ​​the VR device (such as the lower right corner or lower left corner of the visible area) so that the user can enlarge them again.

[0105] In this way, the window control method provided in the embodiment of the present application is applied to control multiple windows at different levels but in the same vertical space. Multiple windows can be operated in batches, which improves the user experience compared to the way users operate a single window.

[0106] According to some embodiments of the present application, multiple windows in the visible area of ​​the VR device are windows at different levels. The window control method of the embodiment of the present application is applied to control the retraction and expansion of multiple windows at different levels. The window control method provided in the embodiment of the present application is applied to control multiple windows at different levels by taking the windows at the first level (recorded as Layer-1 in the embodiment of the present application), the second level (recorded as Layer-2 in the embodiment of the present application) and the third level (recorded as Layer-3 in the embodiment of the present application) as examples. In the embodiment of the present application, multiple windows of different levels refer to windows of the first level, windows of the second level and windows of the third level, which may be in the same position on the x-axis of the three-dimensional space coordinates. It can be understood that the level of the remaining windows in the VR device may also be different from the first to third levels mentioned in the embodiment of the present application, and the level in the visible area of ​​the VR device may also have more levels, which are not limited in the embodiment of the present application.

[0107] In three-dimensional space, Fig. 9AAs shown, the first-level window, the second-level window, and the third-level window may be covered at the same position on the x-axis of the three-dimensional space coordinates. That is, the vertex Layer-70 of the plane area formed by the first-level window, the vertex Layer-80 of the plane area formed by the second-level window, and the vertex Layer-90 of the plane area formed by the third-level window have the same coordinates on the x-axis or partially overlap. That is, the coordinates on the x-axis are all x1. For example, when the user clicks on window D with a finger at the first-level window D, window D is selected, and the coordinates of window D are the same as the coordinates of window F of the second level and the coordinates of window I of the third level. It can be understood that when the coordinates of window F of the second level and the partial coordinates of window I of the third level overlap with the coordinates of window D, it can also be considered that the first-level window, the second-level window, and the third-level window may be covered at the same position on the x-axis of the three-dimensional space coordinates, and the embodiments of the present application are not limited here. Among them, the windows at the first level include but are not limited to window A, window B, window C, and window D. The windows at the second level include, but are not limited to, window E and window F. The windows at the third level include, but are not limited to, window G, window H, and window I.

[0108] like Fig. 9B As shown, the user adopts Figure 2 . For example, the user uses the movement of the finger to select the window D in the first level as the selected target window in the first level (first selection area) by clicking or establishing a selection area. The window F in the second level is selected as the selected target window (second selection area) by clicking or establishing a selection area, and the window I in the third level is selected as the selected target window (third selection area) by clicking or establishing a selection area. In addition, the selected window D, window F and window I present a selection mark 1001 to prompt the user which pages are currently in the selected state and wait for subsequent operations. It is worth noting that when the user clicks on windows of different levels with a finger to select a target window, when the user clicks on the window D of the first level as the selected target window in the first level. Through the first level, when the overlapping area of ​​the window of the second level located at the next level of the first level and the selected window A, window B and window D of the first level exceeds the threshold value (the overlapping area may be 80%), the window of the overlapping area of ​​the second level (such as window F) can be directly selected as the target window. The embodiment of the present application does not limit the method for selecting the target window.

[0109] After the window D, window F and window I are selected, the user can simultaneously reduce or enlarge the window D, window F and window I by retracting and extending the hand. Fig. 9CAs shown, when the fingers of the user's hand are closed, window D, window F and window I are reduced together. The reduction ratio of window D, window F and window I can be determined by the degree of closing of the user's fingers relative to the original position of the fingers. For example, the VR device tracks the movement of the user's fingers in real time. When the user's fingers are completely closed, window D, window F and window I are scaled to the minimum. When the user's fingers are closed to half of the original state, window D, window F and window I are scaled to half of the original size. The reduction ratio of window D, window F and window I can also be determined by other methods, which are not limited in the embodiments of the present application.

[0110] In addition, it can be understood that in the scenario of using a remote control handle, the reduction or enlargement of the control window can be achieved by the case of the remote control handle or by a specific way of waving the remote control handle.

[0111] It is worth noting that the reduced window D can still be displayed in the plane area where the first level of window D before it was reduced is located. That is, the level of window D after reduction can be the same as the level of window D before it was reduced. Window F after reduction can still be displayed in the plane area where the second level of window F before it was reduced is located. Window I after reduction can still be displayed in the plane area where the third level of window I before it was reduced is located. In addition, each reduced window can be displayed in the form of an icon in a non-central position in the visible area of ​​the VR device (such as the lower right corner or lower left corner of the visible area) so that the user can enlarge it again.

[0112] In addition, the selection area created for the target window can also be stored in the memory, so that it can be easily called up and used directly again, thereby avoiding the need to create the selection area again and improving the efficiency of operating the window.

[0113] In this way, the window control method provided in the embodiment of the present application is applied to control multiple windows at different levels, and multiple windows can be operated in batches, which improves the user experience compared to the way users operate a single window.

[0114] Although the above embodiment explains that windows D, F, and I in the first to third level display areas are all selected and reduced, the present application is not limited thereto, and some of the multiple windows may also be selected, for example, selecting windows A and B in the first level, selecting windows E and F in the second level, and selecting windows H and I in the third level.

[0115] According to some embodiments of the present application, the multiple windows in the visible area of ​​the VR device are windows at different levels or at the same level. The window control method of the embodiment of the present application is applied to control the replacement between two windows at the same level. The window control method provided in the embodiment of the present application is applied to control the replacement between two windows at the same level by taking the window at the first level as an example to illustrate that the window control method provided in the embodiment of the present application is applied to control the replacement between two windows at the same level.

[0116] In the embodiment of the present application, windows of different levels are respectively at the first level (which is recorded as Layer-1 in the embodiment of the present application), the second level (which is recorded as Layer-2 in the embodiment of the present application) and the third level (which is recorded as Layer-3 in the embodiment of the present application) as an example. Multiple windows of different levels refer to windows of the first level, windows of the second level and windows of the third level, which can be covered at the same position on the x-axis of the three-dimensional space coordinates. That is to say, the plane area formed by the windows of the first level, the plane area formed by the windows of the second level and the plane area formed by the windows of the third level have the same coordinates on the x-axis or part of the coordinates overlap. For example, when the user clicks on window D with a finger at the first level window D, window D is selected. When the coordinates of the position clicked by the user on window D on the plane of the first level are mapped to the plane of the second level and the plane of the third level, they overlap with window F of the second level and overlap with window I of the third level respectively. In this way, window F of the second level overlaps, and window I of the third level is also selected. In addition, it can be understood that when the coordinates of the second-level window F and part of the coordinates of the third-level window I overlap with the coordinates covered by window D to a preset degree, it can also be considered that the first-level window, the second-level window and the third-level window can be covered at the same position on the x-axis of the three-dimensional space coordinates. For example, when the user clicks on window D with a finger, window D is selected. When it is determined that the range covered by window D is mapped to the plane of the second level and the plane of the third level, the overlap with the second-level window F and the overlap with the third-level window I reach more than 50% of the area covered by window D, then it is determined that the second-level window F overlaps, and the third-level window I is also selected. The embodiment of the present application is not limited here. It can be understood that the level of the remaining windows in the VR device can also be different from the first to third levels mentioned in the embodiment of the present application, and the level in the visible area of ​​the VR device can also have more levels, which are not limited in the embodiment of the present application.

[0117] In three-dimensional space, the positions of the first-level windows, the second-level windows, and the third-level windows in three-dimensional space can be referred to Fig. 9A The embodiments of the present application will not be described in detail here.

[0118] like Fig. 10AAs shown, the user adopts Figure 2 . For example, the user uses the movement of a finger to select window A in the first level as the first target window by clicking or creating a selection. The user uses the movement of a finger to select window B in the first level as the second target window by clicking or creating a selection. In addition, a selection mark 1001 is displayed in the selected windows A and B to prompt the user which pages are currently in the selected state and wait for subsequent operations.

[0119] After window A and window B are selected, Fig. 10B As shown, the user moves window B with a finger toward the display area where window A is located, so as to move window B to the position of window A and window A to the position of window B.

[0120] like Fig. 10C As shown, after the positions of window A and window B are replaced, window B is located in the display area of ​​window A, and window A is located in the display area of ​​window B. It is worth noting that when the area of ​​the display area of ​​window A is inconsistent with the area of ​​the display area of ​​window B, after window A and window B are swapped, the window area of ​​window B is adjusted to be the same as the area of ​​the display area at the position of window A. The window area of ​​window A is adjusted to be the same as the area of ​​the display area at the position of window B. The area of ​​the first target window is adapted to the display area at the position of the second target window, and the area of ​​the second target window is adapted to the display area at the position of the first target window.

[0121] The above is to trigger the replacement by moving some windows toward other windows and approaching them with fingers. According to the embodiment of the present application, it can also be triggered by a specific gesture after selecting both windows as replacement objects, such as rotating the palm. In addition, it can be understood that in the scenario of using a remote control handle, the replacement of the control window can be achieved by pressing the buttons of the remote control handle or by a specific way of waving the remote control handle.

[0122] In this way, the window control method provided in the embodiment of the present application is applied to control multiple windows at the same level, and multiple windows can be operated in batches, which improves the user experience compared to the way users operate a single window.

[0123] Although the above embodiment explains that both window A and window B are selected in the first-level display area, and the positions of window A and window B are replaced with each other, the present application is not limited thereto, and the remaining windows of the first level or windows of other levels may also be selected, for example, selecting window A and window D of the first level, selecting window E and window F of the second level, or selecting window H and window I of the third level.

[0124] According to some embodiments of the present application, the multiple windows in the visible area of ​​the VR device are windows at different levels or at the same level. The window control method of the embodiment of the present application is applied to control the replacement between multiple windows at different levels. The window control method provided in the embodiment of the present application is applied to control the replacement between multiple windows at different levels by taking the window at the first level and the window at the second level as examples to illustrate that the window control method provided in the embodiment of the present application is applied to control the replacement between multiple windows at different levels.

[0125] In the embodiment of the present application, windows at different levels are taken as examples of windows at the first level (which is recorded as Layer-1 in the embodiment of the present application), the second level (which is recorded as Layer-2 in the embodiment of the present application), and the third level (which is recorded as Layer-3 in the embodiment of the present application).

[0126] Can be combined with the above FIG. 9A to FIG. 9B The method of selecting windows at different levels explained above selects multiple windows at different levels.

[0127] In three-dimensional space, the positions of the first-level windows, the second-level windows, and the third-level windows in three-dimensional space can be referred to Fig. 9A The embodiments of the present application will not be described in detail here.

[0128] like Fig.11A As shown, the user adopts Figure 2 . For example, the user uses the movement of the finger to select the window A, window B and window C in the first level as the first target window by clicking or creating a selection. The user uses the finger to select the window E in the second level as the second target window by clicking or creating a selection. In addition, the selected window A, window B, window C and window E present a selection mark 1001 to prompt the user which pages are currently in the selected state and wait for subsequent operations.

[0129] After window A, window B, window C, and window E are selected, the user uses a finger to move window A, window B, and window C closer to the display area where window E is located. Window A, window B, and window C are moved to the position of window E, and window E is moved to the position of the display area formed by window A, window B, and window C.

[0130] like Fig. 11B As shown, after the positions of window A, window B, window C and window E are replaced, window E is located in the display area where window A, window B and window C are located, and window A, window B and window C are located in the display area where window E is located.

[0131] It is worth noting that when the area of ​​the display area formed by window A, window B, and window C is inconsistent with the area of ​​the display area of ​​window E, after window A, window B, window C and window E are swapped, the window area of ​​window E is adjusted to be the same as the area of ​​the display area at the location of window A, window B, and window C. The window area of ​​the plane area formed by window A, window B, and window C is adjusted to be the same as the area of ​​the display area at the location of window E. The area of ​​the first target window is adapted to the display area at the location of the second target window, and the area of ​​the second target window is adapted to the display area at the location of the first target window.

[0132] The above is to trigger the replacement by moving some windows toward other windows and approaching them with fingers. According to the embodiment of the present application, it can also be triggered by a specific gesture after selecting both windows as replacement objects, such as rotating the palm. In addition, it can be understood that in the scenario of using a remote control handle, the replacement of the control window can be achieved by pressing the buttons of the remote control handle or by a specific way of waving the remote control handle.

[0133] In this way, the window control method provided in the embodiment of the present application is applied to control multiple windows at different levels, and multiple windows can be operated in batches, which improves the user experience compared to the way users operate a single window.

[0134] Although the above embodiment explains that windows A, B, and C on the first-level display area and window E on the second level are selected, and the positions of windows A, B, and C and window E are replaced with each other, the present application is not limited thereto, and the remaining windows on the first level or windows on other levels may also be selected for position replacement, for example, windows A, B, C, and D on the first level are selected and windows E and F on the second level are selected for replacement, or window G on the third level is selected and window E on the second level is replaced.

[0135] According to some embodiments of the present application, multiple windows in the visible area of ​​the VR device are windows at different levels or at the same level. The window control method of the embodiment of the present application is applied to control the replacement between multiple windows at different levels but in the same vertical space. The window control method provided in the embodiment of the present application is applied to control the replacement between multiple windows at different levels by taking the windows at the first level, the second level, and the third level as examples to illustrate that the window control method provided in the embodiment of the present application is applied to control the replacement between multiple windows at different levels.

[0136] In the embodiment of the present application, windows of different levels are taken as examples of windows at the first level (which is recorded as Layer-1 in the embodiment of the present application), the second level (which is recorded as Layer-2 in the embodiment of the present application), and the third level (which is recorded as Layer-3 in the embodiment of the present application). Multiple windows of different levels refer to windows of the first level, windows of the second level, and windows of the third level, which can be covered at the same position on the y-axis of the three-dimensional space coordinates (that is, windows of the first level, windows of the second level, and windows of the third level are in the same vertical space). In other words, the coordinates of the plane area formed by the windows of the first level, the plane area formed by the windows of the second level, and the plane area formed by the windows of the third level on the y-axis are the same or part of the coordinates overlap. For example, when the user clicks on window D of the first level with a finger, window D is selected, and the coordinates of window D are the same as or part of the coordinates of window H of the second level and window K of the third level. It is understandable that when the coordinates of the second-level window H and part of the coordinates of the third-level window K coincide with the coordinates of the window D, it can also be considered that the first-level window, the second-level window and the third-level window can be covered at the same position on the y-axis of the three-dimensional space coordinates, and the embodiments of the present application are not limited here. It is understandable that the levels of the remaining windows in the VR device can also be different from the first to third levels mentioned in the embodiments of the present application, and the levels in the visible area of ​​the VR device can also have more levels, and the embodiments of the present application are not limited here.

[0137] In three-dimensional space, the positions of the first-level windows, the second-level windows, and the third-level windows in three-dimensional space can be referred to Fig. 8A The embodiments of the present application will not be described in detail here.

[0138] like Fig. 12A As shown, the user adopts Figure 2 The selection area is established by any of the methods shown in . For example, the user uses the movement of the finger to select the window B and window C in the first level, the window F in the second level, and the window J in the third level as the first target window (represented by group1 in the figure). The user uses the finger to select the window D in the first level, the window G and window H in the second level, and the window K in the third level as the second target window (represented by group2 in the figure) by clicking or establishing a selection area. In addition, the selected windows B, C, F, J, D, G, H, and K present a selection mark 1001 to prompt the user which pages are currently in the selected state and wait for subsequent operations.

[0139] After window B, window C, window F, window J, and window D, window G, window H, and window K are selected, the user moves window B, window C, window F, and window J with a finger to approach the display area where window D, window G, window H, and window K are located. Window B and window C are moved to the display area where window D is located, window F is moved to the display area where window H and window G are located, and window J is moved to the display area where window K is located. Window D is moved to the display area where window B and window C are located, window H and window G are moved to the display area where window F is located, and window K is moved to the display area where window J is located.

[0140] like Fig. 12B As shown, after the positions of the four windows B, C, F, and J and the windows D, G, H, and K are replaced, the display area where window B and C are located at the position of window D, the display area where window F is located at the position of window H and G, and the display area where window J is located at the position of window K. Window D is located at the display area where window B and window C are located, the display area where window H and window G are located at the position of window F, and the display area where window K is located at the position of window J.

[0141] It is worth noting that after the positions of windows B and C are swapped with window D, when the area of ​​the display area formed by windows B and C is inconsistent with the area of ​​the display area of ​​window D, the window area of ​​the plane area formed by windows B and C is adjusted to be the same as the area of ​​the display area at the position of window D. The window area of ​​window D is adjusted to be the same as the area of ​​the display area of ​​the plane area formed by windows B and C.

[0142] After the positions of window F and window G and window H are swapped, when the window area of ​​window F is inconsistent with the area of ​​the plane area formed by window G and window H, the window area of ​​window F is adjusted to be the same as the display area of ​​the plane area formed by window G and window H. The area of ​​the plane area formed by window G and window H is adjusted to be the same as the area of ​​the display area at the position of window F.

[0143] After the positions of window K and window J are swapped, when the window area of ​​window K is inconsistent with the area of ​​the display area where window J is located, the window area of ​​window K is adjusted to be the same as the area of ​​the display area where window J is located. The window area of ​​window J is adjusted to be the same as the area of ​​the display area where window K is located.

[0144] In addition, it is understandable that, in the scenario of using a remote controller, the replacement of the control window can be achieved by pressing the buttons of the remote controller or by waving the remote controller in a specific manner. Further, after the positions of the four windows B, C, F, J and D, G, H and K are replaced, in order to prompt the user which pages are now in the selected state and replaced, the mark 1001 can continue to be displayed on the selected window.

[0145] In this way, the window control method provided in the embodiment of the present application is applied to control multiple windows at different levels, and multiple windows can be operated in batches, which improves the user experience compared to the way users operate a single window.

[0146] The following describes a process of applying a window control method provided in an embodiment of the present application to control the retraction and expansion of windows at different levels or the same level.

[0147] See also Fig.13 , Fig.13 A window control method provided in an embodiment of the present application is a flow chart showing the application of controlling the retraction and expansion of windows at different levels or the same level.

[0148] The method includes steps S130 to S133.

[0149] Step S130: receiving a batch operation instruction, the batch operation instruction is used to instruct to perform a retracting and expanding operation on multiple windows in the visible area of ​​the VR device, wherein the multiple windows may be multiple windows at the same level, for example Fig. 6A The windows shown are window A, window B, window C, and window D at the same level. The multiple windows may also be multiple windows at different levels, for example Fig. 7A The first to third levels are shown as windows A, B, C and D, E and F, G, H and I. Fig. 8A The first level is shown as window A, window B, window C, and window D. The second level is shown as window E, window F, window G, and window H. The third level is shown as window I, window J, and window K. And Fig. 9A The first level of windows A, B, C and D are shown. The second level of windows E and F. The third level of windows G, H and I. Among them, the batch operation instruction can be issued by the user using gestures or pointing components such as remote control handles. For example, the user triggers the zoom instruction or the shrink instruction (the batch operation instruction includes the zoom instruction or the shrink instruction) by spreading or closing five fingers in the visible area of ​​the VR device. Or the user triggers the zoom instruction or the shrink instruction by operating the button on the remote control handle (the batch operation instruction includes the zoom instruction or the shrink instruction).

[0150] Step S131: Identify batch operation instructions. Among them, batch operation instructions include but are not limited to zoom-in instructions for zooming in on windows in the visible area of ​​the VR device or shrinking instructions for shrinking windows. Among them, the zoom-in ratio and the shrinking ratio of the zoom-in instruction or the shrinking instruction can be determined by the degree of expansion or closure of the user's fingers, or can also be determined by the buttons on the remote control handle. For example, the VR device tracks the movement of the user's fingers in real time, and when the user's fingers are completely closed, the selected target window is scaled to the minimum. When the user's fingers are closed to half of the original state, the selected target window is scaled to half of the original size. The reduction ratio of the selected target window can also be determined by other methods, which is not limited in the embodiments of the present application.

[0151] Step S132: Determine the selection range. The selection range may be the plane area where the selected target window is located. The target window may be selected by the user using a selection area established by the user's hand or a remote controller or by clicking a single window. For example, the user may use a button provided on the remote controller, or perform a tapping or clicking action by holding the remote controller to click one or more windows presented in the visible area of ​​the VR device. The clicked window may be used as the selected target window. For example, Figure 7B As shown, the user uses the movement of the finger to select windows A, B and D in the first level as the selected target windows in the first level (first selection area) by clicking or creating a selection area. The user uses the movement of the finger to select window F in the second level as the selected target window (second selection area) by clicking or creating a selection area, and uses the movement of the finger to select window I in the third level as the selected target window (third selection area).

[0152] Or the user waves his hand to select the windows in the area where the hand is waved, and use them as the selected target windows. The hand wave can move within the visible area of ​​the VR device. When the moving trajectory forms a closed figure, the window covered by the closed figure is the target window. When the hand moves within the visible area of ​​the VR device, the moving trajectory forms a curve or a straight line, and the window covered by the curve or the straight line can be selected as the target window. For example, Fig. 6A As shown, the user creates a rectangular moving track by moving his hand, and the rectangle formed by the moving track covers window A, window B, window C, and window D as the selected target windows.

[0153] It is understandable that the user can also use the remote control handle to move within the visible area of ​​the VR device to create a closed figure to select the target window. The closed figure formed can be Figure 3BThe regular rectangle 1002 or triangle 1003 or irregular shape 1004 shown in the figure can be, for example, a closed figure formed by the moving track to determine the selected target window can be: a window that is completely located inside the closed figure is selected as the target window, and a window that is partially located inside the closed figure is regarded as the target window (for example, a window that is 50% of its area is located inside the closed figure is regarded as the target window).

[0154] Step S133: Execute the operation indicated by the batch operation instruction. After determining the selection range, after the zoom-in operation or zoom-out operation in the batch operation instruction is identified, the target window in the selection range is zoomed in or out. The above zoom-in operation or zoom-out operation is an example of an operation instruction, and the operation instruction may also include other instructions for controlling windows, such as closing a window, etc.

[0155] In this way, the window control method provided in the embodiment of the present application is applied to control multiple windows at different levels or the same level, and multiple windows can be operated in batches, which improves the user experience compared to the way users operate a single window.

[0156] Next, the process of applying a window control method provided in an embodiment of the present application to control the replacement of windows at different levels or the same level is described below.

[0157] See also Fig.14A , Fig.14A A flowchart of a window control method provided in an embodiment of the present application for controlling the replacement of windows at different levels or the same level.

[0158] The method includes steps S140 to S144.

[0159] Step S140: receiving a batch operation instruction, the batch operation instruction is used to instruct to replace multiple windows in the visible area of ​​the VR device, the multiple windows can be multiple windows at the same level, or multiple windows at different levels, such as Fig. 9A The first level of windows A, B, C and D are shown. The second level of windows E and F. The third level of windows G, H and I. Among them, the batch operation instruction can be issued by the user using gestures or pointing components such as remote controllers. For example, the user triggers the replacement instruction (the batch operation instruction includes the replacement instruction) by moving a finger in the visible area of ​​the VR device. Or the user triggers the replacement instruction by operating the button on the remote controller.

[0160] Step S141: Identify batch operation instructions. The batch operation instructions include but are not limited to instructions for replacing at least two windows in the visible area of ​​the VR device. For example, the VR device tracks the movement of the user's fingers in real time. When the user's fingers press and hold, Fig. 10B The window B shown moves toward the window A until it moves to the display area where the window A is located. The batch operation instruction at this time is a replacement instruction between target windows.

[0161] Step S142: Determine the selection range. The selection range may be a plane area where the first target window and the second target window of the same level or different levels are selected. The selection of the target window may be determined by a selection area established by the user using the hand or remote controller or by clicking a single window. For example, the user may use the buttons provided on the remote controller, or hold the remote controller and perform actions similar to tapping or clicking, to click one or more windows presented in the visible area of ​​the VR device. The clicked window may be used as the selected target window. For example, Fig. 10A As shown, the user uses the movement of the finger to select window A in the first level as the first target window by clicking or creating a selection area. The user uses the movement of the finger to select window B in the first level as the second target window by clicking or creating a selection area. Fig.11A As shown, the user uses the finger to move, click or create a selection to select window A, window B and window C in the first level as the first target window. The user uses the finger to click or create a selection to select window E in the second level as the second target window. Fig. 12A As shown, the user uses the movement of the finger to select the window B and window C in the first level, the window F in the second level, and the window J in the third level as the first target window (represented by group1 in the figure). The user uses the movement of the finger to select the window D in the first level, the window G and window H in the second level, and the window K in the third level as the second target window (represented by group2 in the figure).

[0162] Step S143: Identify the coordinates of the first target window. Fig. 14B, window A and window B at the same level are selected as the first target window and the second target window. Corresponding to the three-dimensional space, window A and window B are both in the area formed by the positive direction of the x-axis and the positive direction of the z-axis. Corresponding to the rectangular window A, the four vertices of window A have four corresponding coordinates in the three-dimensional space, namely A1 (x1, 0, z1), A2 (x1, 0, z2), A3 (x2, 0, z1), and A4 (x2, 0, z2). Of course, the first target window can also be located at other positions in the three-dimensional space, and the number of first target windows is not limited to one, and the embodiments of the present application are not limited thereto.

[0163] Step S144: Identify the coordinates of the second target window. Fig. 14B , corresponding to the rectangular window B in the three-dimensional space, the four vertices of window B have four corresponding coordinates in the three-dimensional space, namely B1 (x3, 0, z3), B2 (x3, 0, z4), B3 (x4, 0, z3), and B4 (x4, 0, z4). Of course, the second target window can also be located at other positions in the three-dimensional space, and the number of second target windows is not limited to one. The first target window and the second target window can also be windows of different levels, which is not limited in the embodiments of the present application.

[0164] Step S145: Execute the operation indicated by the batch operation instruction. After determining the coordinates of the first target window and the second target window within the selection range, replace the positions of the first target window and the second target window within the selection range for the replacement operation in the identified batch operation instruction. That is, move the first target window to the coordinates of the second target window, and move the second target window to the coordinates of the first target window. When the coordinates of the first target window do not overlap with the coordinates of the second target window, adjust the area of ​​the first target window so that the coordinates at the vertices of the first target window overlap with the coordinates at the vertices of the second target window. When the coordinates of the second target window do not overlap with the coordinates of the first target window, adjust the area of ​​the second target window so that the coordinates at the vertices of the second target window overlap with the coordinates at the vertices of the first target window.

[0165] In this way, the window control method provided in the embodiment of the present application is applied to control multiple windows at different levels or the same level, and multiple windows can be operated in batches, which improves the user experience compared to the way users operate a single window.

[0166] The following is a description of a window control device provided in an embodiment of the present application. Fig.15 , Fig.15 A schematic diagram of the structure of a window control device disclosed in an embodiment of the present application.

[0167] Fig.15 The window control device 2 shown includes: a receiving module 150 , a collection module 151 , a collision detection module 152 , a region identification module 153 , a grouping module 154 , a display module 155 and an output module 156 .

[0168] The receiving module 150 is used to receive Fig.13 and Fig.14A The batch operation instructions shown are used to receive the position of the selected target window in the visible area 100 detected by the collision detection module 152. The batch operation instructions can be window retraction instructions or window replacement instructions issued by the user using hand movements or remote control handles.

[0169] The acquisition module 151 is used to acquire the gestures, actions, and areas pointed by the pointing component, so as to determine the movement track of the pointing component.

[0170] The collision detection module 152 is used to detect the direction of the remote control handle. Figure 1A or Figure 1B The rays emitted from the visible area 100 shown collide with the partial window 1000 in the visible area 100. Or the user points to the part of the window 1000 in the visible area 100 by means of gloves or gestures. Figure 1A or Figure 1B The window 1000 in the visible area 100 shown in FIG. 1 is the window that the ray emitted by the remote control handle hits or the window in the direction pointed by the user's finger is the target window. Figure 3B The window covered by the closed figure shown can also be identified by the collision detection module 152 and determined as the target window.

[0171] The area recognition module 153 is used to recognize the area of ​​the window selected by the user using the hand or the remote control handle. For example, the user uses the movement of the hand to establish Figure 3B The region identification module 153 is used to identify the shape and size of the selected region formed by the closed figure and the window covered by the closed figure. Alternatively, the user uses the movement of the hand to establish the selected region. Figure 3C The area recognition module 153 is used to recognize the length of the straight line or curve and the window through which the straight line or curve passes.

[0172] The grouping module 154 allows the user to group and manage windows.

[0173] The display module 155 is used to display the selected target window, for example, using Figure 3AThe mark 1001 shown in the figure indicates that the position of the target window after the retraction or replacement is completed is shown in the figure. Figure 6B , Figure 7C , Figure 8C , Fig. 9C The window shown is displayed in the visual area 100 after being reduced. Fig. 10C , Fig. 11B as well as Fig. 12B The display position of the target window or target window group after replacement is completed.

[0174] The output module 156 is used to execute the actions of window batch operation interaction. The actions of window batch operation interaction include but are not limited to moving windows, splicing windows, closing windows, maximizing and minimizing windows, etc.

[0175] In some embodiments of the present application, an electronic device is also provided. Fig.16 The electronic device in the embodiment of the present application is introduced. Fig.16 A schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application.

[0176] For at least one embodiment, the controller hub 804 communicates with the processor 801 via a multi-drop bus such as a front-side bus (FSB), a point-to-point interface such as a QuickPath Interconnect (QPI), or a similar connection. The processor 801 executes instructions that control general types of data processing operations. In one embodiment, the controller hub 804 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown) and an input / output hub (IOH) (which can be on a separate chip) (not shown), wherein the GMCH includes a memory and a graphics controller and is coupled to the IOH.

[0177] The electronic device 800 may also include a coprocessor 806 and a memory 802 coupled to the controller hub 804. Alternatively, one or both of the memory 802 and the GMCH may be integrated within the processor 801 (as described in this application), with the memory 802 and the coprocessor 806 being directly coupled to the processor 801 and the controller hub 804, with the controller hub 804 being in a single chip with the IOH.

[0178] In one embodiment, the memory 802 may be, for example, a dynamic random access memory (DRAM), a phase change memory (PCM), or a combination of the two. The memory 802 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. The computer-readable storage medium stores instructions, specifically, temporary and permanent copies of the instructions.

[0179] In one embodiment, coprocessor 806 is a special purpose processor, such as, for example, a high throughput MIC processor, a network or communication processor, a compression engine, a graphics processor, a GPU, or an embedded processor, etc. The optional nature of coprocessor 806 is indicated by dashed lines in FIG. Fig.16 middle.

[0180] In one embodiment, the electronic device 800 may further include a network interface (NIC) 803. The network interface 803 may include a transceiver for providing a radio interface for the device 800 to communicate with any other suitable device (such as a front-end module, an antenna, etc.). In various embodiments, the network interface 803 may be integrated with other components of the electronic device 800. The network interface 803 may implement the functions of the communication unit in the above embodiments.

[0181] In one embodiment, Fig.16 As shown, the electronic device 800 may further include an input / output (I / O) device 805. The input / output (I / O) device 805 may include: a user interface designed to enable a user to interact with the electronic device 800; a peripheral component interface designed to enable peripheral components to interact with the electronic device 800; and / or a sensor designed to determine environmental conditions and / or location information related to the electronic device 800.

[0182] It is worth noting that Fig.16 This is for illustrative purposes only. Fig.16 It is shown that the electronic device 800 includes multiple devices such as a processor 801, a controller hub 804, and a memory 802. However, in actual applications, the devices using the methods of the present application may only include a part of the devices of the electronic device 800, for example, it may only include the processor 801 and the NIC 803. Fig.16 The properties of optional devices are shown with dashed lines.

[0183] In some embodiments of the present application, the instructions stored in the computer-readable storage medium of the electronic device 800 may include: instructions that, when executed by at least one unit in the processor, cause the device to implement the wireless charging alignment detection method mentioned in the above embodiment. When the instructions are executed on the computer, the computer executes the window control method mentioned in the above embodiment.

[0184] Now refer to Figure 17, Fig.17 This is a schematic diagram of the structure of a SOC disclosed in an embodiment of the present application, showing a block diagram of an SoC (System on Chip) 1000 according to an example of an embodiment of the present application. Fig.17In the figure, similar components have the same reference numerals. In addition, the dotted box is an optional feature of a more advanced SoC. The SoC can be used in an electronic device according to an embodiment of the present application, and corresponding functions can be implemented according to the instructions stored therein.

[0185] exist Fig.17 In the embodiment, SoC 1000 includes: an interconnect unit 1002, which is coupled to processor 1001; a system agent unit 1006; a bus controller unit 1005; an integrated memory controller unit 1003; a group or one or more coprocessors 1007, which may include integrated graphics logic, image processors, audio processors, and video processors; a static random access memory (SRAM) unit 1008; and a direct memory access (DMA) unit 1004. In one embodiment, coprocessor 1007 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a GPU, a high throughput MIC processor, or an embedded processor, etc.

[0186] The static random access memory (SRAM) unit 1008 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, and more specifically, temporary and permanent copies of the instructions.

[0187] When the SoC 1000 is applied to an electronic device according to the present application, the instructions stored in the computer-readable storage medium may include: instructions that, when executed by at least one unit in the processor, cause the electronic device to implement the wireless charging alignment detection method mentioned in the above embodiment. When the instructions are executed on a computer, the computer executes the window control method mentioned in the above embodiment.

[0188] In addition, an embodiment of the present application further discloses a computer-readable storage medium, on which a processing program is stored. When the processing program is executed by a processor, the window control method mentioned in the above embodiment is implemented.

[0189] The computer-readable storage medium may be a read-only memory, a random access memory, a hard disk, or an optical disk.

Claims

1. A window control method, applied to an electronic device based on virtual reality technology or augmented reality, wherein the electronic device is associated with a pointing component, It is characterized in that The window control method is used to control multiple window layers displayed in a visible area of ​​the electronic device, each window layer is arranged in sequence along a direction away from a user side of the electronic device, and each window layer includes multiple windows, wherein the window control method includes: detecting movement of the pointing component; Determining a selection area based on the movement trajectory of the pointing component; Determining at least one target window from the plurality of window layers according to the selected area; Simultaneously executing the operation indicated by the pointing component on at least one of the target windows; Determining at least one target window from the plurality of window layers according to the selected area comprises: Based on the selected area, determining a plurality of target windows in a plurality of window layers; The target window at the first level is mapped to other levels, and the overlapping area with the target window at the other levels is greater than or equal to a first threshold.

2. The window control method according to claim 1, It is characterized in that The movement of the pointing component is a click on the window, and the selection area is determined based on the click position of the pointing component.

3. The window control method according to claim 1, It is characterized in that The movement of the pointing component is to move along a predetermined trajectory, The selection area is determined based on a track formed when the pointing component moves.

4. The window control method according to claim 3, It is characterized in that The track formed by the movement of the pointing component forms a closed figure, and the electronic device determines the selection area based on the area covered by the closed figure, and the window within the area of ​​the closed figure as the selection area is the selected target window.

5. The window control method according to any one of claims 1 to 4, It is characterized in that The retracting and releasing operations indicated by the pointing component include reducing, enlarging, and moving the window position of each target window.

6. The window control method according to any one of claims 1 to 4, It is characterized in that Determining at least one target window from the plurality of window layers according to the selected area comprises: At least one target window in the same window layer is determined based on the selected area.

7. The window control method according to any one of claims 1 to 4, It is characterized in that At least one window in each window layer is selected.

8. The window control method according to claim 6, It is characterized in that The range of the closed figure determined by the selection area covers multiple windows in multiple window layers, and the multiple windows are determined as the target windows.

9. The window control method according to claim 6, It is characterized in that The selected area selects a window on one window layer among the plurality of window layers, and the selected area is mapped to at least one window determined on other window layers as the target window.

10. The window control method according to claim 1, It is characterized in that In a case where the at least one target window includes at least two windows, performing the operation indicated by the pointing component on the multiple target windows includes: The positions of the at least two windows are swapped with each other.

11. The window control method according to claim 10, It is characterized in that The at least two windows are divided into two window groups, and performing the operation indicated by the pointing component on the multiple target windows includes: The positions of the two window groups are swapped with each other.

12. The window control method according to claim 10 or 11, It is characterized in that The area of ​​the first target window of at least two of the windows is adjusted to fit with the display area at the position of the second target window, or the area of ​​the second target window of the at least two windows is adjusted to fit with the display area at the position of the first target window.

13. An electronic device, It is characterized in that The electronic device is associated with a pointing component, and the electronic device is an electronic device based on virtual reality technology or augmented reality, and the electronic device includes: A memory, wherein the memory is used to store window control instructions; A processor, wherein the processor implements the following steps when executing the window control instruction: detecting movement of the pointing component; Determining a selection area based on the movement trajectory of the pointing component; determining at least one target window from a plurality of window layers according to the selected area; executing the operation indicated by the pointing component on at least one of the target windows simultaneously; Determining at least one target window from the plurality of window layers according to the selected area comprises: Based on the selected area, determining a plurality of target windows in a plurality of window layers; The target window at the first level is mapped to other levels, and the overlapping area with the target window at the other levels is greater than or equal to a first threshold.

14. The electronic device according to claim 13, It is characterized in that When executing the window control instruction, the processor is further configured to determine the selection area based on the click position of the pointing component.

15. The electronic device according to claim 13, It is characterized in that When executing the window control instruction, the processor is further configured to determine the selection area based on a track formed when the pointing component moves.

16. The electronic device according to any one of claims 13 to 15, It is characterized in that When executing the window control instruction, the processor is further configured to reduce, enlarge, and move the position of each target window based on the retraction and expansion operations indicated by the pointing component.

17. The electronic device according to any one of claims 13 to 15, It is characterized in that At least one window in each window layer is selected.

18. The electronic device according to claim 17, It is characterized in that The memory is further configured to store the selected area so that the processor can call the selected area.

19. The electronic device according to any one of claims 13 to 15, It is characterized in that When executing the window control instruction, the processor is further configured to swap positions of at least two windows or swap positions of two window groups.

20. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a storage window control instruction, and when the storage window control instruction is executed by a processor, the window control method according to any one of claims 1 to 12 is implemented.

Images