Method and apparatus for controlling graphical user interface, electronic device, and medium

Abstract

The present disclosure provides a method and an apparatus for controlling a graphical user interface, an electronic device, and a medium. A specific implementation solution comprises: in response to determining that a mouse interaction mode has been called, acquiring screen information of a screen for displaying a graphical user interface, the screen information comprising screen identifier information; on the basis of the screen identifier information, determining a target screen; on the basis of pose information of an interaction device, determining a target position of a simulated mouse in the target screen, the interaction device being used to control, by means of the simulated mouse, the graphical user interface displayed by the target screen; creating the simulated mouse at the target position of the target screen; and, by means of receiving, by the interaction device, an operation by a user for the simulated mouse, generating a mouse instruction, so as to control the graphical user interface displayed by the target screen.

Description

Method, device, electronic device and medium for controlling a graphical user interface

[0001] This disclosure claims priority to the Chinese patent application filed with the China Patent Office on December 29, 2023, with application number CN202311865643.8, and invention name “Method, device, electronic device and medium for controlling a graphical user interface”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The present disclosure relates to the technical fields of virtual reality and augmented reality, and in particular to a method, device, electronic device, and medium for controlling a graphical user interface. Background Art

[0003] Head-mounted display devices (HMDs) have important applications in augmented reality (AR), virtual reality (VR), and other fields. In these fields, users can interact with HMDs in various ways to control the graphical user interface (GUI) displayed by the HMDs. For example, in specific scenarios, such as when the HMDs are equipped with a camera that captures gestures, users can control the HMD display through gestures. To enhance the user experience, a universal interaction method for controlling the HMD display is urgently needed, applicable to various scenarios. Summary of the Invention

[0004] In a first aspect, an embodiment of the present disclosure provides a method for controlling a graphical user interface, comprising: in response to determining that a mouse interaction mode is called, obtaining screen information of a screen for displaying a graphical user interface, wherein the screen information includes screen identification information; determining a target screen based on the screen identification information; determining a target position of a simulated mouse in the target screen according to posture information of an interactive device, wherein the interactive device is used to control the graphical user interface displayed on the target screen through the simulated mouse; creating a simulated mouse at a target position on the target screen; receiving user operations on the simulated mouse through the interactive device, and generating mouse instructions; and controlling the graphical user interface displayed on the target screen.

[0005] In a second aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program for executing the method for controlling a graphical user interface provided by any of the above embodiments of the present disclosure.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the method for controlling a graphical user interface provided in any of the above embodiments of the present disclosure.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a computer program product, comprising computer program instructions. When the computer program instructions are executed by a processor, the processor executes the method for controlling a graphical user interface provided by any of the above embodiments.

[0008] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other purposes, features, and advantages of the present disclosure will become more apparent through a more detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and are not intended to limit the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0010] FIG1 is an exemplary system architecture of an embodiment of a method or apparatus for controlling a graphical user interface that may be applied to the present disclosure;

[0011] FIG2 is a flow chart of a method for controlling a graphical user interface provided by an exemplary embodiment of the present disclosure;

[0012] FIG3 is a schematic diagram of a process of creating a simulated mouse according to an exemplary embodiment of the present disclosure;

[0013] FIG4 is a schematic diagram of a process of creating a simulated mouse provided by another exemplary embodiment of the present disclosure;

[0014] FIG5 is a flow chart of a method for controlling a graphical user interface provided by another exemplary embodiment of the present disclosure;

[0015] FIG6 is a schematic diagram of a flow chart of generating a mouse instruction according to an exemplary embodiment of the present disclosure;

[0016] FIG7 is a schematic diagram of a process of converting a user operation into a touch event according to an exemplary embodiment of the present disclosure;

[0017] FIG8 is a schematic diagram of a flow chart of controlling the movement of a simulated mouse position by an interactive device according to an exemplary embodiment of the present disclosure;

[0018] FIG9 is a schematic diagram of a process of simulating a mouse operation on an application page through key control of an interactive device according to an exemplary embodiment of the present disclosure;

[0019] FIG10 is a schematic structural diagram of an apparatus for controlling a graphical user interface provided by an exemplary embodiment of the present disclosure;

[0020] FIG11 is a schematic structural diagram of an apparatus for controlling a graphical user interface provided by another exemplary embodiment of the present disclosure;

[0021] FIG12 is a schematic structural diagram of an application embodiment of the electronic device disclosed herein. DETAILED DESCRIPTION

[0022] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0023] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0024] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0025] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0026] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0027] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices and servers, which can operate with many other general or special computing system environments or configurations. Electronic devices such as terminal devices and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, target programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, in which tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on a local or remote computing system storage medium including a storage device.

[0028] The following describes the specific embodiments of the present disclosure in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure and are not intended to limit the present disclosure.

[0029] Figure 1 illustrates an exemplary system architecture applicable to embodiments of the method or apparatus for controlling a graphical user interface disclosed herein. The system architecture may include a head-mounted display device 1, a network 2, and a terminal device 3. Network 2 may provide a communication link between head-mounted display device 1 and terminal device 3. Network 2 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0030] The head-mounted display device 1 can be an electronic device with an image display function. Users can view two-dimensional or three-dimensional images or videos through the head-mounted display device. Head-mounted display devices can include, but are not limited to, augmented reality (AR) devices, virtual reality (VR) devices, mixed reality (MR) devices, etc.

[0031] The head-mounted display device 1 generally includes an optical imaging system and a frame that supports the optical imaging system. The optical imaging system allows the user to view images of appropriate size. The frame allows the user to wear the head-mounted display device 1 on their head to view images through the optical imaging system. The frame can take the form of glasses, a headband, a helmet, or the like.

[0032] In some optional embodiments, the head-mounted display device 1 may be an all-in-one device that integrates the functionality of a computing unit. This computing unit provides task processing capabilities for the head-mounted display device 1, and various client applications installed on the head-mounted display device 1 are actually installed in the computing unit. In this case, the interactive device may be an external input device for the head-mounted display device, used to indicate the coordinate positioning of the head-mounted display device's display system.

[0033] In some optional embodiments, the head-mounted display device 1 can also be a split device. The split head-mounted display device 1 may include a head-mounted display and a computing unit that is separately provided from the head-mounted display. The display screen of the head-mounted display is used to display images processed by the computing unit. The computing unit in the split head-mounted display device 1 can be a terminal device, as shown in Figure 1. The terminal device can include but is not limited to a smartphone, a tablet computer, an e-book reader, a laptop computer, a desktop computer, a screen projector, and the like. The screen projector can be configured with an operating system such as Android and can also install various applications. Optionally, the interactive device and the terminal device where the computing unit is located can be the same electronic device, or the two can also be different electronic devices.

[0034] Optionally, the projector itself may not have a screen; instead, it may be combined with a head-mounted display device for display. Users can interact with the head-mounted display device via buttons, scroll wheels, knobs, etc. on the projector to control the display of the graphical user interface. The specific operation method of the projector is not limited to a single method.

[0035] The terminal device 3 is an electronic device that can provide various services. In Figure 1, the interactive device and the computing unit are the same electronic device, a screen projector, as an example. The screen projector can be provided with buttons with different control functions. In response to determining to call the mouse interaction mode, the screen projector can obtain the screen identification information of the screen used to display the graphical user interface, thereby determining the target screen, and then determine the target position of the simulated mouse in the target screen according to the posture information of the interactive device, and create a simulated mouse at the target position of the target screen, so that the user can control the graphical user interface displayed on the target screen through the simulated mouse. Accordingly, the device for controlling the graphical user interface can be provided in the terminal device 3.

[0036] It should also be noted that the method for controlling a graphical user interface provided by the present disclosure can also be applied to a server (not shown in the figure), which can be a backend server or a cloud service. Accordingly, the device for controlling the graphical user interface is provided in the server. This is not a single limitation.

[0037] Exemplary Methods

[0038] FIG2 is a flow chart of a method for controlling a graphical user interface provided by an exemplary embodiment of the present disclosure. The embodiment of the present disclosure can be run on an electronic device. As shown in FIG2 , the method includes the following steps:

[0039] Step 210 : In response to determining that the mouse interaction mode is invoked, screen information of a screen for displaying a graphical user interface is acquired.

[0040] In some optional embodiments of the present disclosure, the screen information may include but is not limited to screen identification information.

[0041] In some optional embodiments of the present disclosure, if the interactive device only supports the mouse interaction mode, it can be directly determined that it is currently in the mouse interaction mode. If the interactive device supports other interaction modes such as the key interaction mode in addition to the mouse interaction mode, it can be determined whether it is currently in the mouse interaction mode. Any feasible method can be used here to determine whether it is in the mouse interaction mode. For example, when the system sets the mouse interaction mode by default, it can be directly determined that it is currently in the mouse interaction mode. For another example, the user can also actively call the mouse interaction mode, and the calling of the mouse interaction mode can be determined based on the user setting information.

[0042] In some optional embodiments of the present disclosure, the screen information for displaying the graphical user interface can be determined according to the specific conditions of the graphical user interface display. For example, the screen information of the screen can be determined according to the size of the graphical user interface, the type of application to which it belongs, etc.

[0043] In some optional embodiments of the present disclosure, the screen identification information may be a unique identifier for the screen. The screen identification information may be represented as a Display ID. Different IDs represent different screens. For example, Display ID = 0 represents the system screen, and Display ID = 1 represents a virtual screen. Optionally, the number of virtual screens may be at least one. Different virtual screens may be represented by different screen identification information. For example, Display ID = 1 represents virtual screen 1, and Display ID = 2 represents virtual screen 2. Of course, the screen identification information may also be represented in other ways, which are not limited by the embodiments of the present disclosure.

[0044] The execution subject of the method for controlling a graphical user interface (which may be a terminal device, such as the screen projector shown in FIG1 ) can obtain screen information of the screen used to display the graphical user interface when determining that it is currently in a state of calling a mouse interaction mode. It can be understood that for a currently running application, the screen for display can be called to display the corresponding graphical user interface through a head-mounted display device, and the called screen here can be configured with corresponding screen information. The above-mentioned execution subject can obtain the screen information of the called screen. As an example, the called screen can be a system screen with Display ID=0, and the screen information obtained by the above-mentioned execution subject includes Display ID=0.

[0045] Step 220: Determine the target screen based on the screen identification information.

[0046] In some optional embodiments of the present disclosure, the target screen is a screen that can be displayed on the physical display screen of a head-mounted display device. For example, the target screen can be a system screen displayed by the head-mounted display device, a spatial screen hosting a virtual screen, and so on. Different screen identification information corresponds to different target screens.

[0047] Step 230: Determine the target position of the simulated mouse on the target screen based on the posture information of the interactive device.

[0048] In some optional embodiments of the present disclosure, the interactive device may be an external input device for a head-mounted display device. The interactive device may be an indicator of the vertical and horizontal coordinates of the head-mounted display system. The interactive device may simulate a mouse to operate a graphical user interface displayed on a target screen.

[0049] In some optional embodiments of the present disclosure, there is a correlation between the posture information of the interactive device and the position of the simulated mouse displayed on the target screen, so that changes in the posture of the interactive device can represent changes in the position of the simulated mouse on the target screen. Therefore, the target position of the simulated mouse on the target screen can be determined based on the posture information of the interactive device.

[0050] For example, a relationship can be established between the initial position of the interactive device and the initial position of the simulated mouse on the target screen. Then, as the position of the interactive device changes, the target position of the simulated mouse on the target screen can be determined in real time.

[0051] In some optional embodiments of the present disclosure, the position information of the interactive device can be determined using data collected by sensors on the interactive device. The position information can include at least one of position and attitude information. For example, the position information of the interactive device can be determined using gyroscope data on the interactive device. The position information of the interactive device can be stored in the form of quaternions.

[0052] In some optional embodiments of the present disclosure, the simulated mouse is different from the real mouse used in the system. The simulated mouse can be a graphic added to the screen to simulate a mouse, so that similar functions as a mouse can be realized by operating the simulated mouse. For example, the user's operation on the simulated mouse can be converted into mouse operation instructions, thereby realizing the mouse function. Therefore, the simulated mouse can follow the screen identifier and draw the simulated mouse on the screen corresponding to the screen identifier.

[0053] Step 240: Create a simulated mouse at a target location on the target screen.

[0054] In some optional embodiments of the present disclosure, after determining the target position of the simulated mouse on the target screen, the simulated mouse can be created at the target position of the target screen to display the simulated mouse on the target screen so that the user can intuitively see the position of the simulated mouse.

[0055] In some optional embodiments of the present disclosure, a simulated mouse can be created at a target location on a target screen using any feasible method. For example, a simulated mouse with corresponding attributes can be created at the target location on the target screen based on preset attributes such as the shape, size, and color of the simulated mouse. The specific shape, size, and color of the simulated mouse are not intended to be exclusive. As an example, the simulated mouse can be displayed as a cursor point.

[0056] Step 250: receiving user operations on the simulated mouse through the interactive device, and generating mouse instructions to control the graphical user interface displayed on the target screen.

[0057] In some optional embodiments of the present disclosure, an interactive device is provided with a user operation area, which may include an operation object such as a button, scroll wheel, or knob with at least one function. The user can trigger an operation on the simulated mouse through the operation object. The interactive device can receive the user's operation on the simulated mouse and generate mouse commands to control the graphical user interface displayed on the target screen. For example, a user long-pressing a button on the interactive device may generate a mouse command to control the graphical user interface displayed on the target screen to turn pages, scroll, return to the previous page, etc.

[0058] The method for controlling a graphical user interface provided by an embodiment of the present disclosure can obtain screen information of a screen for displaying a graphical user interface when it is determined that a mouse interaction mode is called, and the screen information includes screen identification information. The target screen can be determined based on the screen identification information. According to the posture information of the interactive device, the target position of the simulated mouse in the target screen is determined. A simulated mouse is created at the target position of the target screen. Then, the user's operation on the simulated mouse can be received through the interactive device, and mouse instructions can be generated to control the graphical user interface displayed on the target screen. The embodiment of the present disclosure can create a simulated mouse on the screen indicated by different screen identifiers, so that the simulated mouse can be created following the screen for display, so that no matter which screen the graphical user interface is displayed on, the display of the graphical user interface can be controlled by the simulated mouse. It effectively solves the problem in the related art that the native mouse can only be drawn on the main screen and cannot be used on other screens.

[0059] In some optional embodiments of the present disclosure, the target position of the simulated mouse on the target screen can be determined by the following steps: in response to determining that the posture information of the interactive device is posture information at an initial moment, determining the initial position of the simulated mouse on the target screen as the target position. In response to determining that the posture information of the interactive device is posture information at a non-initial moment, determining the target position of the simulated mouse on the target screen based on the posture information of the interactive device.

[0060] In some optional embodiments of the present disclosure, it can be determined first whether the posture information of the interactive device is the posture information at the initial moment. The initial moment here can be the moment when the mouse interaction mode is started to be called. Alternatively, the initial moment can also be the moment when the user's operation on the simulated mouse is received again when no operation on the simulated mouse is received for more than a preset time. In the case of determining that the posture information of the interactive device is the posture information at the initial moment, a correspondence between the posture information of the interactive device at the initial moment and the initial position in the target screen can be established, and the initial position of the simulated mouse on the target screen can be the target position corresponding to the posture of the interactive device at the initial moment.

[0061] In some optional embodiments of the present disclosure, the initial position of the simulated mouse on the target screen can be a preset position in the target screen. For example, the initial position of the simulated mouse on the target screen can be the center position of the target screen or other preset positions.

[0062] In some optional embodiments of the present disclosure, if the posture information of the interactive device is not the posture information at the initial moment, the target position of the simulated mouse on the target screen is determined based on the posture information of the interactive device and the posture information at the initial moment. For example, the posture change information of the interactive device can be calculated based on the current posture information of the interactive device and the posture information at the initial moment. Based on the posture change information of the interactive device, the coordinate change information of the simulated mouse can be calculated. The target position on the target screen can be obtained by superimposing the coordinate change information on the coordinates of the initial position.

[0063] In some optional embodiments of the present disclosure, if the posture information of the interactive device is not the posture information at the initial moment, the target position of the simulated mouse in the target screen can also be determined based on the posture information of the interactive device at the current moment and the posture information at the previous moment.

[0064] This embodiment effectively determines the target position of the simulated mouse on the target screen by determining whether the interactive device's posture information is the initial posture information, thereby improving the accuracy of the target position. Furthermore, determining the initial position on the target screen facilitates the user's ability to quickly locate the simulated mouse on the target screen at the initial moment, improving the user experience. This solves the problem of the system's built-in mouse being unable to meet the requirements of this solution for drawing the mouse based on the target screen's identification.

[0065] In some optional embodiments, a method for controlling a graphical user interface is provided, the method being used to control a graphical user interface displayed on a virtual screen of a head-mounted display device via an interactive device. The method comprises: in response to determining that the head-mounted display device is in a mouse interaction mode, obtaining a change in the posture of the interactive device; determining a change in the position of a simulated mouse on the virtual screen of the head-mounted display device based on the change in the posture of the interactive device; wherein a preset association exists between the posture information of the interactive device and the position information of the simulated mouse on the virtual screen; displaying a graphic of the simulated mouse at the position of the simulated mouse on the virtual screen after the position change; and receiving an operation input by a user on the simulated mouse via the interactive device, and controlling the graphical user interface displayed on the virtual screen to update in response to the user operation.

[0066] In some optional embodiments, a method for controlling a graphical user interface is provided, comprising: detecting posture change information of an interactive device in real time; determining, in response to the posture change information of the interactive device, a position change of a simulated mouse according to a preset association relationship between the posture information of the interactive device and position information of a simulated mouse in a virtual screen; determining a screen identifier of the virtual screen of a head-mounted display device; drawing the simulated mouse after the position change in the virtual screen indicated by the determined screen identifier; receiving an operation input by a user for the simulated mouse through the interactive device, and controlling the graphical user interface displayed on the virtual screen to update in response to the user operation.

[0067] FIG3 is a schematic diagram of a flow chart of creating a simulated mouse according to an exemplary embodiment of the present disclosure.

[0068] In some optional embodiments of the present disclosure, a simulated mouse may be created by the following steps:

[0069] Step 24110: Create a window of a preset size at a target location on the target screen.

[0070] In some optional embodiments of the present disclosure, the preset size can be set to any size according to actual needs. The preset size of the window can be expressed as h*w. h can represent the height of the window (i.e., the number of pixels occupied by the window in the height direction of the target screen), and w can represent the width of the window (i.e., the number of pixels occupied by the window in the width direction of the target screen). For example, the preset size of the window can be 50*50.

[0071] Step 24120, draw the graphics of the simulated mouse in the window to create a simulated mouse.

[0072] In some optional embodiments of the present disclosure, the graphic simulating the mouse can be a graphic of any shape and size. For example, the graphic simulating the mouse can be a circle with a preset radius, a square with a preset side length, a triangle, an arrow shape, etc.

[0073] In some optional embodiments of the present disclosure, the execution entity may create a window at a higher level than the target screen above the target screen, and then draw a simulated mouse graphic within the window. This embodiment can separate the target screen and the simulated mouse into different levels, thereby facilitating the simulated mouse to follow the target screen for display. It is understood that the execution entity may also create a simulated mouse in other ways, for example, by directly drawing a simulated mouse graphic at the target location.

[0074] This embodiment creates a window of preset size at the target position of the target screen and draws a simulated mouse graphic in the window, thereby achieving the drawing of the simulated mouse that changes with the position of the interactive device, making it easier to control the display of the graphical user interface through the simulated mouse.

[0075] In some optional embodiments of the present disclosure, the target screen may be determined by the following steps: in response to determining that the screen identification information includes a preset identifier, determining the system screen as the target screen. In response to determining that the screen identification information includes a non-preset identifier, determining the spatial screen corresponding to the virtual screen indicated by the non-preset identifier as the target screen.

[0076] In some optional embodiments of the present disclosure, the target screen corresponding to the virtual screen is obtained in the following manner: creating a virtual screen and setting a unique screen identifier for the virtual screen; determining the spatial screen corresponding to the virtual screen based on the posture information of the head-mounted display device, the screen information of the spatial screen including the screen identifier information of the virtual screen and the posture information of the spatial screen.

[0077] In some optional embodiments of the present disclosure, the preset identifier may be a screen identifier corresponding to a system screen, and thus may be used to indicate the system screen. As an example, the system screen may be the home screen of an Android system. For example, the preset identifier may include Display ID=0. If it is determined that the screen identification information includes a preset identifier, the system screen may be determined as the target screen for displaying a graphical user interface. Typically, the content displayed on the system screen may be displayed on the head (or mirrored) to the physical screen of the head-mounted display device.

[0078] In some optional embodiments of the present disclosure, if it is determined that the screen identification information includes a non-preset identification, the non-preset identification may indicate that the screen used to display the graphical user interface is not a system screen. Alternatively, the non-preset identification may also indicate that the screen used to display the graphical user interface is a created virtual screen. In this case, the spatial screen corresponding to the virtual screen indicated by the non-preset identification may be determined as the target screen.

[0079] In some optional embodiments of the present disclosure, a virtual screen may be created through a pre-installed virtual graphics card driver, and a unique screen ID may be set for the virtual screen, for example, the screen ID of the virtual screen may be set to Display ID=1.

[0080] In some optional embodiments of the present disclosure, the position and posture information of the head-mounted display device can be determined using any feasible method. For example, the head-mounted display device can be positioned using sensor data collected by sensors on the head-mounted display device to obtain first device position and posture information of the head-mounted display device. For example, the position and posture information of the head-mounted display device can be determined using a simultaneous localization and mapping (SLAM) method.

[0081] In some optional embodiments of the present disclosure, a spatial screen is a screen used to carry image data displayed on a virtual screen to display a spatial image. For example, the spatial screen can be a screen with certain position information in the virtual space viewed by a head-mounted display device. For the virtual screen and its opened graphical user interface (which may include a simulated mouse), the corresponding data stream can be processed as a virtual scene, and then the processed image (also referred to as a virtual scene image) can be displayed on the physical screen of the head-mounted display device. The content displayed on the physical screen can be projected into space by the head-mounted display device to obtain a spatial screen corresponding to the virtual screen.

[0082] In some optional embodiments, processing the virtual scene may include using a 3D image rendering engine (e.g., Unity3D) to render the data stream into a constructed virtual space. Alternatively, processing the virtual scene may also include rendering the data stream or the image rendering result. This may result in a virtual scene image of the virtual screen and its open graphical user interface.

[0083] In some optional embodiments of the present disclosure, the pose information of the head-mounted display device may be pose information in a reference coordinate system, such as a world coordinate system, a reference coordinate system with the initial position of the head-mounted display device as the origin, or the like. The reference coordinate system is associated with the virtual space, and therefore, the pose information of the virtual camera observing the virtual space can be determined based on the pose information of the head-mounted display device. Based on the pose information of the virtual camera, the pose information of the spatial screen can be determined, thereby enabling 3D0F or 6D0F display. For example, the spatial screen is a screen in hovering mode at a preset distance in front of the head-mounted display device.

[0084] In some optional embodiments of the present disclosure, screen identification information for displaying the graphical user interface of each application (or simply application) can be set for each application based on the performance of the application in the system. For example, for an application with high performance consumption, the graphical user interface of the application can be displayed on the system screen, and a simulated mouse can be created on the system screen. For an application with low performance consumption, the graphical user interface of the application can be displayed on a virtual screen, and a simulated mouse can be created on the virtual screen.

[0085] In this embodiment, when it is determined that the screen identification information includes a non-preset identification, the spatial screen corresponding to the virtual screen can be determined based on the posture information of the head-mounted display device, and the spatial screen can be determined as the target screen to realize the display of the virtual screen and the display of the graphical user interface of the virtual screen by simulating a mouse.

[0086] FIG4 is a schematic diagram of a flow chart of creating a simulated mouse provided by another exemplary embodiment of the present disclosure.

[0087] In some optional embodiments of the present disclosure, a simulated mouse may be created by the following steps:

[0088] Step 24210: Determine the target position on the virtual screen corresponding to the target screen based on the screen identification information of the virtual screen.

[0089] In some optional embodiments of the present disclosure, context information of the virtual screen indicated by the virtual screen can be obtained based on the screen identification information of the virtual screen to determine the target position on the virtual screen. The context information of the virtual screen may include the size information (resolution) of the virtual screen and other relevant information. For example, the virtual screen is W*H, where H represents the number of pixels in the height direction of the virtual screen and W represents the number of pixels in the width direction of the virtual screen. For example, the virtual screen is a 1920*1080 screen.

[0090] In some optional embodiments of the present disclosure, if the target position of the simulated mouse on the target screen is the initial position, the preset position on the virtual screen is determined as the target position. The preset position can be, for example, the center position of the virtual screen. For example, if the virtual screen is a 1920*1080 screen, the target position is the position (960,540) in the center of the virtual screen. If the target position of the simulated mouse on the target screen is not the initial position, the target position can be determined on the virtual screen based on the posture change information of the interactive device.

[0091] Step 24220, create a window layer simulating a mouse at the target position.

[0092] In some optional embodiments of the present disclosure, the layer level of the window simulating the mouse is higher than the layer level of the graphical user interface.

[0093] In some optional embodiments of the present disclosure, after the virtual screen determines the target position, a window layer of a simulated mouse can be created at the target position on the virtual screen for drawing the simulated mouse. The window layer level of the simulated mouse is higher than the layer level of the graphical user interface. This allows the simulated mouse to cover a local window area of ​​the graphical user interface after the virtual screen and its content are rendered to the spatial screen, so that the user can see the position of the simulated mouse in the graphical user interface, making it easier to determine the corresponding operation position of the user's operation on the simulated mouse in the graphical user interface.

[0094] Step 24230, draw the simulated mouse in the simulated mouse window layer.

[0095] In some optional embodiments of the present disclosure, after creating a window layer for simulating a mouse at a target location, a simulated mouse can be drawn in the window layer for simulating a mouse. For example, a graphic of a certain shape and size can be drawn in the window layer for simulating a mouse to represent the simulated mouse.

[0096] In this embodiment, a target position is determined on a virtual screen corresponding to a target screen, a simulated mouse window layer is created at the target position, and a simulated mouse is drawn in the simulated mouse window layer, thereby creating a simulated mouse on the virtual screen, making the drawing of the simulated mouse more convenient. Then, based on the mapping relationship between the virtual screen and the spatial screen, a simulated mouse is created at the target position on the target screen, enabling the simulated mouse to control the display of a graphical user interface not only on the system screen but also on the spatial screen corresponding to the virtual screen, following the screen identifier.

[0097] FIG5 is a flowchart of a method for controlling a graphical user interface provided by another exemplary embodiment of the present disclosure.

[0098] In some optional embodiments of the present disclosure, the method of the embodiment of the present disclosure further includes:

[0099] Step 310: Acquire the image data of the simulated mouse drawn in the window layer of the simulated mouse.

[0100] In some optional embodiments of the present disclosure, the image data of the simulated mouse can be set to image data of any shape and size according to actual needs. For example, the image data of the simulated mouse can be a circular image with a preset radius, a square image with a preset side length, an arrow image, etc.

[0101] In some optional embodiments of the present disclosure, the image data of the simulated mouse can be pre-set and stored, so that the drawn image data of the simulated mouse can be obtained from the corresponding storage area.

[0102] Step 320: Render the image data of the simulated mouse and the image data of the graphical user interface to the spatial screen, and display the spatial screen through the head-mounted display device.

[0103] In some optional embodiments of the present disclosure, the graphical user interface image data can be determined based on interface changes caused by user operations. For example, if a user clicks Application A on the system homepage, the graphical user interface image data corresponding to Application A is obtained, and the simulated mouse image data and the graphical user interface image data are rendered to the spatial screen, which can then be displayed via the head-mounted display device.

[0104] In some optional embodiments of the present disclosure, the image data of the simulated mouse and the image data of the graphical user interface can be rendered to the spatial screen according to the layer hierarchy of the graphical user interface and the layer hierarchy of the window of the simulated mouse. For example, according to the layer hierarchy of the graphical user interface and the layer hierarchy of the window of the simulated mouse, the image data of the graphical user interface is first rendered to the spatial screen, and then the image data of the simulated mouse is rendered to the spatial screen, covering the area of ​​the window size of the simulated mouse at the target position of the layer of the graphical user interface. Optionally, the layer of the graphical user interface can also be merged with the window layer of the simulated mouse to obtain a graphical user interface including the simulated mouse, and the corresponding data stream can be rendered to the spatial screen.

[0105] In this embodiment, by rendering the image data of the simulated mouse and the image data of the graphical user interface to the spatial screen, the spatial screen can be displayed through the head-mounted display device, so that the graphical user interface of the virtual screen and the simulated mouse are displayed on the display screen of the head-mounted display device, thereby facilitating the control of the graphical user interface of the virtual screen through the simulated mouse.

[0106] FIG6 is a schematic diagram of a flow chart of generating a mouse instruction according to an exemplary embodiment of the present disclosure.

[0107] In some optional embodiments of the present disclosure, the mouse command may be generated by the following steps:

[0108] Step 2510: In response to receiving a user operation on a simulated mouse through the interactive device, convert the user operation into a touch event.

[0109] In some optional embodiments of the present disclosure, the user's operation on the simulated mouse can be determined based on the specific circumstances of the interactive device. For example, if the interactive device's operation object is a key, then the user's operation is a key operation, and a key event can be generated. The interactive device's operation object can also be an object such as a scroll wheel or knob, and the user's operation is an operation corresponding to the operation object. It is understood that if the interactive device's operation object is a touch screen, then a touch event can be directly generated based on the user's operation.

[0110] In some embodiments of the present disclosure, a touch event (Motion Event) may be an event similar to a touch operation on a touch screen, supported by systems such as Android. For interactive devices that do not include a touch screen, such as a button-type interactive device, the native interaction method in the related art is usually to move the selection box and click through the buttons on the interactive device, that is, to select an element and click on the element in the displayed interface. Button interaction cannot achieve operations such as sliding and dragging the page. And button interaction usually requires continuous traversal of certain elements on the page to select a certain element. For example, the leftmost button is currently selected on the page. If you want to select the rightmost button, there are one or more other elements between the rightmost button and the leftmost button. You need to move from the leftmost button one by one through the other elements in the middle to reach the rightmost button, and the user operation efficiency is low. The embodiment of the present disclosure can convert the user's operation into a touch event, which is convenient for producing an operation effect similar to that of a touch screen touch, and helps to improve the convenience and efficiency of user operations.

[0111] Step 2520: inject the touch event into the target screen and generate a mouse instruction for controlling the graphical user interface displayed on the target screen.

[0112] In some optional embodiments of the present disclosure, touch events are injected into a target screen to produce the effect of a touch operation on the target screen. Mouse commands for controlling a graphical user interface displayed on the target screen can be generated based on the touch events. Touch event injection herein can be understood as inputting simulated touch events into the target screen.

[0113] In some optional embodiments of the present disclosure, if the target screen is a system screen, the touch event can be injected into the system screen. If the target screen is a space screen, the touch event can be injected into the virtual screen corresponding to the space screen.

[0114] This embodiment converts the user's operations into touch events, injects the touch events into the target screen, and generates mouse instructions for controlling the graphical user interface displayed on the target screen. This helps to convert the user's operations on an interactive device without a touch screen into touch operations on a touch screen, so as to control the graphical user interface displayed on the target screen through touch operations, facilitate the achievement of operations such as sliding and dragging the page, and improve the user operation experience.

[0115] It is understandable that if other events can generate mouse commands, the execution subject can also convert other user operations for simulating the mouse into other events. Here, the event to be converted by the user operation can be determined according to the actual solution.

[0116] FIG7 is a schematic diagram of a flow chart of converting a user operation into a touch event provided by an exemplary embodiment of the present disclosure.

[0117] In some optional embodiments of the present disclosure, the user's operation may be converted into a touch event through the following steps:

[0118] Step 25110: Receive the user's operation on the preset button of the interactive device, and obtain the button identifier corresponding to the user's operation.

[0119] In some optional embodiments of the present disclosure, preset buttons of the interactive device can be set according to the specific situation of the interactive device. For example, the preset buttons are buttons with functions such as "confirm" and "select", buttons indicating directions (such as up, down, left, right, etc.), etc.

[0120] In some optional embodiments of the present disclosure, the key identifier is information that uniquely identifies a key. The key identifier can be represented by a KeyCode, and different KeyCodes can represent different keys.

[0121] In some optional embodiments of the present disclosure, the key identifier corresponding to the user operation can be obtained through the system desktop process. For example, the system desktop process can detect the key press behavior of the interactive device through the system key event callback and distinguish which key is currently pressed through the key identifier. The type of key event can also be determined through the action acquisition function of the key event (e.g., getAction). The key event type can include press and release.

[0122] Step 25120: In response to the key identifier being a preset key identifier, convert the key event generated by the user's operation into a touch event.

[0123] In some optional embodiments of the present disclosure, a preset key identifier for a key identifier can be set based on the specific key conditions of the interactive device. For example, the preset key identifiers may be KeyCode = "Confirm Key" or KeyCode = "Direction Key." When the key identifier is determined to be a preset key identifier, the key event generated by the user's operation is converted into a touch event, thereby enabling touch operation through key operation on the interactive device, thereby improving the user's operating experience.

[0124] In some optional embodiments of the present disclosure, the method of the embodiment of the present disclosure may further include: in response to receiving the first key event, injecting touch events into the target screen at a preset frequency; and in response to receiving the second key event, ending the touch event.

[0125] In some optional embodiments of the present disclosure, the first key event may be an event of pressing a key, which may be represented as a down event. The second key event may be an event of lifting a key, which may be represented as an up event. After receiving the first key event, touch events may be injected into the target screen at a preset frequency, and each touch event corresponds to a touch position on the target screen (i.e., the position of the simulated mouse on the target screen). The touch position may be determined based on the posture information of the interactive device. Based on the touch events injected into the target screen, the position change of the simulated mouse on the target screen may be calculated, and then, based on the position change of the simulated mouse on the target screen, a mouse instruction may be generated to control the display of the graphical user interface. After receiving the second key event, the injection of the touch event is ended.

[0126] In some optional embodiments of the present disclosure, at least one touch event can be injected into the target screen based on the duration of the user's key press and the preset frequency, enabling operations such as clicking, sliding, and dragging the page. It is understood that if the user long presses the key, multiple touch events can be injected into the target screen at a preset frequency. If the user clicks the key, it can be directly converted into a touch event, that is, a single touch event is injected into the target screen.

[0127] In this embodiment, when a first key event is received, a touch event is injected into the target screen at a preset frequency, and when a second key event is received, the touch event is ended. This can convert the key operation of the interactive device into a touch operation on the target screen, and helps to achieve the touch operation effect on the target screen through the key operation of the interactive device, and facilitates the sliding, dragging and other controls on the page displayed on the target screen, thereby improving the user operation experience.

[0128] In some optional embodiments of the present disclosure, mouse instructions can be generated through the following steps: in response to a key event being a preset operation event for a key identified by a first preset key, a touch event is injected into the target screen at a preset frequency, and an interface drag instruction for dragging a graphical user interface is generated.

[0129] In some optional embodiments of the present disclosure, the first preset button identifier can be set according to the specific button function of the interactive device. For example, the first preset button identifier is the button identifier of a button on the interactive device with functions such as "Confirm," "Select," or "OK." By performing a preset operation such as long pressing the button with the first preset button identifier, touch events can be injected into the target screen at a preset frequency, generating an interface drag instruction for dragging the graphical user interface. According to the interface drag instruction, the graphical user interface displayed on the target screen can be dragged.

[0130] In some optional embodiments of the present disclosure, the preset frequency can be set to any frequency according to actual needs. For example, a touch event is injected into the target screen every preset time (e.g., 10 milliseconds, 15 milliseconds, 20 milliseconds, etc.).

[0131] In this embodiment, when the key event is a preset operation event such as a long press of a key for a first preset key identifier, the touch event can be injected into the target screen at a preset frequency to generate an interface drag instruction for dragging the graphical user interface, which helps to drag the graphical user interface by long pressing the key, thereby improving the convenience and efficiency of user operations.

[0132] In some optional embodiments of the present disclosure, mouse instructions can be generated through the following steps: in response to a key event being a preset operation event for a key identified by a second preset key, a touch event is injected into the target screen at a preset frequency to generate an interface sliding event for sliding a graphical user interface.

[0133] In some optional embodiments of the present disclosure, the second preset button identifier can be set based on the specific circumstances of the buttons on the interactive device. For example, the button identifier of the direction button on the interactive device can be set to the second preset button identifier. The direction button can include a direction button for at least one direction. For example, the interactive device can include direction buttons for the four directions of up, down, left, and right. Direction buttons for different directions can be distinguished by different button identifiers, or by the direction type (or direction attribute) set for the direction button.

[0134] In some optional embodiments of the present disclosure, when a preset operation such as a long press is performed on a key identified by a second preset key, a touch event can be injected into the target screen at a preset frequency to generate an interface sliding event for sliding the graphical user interface. The sliding of the graphical user interface displayed on the target screen can be controlled according to the interface sliding event, thereby realizing page turning.

[0135] In some optional embodiments of the present disclosure, the preset operations for the keys with the first preset key identifier and the second preset key identifier may inject touch events into the target screen using the same or different preset frequencies. Alternatively, the preset operations for the keys with the first preset key identifier and the second preset key identifier may be the same or different operations.

[0136] In this embodiment, when the key event is a preset operation event such as a long press of a key identified by a second preset key, the touch event can be injected into the target screen at a preset frequency to generate an interface sliding event for sliding the graphical user interface, which helps to achieve the effect of a touch sliding interface through the key operation of the interactive device.

[0137] In some optional embodiments of the present disclosure, after generating an interface sliding event for sliding a graphical user interface, the method of the embodiment of the present disclosure may further include: determining an event position corresponding to each touch event on the target screen based on the starting position of the simulated mouse, the direction attribute of the key identified by the second preset key, and the sliding step size. Based on the event position corresponding to each touch event, generating an interface sliding instruction for controlling the graphical user interface displayed on the target screen to slide in the direction indicated by the direction attribute of the key identified by the second preset key.

[0138] In some optional embodiments of the present disclosure, the starting position of the simulated mouse can be the position of the simulated mouse at the moment when the key event of the second preset key identifier is triggered. For example, at the moment when the first key event (down event) of the key identified by the second preset key is received (this moment can be the triggering moment), the position of the current simulated mouse in the target screen is recorded as the starting position of the simulated mouse. The starting position of the simulated mouse can be expressed as (initX, initY), for example. The direction attribute of the key identified by the second preset key can be any one of up, down, left, and right. The direction attribute can be obtained through the system. The sliding step size is the pixel step size used for gradual accumulation, and the specific sliding step size can be set according to actual needs.

[0139] Optionally, the sliding step size can be expressed as delta. The event position corresponding to any touch event on the target screen can be expressed as (touchX, touchY). As touch events are continuously injected, the event position corresponding to the touch event on the target screen is gradually accumulated on the basis of the starting position of the simulated mouse. It can be understood that the event position corresponding to the touch event on the target screen here may not be the real-time position of the simulated mouse, but the position obtained by gradually accumulating the pixel step size according to the direction to be slid based on the starting position of the simulated mouse. As an example, the event position corresponding to the touch event on the target screen is initially assigned touchX=initX, touchY=initY. According to the different direction attributes of the keys identified by the second preset key, the event position corresponding to the touch event is accumulated in different directions. The event position corresponding to any touch event can be expressed as follows:

[0140] Left button: touchX+=delta; touchY remains unchanged. This means that delta is gradually accumulated in the X direction.

[0141] Right click: touchX -= delta; touchY remains unchanged. This means that delta is gradually accumulated in the opposite direction of X.

[0142] Up key: touchY+=delta; touchX remains unchanged. This means that delta is gradually accumulated in the Y direction.

[0143] Down key: touchY-=delta; touchX remains unchanged. This means that delta is gradually accumulated in the opposite direction of Y.

[0144] In this way, continuous touch events in a certain direction can be injected into the target screen. According to the event position corresponding to each touch event on the target screen, an interface sliding instruction can be generated to control the graphical user interface displayed on the target screen to slide along the direction attribute of the key identified by the second preset key. According to the interface sliding instruction, the graphical user interface can be controlled to slide along the direction corresponding to the direction attribute until the touch event ends, thereby achieving the effect of continuously sliding the page along this direction.

[0145] In some optional embodiments of the present disclosure, the method of the embodiments of the present disclosure also includes: in response to the event position of the last touch event reaching the boundary of the target screen, determining the starting position of the simulated mouse as the event position of the last touch event; determining the current event position of the current touch event on the target screen based on the event position of the last touch event, the direction attribute of the key identified by the second preset key, and the sliding step size.

[0146] Since the event position corresponding to the touch event is the result of gradual accumulation based on the starting position of the simulated mouse, as the accumulation continues, the event position will gradually reach the edge of the target screen, causing the page to stop sliding after sliding for a certain distance. Moreover, if the starting position of the simulated mouse is close to the edge of the target screen, and continues to slide in the direction of the edge to reach or exceed the edge, the coordinates of the simulated mouse cannot be accumulated or subtracted, which makes it easy to misidentify it as a click operation on the page. In response to the above problem, the embodiment of the present disclosure can use the starting position of the simulated mouse as the event position of the last touch event when the event position of the last touch event reaches the boundary of the target screen, and continue to slide based on the starting position of the simulated mouse to determine the current event position of the current touch event on the target screen. As an example, the event position (touchX, touchY) of the last touch event reaches the boundary of the target screen, and touchX and touchY are reassigned to touchX=initX, touchY=initY, (initX, initY) can represent the starting position of the simulated mouse. Based on touchX=initX and touchY=initY, the current event position of the current touch event on the target screen can be determined according to the direction attribute and sliding step size of the case identified by the second preset key. For example, the current event position of the current touch event can be determined according to the accumulation method corresponding to the left key, right key, up key, and down key.

[0147] In some optional embodiments of the present disclosure, when the event position of the last touch event reaches the boundary of the target screen, a touch event for canceling a touch slide (e.g., ACTION_CANCEL) can be injected into the target screen to indicate that the touch event is terminated, and the event position of the touch event is reassigned to the starting position of the simulated mouse, and a touch event for starting a touch slide (e.g., ACTION_DOWN) is continuously injected into the target screen to start a new slide and continue to accumulate or subtract event positions. When the simulated mouse reaches the edge of the page, replacing the ACTION_UP key event with the ACTION_CANCEL touch event can effectively ensure that the page does not jump back and forth, so that the page can continue to slide continuously and stably in the original direction.

[0148] In this embodiment, when the event position of the previous touch event reaches the boundary of the target screen, the starting position of the simulated mouse is used as the event position of the previous touch event. Furthermore, based on the event position of the previous touch event, the direction attribute of the key identified by the second preset key, and the sliding step length, the current event position of the current touch event can be determined. This can avoid stopping page sliding due to the event position reaching the boundary of the target screen, thereby improving the continuity of page sliding. Furthermore, it can prevent the simulated mouse from being mistakenly recognized as a click operation when it reaches the boundary of the target screen and continues to slide toward the boundary, helping to improve the accuracy of operation recognition and thus enhance the user operation experience.

[0149] In some optional embodiments of the present disclosure, the method of the embodiment of the present disclosure further includes: in response to the interactive device receiving a preset operation event for a key with a third preset key identifier, generating a simulated mouse reset instruction to reset the position of the simulated mouse in the target screen to a preset initial position. In response to the interactive device receiving a preset operation event for a key with a fourth preset key identifier, generating an interface return simulated mouse instruction to return the graphical user interface displayed on the target screen to the previous level interface. Here, the preset operation events for the key with the third preset key identifier and the key with the fourth preset key identifier can be the same operation or different operations. As an example, the operation for the key with the third preset key identifier can be a long press operation, and the operation for the key with the fourth preset key identifier can be a click operation.

[0150] In some optional embodiments of the present disclosure, the third preset button identifier can be set according to the specific button function of the interactive device. For example, the third preset button identifier can be the button identifier of the button representing "mode" on the interactive device. If the user long presses the button with the third preset button identifier, a simulated mouse reset instruction can be generated based on the user's key event. The simulated mouse reset instruction is used to reset the position of the simulated mouse in the target screen, resetting the position of the simulated mouse in the target screen to a preset initial position. The preset initial position can be set according to actual needs. For example, the preset initial position is the center position of the target screen.

[0151] In some optional embodiments of the present disclosure, the fourth preset button identifier can be set based on the specific button function of the interactive device. For example, the fourth preset button identifier can be the button identifier of the button representing "Back" on the interactive device. If the user clicks the button with the fourth preset button identifier, a simulated mouse command for returning to the interface can be generated based on the user's key event, returning the graphical user interface displayed on the target screen to the previous interface.

[0152] In some optional embodiments of the present disclosure, Figure 8 is a flow chart of the interactive device controlling the movement of the simulated mouse position provided by an exemplary embodiment of the present disclosure. As shown in Figure 8, when the user clicks the application icon on the system desktop, the screen on which the application is started can be determined based on the application. If it is started on the system screen, the screen information of the screen used to display the graphical user interface of the application is obtained as the screen information of the system screen, that is, the screen information includes the screen identification information mDisplayID=0, and the system screen is determined as the target screen. If it is started on the virtual screen, a virtual screen is created, and the unique screen identification of the virtual screen is set to virDisplayID. The screen information obtained includes the screen identification mDisplayID=virDisplayID. The screen indicated by mDisplayID is the target screen. Then enter the initialization process of the simulated mouse, obtain the context information (Context) of the target screen (system screen or virtual screen) according to the screen identification information, create a window of a preset size at a preset initial position of the screen according to the context information of the target screen, and draw the graphics of the simulated mouse in the window. After initializing the simulated mouse, the process for moving the simulated mouse can begin: starting the interactive device's gyroscope data acquisition thread, acquiring gyroscope data through the gyroscope data acquisition thread, calculating the simulated mouse's target position on the screen based on the gyroscope data, and updating the window position to the target position. The simulated mouse can then be used to control the display of the application page. If the screen is a system screen, the system screen can be displayed on the head-mounted display device's display screen. If the screen is a virtual screen, the graphical user interface image data and the simulated mouse image data on the virtual screen can be rendered to a spatial screen corresponding to the virtual screen. This spatial screen is then displayed on the head-mounted display device's display screen, allowing the user to view the application page with the simulated mouse drawn on it. The user can control the position of the simulated mouse on the screen by changing the interactive device's posture information. This allows the simulated mouse to be moved according to the aforementioned process to control the display effects of clicks, slides, drags, and other aspects of the application page. After closing the application, the simulated mouse can also be removed. This is merely an exemplary application scenario for the method of the disclosed embodiments.

[0153] In some optional embodiments of the present disclosure, the posture change information of the interactive device relative to the initial posture information can be calculated based on the real-time posture information of the interactive device and the initial posture information of the interactive device. For example, the initial posture information of the interactive device is obtained by the gyroscope, and the initial gyroscope data is represented as the quaternion mOriginQura. During the shaking process of the interactive device, the real-time gyroscope data is represented as mCurrentQura, and the rotation matrix can be calculated by the quaternion. For example, R0 is the rotation matrix of mOriginQura, R tis the rotation matrix of mCurrentQura. Z represents the distance from the screen (system screen or space screen) viewed by the head-mounted display device to the user's eyes (for example, 2 meters). t R can be calculated t The increment relative to R0. Based on this increment and Z, the real-time position of the simulated mouse on the screen can be calculated. For example, based on the increment and Z, the spatial coordinate information of the simulated mouse on the spatial plane at a distance Z from the head-mounted display device can be calculated, which can be expressed as (x, y). Based on (x, y) and the pixel resolution of the screen, the position of the simulated mouse on the screen, Pos(x, y), can be calculated. Pos(x, y) is the pixel coordinate of the simulated mouse on the screen.

[0154] In some optional embodiments of the present disclosure, Figure 9 is a flow chart of simulating mouse operation on an application page through key control of an interactive device provided by an exemplary embodiment of the present disclosure. As shown in Figure 9, the user's key operation on the interactive device can be detected through the key event callback by registering a key event callback. onKeyEvent(keyCode, action) can represent a key event callback function. keyCode can represent a key identifier, and Action represents a key action. keyCode = confirmation key, indicating that the key identifier is the first preset key identifier. keyCode = up, down, left, and right keys, indicating that the key identifier is the second preset key identifier. Action == Down indicates whether the action of the key is the first key event (i.e., pressing), isKeyDown = true indicates that it is the first key event, and isKeyDown = false can indicate that it is the second key event (i.e., lifting). In the case where the key identifier is the confirmation key, if the key action is pressing, the real-time position of the simulated mouse can be obtained according to the preset frequency, and a touch event can be injected into the real-time position. If the key action is lifting, the touch event is ended. In the case where the button is identified as the up, down, left, or right button, if the button action is pressed, the starting position of the simulated mouse when it is pressed can be recorded. Based on the starting position, according to the direction attribute and sliding step size of the button, a touch event is injected into the screen at a preset frequency. If the button action is lifted, the touch event ends. Through the button operation of the interactive device, the effects of dragging and continuously sliding the application page can be achieved, improving the convenience and efficiency of user operation and enhancing the user experience. This is only an exemplary application scenario of the disclosed solution.

[0155] The method of the embodiment of the present disclosure can be applied to an interactive device with buttons, converting button events into touch events, and realizing the effects of continuous page turning, long press and dragging of the graphical user interface by simulating a mouse, thereby realizing the effect of touch sliding. Moreover, dragging the page by long pressing the button with the first preset button identifier is actually also an act of touch sliding the page, which can make up for the problem that the function of continuously turning the page by clicking the button with the second preset button identifier cannot slide a shorter distance. For example, when the up, down, left, and right keys are pressed and then lifted, the step length of one slide is usually not less than a threshold value (for example, 200 pixels). When the user needs to slide less than the threshold value, a short-distance sliding operation can be achieved by long pressing the button with the first preset button identifier and dragging the page.

[0156] The above-mentioned embodiments or optional examples of the present disclosure may be implemented separately or in any combination without conflict. The specific configuration may be based on actual needs and is not limited by the present disclosure.

[0157] Any method for controlling a graphical user interface provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to a terminal device and a server. Alternatively, any method for controlling a graphical user interface provided in the embodiments of the present disclosure can be executed by a processor, such as by invoking corresponding instructions stored in a memory to execute any method for controlling a graphical user interface mentioned in the embodiments of the present disclosure. This will not be further described below.

[0158] Exemplary devices

[0159] Figure 10 is a schematic diagram of the structure of an apparatus for controlling a graphical user interface, provided by an exemplary embodiment of the present disclosure. The apparatus for controlling a graphical user interface of this embodiment can be used to implement the corresponding method embodiment for controlling a graphical user interface of the present disclosure. As shown in Figure 10, the apparatus includes an acquisition module 410, a first processing module 420, a second processing module 430, a third processing module 440, and a fourth processing module 450.

[0160] The acquisition module 410 is configured to acquire screen information of a screen for displaying a graphical user interface in response to determining that the mouse interaction mode is invoked, wherein the screen information includes screen identification information.

[0161] The first processing module 420 is configured to determine a target screen based on the screen identification information.

[0162] The second processing module 430 is configured to determine a target position of the simulated mouse on the target screen according to the position information of the interactive device, wherein the interactive device is configured to control a graphical user interface displayed on the target screen through the simulated mouse.

[0163] The third processing module 440 is configured to create a simulated mouse at a target position on the target screen.

[0164] The fourth processing module 450 is configured to receive user operations on the simulated mouse through the interactive device, and generate mouse instructions to control the graphical user interface displayed on the target screen.

[0165] In some optional embodiments of the present disclosure, the second processing module 430 is specifically configured to: in response to determining that the posture information of the interactive device is posture information at an initial moment, determine the initial position of the simulated mouse on the target screen as the target position. In response to determining that the posture information of the interactive device is posture information at a non-initial moment, determine the target position of the simulated mouse on the target screen based on the posture information of the interactive device.

[0166] In some optional embodiments of the present disclosure, the third processing module 440 is specifically configured to: create a window of a preset size at a target position on the target screen, and draw a graphic simulating a mouse in the window to create a simulated mouse.

[0167] In some optional embodiments of the present disclosure, the first processing module 420 is specifically configured to: in response to determining that the screen identification information includes a preset identifier, determine the system screen as the target screen. In response to determining that the screen identification information includes a non-preset identifier, determine the spatial screen corresponding to the virtual screen indicated by the non-preset identifier as the target screen.

[0168] In some optional embodiments of the present disclosure, the first processing module 420 is specifically used to obtain a target screen corresponding to the virtual screen in the following manner: create a virtual screen and set a unique screen identifier for the virtual screen; determine the spatial screen corresponding to the virtual screen based on the posture information of the head-mounted display device, and the screen information of the spatial screen includes the screen identifier information of the virtual screen and the posture information of the spatial screen.

[0169] In some optional embodiments of the present disclosure, the third processing module 440 is specifically configured to: determine a target position on the virtual screen corresponding to the target screen based on the screen identification information of the virtual screen; create a window layer for simulating a mouse at the target position; wherein the window layer for simulating a mouse is at a higher level than the layer level of the graphical user interface; and draw the simulated mouse in the window layer for simulating a mouse.

[0170] FIG11 is a schematic structural diagram of an apparatus for controlling a graphical user interface provided by another exemplary embodiment of the present disclosure.

[0171] In some optional embodiments of the present disclosure, the apparatus of the embodiment of the present disclosure further includes:

[0172] The first acquisition module 510 is configured to acquire image data of the simulated mouse drawn in the window layer of the simulated mouse.

[0173] The rendering module 520 is used to render the image data of the simulated mouse and the image data of the graphical user interface to the spatial screen, and display the spatial screen through the head-mounted display device.

[0174] In some optional embodiments of the present disclosure, the fourth processing module 450 is specifically configured to: in response to receiving a user operation on a simulated mouse via the interactive device, convert the user operation into a touch event, inject the touch event into a target screen, and generate a mouse command for controlling a graphical user interface displayed on the target screen.

[0175] In some optional embodiments of the present disclosure, the fourth processing module 450 is specifically configured to: receive a user operation on a preset key of the interactive device, obtain a key identifier corresponding to the user operation, and, in response to the key identifier being the preset key identifier, convert a key event generated by the user operation into a touch event.

[0176] In some optional embodiments of the present disclosure, the fourth processing module 450 is further configured to: in response to receiving the first key event, inject a touch event into the target screen at a preset frequency; and in response to receiving the second key event, end the touch event.

[0177] In some optional embodiments of the present disclosure, the fourth processing module 450 is specifically configured to: in response to the key event being a preset operation event for a key with a first preset key identifier, inject a touch event into the target screen at a preset frequency, and generate an interface drag instruction for dragging the graphical user interface.

[0178] In some optional embodiments of the present disclosure, the fourth processing module 450 is specifically configured to: in response to the key event being a preset operation event for a key with a second preset key identifier, inject a touch event into the target screen at a preset frequency, and generate an interface sliding event for sliding the graphical user interface.

[0179] In some optional embodiments of the present disclosure, after generating an interface sliding event for sliding a graphical user interface, the fourth processing module 450 is further configured to: determine an event position corresponding to each touch event on the target screen based on the starting position of the simulated mouse, the direction attribute of the key identified by the second preset key, and the sliding step size. Based on the event position corresponding to each touch event, generate an interface sliding instruction for controlling the graphical user interface displayed on the target screen to slide in the direction indicated by the direction attribute of the key identified by the second preset key.

[0180] In some optional embodiments of the present disclosure, the fourth processing module 450 is further configured to:

[0181] In response to the event position of the last touch event reaching the boundary of the target screen, the starting position of the simulated mouse is determined as the event position of the last touch event; based on the event position of the last touch event, the direction attribute of the key identified by the second preset key, and the sliding step size, the current event position of the current touch event on the target screen is determined.

[0182] In some optional embodiments of the present disclosure, the fourth processing module 450 is further configured to: in response to the interactive device receiving a preset operation event of a user pressing a key with a third preset key identifier, generate a simulated mouse reset instruction to reset the position of the simulated mouse on the target screen to a preset initial position. In response to the interactive device receiving a preset operation event of a user pressing a key with a fourth preset key identifier, generate an interface return simulated mouse instruction to return the graphical user interface displayed on the target screen to the previous interface.

[0183] It should be noted that the specific implementation of the device for controlling the graphical user interface in the embodiment of the present disclosure is similar to the specific implementation of the method for controlling the graphical user interface in the embodiment of the present disclosure. Please refer to the part on the method for controlling the graphical user interface for details. In order to reduce redundancy, it will not be described here.

[0184] Exemplary electronic devices

[0185] An embodiment of the present disclosure further provides an electronic device, including: a processor, and a memory for storing instructions executable by the processor.

[0186] The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method for controlling the graphical user interface described in any of the above embodiments of the present disclosure.

[0187] FIG12 is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. In this embodiment, the electronic device 100 includes one or more processors 110 and a memory 120.

[0188] The processor 110 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0189] The memory 120 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 110 may execute the program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0190] In one example, the electronic device 100 may further include an input device 130 and an output device 140 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0191] For example, the input device 130 may be the aforementioned microphone or microphone array, configured to capture input signals from a sound source.

[0192] In addition, the input device 130 may also include, for example, a keyboard, a mouse, and the like.

[0193] The output device 140 can output various information to the outside, including determined distance information, direction information, etc. The output device 140 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0194] Of course, for simplicity, FIG12 only shows some of the components related to the present disclosure in the electronic device 100, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 100 may further include any other appropriate components according to specific application scenarios.

[0195] Exemplary computer program products and computer-readable storage media

[0196] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0197] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0198] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0199] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0200] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0201] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for controlling a graphical user interface, comprising: In response to determining to invoke a mouse interaction mode, obtaining screen information of a screen for displaying the graphical user interface, where the screen information includes screen identification information; Based on the screen identification information, determining a target screen; According to the pose information of an interaction device, determining a target position of a simulated mouse on the target screen, where the interaction device is used to control the graphical user interface displayed on the target screen through the simulated mouse; Creating a simulated mouse at the target position on the target screen; Receiving, through the interaction device, an operation of a user on the simulated mouse and generating a mouse instruction; Controlling the graphical user interface displayed on the target screen.

2. The method according to claim 1, wherein, The creating a simulated mouse at the target position on the target screen includes: Creating a window with a preset size at the target position on the target screen; Drawing a graphic of the simulated mouse in the window; Creating the simulated mouse.

3. The method according to claim 1 or 2, wherein The determining a target position of a simulated mouse on the target screen according to the pose information of an interaction device includes: In response to determining that the pose information of the interaction device is the pose information at the initial moment, determining the initial position of the simulated mouse on the target screen as the target position; In response to determining that the pose information of the interaction device is not the pose information at the initial moment, determining the target position of the simulated mouse on the target screen based on the pose change information of the interaction device.

4. According to the method described in any one of claims 1-3, wherein The determining a target screen based on the screen identification information includes: In response to determining that the screen identification information includes a preset identifier, determining the system screen as the target screen; In response to determining that the screen identification information includes a non-preset identifier, determining a spatial screen corresponding to a virtual screen indicated by the non-preset identifier as the target screen; The target screen corresponding to the virtual screen is obtained by the following method: creating a virtual screen and setting a unique screen identifier for the virtual screen; according to the pose information of a head-mounted display device, determining a spatial screen corresponding to the virtual screen, where the screen information of the spatial screen includes the screen identification information of the virtual screen and the pose information of the spatial screen.

5. The method according to claim 4, wherein The creating a simulated mouse at the target position on the target screen includes: Based on the screen identification information of the virtual screen, determining the target position on the virtual screen corresponding to the target screen; Creating a window layer of the simulated mouse at the target position, where the layer level of the window layer of the simulated mouse is higher than the layer level of the graphical user interface; Drawing the simulated mouse in the window layer of the simulated mouse.

6. The method according to claim 5, wherein The method further includes: Obtaining image data of the simulated mouse drawn in the window layer of the simulated mouse; Rendering the image data of the simulated mouse and the image data of the graphical user interface to the spatial screen; Displaying the spatial screen through the head-mounted display device.

7. According to the method described in any one of claims 1-6, wherein, The receiving, through the interaction device, an operation of a user on the simulated mouse and generating a mouse instruction to control the graphical user interface displayed on the target screen includes: In response to receiving the user's operation on the analog mouse through the interaction device, convert the user's operation into a touch event; Inject the touch event into the target screen to generate a mouse instruction for controlling the graphical user interface displayed on the target screen.

8. The method according to claim 7, wherein, The converting the user's operation into a touch event includes: Receiving the user's operation on a preset button of the interaction device; Obtaining a button identifier corresponding to the user's operation; In response to the button identifier being a preset button identifier, converting the button event generated by the user's operation into a touch event.

9. The method according to claim 7, wherein The method further includes: In response to receiving a first button event, injecting the touch event into the target screen at a preset frequency; In response to receiving a second button event, ending the touch event.

10. The method according to any one of claims 7-9, wherein, The injecting the touch event into the target screen to generate a mouse instruction for controlling the graphical user interface displayed on the target screen includes: In response to the button event being a preset operation event for a button with a first preset button identifier, injecting the touch event into the target screen at a preset frequency; Generating an interface drag instruction for dragging the graphical user interface.

11. According to the method of any one of claims 7-9, wherein, The injecting the touch event into the target screen to generate a mouse instruction for controlling the graphical user interface displayed on the target screen includes: In response to the button event being a preset operation event for a button with a second preset button identifier, injecting the touch event into the target screen at a preset frequency; Generating an interface sliding event for sliding the graphical user interface.

12. The method according to claim 11, wherein, After generating the interface sliding event for sliding the graphical user interface, the method further includes: Determining the event positions corresponding to each touch event on the target screen according to the starting position of the analog mouse, the direction attribute of the button with the second preset button identifier, and the sliding step length; Generating an interface sliding instruction for controlling the graphical user interface displayed on the target screen to slide in the direction indicated by the direction attribute of the button with the second preset button identifier according to the event positions corresponding to each touch event.

13. The method according to claim 12, wherein, The method further includes: In response to the event position of the previous touch event reaching the boundary of the target screen, determining the starting position of the analog mouse as the event position of the previous touch event; Determining the current event position of the current touch event on the target screen according to the event position of the previous touch event, the direction attribute of the button with the second preset button identifier, and the sliding step length.

14. A computer-readable storage medium storing a computer program for executing the method for controlling a graphical user interface according to any one of claims 1-13 above.

15. An electronic device, the electronic device includes: A processor; A memory for storing executable instructions of the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for controlling a graphical user interface according to any one of claims 1-13 above.

Images