Method and apparatus for displaying application interface of head-mounted display device

Abstract

The method and device for displaying an application interface of a head-mounted display device are disclosed, and the method comprises the following steps: in response to receiving a starting operation instruction of a target application, a first position is acquired; a second position spaced apart from the first position by a preset interval is determined based on the first position; an estimated position of a pointer at a target time is acquired; a distance between the second position and the estimated position is compared with a preset interval threshold value, and a display position of an interface image of the target application at the target time is determined based on a comparison result; and the interface image of the target application is displayed at the display position at the target time. The interface image of the started application can be displayed at a reasonable position, so that the interface image is located in a user's field of view.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method and apparatus for displaying an application interface on a head-mounted display device. Background Technology

[0002] In scenarios such as Virtual Reality (VR), Augmented Reality (AR), or Mixed Reality (MR), the terminal provides users with an interactive and immersive experience by constructing a virtual environment.

[0003] After a user activates the head-mounted display device, they can launch applications by clicking on the application icon in the virtual interface. How to display the launched application's interface in a suitable location within the virtual interface, ensuring it is within the user's field of vision, is a problem that urgently needs to be solved. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a method and apparatus for displaying an application interface on a head-mounted display device.

[0005] According to a first aspect of the present disclosure, a method for displaying an application interface on a head-mounted display device is provided, comprising:

[0006] In response to receiving a launch operation command from the target application, a first position is obtained, wherein the first position is one of the current position of the pointer and the position of the target application's icon;

[0007] Based on the first position, determine a second position that is spaced at a preset distance from the first position;

[0008] Obtain the estimated position of the pointer at the target time;

[0009] The distance between the second position and the estimated position is compared with a preset distance threshold. Based on the comparison result, the display position of the target application's interface image at the target time is determined. The display position is one of the second position and the estimated position.

[0010] At the target time, display the interface image of the target application at the display location.

[0011] According to a second aspect of the present disclosure, an apparatus for displaying an application interface on a head-mounted display device is provided, comprising:

[0012] The first position acquisition module is used to acquire the first position in response to receiving the launch operation instruction of the target application, wherein the first position is one of the current position of the pointer and the position of the icon of the target application;

[0013] The second position acquisition module is used to determine a second position that is spaced at a preset distance from the first position, based on the first position;

[0014] The position estimation module is used to obtain the estimated position of the pointer at the target time;

[0015] The display position determination module is used to compare the distance between the second position and the estimated position with a preset distance threshold, and determine the display position of the interface image of the target application at the target time based on the comparison result. The display position is one of the second position and the estimated position.

[0016] The display module is used to display the interface image of the target application at the target time and display location.

[0017] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the method of the first aspect for displaying an application interface on a head-mounted display device.

[0018] According to a fourth aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0019] processor;

[0020] Memory used to store processor-executable instructions;

[0021] A processor for reading executable instructions from memory and executing the instructions to implement the method of displaying an application interface for a head-mounted display device as described in the first aspect above.

[0022] According to a fifth aspect of the present disclosure, a head-mounted display device is provided, including the means described in the second aspect for displaying an application interface on the head-mounted display device.

[0023] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0024] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0025] Figure 1 This is an exemplary system architecture diagram that can be applied to embodiments of the method or apparatus for displaying an application interface on a head-mounted display device disclosed herein;

[0026] Figure 2 This is a flowchart illustrating a method for displaying an application interface on a head-mounted display device according to one embodiment of the present disclosure;

[0027] Figure 3 This is a flowchart illustrating step S4 in one embodiment of the present disclosure;

[0028] Figure 4 This is a schematic diagram of the second position in one example of this disclosure;

[0029] Figure 5 This is a schematic diagram of the second position in another example of this disclosure;

[0030] Figure 6 This is a schematic diagram of the second position in yet another example of this disclosure;

[0031] Figure 7 This is a flowchart illustrating step S6 in one embodiment of the present disclosure;

[0032] Figure 8 This is a flowchart illustrating step S10 in one embodiment of this disclosure;

[0033] Figure 9 This is a structural block diagram of an apparatus for displaying an application interface on a head-mounted display device according to one embodiment of the present disclosure;

[0034] Figure 10 This is a structural block diagram of an electronic device in one embodiment of the present disclosure. Detailed Implementation

[0035] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. It is obvious that the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0036] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0037] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0038] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0039] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0040] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0041] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0042] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0043] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0044] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0045] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0046] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0047] Figure 1 This is an exemplary system architecture diagram that can be applied to embodiments of the methods or apparatus disclosed herein for displaying application interfaces on head-mounted display devices.

[0048] like Figure 1 As shown, the system architecture may include a head-mounted display device 1, a network 2, and a server 3. Network 2 serves as the medium for communication links between the head-mounted display device 1 and the server 3. Network 2 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0049] The aforementioned head-mounted display device 1 can be an electronic device with image display capabilities, including but not limited to AR smart glasses, VR smart glasses, etc. The head-mounted display device 1 can be an all-in-one device, meaning it can install various client applications. Alternatively, the head-mounted display device 1 can also be a separate device used in conjunction with a terminal device to achieve image display functionality; in this case, the terminal device can perform the computing functions of the head-mounted display device. Here, for ease of representation, both all-in-one devices and separate devices can be collectively referred to as head-mounted display devices.

[0050] The aforementioned head-mounted display device 1 can provide various services, such as acquiring the source data of the target application when it receives a startup command from the target application, and processing the source data of the target application to generate the interface image of the target application.

[0051] It should be noted that the method for displaying an application interface on a head-mounted display device provided in this disclosure is generally executed by the head-mounted electronic device 1, and correspondingly, the apparatus for displaying an application interface on a head-mounted display device is generally disposed in the head-mounted electronic device 1.

[0052] Optionally, the method for displaying an application interface on a head-mounted display device provided in this embodiment of the present disclosure can also be executed by a terminal device connected to the head-mounted electronic device 1, and correspondingly, the apparatus for displaying an application interface on a head-mounted display device can also be disposed in the terminal device.

[0053] It should also be noted that while the solution disclosed herein can be applied to head-mounted display devices, it does not preclude its application to server 3, which can be a backend server. When the solution disclosed herein is applied to server 3, upon receiving a startup command from the target application, the server can obtain a first position in response to the received command. The server can determine the window to be displayed and, based on the first position, determine a second position at a preset distance from the first position. The server can also obtain the estimated position of the pointer at the target time, compare the distance between the second position and the estimated position with a preset distance threshold, and determine the display position of the target application's interface image at the target time based on the comparison result. Then, at the target time, the interface image of the target application is displayed at the display position. In this case, the information display method for the head-mounted display device can be executed by server 3, and correspondingly, the device for displaying the application interface of the head-mounted display device can also be provided in server 3.

[0054] Exemplary methods

[0055] Figure 2 This is a flowchart illustrating a method for displaying an application interface on a head-mounted display device according to one embodiment of this disclosure. Embodiments of this disclosure can be applied to electronic devices, such as… Figure 2 As shown, it includes the following steps:

[0056] S2: In response to receiving a launch command from the target application, obtain the first position. The first position is either the current position of the pointer or the position of the target application's icon.

[0057] In one alternative approach, after the user wears and turns on the head-mounted display, they generate launch instructions for the target application by pointing at its icon.

[0058] The term "pointer" as used in this article refers to a tool that allows users to instruct or manipulate the displayed content, without limiting its specific form.

[0059] In another alternative approach, after the user wears and turns on the head-mounted display device, they select the launch command for the target application from the operating instructions provided by the operating system of the head-mounted display device.

[0060] After receiving the launch command from the target application, the first position is obtained. The first position can represent the user's current focus.

[0061] S4: Based on the first position, determine a second position that is spaced at a preset distance from the first position.

[0062] Within a spatial location separated from the first position by a preset distance—that is, on the surface of a sphere centered at the first position—a second position can be randomly selected or a specific position can be chosen. The second position can be the expected initial display position of the target application's interface image. By setting a preset distance, the second position can be closer to the first position. For example, if the user is focusing on the same position, both the first and second positions can be near that position, allowing the user to observe both positions without switching their focus.

[0063] S6: Obtain the estimated position of the pointer at the target time.

[0064] As an alternative approach, the pointer's position at the target time can be estimated based on the pointer's position sequence before the target time, thus obtaining the pointer's estimated position at the target time.

[0065] S8: Compare the distance between the second position and the estimated position with a preset distance threshold, and determine the display position of the target application's interface image at the target time based on the comparison result. The display position is one of the second position and the estimated position.

[0066] Since acquiring the source data of the target application and generating its interface image takes time, the pointer position at the target time may differ from the first position. The pointer position typically reflects the user's focus point; therefore, the display position of the target application's interface image can be adjusted based on the pointer position at the target time, ensuring it appears near the user's focus point for easy operation. A preset spacing threshold can be used to assess the magnitude of the pointer position change. The display position of the target application's interface image at the target time is selected based on a comparison between the distance between the second position and the estimated position and the preset spacing threshold.

[0067] S10: At the target time, display the interface image of the target application at the display location.

[0068] In this article, "target time" can refer to a moment after a preset time interval following the generation of the target application's startup command, within which sufficient time is available to generate the target application's interface image. In this article, "target time" refers to the display location where the target application's interface image is displayed, allowing the user to interact with it.

[0069] In this embodiment, upon receiving a launch command from the target application, a first position representing the user's current visual focus is obtained, and a second position, spaced at a preset distance from the first position, is selected as the expected initial display position of the target application's interface image. Since acquiring the target application's source data and generating its interface image takes time, the pointer's position is estimated to determine its predicted position at the target time. Based on a comparison between the distance between the second position and the predicted position and a preset distance threshold, either the second position or the predicted position is selected as the display position of the target application's interface image at the target time. This ensures the display position at the target time is near the user's current visual focus, facilitating subsequent user operations.

[0070] Figure 3 This is a flowchart illustrating step S4 in one embodiment of this disclosure. Figure 3 As shown, step S4 includes:

[0071] S4-2: Obtain the controller's position. The pointer is controlled by the controller.

[0072] In one alternative approach, the user can control the pointer using a handheld mobile control terminal, which acts as the controller. The mobile control terminal can be a companion control terminal connected to the head-mounted display device, or a portable external device connected to the head-mounted display device, such as a mobile phone. In another alternative approach, the user can control the pointer with their hand, which can be understood as the controller. Methods for obtaining the position of the controller include, but are not limited to, calculating the controller's position using data collected by the controller's own devices, calculating the controller's position by acquiring images of the controller using an image acquisition device, and a combination of both.

[0073] In one alternative approach, the position of the controller can be preset in a virtual space coordinate system. In this example, it can be assumed that the controller has a fixed initial position.

[0074] Because the controller is equipped with a positioning device, its position can be obtained through this device. The controller's position can then represent the user's position.

[0075] S4-4: In the direction of the line connecting the controller position and the first position, the position that is separated from the first position by a preset distance is determined as the second position.

[0076] In one alternative approach, a virtual spatial coordinate system is pre-established. The direction of the connecting line (e.g., along the straight line connecting the first position and the controller) can be determined based on the three-dimensional coordinates of the first position and the controller's coordinates in the virtual spatial coordinate system. Then, a position at a preset distance from the first position is selected along this connecting line as the second position. Optionally, the second position can be selected along the line connecting the controller's position and the first position, at a preset distance from the first position. Alternatively, the second position can be selected along the extension of the line connecting the controller's position and the first position, at a preset distance from the first position.

[0077] In this embodiment, when determining the second position as the expected initial display position of the interface image of the target application, the position of the controller used to represent the user's position is first obtained, and then the line connecting the position of the controller and the first position is obtained. This line can represent the user's line of sight from the user's position to the direction of the user's current focus. Selecting the second position from the direction of this line (e.g., on the straight line connecting the two) can ensure that the second position is within the user's field of vision and near the user's line of sight focus.

[0078] In one embodiment of this disclosure, step S4-4 may specifically include: determining a position at a preset distance from the first position as a second position on the extension line of the line connecting the position of the controller and the first position.

[0079] Figure 4 This is a schematic diagram illustrating the determination of the second position in one example of this disclosure. For example... Figure 4 As shown, C represents the controller's position, P1 represents the first position, and P2 represents the second position. P2 is located on the extension of the line connecting C and P1, and the distance between P2 and P1 is a preset distance. It can be seen that P2 is located on the extension of the line connecting C and P1, at a position farther from C, meaning P2 is farther from the user relative to P1.

[0080] In this embodiment, since the interface image of the target application is larger than the icon size of the target application, a second position is selected on the extension line connecting the controller position and the first position. When the interface image of the target application is displayed at the second position, the interface image can be placed near the user's line of sight, and the user's visual adaptation can be improved by displaying the interface image in an appropriate position.

[0081] In another embodiment of this disclosure, step S4 may include: determining a preset position that is spaced at a preset distance from the first position and has a distance from the controller that is less than the distance between the first position and the controller, as the second position. The pointer is controlled by the controller.

[0082] Figure 5This is a schematic diagram of the second position in another example of this disclosure. For example... Figure 5 As shown, C represents the controller position, P1 represents the first position, and P2 represents the second position. The distance between P2 and P1 is a preset distance, and the distance between P2 and C is less than the distance between P1 and C.

[0083] Optionally, the P2 position can be located on either the left or right side of the line connecting P1 and C, determined based on whether the user is operating the controller with their left or right hand. A step to confirm left-hand or right-hand mode can be added before step S4. After confirming left-hand or right-hand mode, the P2 position can be set to the right side of the line connecting P1 and C based on left-hand mode, for example... Figure 5 As shown. Similarly, based on right-hand mode, the position of P2 can be set to the left of the line connecting P1 and C, so that P2 is closer to the user's direct line of sight. The "left-hand mode" described here can be understood as operating the controller with the left hand or using the left hand as the controller, and the same applies to the right hand. The "left" and "right" mentioned above are based on the user's perspective.

[0084] In this embodiment, near the first position which is spaced at a preset distance from the first position, a position that is closer to the user than the first position is selected as the second position. This allows the user to see the interface image of the target application immediately, so that the user can quickly operate on the interface image of the target application, such as zooming in, zooming out, or moving away from the interface image of the target application, thereby improving the user experience.

[0085] In another embodiment of this disclosure, step S4 includes: determining a preset position at a preset distance from the first position as the second position in the interface where the icon of the target application is located.

[0086] Figure 6 This is a schematic diagram of the second position in yet another example of this disclosure. For example... Figure 6 As shown, C represents the position of the controller, P1 represents the first position, P2 represents the second position, and the box represents the interface where the target application's icon is located. The spacing between P2 and P1 is a preset spacing, and P2 is located within the interface where the target application's icon is located.

[0087] In this embodiment, within the interface where the target application's icon is located, a position with a preset distance from the first position is selected as the second position. This ensures that the user's visual distance to the second position is similar to their visual distance to the first position. This makes it easier for the user to adapt to the target application's interface image when it is displayed at the second position, thereby improving the user experience. Furthermore, displaying the target application's interface image directly within the interface where the target application's icon is located eliminates the need to calculate position depth, simplifying the operation.

[0088] In one embodiment of this disclosure, step S8 includes:

[0089] If the distance between the second position and the estimated position is less than a preset distance threshold, the second position is determined as the display position of the target application's interface image at the target time.

[0090] If the distance between the second position and the estimated position is greater than or equal to a preset distance threshold, the estimated position is determined as the display position of the target application's interface image at the target time.

[0091] As mentioned above, since the process of acquiring the source data of the target application and generating the interface image of the target application takes some time, if the pointer position changes little during this process, the second position is usually near the user's focus at the target time, and can be used as the display position of the target application's interface image at the target time. If the pointer position changes significantly during this process, it can be assumed that the user's focus has changed significantly, and the estimated position can be used as the display position of the target application's interface image at the target time. The preset spacing threshold can be a spacing threshold set to detect whether the distance between the second position and the estimated position is too small.

[0092] In this embodiment, when the distance between the second position and the estimated position is less than a preset distance threshold, it can be considered that the distance between the second position and the estimated position is small. In this case, selecting the second position as the display position ensures that the display position at the target time is within the user's field of vision and close to the user's current point of focus, while also reducing system resource consumption because the expected display position of the target application's interface image is not changed. When the distance between the second position and the estimated position is greater than or equal to the preset distance threshold, it can be considered that the distance between the second position and the estimated position is large. In this case, selecting the estimated position as the display position ensures that the generated interface image of the target application is within the user's field of vision and close to the user's current point of focus, facilitating user operation.

[0093] Figure 7 This is a flowchart illustrating step S6 in one embodiment of this disclosure. Figure 7As shown, step S6 includes:

[0094] S6-2: Acquire at least one of an attitude data sequence and a position data sequence that indicates the controller's positioning before the target time. The pointer is controlled by the controller.

[0095] In one alternative approach, the user can control the pointer using a handheld mobile control terminal, which acts as the controller. The mobile control terminal can be a companion control terminal connected to the head-mounted display device, or an external device connected to the head-mounted display device, such as a mobile phone. The attitude data sequence of the controller in the real world can be obtained through an Inertial Measurement Unit (IMU) installed within the controller. By combining this with the attitude mapping relationship between the controller in the real world and the controller in the virtual space coordinate system, the attitude data sequence indicating the controller's positioning in the virtual space coordinate system can be obtained. Optionally, the position data sequence of the mobile control terminal can be calculated by acquiring images from the mobile control terminal. Further, the pose data sequence of the mobile terminal can be calculated by combining the aforementioned IMU data. This results in the pose data sequence indicating the controller's positioning in the virtual space coordinate system.

[0096] In an alternative approach, the user's hand can control the pointer, which can be understood as the controller. By acquiring images of the user's hand and performing gesture recognition and hand pose calculation, a sequence of position data, an attitude data sequence, and a pose data sequence indicating the controller's positioning can be obtained.

[0097] Understandably, the device used to acquire images (such as a mobile control terminal or a user's hand) can be the head-mounted display itself or other external devices with image acquisition capabilities.

[0098] When using the virtual controller's position data sequence alone, a three-degree-of-freedom (DOF) position sequence (horizontal, vertical, and forward / backward) can be obtained. When using the controller's attitude data sequence alone, a three-degree-of-freedom (DOF) attitude sequence (left / right, up / down, and forward / backward) can be obtained. When using both the controller's attitude data sequence and position data sequence simultaneously, a six-degree-of-freedom (DOF) pose sequence can be obtained.

[0099] In one optional approach, the user can see a ray originating from the controller in the virtual coordinate system, with the endpoint of this ray (the intersection with the display interface) indicating the pointer's position. Optionally, an anchor point can be preset in the virtual coordinate system, and the controller in the virtual coordinate system responds by moving around this anchor point based on the acquired data. Optionally, when using attitude data sequences, the controller in the virtual coordinate system rotates around the anchor point based on the acquired attitude data sequence, thus moving the pointer's position. Optionally, when using position data sequences, the controller in the virtual coordinate system moves around the anchor point based on the acquired attitude data sequence, thus moving the pointer's position. Optionally, using both pose data sequences and position data sequences allows for more precise pointer control.

[0100] S6-4: Based on at least one of the attitude data sequence and the position data sequence, perform position prediction on the pointer to obtain the predicted position of the pointer at the target time.

[0101] In one alternative approach, when both attitude and position data sequences are used simultaneously, the predicted position of the pointer at the target time can be accurately estimated using a six-DOF pose sequence. When only the attitude or position data sequence is used, the predicted position of the pointer at the target time can be roughly estimated.

[0102] In this embodiment, by using at least one of the attitude data sequence and position data sequence of the controller before the target time, the estimated position of the pointer at the target time can be estimated, so that subsequent steps can be processed based on the estimated position, ultimately ensuring that the interface image of the target application displayed at the display position at the target time is within the user's field of vision, resulting in a good user experience.

[0103] Figure 8 This is a flowchart illustrating step S10 in one embodiment of this disclosure. Figure 8 As shown, step S10 includes:

[0104] S10-2: Before the target time, display the interface image of the target application at the second position with the first image size.

[0105] After receiving the launch command from the target application, the source data of the target application can be obtained from the server, and the interface image of the target application can be generated based on the source data of the target application.

[0106] After determining the second position and before the target time, the target application's interface image at the second position can be displayed at the second position with a first image size. The first image size is the default size of the target application's interface image.

[0107] S10-4: At the target time, display the interface image of the target application at the display location with a second image size. The second image size is larger than the first image size.

[0108] Before the target time, the interface image of the target application can be enlarged from a first image size to a second image size. This ensures that the interface image of the target application can be displayed at the display position determined in step S8 at the target time, using the second image size.

[0109] After displaying the target application's interface image at the display position determined in step S8 with the second image size, the ray between the controller and the pointer in the virtual space coordinate system can be changed to another display form, such as a fishing line-like shape. In the fishing line shape, the user can receive predetermined operation commands for the target application's interface image, such as zoom in, zoom out, or pull away, thereby facilitating the user to adjust the size and position of the target application's interface image.

[0110] In this embodiment, after receiving the launch command of the target application, the system can acquire the source data of the target application and generate its interface image. This interface image is then displayed at a second position with a smaller first image size. Simultaneously, the display position of the pointer at the target time can be determined. Then, at the target time, the interface image of the target application can be displayed at the display position with a larger second image size. The system can then prepare to receive the user's predetermined operation commands for the target application, resulting in a better user experience.

[0111] In one alternative approach, after the target application's interface image is displayed at the second position at the first image size, but before the target time, the image size of the interface image can be changed from the first image size to the second image size. The size change can be gradual, step-like, stepless, etc.

[0112] If the estimated position is determined to be the display position of the target application's interface image at the target time, the target application's interface image can be moved from the second position to the display position after it has been displayed at the second position with the first image size, but before the target time. During this movement, the user can observe the movement of the interface image. Then, at the target time, the target application's interface image is displayed at the estimated position with the second image size.

[0113] The following embodiments can be used to calculate the position, orientation, or pose of objects in real space (such as mobile terminals or head-mounted displays acting as controllers) in a virtual space coordinate system.

[0114] In one embodiment of this disclosure, a preset reference point can be set, and the connection between the same object in the virtual space coordinate system and the real space coordinate system can be established through the connection of the preset reference point in the virtual space coordinate system and the real space coordinate system. Initial coordinates are established for the preset reference point in the virtual space coordinate system, and initial coordinates are also established for the preset reference point in the real space coordinate system. The real space coordinate system is established with a certain position as the origin. Therefore, the coordinates of the preset reference point in the virtual space coordinate system and the coordinates in the real space coordinate system in the initial state are both known.

[0115] Based on at least one of the sensors and image acquisition and processing devices equipped with the head-mounted display device, the current displacement of the head-mounted display device relative to a preset reference point in the real-space coordinate system is determined. The sensors equipped with the head-mounted display device may include a position sensor or a gyroscope. This displacement can be obtained through the position sensor, or it can be determined by capturing and analyzing images using the image acquisition and processing device, or by combining data from the gyroscope with images captured and analyzed by the image acquisition and processing device.

[0116] Based on this actual displacement and the pre-established mapping relationship between virtual space coordinates and real space coordinates, the virtual displacement of the current position of the head-mounted display device relative to the preset reference point in the virtual space coordinate system is determined.

[0117] Based on this virtual displacement and the coordinates of the preset reference point in the virtual space coordinate system, the current coordinates of the head-mounted display device in the virtual space coordinate system can be calculated.

[0118] The location (position, posture, or pose) of other objects in real space in virtual space coordinates can be established by referring to their positional relationship with the head-mounted display device, or obtained through other methods.

[0119] In one embodiment, the content displayed in the virtual space coordinate system can be created in the development engine. The positioning (position, attitude, or pose) of the content displayed in the virtual space coordinate system can be obtained through the development engine. The development engine can be any engine capable of creating virtual content, such as a 3D space engine.

[0120] In one embodiment of this disclosure, a method for displaying an application interface on a head-mounted display device may include:

[0121] A: In response to receiving a launch command from the target application, obtain the first position. The first position is either the current position of the pointer or the position of the target application's icon.

[0122] In one alternative approach, after the user wears and turns on the head-mounted display, they generate launch instructions for the target application by pointing at its icon.

[0123] In another alternative approach, after the user wears and turns on the head-mounted display device, they select the launch command for the target application from the operating instructions provided by the operating system of the head-mounted display device.

[0124] After receiving the launch command from the target application, the first position is obtained. The first position can represent the user's current focus.

[0125] C: On the extension of the line connecting the controller position and the first position, the position that is spaced at a preset distance from the first position is determined as the second position.

[0126] In one alternative approach, a virtual spatial coordinate system is pre-established. The direction of the connecting line (e.g., on the straight line connecting the two) can be determined based on the three-dimensional coordinates of the first position in the virtual spatial coordinate system and the coordinates of the controller in the virtual spatial coordinate system. Then, a position at a preset distance from the first position is selected on the extension line of the connecting line as the second position.

[0127] E: Obtain the estimated position of the pointer at the target time.

[0128] As an alternative approach, the pointer's position at the target time can be estimated based on the pointer's position sequence before the target time, thus obtaining the pointer's estimated position at the target time.

[0129] G: If the distance between the second position and the estimated position is less than a preset distance threshold, the second position is determined as the display position of the target application's interface image at the target time; if the distance between the second position and the estimated position is greater than or equal to the preset distance threshold, the estimated position is determined as the display position of the target application's interface image at the target time.

[0130] Since the process of acquiring the source data of the target application and generating the interface image of the target application takes some time, if the pointer position changes little during this process, the second position is usually near the user's focus at the target time, and can be used as the display position of the target application's interface image at the target time. If the pointer position changes significantly during this process, it can be assumed that the user's focus has changed significantly, and the estimated position can be used as the display position of the target application's interface image at the target time. The preset spacing threshold can be a spacing threshold set to detect whether the distance between the second position and the estimated position is too small.

[0131] I: Before the target time, display the interface image of the target application at a second position with a first image size; at the target time, display the interface image of the target application at a display position with a second image size.

[0132] After receiving the launch command from the target application, the source data of the target application can be obtained from the server, and the interface image of the target application can be generated based on the source data of the target application.

[0133] After determining the second position and before the target time, the target application's interface image at the second position can be displayed at the second position with a first image size. The first image size is the default size of the target application's interface image.

[0134] Before the target time, the interface image of the target application can be enlarged from a first image size to a second image size. This ensures that the interface image of the target application can be displayed at the display position determined in step G at the target time, using the second image size.

[0135] After displaying the target application's interface image at the display position determined in step G with the second image size, the ray between the controller and the pointer in the virtual space coordinate system can be changed to another display form, such as a fishing line-like shape. In the fishing line shape, the user can receive predetermined operation commands for the target application's interface image, such as zoom in, zoom out, or pull away, thereby facilitating the user to adjust the size and position of the target application's interface image.

[0136] In this embodiment, upon receiving a startup command from the target application, a first position representing the user's current visual focus is obtained. Then, a second position, spaced at a preset distance from the first position, is selected on the extension of the line connecting the controller's position and the first position as the expected initial display position of the target application's interface image. Since acquiring the target application's source data and generating its interface image takes time, the pointer's position is estimated to determine its expected position at the target time. When the distance between the second position and the estimated position is less than a preset distance threshold, the distance is considered small. Selecting the second position as the display position at the target time ensures that the display position is within the user's field of vision and close to their current visual focus, while also reducing system resource consumption because the expected display position of the target application's interface image remains unchanged. When the distance between the second position and the estimated position is greater than or equal to the preset distance threshold, the distance is considered large. Selecting the estimated position as the display position at the target time ensures that the generated interface image of the target application is within the user's field of vision and close to their current visual focus, facilitating user operation. Before the target time, the target application's interface image can be displayed in a second position with a smaller first image size. At the target time, the target application's interface image can be displayed in the display position with a larger second image size. Then, it can be prepared to receive the user's pre-defined operation instructions for the target application, resulting in a better user experience.

[0137] Any of the methods for displaying an application interface on a head-mounted display device provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the methods for displaying an application interface on a head-mounted display device provided in this disclosure can be executed by a processor, such as by a processor executing any of the methods for displaying an application interface on a head-mounted display device mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated upon below.

[0138] Exemplary device

[0139] Figure 9 This is a structural block diagram of an apparatus for displaying an application interface on a head-mounted display device, according to one embodiment of this disclosure. Figure 9 As shown, a device for displaying an application interface on a head-mounted display device includes:

[0140] The first position acquisition module 100 is used to acquire a first position in response to receiving a launch operation instruction from the target application, wherein the first position is one of the current position of the pointer and the position of the icon of the target application in the virtual interface.

[0141] The second position acquisition module 200 is used to determine a second position that is spaced at a preset distance from the first position based on the first position;

[0142] The position estimation module 300 is used to obtain the estimated position of the pointer at the target time;

[0143] The display position determination module 400 is used to compare the distance between the second position and the estimated position with a preset distance threshold, and determine the display position of the interface image of the target application at the target time based on the comparison result, wherein the display position is one of the second position and the estimated position;

[0144] Display module 500 is used to display the interface image of the target application at the display position at the target time.

[0145] In one embodiment of this disclosure, the second position acquisition module 200 is used to acquire the position of the controller, wherein the pointer is controlled by the controller; the second position acquisition module 200 is also used to determine the position that is spaced from the first position by a preset distance along the line connecting the position of the controller and the first position as the second position.

[0146] In one embodiment of this disclosure, the second position acquisition module 200 is used to determine the position that is spaced from the first position by a preset distance on the extension of the line connecting the position of the controller and the first position as the second position.

[0147] In another embodiment of this disclosure, the second position acquisition module 200 is used to determine a preset position that is spaced apart from the first position by the preset distance and whose distance from the controller is less than the distance between the first position and the controller as the second position; wherein the pointer is controlled by the controller.

[0148] In another embodiment of this disclosure, the second location acquisition module 200 is used to determine a preset location that is spaced from the first location by a preset distance in the interface where the icon of the target application is located as the second location.

[0149] In another embodiment of this disclosure, the display position determination module 400 is configured to determine the second position as the display position of the interface image of the target application at the target time if the distance between the second position and the estimated position is less than the preset distance threshold; the display position determination module 400 is further configured to determine the estimated position as the display position of the interface image of the target application at the target time if the distance between the second position and the estimated position is greater than or equal to the preset distance threshold.

[0150] In another embodiment of this disclosure, the position estimation module 300 is used to acquire at least one of an attitude data sequence and a position data sequence that indicates the positioning of the controller before the target time, wherein the pointer is controlled by the controller; the position estimation module 300 is further used to perform position estimation on the pointer based on at least one of the attitude data sequence and the position data sequence to obtain the estimated position of the pointer at the target time.

[0151] In another embodiment of this disclosure, the display module 500 is configured to display the interface image of the target application at the second position with a first image size before the target time; the display module 500 is also configured to display the interface image of the target application at the display position with a second image size at the target time; wherein the second image size is larger than the first image size.

[0152] It should be noted that the specific implementation of the apparatus for displaying an application interface on a head-mounted display device in this disclosure is similar to the specific implementation of the method for displaying an application interface on a head-mounted display device in this disclosure. For details, please refer to the method section for displaying an application interface on a head-mounted display device. To reduce redundancy, further details will not be provided.

[0153] Exemplary electronic devices

[0154] Below, for reference Figure 10 To describe an electronic device according to embodiments of this disclosure. For example... Figure 10 As shown, the electronic device includes one or more processors 10 and memory 20.

[0155] The processor 10 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0156] The memory 20 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 10 may execute the program instructions to implement the methods for displaying application interfaces on a head-mounted display device according to the various embodiments of this disclosure described above, and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0157] In one example, the electronic device may also include an input device 30 and an output device 40, which are interconnected via a bus system and / or other forms of connection mechanism (not shown). The input device 30 may be, for example, a keyboard, a mouse, etc. The output device 40 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, etc.

[0158] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0159] Exemplary computer-readable storage media

[0160] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0161] Furthermore, this disclosure also protects a head-mounted display device that includes the means for displaying an application interface as described in the above embodiments.

[0162] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0163] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0164] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0165] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0166] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0167] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0168] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for displaying an application interface on a head-mounted display device, comprising: In response to receiving a launch operation command from the target application, a first position is obtained, wherein the first position is one of the current position of the pointer and the position of the icon of the target application; Based on the first position, a second position is determined that is spaced at a preset distance from the first position; Obtain the estimated position of the pointer at the target time; The distance between the second position and the estimated position is compared with a preset distance threshold. Based on the comparison result, the display position of the interface image of the target application at the target time is determined. The display position is one of the second position and the estimated position. At the target time, the interface image of the target application is displayed at the display location; Wherein, obtaining the estimated position of the pointer at the target time includes: Acquire at least one of an attitude data sequence and a position data sequence that indicates the positioning of the controller before the target time, wherein the pointer is controlled by the controller; The pointer's position is estimated based on at least one of the attitude data sequence and the position data sequence to obtain the pointer's estimated position at the target time.

2. The method according to claim 1, wherein, Determining a second position at a preset distance from the first position based on the first position includes: Obtain the position of the controller, wherein the pointer is controlled by the controller; In the direction of the line connecting the controller position and the first position, the position that is separated from the first position by the preset distance is determined as the second position.

3. The method according to claim 2, wherein, The step of determining the position at a preset distance from the first position as the second position along the line connecting the position of the controller and the first position includes: On the extension of the line connecting the controller position and the first position, the position that is spaced from the first position by the preset distance is determined as the second position.

4. The method according to claim 1, wherein, Determining a second position at a preset distance from the first position based on the first position includes: The preset position that is spaced apart from the first position by the preset distance and whose distance from the controller is less than the distance between the first position and the controller is determined as the second position; The pointer is controlled by the controller.

5. The method according to claim 1, wherein, Determining a second position at a preset distance from the first position based on the first position includes: In the interface where the icon of the target application is located, a preset position that is separated from the first position by the preset distance is determined as the second position.

6. The method according to any one of claims 1-5, wherein, The step of comparing the distance between the second position and the estimated position with a preset distance threshold, and determining the display position of the target application's interface image at the target time based on the comparison result, includes: If the distance between the second position and the estimated position is less than the preset distance threshold, the second position is determined as the display position of the interface image of the target application at the target time; If the distance between the second position and the estimated position is greater than or equal to the preset distance threshold, the estimated position is determined as the display position of the interface image of the target application at the target time.

7. The method according to any one of claims 1-5, wherein, The step of displaying the interface image of the target application at the display location at the target time includes: Before the target time, the interface image of the target application is displayed at the second position at the first image size; At the target time, the interface image of the target application is displayed at the display position at a second image size; The second image size is larger than the first image size.

8. An apparatus for displaying an application interface on a head-mounted display device, comprising: The first position acquisition module is used to acquire a first position in response to receiving a launch operation instruction from the target application, wherein the first position is one of the current position of the pointer and the position of the icon of the target application; The second position acquisition module is used to determine a second position that is spaced at a preset distance from the first position, based on the first position; A position estimation module is used to obtain the estimated position of the pointer at the target time; The display position determination module is used to compare the distance between the second position and the estimated position with a preset distance threshold, and determine the display position of the interface image of the target application at the target time based on the comparison result, wherein the display position is one of the second position and the estimated position; The display module is used to display the interface image of the target application at the display position at the target time; Specifically, the location estimation module is used for: Acquire at least one of an attitude data sequence and a position data sequence that indicates the positioning of the controller before the target time, wherein the pointer is controlled by the controller; The pointer's position is estimated based on at least one of the attitude data sequence and the position data sequence to obtain the pointer's estimated position at the target time.

9. A computer-readable storage medium storing a computer program for performing the method for displaying an application interface for a head-mounted display device according to any one of claims 1-7.

10. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for displaying an application interface for a head-mounted display device as described in any one of claims 1-7.

11. A head-mounted display device, comprising the means for displaying an application interface on the head-mounted display device as described in claim 8.

Images