Display method, device, system and electronic device for head-mounted display device

Abstract

This application provides a display method, apparatus, system, and electronic device for a head-mounted display device. The method includes: acquiring current movement information of an external positioning device; invoking a preset image rendering thread to store the current movement information in local storage; if the local storage includes at least one historical movement information of the external positioning device, determining target movement information of the external positioning device based on the current movement information and the historical movement information; rendering a location icon of the external positioning device using the image rendering thread based on the target movement information; and displaying the rendered location icon on the spatial screen of the head-mounted display device.

Description

Technical Field

[0001] This application relates to the field of head-mounted display device technology, and in particular to a display method, apparatus, system and electronic device for head-mounted display devices. Background Technology

[0002] Typically, head-mounted display devices can present a spatial screen, displaying images on the spatial screen to meet users' entertainment needs such as watching images through the head-mounted display device.

[0003] External positioning devices, as external input devices for electronic devices, can input external operating information into the electronic device and display it through a display system. The use of external positioning devices can make the operation of electronic devices simpler and faster, including but not limited to mice. To facilitate user interaction with head-mounted displays, external positioning devices can be connected to them. Therefore, a new technical solution is needed to achieve compatibility between head-mounted displays and external positioning devices. Summary of the Invention

[0004] This application provides a display method, apparatus, system, and electronic device for head-mounted display devices.

[0005] According to a first aspect of the embodiments of this application, a display method for a head-mounted display device is provided, comprising: acquiring current movement information of an external positioning device; calling a preset image rendering thread to store the current movement information in local storage; if the local storage includes at least one historical movement information of the external positioning device, determining target movement information of the external positioning device based on the current movement information and the historical movement information; rendering a location icon of the external positioning device using the image rendering thread based on the target movement information; and displaying the rendered location icon on the spatial screen of the head-mounted display device.

[0006] According to a second aspect of the embodiments of this application, a display device for a head-mounted display device is provided, comprising: an acquisition module for acquiring current movement information of an external positioning device; a calling module for calling a preset image rendering thread to store the current movement information in local storage; a determination module for determining target movement information of the external positioning device based on the current movement information and the historical movement information if the local storage includes at least one historical movement information of the external positioning device; a rendering module for rendering a location icon of the external positioning device using the target movement information and an image rendering thread; and a display module for displaying the rendered location icon on the spatial screen of the head-mounted display device.

[0007] According to a third aspect of the embodiments of this application, a display system for a head-mounted display device is provided, comprising: an external positioning device configured to move its position in response to a user's operation; a controller configured to acquire current movement information of the external positioning device; invoke a preset image rendering thread to store the current movement information in local storage; if the local storage includes at least one historical movement information of the external positioning device, determine target movement information of the external positioning device based on the current movement information and the historical movement information; render a location icon of the external positioning device using the image rendering thread based on the target movement information; and a head-mounted display device configured to display the rendered location icon on a spatial screen.

[0008] According to a fourth aspect of the embodiments of this application, an electronic device is provided, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store a computer program; and the processor is used to execute the method of any one of the first aspects by running the computer program stored in the memory.

[0009] According to a fifth aspect of the embodiments of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method as described in any of the first aspects.

[0010] According to a sixth aspect of the embodiments of this application, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the method as described in any of the first aspects. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0012] Figure 1A This is a schematic diagram of some optional display systems for head-mounted display devices in the embodiments of this application.

[0013] Figure 1B This is a schematic diagram showing the connection of some optional display systems in the embodiments of this application.

[0014] Figure 2 This is a flowchart illustrating some optional display methods for head-mounted display devices in embodiments of this application.

[0015] Figure 3The diagram shows some examples of the locked state of the system screen on the host side.

[0016] Figure 4 Here are some optional flowcharts showing the methods for obtaining current mobility information in the embodiments of this application.

[0017] Figure 5 Here are some optional flowcharts illustrating the methods for determining target movement information in embodiments of this application.

[0018] Figure 6 The following are some optional flowcharts illustrating the methods for rendering location icons in the embodiments of this application.

[0019] Figure 7 This is a structural block diagram of some optional display devices for head-mounted display devices in the embodiments of this application.

[0020] Figure 8 This is a structural block diagram of some optional electronic devices in the embodiments of this application.

[0021] Explanation of reference numerals in the attached figures:

[0022] 100. Display system for head-mounted display devices; 102. External positioning device; 104. Controller; 106. Head-mounted display device; 108. Host terminal; 700. Display device for head-mounted display devices; 702. Acquisition module; 704. Calling module; 706. Determination module; 708. Rendering module; 710. Display module; 800. Electronic device; 802. Processor; 804. Communication interface; 806. Memory; 808. Communication bus; 810. Computer program. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of this application. It should be understood that the various steps described in the method implementation of this disclosure can be performed in different orders and / or in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0024] Typically, head-mounted displays (HUDs) can present a spatial screen, displaying images to meet users' entertainment needs. External positioning devices, as external input devices for electronic devices, can input external operating information to assist in controlling the HUD's display system. The use of external positioning devices makes operating electronic devices simpler and faster; these devices include, but are not limited to, mice. To make HUDs more convenient for users, external positioning devices can be connected to them.

[0025] This application provides a new technical solution that can effectively achieve the adaptation between head-mounted display devices and external positioning devices, enabling the external positioning device to accurately control the rendered content on the spatial screen of the head-mounted display device, thereby improving the user experience.

[0026] To better illustrate the display solution for head-mounted display devices described in this application, an exemplary description is provided below, using a display system suitable for this solution as an example. For instance... Figure 1A As shown, the display system 100 for a head-mounted display device provided in this application may include: an external positioning device 102, a controller 104, and a head-mounted display device 106.

[0027] In this application, the external positioning device 102 can be configured to move in response to user operations. The external positioning device 102 can be any electronic device. In some optional embodiments, the external positioning device 102 may include, but is not limited to, a mouse, an electronic pen, etc. For example, a mouse can be moved by the user sliding it across a support surface (e.g., a desktop); similarly, an electronic pen can be moved by the user writing, and so on. For ease of explanation, a mouse can be used as an example of the external positioning device 102 in this application.

[0028] In this application, the controller 104 can be configured to: acquire the current movement information of the external positioning device 102; call a preset image rendering thread to store the current movement information in local storage; if the local storage includes at least one historical movement information of the external positioning device, determine the target movement information of the external positioning device based on the current movement information and the historical movement information; and render the location icon of the external positioning device using the image rendering thread based on the target movement information.

[0029] Optionally, the controller 104 may include at least one processing module capable of data processing, such as a chip. The controller 104 may include, but is not limited to, a CPU (Central Processing Unit), an MCU (Microcontroller Unit), a GPU (Graphics Processing Unit), or a FPGA (Field Programmable Gate Array).

[0030] The location icon can be an icon used to indicate the position of the external positioning device 102 on the rendered graphical user interface. For example, if the external positioning device 102 is a mouse, the location icon can also be called an air mouse icon or a mouse cursor, etc.

[0031] In this application, the head-mounted display device 106 is configured to display rendered location icons on a spatial screen. The spatial screen of the head-mounted display device 106 can display rendered location icons. After a user wears the head-mounted display device 106, the head-mounted display device 106 can present the spatial screen to the user. The spatial screen can be a virtual screen (e.g., a virtual reality screen), and images can be displayed on the spatial screen to meet the user's entertainment needs such as viewing images through the head-mounted display device 106. For example, the head-mounted display device 106 can include, but is not limited to, AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, etc.

[0032] It should be understood that the external positioning device 102 can be connected to the controller 104 in any way, such as by wired connection or by wireless connection (e.g., including but not limited to Bluetooth connection, cellular connection, WIFI connection, etc.).

[0033] In some alternative implementations, the head-mounted display device 106 and the controller 104 can be an integrated structure. That is, the controller 104 can be housed within the head-mounted display device 106. For example, the head-mounted display device 106 can be a VR all-in-one device. In this case, the external positioning device 102 can be connected to the head-mounted display device 106.

[0034] In some alternative embodiments, the head-mounted display device 106 and the controller 104 can be a separate structure. For example, see reference... Figure 1BAs shown, the display system 100 may also include a host terminal 108. A controller 104 may be located on the host terminal 108, which is electrically connected to the head-mounted display device 106. It is understood that the controller 104 may also be a separate device communicatively connected to the host terminal 108; there is no single limitation. The host terminal 108 may be various electronic devices with data computing capabilities, including but not limited to: handheld terminals (e.g., mobile phones, handheld projectors, etc.), tablet computers, algorithm boxes, computers, vehicle-mounted systems, servers, and other electronic devices. The head-mounted display device 106 and the host terminal 108 may be connected via wired or wireless means. Optionally, the host terminal 108 may provide at least some computing resources to the head-mounted display device 106 through the controller 104. Optionally, the host terminal 108 may also provide data for display to the head-mounted display device 106. (See reference...) Figure 1B As shown, the controller 104 of the host 108 can also be connected to an external positioning device 102, such as through wired or wireless connection.

[0035] Optionally, when the display system includes both the host terminal 108 and the external positioning device 102, the user can control the display of the head-mounted display device 106 by operating the external positioning device 102, or by directly operating the host terminal 108. Alternatively, the display of the head-mounted display device 106 can be controlled by the combined operation of the external positioning device 102 and the host terminal 108; there is no single limitation here.

[0036] The display system 100 described in this application uses target movement information determined by the current movement information of the external positioning device 102 and at least one historical movement information when the controller 104 renders each frame of the image using the image rendering thread to render the position icon. Therefore, it can ensure that each frame of the rendered image uses the movement information generated by the external positioning device 102 from the previous frame to the current frame, thus ensuring the accuracy of the position icon of the external positioning device 102 in the rendered image. For example, if the acquisition frequency of the movement amount of the external positioning device 102 is different from the image rendering frequency of the image rendering thread, the movement amount of the external positioning device 102 is stored locally during the rendering of adjacent frame images. The display method provided in this embodiment can avoid the omission or loss of the acquired movement amount information. When the image rendering thread renders image frames, it can ensure that the movement amount information acquired by the external positioning device 102 after the end of the previous frame image can be reflected in the movement of the position icon of the image frame rendered by the image rendering thread. This improves the accuracy of the display position of the position icon of the external positioning device 102 on the spatial screen of the head-mounted display device 106, meets the user's needs for using the head-mounted display device 106, and effectively improves the user experience. Through this technical solution, the adaptation between the head-mounted display device and the external positioning device can be effectively realized, enabling the external positioning device to accurately control the rendering content on the spatial screen of the head-mounted display device, thereby facilitating the improvement of the user experience.

[0037] In some optional embodiments, the controller 104 is further configured to control the movement of a control cursor on the spatial screen of the head-mounted display device 106 in response to changes in the spatial distribution of the controller 104. Optionally, changes in the spatial distribution of the controller 104 may refer to changes in the spatial distribution information of the controller 104, which may include at least one of position information and attitude information. The spatial distribution information of the controller 104 can be determined by sensor data collected by any sensor in the controller 104. For example, the sensor data may include at least one of data collected by an IMU (Inertial Measurement Unit), a gyroscope, a camera, etc. Optionally, the control cursor can be used to locate the pointing position of the controller 104 on the spatial screen.

[0038] Optionally, the control cursor indicated by the spatial distribution information of the controller 104 is different from the position icon of the external positioning device 102. The control cursor may be controlled by the user's operation of the controller 104, rather than by the user's operation of the external positioning device 102. In some embodiments, the control cursor and the position icon have different shapes in the images displayed on the spatial screen so that the user can distinguish between the two. Optionally, a straight line or curve may be rendered between the controller 104 and the control cursor to visually represent the positional correspondence between the controller 104 in space and the control cursor on the spatial screen. Alternatively, the control cursor may also take other forms; there is no single limitation.

[0039] Optionally, if the controller 104 is located on the head-mounted display device 106, then the various sensors described above can be located on the head-mounted display device. For example, the controller 104 may include sensors for eye tracking (such as infrared sensors). When the user wears the head-mounted display device 106, changes in the user's gaze detected by the eye-tracking sensors can characterize changes in the spatial distribution of the controller, and the movement of the control cursor on the spatial screen of the head-mounted display device 106. In response to detecting preset operations such as blinking, the controller 104 can update the display interface of the spatial screen after detecting the preset operation.

[0040] Optionally, if the controller 104 is located on the host terminal 108, then the various sensors mentioned above can be located on the host terminal 108. For example, the controller 104 may include sensors such as an IMU and a gyroscope. When the user wears the head-mounted display device 106, if the user shakes the host terminal 108 (for example, a handheld terminal), the controller 104 will shake along with the host terminal 108, causing a change in the spatial distribution of the controller 104. The controller 104 can control the control cursor on the spatial screen of the head-mounted display device 106 to move accordingly based on the detected change in spatial distribution, thereby changing the position of the control cursor on the spatial screen so that the displayed content on the spatial screen can be adjusted as needed by controlling the cursor.

[0041] It should be understood that the above optional methods allow users to adjust the position and orientation of the controller 104, thereby enabling flexible movement and control of the control cursor on the spatial screen of the head-mounted display device 106. This allows the controller 104 to adjust the content displayed on the spatial screen as needed, thereby improving the user experience.

[0042] In some optional embodiments, the controller 104 is further configured to update the display interface of the spatial screen in response to detecting a preset operation by the user on the controller 104. This allows the user to flexibly control the spatial screen display interface of the head-mounted display device by operating the controller 104, thereby improving the user experience.

[0043] Optionally, preset operations can be set as needed and can have various forms of expression, without specific limitations here.

[0044] For example, preset operations can be certain specific actual actions performed by the user on the controller 104. These include, but are not limited to, tilting to the left a predetermined distance or tilting to the right a predetermined distance. When such an action is detected, the display interface of the spatial screen can be updated according to the corresponding action, thereby enabling the user to control the spatial screen display interface of the head-mounted display device by operating the controller 104.

[0045] For example, preset operations can also be operations sent directly by the user to the controller 104, or indirectly through other hardware. For instance, the controller 104 is set on the host 108, which may include a display screen. The display screen may have one or more control buttons. The user can send operation commands to the controller 104 by manipulating the control buttons. When the controller 104 detects such operation commands, it can update the display interface of the spatial screen according to the corresponding operation commands, thereby enabling the user to control the spatial screen display interface of the head-mounted display device 106 by operating the controller 104.

[0046] In some optional embodiments, controller 104 is specifically configured to: construct a spatial screen for the head-mounted display device, set a screen identifier for the spatial screen, and render a location icon using an image rendering thread based on the target movement information. Head-mounted display device 106 is further configured to: display the rendered location icon on the spatial screen with the set screen identifier.

[0047] In some optional embodiments, the movement amount information may include a first movement amount along a first direction and a second movement amount along a second direction. Optionally, the first direction and the second direction may intersect. The controller 104 is further configured to: superimpose the first movement amount in the current movement amount information and the first movement amount in the historical movement amount information to obtain a first target movement amount; superimpose the second movement amount in the current movement amount information and the second movement amount in the historical movement amount information to obtain a second target movement amount; and determine the first target movement amount and the second target movement amount as target movement amount information.

[0048] In some optional embodiments, the controller 104 is further configured to: after determining the target mobility information of the external positioning device based on the current mobility information and the historical mobility information, clear the mobility information stored in the local storage.

[0049] In some alternative embodiments, the controller 104 is specifically configured to: dispatch a movement event of the external positioning device in response to determining that movement has occurred in the external positioning device; and obtain current movement information based on the movement event.

[0050] In some optional embodiments, the local storage also includes button information of the external positioning device; the controller 104 is further configured to: obtain the button information of the external positioning device from the local storage; execute the button command indicated by the button information for the position icon of the interface displayed on the spatial screen of the head-mounted display device; and update the content screen displayed on the spatial screen of the head-mounted display device using an image rendering thread.

[0051] In some optional embodiments, the external positioning device 102 is communicatively connected to the host terminal 108. The controller 104 is further configured to: after the external positioning device is communicatively connected to the host terminal, control the system screen of the host terminal to enter a screen lock state and hide the pointer on the system screen corresponding to the external positioning device; wherein, if the system screen enters the screen lock state, at least one interface element on the system screen is in a state that cannot be triggered by the external positioning device.

[0052] In some alternative embodiments, the controller 104 is further configured to: render a touch panel on the system screen, the touch panel being used to control the display of the head-mounted display device; and, in response to receiving an interface lock operation from a user, control the touch panel interface to enter an interface lock state to lock the touch panel interface on the system screen.

[0053] The various optional embodiments described above will be specifically described in the method embodiments below. The specific content and beneficial effects can be understood by referring to the following text, and will not be repeated here.

[0054] According to one aspect of the embodiments of this application, a display method for a head-mounted display device is provided. Figure 2 A flowchart illustrating some optional display methods for head-mounted display devices in embodiments of this application is shown. (Refer to...) Figure 2 As shown, the method includes steps S202, S204, S206, S208, and S210:

[0055] S202: Obtain the current movement information of the external positioning device.

[0056] It is understandable that, as some optional types of external positioning device 102 have been introduced above, the aforementioned external positioning device 102 can also be other types of external input devices; there is no single limitation. Here, taking a mouse as an example, the user can move the mouse on a support (such as a desktop). The current movement information of the external positioning device 102 can be obtained in real time. Optionally, the movement information can include information related to the movement in each direction under a preset coordinate system. For example, the preset coordinate system can be a two-dimensional coordinate system established on the movement plane of the external positioning device (such as a mouse).

[0057] In some optional embodiments, the movement information may include a first movement along a first direction and a second movement along a second direction. Optionally, the first and second directions intersect. Optionally, the first and second directions may be perpendicular to each other. For example, the first and second directions may be the two coordinate axes of a two-dimensional coordinate system established in the movement plane of the external positioning device (such as a mouse). In this embodiment, by collecting the first movement of the external positioning device 102 in the first direction and the second movement in the second direction, it is convenient to determine the movement of the position icon corresponding to the external positioning device 102 on the spatial screen.

[0058] Optionally, when the head-mounted display device 106 is a separate device, the external positioning device 102 can communicate with the host device 108. The host device 108 may include a system screen (or main screen), which may be, for example, a physical screen integrated into the host device 108. The host device 108 can serve as the main device or data source device for the head-mounted display device 106, and can be connected to the head-mounted display device 106 via wired or wireless means. The external positioning device 102 can communicate with the host device 108 via wired or wireless means. Taking the communication connection between the external positioning device 102 (e.g., a mouse) and the host device 108 (e.g., a handheld terminal) (e.g., Bluetooth connection) as an example, the host device 108 can typically calculate the position of the pointer icon (e.g., a mouse pointer) corresponding to the external positioning device 102 on the system screen. In some optional implementations, the aforementioned movement information can be movement information in system screen coordinates. The first movement amount deltaX in the above movement information can be determined as: the movement amount of the pointer icon corresponding to the external positioning device 102 along the first direction in the coordinate system of the system screen; the second movement amount deltaY can be determined as: the movement amount of the pointer icon corresponding to the external positioning device 102 along the second direction in the coordinate system of the system screen. Here, the first and second directions can be the two coordinate axes of the system screen coordinate system, such as the x and y directions respectively, or understood as the horizontal and vertical directions. Optionally, the first and second movement amounts can be in pixels of the system screen.

[0059] It should be understood that for the rendering of any frame of the head-mounted display device 106, the movement direction and distance of the external positioning device 102 between the previous frame and the current frame can be determined by accumulating the movement information of the external positioning device between the previous frame and the current frame. Therefore, the movement direction and distance of the position icon corresponding to the external positioning device 102 on the spatial screen can be determined for any adjacent frame without calculating the change in the coordinate position of the external positioning device 102 on the host system screen. This solution is not limited by the edge of the host system screen. Therefore, even if the coordinate position of the external positioning device 102 moves to or exceeds the edge of the system screen during movement, the solution disclosed in this embodiment can continue to detect and calculate the movement information of the external positioning device 102 in real time, avoiding the omission or loss of movement information.

[0060] In some optional embodiments, the display method further includes: after the external positioning device 102 is communicatively connected to the host terminal 108, controlling the system screen of the host terminal 108 to enter a screen lock state, and hiding the pointer icon on the system screen corresponding to the external positioning device 102. In this embodiment, if the system screen enters a screen lock state, at least one interface element on the system screen is in a state that cannot be triggered by the external positioning device 102. It is understood that the aforementioned screen lock state is used to control the response of the system screen to the external positioning device 102. Therefore, after the host terminal 108 controls the system screen to enter a screen lock state, the system screen may neither respond to the operation received by the external positioning device 102 nor display the pointer icon corresponding to the external positioning device 102. For example, in some optional embodiments, the host terminal 108 may run an application for controlling the head-mounted display device. In this case, the pointer icon on the system screen corresponding to the external positioning device 102 can be hidden, so that the pointer is neither displayed nor can it trigger the interface elements of the system screen.

[0061] Based on this, by controlling the system screen of the host device 108 to enter a screen lock state after the external positioning device 102 establishes a communication connection with the host device 108, at least one interface element on the system screen of the host device 108 can be prevented from responding to the operation of the external positioning device 102 on the spatial screen of the head-mounted display device 106 (for example, the operation on the external positioning device 102 could be a single click or double click). This ensures the normal display of the host device 108 and the user experience of the head-mounted display device 106. Furthermore, by hiding the pointer icon corresponding to the external positioning device 102 on the system screen, the external positioning device 102 only displays the position icon on the spatial screen of the head-mounted display device 106 during use, and does not display the pointer icon on the system screen of the host device 108. This allows for convenient operation of the spatial screen of the head-mounted display device 106 without affecting the display content of the system screen of the host device 108, further improving the user experience.

[0062] In this application, the pointer icon corresponding to the external positioning device 102 can be hidden in any way. One optional method is to change the transparency of the pointer icon, making it invisible to the user's eye, thus achieving hiding. For example, the transparency of the pointer icon can be set to 0, or set below a certain threshold. Another optional method is to perform a non-rendering operation on the pointer icon corresponding to the external positioning device 102 to prevent it from being displayed on the system screen, thus achieving hiding. One or more of these methods can be chosen to hide the pointer icon corresponding to the external positioning device 102.

[0063] In some alternative embodiments, such as Figure 3 As shown, an application can be installed on the host 108, and the application's interface can be displayed on the system screen of the host 108. The interface can include multiple interface elements. For example... Figure 3The interface elements include multiple buttons, such as a QR code scanning button, a screen recording button, a screenshot button, an exit button, etc. (for example only). Optionally, when the external positioning device 102 is not connected to the host device 108, these buttons on the system screen can be triggered (e.g., by user touch) to perform corresponding functions. When the external positioning device 102 is connected to the host device 108, the controller 104 can control the system screen of the host device 108 to enter a screen lock state. After hiding the pointer icon on the system screen corresponding to the external positioning device 102, the buttons displayed on the program interface on the system screen will not be triggered by the external positioning device 102 (e.g., by clicking or double-clicking the external positioning device 102). That is, the buttons on the system screen do not respond to the instructions of the external positioning device 102, thereby avoiding accidental button triggering when operating the spatial screen of the head-mounted display device 106 through the external positioning device 102, thus ensuring the normal display of the host device 108 and the user experience of the head-mounted display device 106.

[0064] In some optional embodiments, the display method may further include: rendering a touch panel on a system screen, the touch panel being used to control the display of a head-mounted display device; and, in response to receiving an interface lock operation from a user, controlling the touch panel interface to enter an interface lock state to lock the touch panel interface on the system screen. It is understood that having the touch panel interface in the interface lock state also prevents the touch panel interface from exiting the system screen.

[0065] Optionally, after the head-mounted display device 106 is connected to the host terminal 108, the host terminal 108 can act as a control device for the head-mounted display device 106. In this case, after the host terminal 108 runs the corresponding application, the system screen can display a touch panel for controlling the head-mounted display device 106. If the system screen of the host terminal 108 currently displays the touch panel of the head-mounted display device 106, the control panel interface will not respond to the operation of the external positioning device 102, while allowing the user to perform touch operations through the system screen to assist in controlling the display of the spatial screen of the head-mounted display device 106. The host terminal 108, as a control device, can cooperate with the external positioning device 102 to achieve stable and accurate display control of the head-mounted display device 106, thereby improving the diversity of interaction of the head-mounted display device 106. Optionally, by locking the interface, the touch panel interface can be kept on the system screen without exiting, and the operation of the external positioning device 102 will not affect the touch panel interface displayed on the system screen. This improves the stability of the host 108 as a control device for the head-mounted display device 106 and makes it easier for users to operate the head-mounted display device 106 through the host.

[0066] In some alternative embodiments, it is still based on Figure 3 Using the example shown, the touch panel interface rendered on the system screen can be the program interface of the application used to support the operation of the head-mounted display device 106, which supports touch-based control of the head-mounted display device 106's display functions. When the external positioning device 102 is connected to the host 108, at least one button (interface element) on the touch panel interface cannot be triggered by the external positioning device 102, but the user can still operate it via touch to trigger the button or perform other operations. Optionally, in response to the user's interface locking operation, the system interface of the host 108 can be locked to the touch panel interface of the head-mounted display device 106. In this case, such as Figure 3 The swipe-up menu trigger area and drop-down menu trigger area shown cannot be triggered, so that the swipe-up menu and drop-down menu will not be triggered and woke up by the external positioning device 102, nor will they be brought up by the user's swipe-up or swipe-down.

[0067] In this application, current mobility information can be obtained in any way. In some optional embodiments, refer to... Figure 4 The flowchart shown can be implemented through the following steps S2022 and S2024, specifically:

[0068] S2022: In response to determining that movement has occurred in the external positioning device, dispatch a movement event for the external positioning device.

[0069] For example, the controller 104 (e.g., it can be set on the host 108 or the head-mounted display device 106) can run an operating system (e.g., Android or Linux). When the controller 104 determines that the external positioning device 102 has moved, it can call a preset function (e.g., the dispatchGenericMotionEvent function) through the operating system to dispatch movement events of the external positioning device 102, such as mouse movement events (MotionEvent), so as to obtain the movement amount information of the external positioning device 102.

[0070] S2024: Obtain current mobility information based on mobility events.

[0071] For movement events such as mouse movement events (MotionEvent), the first movement amount deltaX and the second movement amount deltaY of the external positioning device 102 moving in the first direction at the current moment can be obtained, and the obtained first movement amount deltaX and second movement amount deltaY can be determined as the current movement amount information of the external positioning device 102.

[0072] Based on this, by implementing the above steps S2022 to S2024, the current movement information of the external positioning device 102 can be effectively determined, which facilitates the accurate rendering of the location icon of the external positioning device 102 and its accurate display on the front screen of the head-mounted display device, thereby improving the user experience.

[0073] S204: Call the preset image rendering thread to store the current movement information in local storage.

[0074] The image rendering thread can be implemented using any image rendering engine. For example, the Unity3D image rendering engine can be used. Using the features of the Unity3D image rendering engine, position icons displayed on the screen can be rendered subsequently.

[0075] After obtaining motion data (current or historical motion data), the image rendering thread can be invoked to store the obtained motion data in local storage. When rendering image frames displayed on the spatial screen of the head-mounted display device, the stored motion data can be quickly retrieved from local storage.

[0076] It should be understood that local storage can be any storage unit, such as a memory, a database, etc. In some optional embodiments, local storage can be a local cache of the image rendering thread. By storing the various movement information between two frames of images displayed on the virtual display screen in the local cache of the image rendering thread, the subsequent image rendering thread can quickly retrieve this data to accurately render the location icon of the external positioning device 102.

[0077] It should be understood that if the frequency of motion measurement by the external positioning device 102 is greater than the frequency of location icon rendering, multiple motion measurement information can be collected between rendering two adjacent frames of location icons. The motion measurement information collected at the current moment can be considered current motion measurement information, while the motion measurement information collected at moments prior to the current moment can be considered historical motion measurement information. In this application, each piece of motion measurement information obtained between rendering two adjacent frames of location icons can be stored locally for later use.

[0078] S206: If the local storage includes at least one historical motion information of an external positioning device, determine the target motion information of the external positioning device based on the current motion information and the historical motion information.

[0079] If the local storage contains at least one historical motion data point (i.e., the number of historical motion data points is greater than 0), and multiple motion data points were collected during the time period from rendering the previous frame to rendering the current frame, then the historical motion data and current motion data generated by the external positioning device from the end of rendering the previous frame to the current moment can be obtained from the local storage. The target motion data can then be determined based on this motion data.

[0080] Since the target movement information uses not only the current movement information but also historical movement information, it can avoid the omission or loss of at least some movement information, thereby ensuring the accuracy of the movement when the position icon rendered based on the target movement information is displayed on the spatial screen.

[0081] Optionally, the target motion information can be determined based on the current motion information stored locally and all historical motion information collected during the time period from the previous frame to the rendering of the current frame, thereby minimizing the omission or loss of historical motion information.

[0082] Optionally, the local storage can store motion data collected during the time period from the previous frame to the rendering of the current frame. In this case, the number of historical motion data in the local storage can be determined first. If it is greater than 0, it indicates that there is historical motion data that needs to be processed. Then, the first and second motion values ​​from each historical motion data can be retrieved from the local storage using a preset function to obtain historical values, so as to calculate the target motion data.

[0083] Optionally, the target movement information of the external positioning device 102 may include at least a first movement (i.e., the first target movement) along a first direction and a second movement (i.e., the second target movement) along a second direction.

[0084] This application does not limit the specific method for determining the target mobility information. Optionally, refer to... Figure 5 The flowchart shown illustrates how target movement information can be determined through the following steps S2062–S2066:

[0085] S2062: The first movement amount in the current movement amount information and the first movement amount in the historical movement amount information are superimposed to obtain the first target movement amount.

[0086] The first movement amount in the current movement amount information and the first movement amount in each historical movement amount information can be superimposed to ensure that the calculated first target movement amount has taken into account the data in the first direction in each movement amount information of the external positioning device 102.

[0087] For example, the first target movement can be denoted as Δx, and the first movement in the current movement information can be denoted as deltaX_0. Assuming there are n historical movement information, the first movement in the n historical movement information can be denoted as deltaX_1, ..., deltaX_n respectively. Then Δx = deltaX_0 + deltaX_1 + ... + deltaX_n.

[0088] As an example, the first movement amount stored in local storage can be denoted as deltaLocalX. The collected movement amount information (including historical and current movement amount information) can be sent to the image rendering thread (e.g., an image rendering engine like Unity3D). Then, the image rendering thread can be called to add the value of the first movement amount deltaX to deltaLocalX, thus achieving the superposition of various first movement amounts. Finally, by adding the first movement amount of the current movement amount information to deltaLocalX, the first target movement amount can be obtained.

[0089] S2064: The second movement amount in the current movement amount information and the second movement amount in the historical movement amount information are superimposed to obtain the second target movement amount.

[0090] The second movement amount in the current movement amount information and the second movement amount in each historical movement amount information can be superimposed to ensure that the calculated second target movement amount has taken into account the data in the second direction in each movement amount information of the external positioning device 102.

[0091] For example, the second target movement can be denoted as Δy, and the second movement in the current movement information can be denoted as deltaY_0. Assuming there are n historical movement information, the second movement in the n historical movement information can be denoted as deltaY_1, ..., deltaY_n respectively. Then Δy = deltaY_0 + deltaY_1 + ... + deltaY_n.

[0092] For example, the second movement amount stored in local storage can be denoted as deltaLocalY. The collected movement amount information (including historical and current movement amount information) can be sent to the image rendering thread (e.g., an image rendering engine such as Unity3D). Then, the image rendering thread can be invoked to add the value of the second movement amount deltaY to deltaLocalY, thus achieving the superposition of various second movement amounts. Finally, by adding the second movement amount from the current movement amount information to deltaLocalY, the second target movement amount can be obtained.

[0093] S2066: Determine the first target movement amount and the second target movement amount as target movement amount information.

[0094] In this way, we can obtain the target movement information: the movement in the first direction, i.e., the first target movement, and the movement in the second direction, i.e., the second target movement.

[0095] Based on this, through the implementation of steps S2062 to S2066 above, the acquired movement information can be conveniently and effectively superimposed to obtain the target movement information. This effectively avoids the omission or loss of at least some movement information collected by the external positioning device 102 during the rendering of the location icon. This ensures the accuracy of the movement when the location icon rendered based on the target movement information is displayed on the spatial screen, thereby improving the display effect of the location icon of the external positioning device 102 on the spatial screen of the head-mounted display device 106, meeting the user's needs for using the head-mounted display device 106, and effectively improving the user experience.

[0096] S208: Based on the target movement information, use the image rendering thread to render the location icon of the external positioning device.

[0097] S210: Display the rendered location icon on the spatial screen of the head-mounted display device.

[0098] After obtaining the target movement information of the external positioning device 102, the image rendering thread can render the corresponding location icon of the external positioning device 102 based on the target movement information. The rendered location icon is then sent to the head-mounted display device 106, enabling the head-mounted display device 106 to display the rendered location icon on its spatial screen. It should be understood that the rendering method here can be implemented using the rendering method in the image rendering engine (such as Unity3D) used by the image rendering thread, or other methods can be used; no single limitation is made here. It is understood that the image rendering thread can render image frames displayed on the spatial screen; therefore, the aforementioned location icon can be displayed on the image frame rendered by the image rendering thread, and then sent to the spatial screen for display. Thus, the location icon can be displayed on the image frame displayed on the spatial screen. For example, the layer containing the location icon can be overlaid on the layer containing the image frame, thereby achieving the goal of displaying the location icon on the image frame.

[0099] Based on this, through the optional implementation described above, the target motion information generated during the adjacent frame image can be determined by using the current motion information of the external positioning device 102 and at least one historical motion information. This allows for the determination of the motion of the corresponding location icon in the adjacent frame image (the motion can include both position and direction dimensions). This solution improves the problem of inaccurate rendering of the location icon using the current motion information, which is caused by the difference between the motion acquisition frequency and the image rendering frequency of the external positioning device 102. It effectively avoids the omission or loss of at least some motion information during the motion acquisition of the external positioning device 102 during location icon rendering, thus ensuring the accuracy of the motion when the location icon rendered based on the target motion information is displayed on the spatial screen. This improves the display effect of the location icon of the external positioning device 102 on the spatial screen of the head-mounted display device 106, meeting the user's needs for using the head-mounted display device 106 and effectively enhancing the user experience. Optionally, this scheme can also acquire the movement information of the external positioning device 102 collected within the time range of adjacent frame images, and determine it as the target movement information for calculating the position icon. This allows the calculation of the movement of the position icon in the current frame image relative to the previous frame image, thereby determining the accurate position of the position icon in the current frame image.

[0100] It is understandable that, in this technical solution, when rendering each frame of an image using an image rendering thread, the target movement information can be determined by the current movement information of the external positioning device 102 and at least one historical movement information to render the position icon. Therefore, it can be ensured that each frame of the rendered image uses the movement information generated by the external positioning device 102 from the previous frame to the current frame, thus ensuring the accuracy of the position icon of the external positioning device 102 in the rendered image. For example, if the acquisition frequency of the movement data of the external positioning device 102 differs from the image rendering frequency of the head-mounted display device 106, the movement data of the external positioning device 102 can be stored locally during the rendering of adjacent frames. This avoids the omission or loss of the acquired movement data. When the image rendering thread renders image frames, it ensures that the movement data acquired by the external positioning device 102 since the end of the previous frame is reflected in the movement of the position icon in the image frame rendered by the image rendering thread. This improves the accuracy of the display position of the position icon of the external positioning device 102 on the spatial screen of the head-mounted display device 106, meeting the user's need for accurate control of the head-mounted display device 106 and enhancing the user experience. This technical solution effectively achieves the adaptation between the head-mounted display device and the external positioning device, enabling the external positioning device to accurately control the rendered content on the spatial screen of the head-mounted display device, thereby improving the user experience.

[0101] In some alternative embodiments, refer to Figure 6 As shown in the flowchart, step S208 above can be implemented by the following steps S2082 to S2084 to render the location icon, specifically:

[0102] S2082: Construct a spatial screen for a head-mounted display device and set a screen identifier for the spatial screen.

[0103] A virtual spatial screen is constructed for the head-mounted display device 106, and a screen identifier is set for the spatial screen. The constructed spatial screen is invoked through the screen identifier, thereby rendering the image to be displayed on the spatial screen, including image frames and location icons.

[0104] Screen logos can take any form, such as using Arabic numerals, English letters, etc.

[0105] In some alternative implementations, the head-mounted display device 106 and the controller 104 are integrated into one unit, which allows the spatial screen to be assigned a screen identifier indicating that it is the main screen, meaning that the spatial screen can be used as the main screen when displaying subsequent images. For example, an identifier of 1 indicates that it is the main screen.

[0106] In some alternative embodiments, the head-mounted display device 106 and the controller 104 are separate structures. The controller 104 is located on the host terminal 108, which also includes a system screen. The host terminal 108 can construct a secondary screen different from the main screen (or system screen) and send location icons, etc., to the constructed secondary screen. This secondary screen can be considered as the aforementioned spatial screen. That is, in subsequent display screens, the secondary screen can be used as a spatial screen, while the main screen of the host terminal 108 corresponds to the system screen. Optionally, a screen identifier indicating that the spatial screen is the main screen can be set, and a screen identifier indicating that the system screen is the secondary screen can be set. That is, in subsequent display screens, the main screen can be used as a spatial screen, while the secondary screen is used as the system screen of the host terminal 108. For example, an identifier of 1 indicates the main screen, and an identifier of 2 indicates the secondary screen.

[0107] Of course, the way the screen identifiers are represented and set here are just examples and are not intended to limit the embodiments of this application.

[0108] S2084: Render the location icon using the image rendering thread based on the target movement information.

[0109] Here, the location icon can be rendered using the image rendering engine (such as Unity3D) used by the image rendering thread, or it can be rendered in other ways; there is no single limitation.

[0110] Optionally, based on steps S2082 to S2084, step S210 may include: displaying the rendered location icon on the spatial screen based on the screen identifier.

[0111] For example, the rendered location icon can be displayed on the spatial screen indicated by the screen identifier to avoid the rendered location icon being displayed in the wrong position (e.g., it should be displayed on the spatial screen but is displayed on the system screen).

[0112] For example, assuming the screen identifier of the space screen is 1, the location icon can be set to be displayed on the screen with screen identifier 1, so that the rendered location icon can be accurately displayed on the space screen.

[0113] Based on this, the above-described embodiments in this application can effectively ensure the accuracy of the rendered position icons and the accuracy of the movement amount when the rendered position icons are displayed on the constructed spatial screen by constructing a spatial screen and displaying the position icons rendered according to the target movement amount information on the constructed spatial screen based on the screen identifier. This improves the display effect of the position icons of the external positioning device 102 on the spatial screen of the head-mounted display device 106, meets the user's needs for using the head-mounted display device 106, and effectively improves the user experience.

[0114] In some optional embodiments, after determining the target mobility information of the external positioning device based on the current mobility information and historical mobility information, the display method of this application further includes clearing the mobility information stored in local storage. For example, both the current mobility information and all historical mobility information in local storage can be cleared.

[0115] Based on this, on the one hand, it can avoid interference from already used motion information when rendering the position icon of the external positioning device 102 in the next frame, and more accurately and quickly calculate the target motion information during the rendering of each frame. On the other hand, clearing the motion information cache can also reduce the storage pressure on local storage, which is beneficial to improving the system's operating speed.

[0116] In some optional embodiments, the display method of this application further includes: obtaining the button information of the external positioning device 102 from local storage.

[0117] Optionally, taking a mouse as an example, the button information may include at least one of the following: the button value information of the left and / or right mouse buttons (e.g., the number of times the button is pressed), release status information (i.e., up status), press status information (i.e., down status), and cancel status information (i.e., cancel status).

[0118] Optionally, taking the external positioning device 102 as a mouse as an example, when a change in the mouse button state is detected, a mouse button event can be dispatched, and the button information of the external positioning device 102 can be obtained through the mouse button event. For example, the mouse button event (MotionEvent) can be dispatched by calling the dispatchTouchEvent function through the operating system (such as Android system). The button value information of the left and / or right mouse buttons (e.g., denoted as buttonState) can be obtained through the button value acquisition function (e.g., getButtonState function) in the mouse button event. At least one of the button release state information (i.e., up state), press state information (i.e., down state), and cancel state information (i.e., cancel state) can be obtained through the action acquisition function (e.g., getAction function).

[0119] Optionally, the obtained button information can be saved to local storage for later use. This local storage can be a local cache of the image rendering thread. For example, the button value information stored in local storage can be denoted as `localButton`. Therefore, after collecting the button value information `buttonState`, it can be sent to the image rendering thread (e.g., an image rendering engine like Unity3D), and the image rendering thread can then save `buttonState` as `localButton`. Similarly, the information obtained from the action function (e.g., the `getAction` function) in the button information stored in local storage can be denoted as `localAction`. Therefore, each time at least one of the button's release state information (i.e., up state), press state information (i.e., down state), or cancel state information (i.e., cancel state) is obtained, it can be sent to the image rendering thread (e.g., an image rendering engine like Unity3D), and the image rendering thread can then save it as `localAction`.

[0120] Optionally, in step S208 above, when rendering the position icon using the image rendering thread, the position icon can be rendered based on the key information and target movement information. This allows the rendered position icon displayed on the spatial screen of the head-mounted display device to be used to control the display interface of the spatial screen according to the key information. It is understood that key information generated between adjacent frames can be stored locally. Therefore, when rendering the current frame, key information generated from the previous frame to the current moment can be retrieved and corresponding instructions executed, thus avoiding the omission of key operations from the external positioning device. For example, controlling the display interface of the spatial screen can involve manipulating one or more interface elements displayed on the screen. These elements include, but are not limited to, keys, images, and video elements. Corresponding manipulations can include, but are not limited to, triggering keys (e.g., left / right mouse button trigger, double-click trigger, etc.), zooming in or out of the image display (e.g., double-clicking with the left / right mouse button to zoom in while the image is zoomed out, and double-clicking again to zoom out, etc.), and playing videos (e.g., left / right mouse button to play, double-clicking to play, etc.). Alternatively, the display interface of the space screen can be controlled, or the entire display interface can be controlled, such as page turning, brightness control, etc. Other feasible methods for controlling the display interface of the space screen are also possible; this application does not limit this to a single method.

[0121] Based on this, by acquiring the button information of the external positioning device 102, and then rendering the position icon using an image rendering thread according to the button information and target movement information, the position icon displayed on the spatial screen of the head-mounted display device can be used to control the display interface of the spatial screen according to the button information. This allows the user to accurately move the position icon in the spatial screen using the external positioning device 102, and flexibly and accurately control the display interface of the spatial screen of the head-mounted display device 106, thereby meeting the user's operation needs of the head-mounted display device 106 and effectively improving the user experience.

[0122] In some optional embodiments, the display method in this application may further include: acquiring the roller information of the external positioning device 102, and controlling the display interface of the spatial screen according to the roller information.

[0123] Taking an external positioning device 102 as a mouse as an example, the scroll information can include the rotation angle information of the mouse wheel. Optionally, when a change in the scroll state of the external positioning device 102 is detected, the scroll information of the external positioning device 102 (e.g., denoted as scrollValue) can be obtained by calling the dispatchGenericMotionEvent function and the getAxisValue(MotionEvent.AXIS_VSCROLL) function in the dispatchTouchEvent function through the operating system. Optionally, the obtained scroll information can be saved to local storage for later use. The local storage here can be the local cache of the image rendering thread. For example, the scroll information stored in local storage can be denoted as localScrollValue. After the scroll information scrollValue is collected and sent to the image rendering thread (e.g., an image rendering engine such as Unity3D), the image rendering thread will save scrollValue as localscrollValue.

[0124] The specific methods for controlling the display interface of the spatial screen via scroll information are not limited here. As examples, this may include, but is not limited to, scaling the display interface, swiping up and down on the display interface, adjusting the distance of the spatial screen relative to the user, and controlling the brightness, font, playback progress, etc.

[0125] It should be understood that by controlling the display interface of the spatial screen through the roller information of the external positioning device 102, the sliding and other operations of the display interface of the spatial screen of the head-mounted display device 106 can be effectively controlled, thereby helping to meet the user's operation needs when using the head-mounted display device 106 and effectively improving the user experience.

[0126] In some optional embodiments, the display method of this application may further include: determining the deflection information of the position icon of the external positioning device displayed on the spatial screen of the head-mounted display device in space based on the target movement information, wherein the deflection information includes at least one of deflection direction information and deflection angle information; rendering the position icon based on the deflection information to send the position icon to the spatial screen for display.

[0127] Based on this, through the optional implementation described above, when rendering the position icon of the external positioning device 102, the position icon can be rendered using deflection information determined by the target movement information of the external positioning device 102, including at least one of deflection direction information and deflection angle information. Therefore, the position icon can adaptively deflect according to the deflection information, adapting to spatial screens of various forms and sizes, effectively ensuring the accuracy of the position icon of the external positioning device 102 in the rendered image, meeting the user's operational needs for various scenarios on different types of spatial screens when using the head-mounted display device, and improving the accuracy of the display position of the position icon of the external positioning device 102 on the spatial screen of the head-mounted display device 106, thus effectively improving the user experience.

[0128] In this application, the target movement information of the external positioning device 102 is used to determine the deflection information of the location icon in space, so that when the location icon rendered according to the deflection information is displayed on the spatial screen, it can be adaptively deflected according to the deflection information.

[0129] The target movement information of the external positioning device 102 may include at least a first movement (i.e., a first target movement) along a first direction and a second movement (i.e., a second target movement) along a second direction. Optionally, the first and second target movements may be obtained based on the target movement information to calculate the deflection information of the location icon.

[0130] It is understandable that the target motion information includes not only the current motion information but also historical motion information collected during adjacent frame images. Using this target motion information to calculate the deflection information of the location icon can avoid the omission or loss of at least some motion information, thereby ensuring the accuracy of the deflection information when the subsequently determined location icon is displayed on the spatial screen.

[0131] It should be understood that the optional solutions in steps S2062 to S2066 above can conveniently and effectively superimpose the target movement information, effectively avoiding the omission or loss of at least part of the movement information collected by the external positioning device 102. This ensures the accuracy of the deflection and movement of the location icon rendered based on the target movement information when displayed on the spatial screen, effectively improving the user experience.

[0132] In this application, deflection information can be determined in any manner. In some optional embodiments, deflection information can be determined as follows: based on a preset reference point in space and the target movement information of an external positioning device, the deflection information of the location icon in a preset spatial coordinate system is calculated.

[0133] Therefore, this application can accurately and effectively determine the deflection information of the position icon of the external positioning device 102 in space, so that when the position icon rendered according to the deflection information is displayed on the spatial screen, it can be adaptively deflected according to the deflection information. This allows the position icon of the external positioning device 102 to be accurately displayed on spatial screens of various shapes and sizes, meeting the needs of users operating various types of spatial screens when using head-mounted display devices. It also helps to improve the accuracy of the display position of the position icon of the external positioning device 102 on the spatial screen of the head-mounted display device 106, meeting the needs of users using the head-mounted display device 106 and effectively improving the user experience.

[0134] The preset reference point can be selected as needed, and it can be any point in space that can be used as a reference. Here, the specific location of the preset reference point in space can be determined by a preset spatial coordinate system. In one embodiment, the preset reference point can be a point on a physical target in the preset spatial coordinate system; in another embodiment, the preset reference point can be a point on a virtual target in the preset spatial coordinate system; in yet another embodiment, the preset reference point can be a point on a coordinate axis of the preset spatial coordinate system; and so on.

[0135] Optionally, after selecting a preset reference point, for the determined target movement information, the reference point can be used as a reference point (the position of the reference point in space remains unchanged) to determine the deflection information of the corresponding position icon relative to the reference point when the external positioning device 102 moves according to the target movement information.

[0136] Optionally, after selecting a preset reference point, it can be understood as creating an empty object based on the preset reference point. A parent-child relationship can be established between the empty object at the preset reference point and the position icon, designating the empty object at the preset reference point as the parent object of the position icon, and the position icon as the child object. The deflection information of the child object can be determined by adjusting the parent object.

[0137] In some optional embodiments, the deflection information of the position icon in the preset spatial coordinate system can be calculated through the following steps S302 to S304:

[0138] S302: Based on the target movement information, determine the first target movement along the first direction and the second target movement along the second direction of the external positioning device.

[0139] After obtaining the target movement information, the first target movement Δx and the second target movement Δy can be determined from the target movement information.

[0140] S304: Based on a preset reference point, determine the second deflection angle of the position icon around the second preset direction in the preset spatial coordinate system according to the first target movement amount of the external positioning device along the first direction, and determine the first deflection angle of the position icon around the first preset direction in the preset spatial coordinate system according to the second target movement amount of the external positioning device along the second direction.

[0141] In this embodiment of the application, the second preset direction intersects with the first preset direction.

[0142] Based on this, this application, based on a preset reference point, and according to the first target movement amount of the external positioning device 102 along the first direction and the second target movement amount along the second direction, can respectively determine the second deflection angle of the position icon around the second preset direction and the first deflection angle around the first preset direction, thereby accurately and effectively obtaining deflection information. This allows the position icon rendered by the subsequent deflection information to be adaptively deflected when displayed on the spatial screen, enabling the position icon of the external positioning device 102 to be accurately displayed on spatial screens of various shapes and sizes. This meets the diverse scenario needs of users operating various types of spatial screens when using head-mounted display devices, and helps improve the accuracy of the display position of the position icon of the external positioning device 102 on the spatial screen of the head-mounted display device 106, thus meeting the needs of users using the head-mounted display device 106 and effectively improving the user experience.

[0143] Optionally, the first target movement of the external positioning device along the first direction can be converted into the rotation of the parent object (an empty object with a preset reference point) in the second preset direction, and the second target movement of the external positioning device along the second direction can be converted into the rotation of the parent object in the first preset direction, so as to determine the second deflection angle of the child object (position icon) driven by the parent object in the second preset direction and the first deflection angle of the child object in the first preset direction.

[0144] Optionally, after selecting a preset reference point, for the determined target movement information, the reference point can be used as a reference point (the position of the reference point in space remains unchanged). When the external positioning device 102 moves according to the target movement information, the deflection information of the corresponding position icon relative to the reference point can be the deflection information of the position icon in space.

[0145] Optionally, the second preset direction and the first preset direction can be perpendicular to each other, which makes it easier to determine the first deflection angle and the second deflection angle.

[0146] Optionally, the second preset direction and the first preset direction can be two coordinate axes of a preset spatial coordinate system. For example, the second preset direction can be the y-axis of the preset spatial coordinate system, while the first preset direction can be the x-axis of the preset spatial coordinate system. Using two coordinate axes makes it easier to determine the first deflection angle and the second deflection angle.

[0147] In some optional embodiments, the second deflection angle θ2 can be determined according to the formula θ2=Δx*a1, where Δx can represent the first target movement amount of the external positioning device 102 along the first direction, and a1 represents the first unit deflection angle corresponding to the unit movement amount of the external positioning device 102 along the first direction.

[0148] As mentioned above, the first and second movement amounts of the movement information in this application can be expressed in pixels of the system screen. Optionally, the unit movement amount of the external positioning device 102 along the first direction can refer to the movement amount of one system screen pixel of the external positioning device 102 along the first direction.

[0149] The first unit deflection angle a1 can be preset as needed. For example, assuming that each p pixels (e.g., the width of the system screen can be p pixels) corresponds to a deflection angle of q degrees, then the first unit deflection angle can be set as a1 = q / p degrees. For example, using the example of p = 1080 and q = 20, the first unit deflection angle can be set as a1 = q / p = 20 / 1080 degrees.

[0150] It should be understood that by determining the product of the first target movement Δx and the first unit deflection angle a1 as the second deflection angle θ2 in the above manner, the second deflection angle θ2 can be accurately determined. This ensures that the deflection of the position icon rendered based on the deflection information in the preset spatial coordinate system around the second preset direction is adapted to the actual movement of the external positioning device along the first direction. As a result, when the position icon is displayed on the spatial screen, it can adaptively deflect according to the deflection information. This helps to ensure the accuracy of the position icon of the external positioning device 102 in the rendered image, meets the needs of users operating various types and sizes of spatial screens when using head-mounted display devices, and improves the accuracy of the display position of the position icon of the external positioning device 102 on the spatial screen of the head-mounted display device 106. This also meets the needs of users using the head-mounted display device 106 and effectively improves the user experience.

[0151] In some optional embodiments, the first deflection angle θ1 can be determined according to the formula θ1=Δy*a2, where Δy can represent the second target movement amount of the external positioning device 102 along the second direction, and a1 represents the second unit deflection angle corresponding to the unit movement amount of the external positioning device 102 along the second direction.

[0152] As mentioned above, the first and second movement amounts of the movement amount information in this application can be expressed in pixels of the system screen. Optionally, the unit movement amount of the external positioning device 102 along the second direction can refer to the movement amount of one system screen pixel of the external positioning device 102 along the second direction.

[0153] The second unit deflection angle a2 can be preset as needed. The second unit deflection angle a2 can be set to be the same as the first unit deflection angle a1, or it can be set to be different from the first unit deflection angle a1. There is no single limitation here.

[0154] It should be understood that by determining the product of the second target movement Δy and the second unit deflection angle a2 as the first deflection angle θ1 in the above manner, the first deflection angle θ1 can be accurately determined. This ensures that the deflection of the position icon rendered based on the deflection information in the preset spatial coordinate system around the first preset direction is adapted to the actual movement of the external positioning device along the second direction. As a result, when the position icon is displayed on the spatial screen, it can adaptively deflect according to the deflection information. This helps to ensure the accuracy of the position icon of the external positioning device 102 in the rendered image, meets the needs of users operating various types and sizes of spatial screens when using head-mounted display devices, and improves the accuracy of the display position of the position icon of the external positioning device 102 on the spatial screen of the head-mounted display device 106. This meets the needs of users using the head-mounted display device 106 and effectively improves the user experience.

[0155] In this application, after obtaining the deflection information, a location icon corresponding to the external positioning device 102 can be rendered based on the deflection information. Then, the rendered location icon is sent to the head-mounted display device 106, so that the head-mounted display device 106 can display the rendered location icon on its spatial screen.

[0156] It is understandable that, since the target movement information can be determined by the current movement information and at least one historical movement information of the external positioning device 102 when rendering each frame of the image using the image rendering thread, and then the deflection information of the position icon can be obtained for rendering, it can be ensured that each frame of the rendered image uses the movement information generated by the external positioning device 102 from the previous frame to the current frame, thereby ensuring the accuracy of the position icon of the external positioning device 102 in the rendered image. For example, if the motion acquisition frequency of the external positioning device 102 differs from the image rendering frequency of the head-mounted display device 106, the motion of the external positioning device 102 can be stored locally during the rendering of adjacent frames. This prevents the omission or loss of acquired motion information. When the image rendering thread renders image frames, it ensures that the motion information acquired by the external positioning device 102 since the end of the previous frame is reflected in the deflection and movement of the position icon in the image frame rendered by the image rendering thread. This improves the accuracy of the position icon's display position on the spatial screen of the head-mounted display device 106, meeting the user's needs and effectively enhancing the user experience. This technical solution effectively achieves compatibility between the head-mounted display device and the external positioning device, enabling the external positioning device to accurately control the rendered content on the spatial screen of the head-mounted display device, thereby improving the user experience.

[0157] In some alternative embodiments, the location icon can be rendered to the current frame rendering image using an image rendering thread based on the deflection information, so that the current frame rendering image containing the location icon can be displayed on the spatial screen of the head-mounted display device.

[0158] It should be understood that the rendering method here can be implemented using the rendering methods of an image rendering engine (such as Unity3D), or other methods; there is no single limitation. It is understood that the image rendering thread can render image frames displayed on the spatial screen, therefore the aforementioned position icon can be displayed on the image frames rendered by the image rendering thread. Thus, the position icon can be accurately displayed on the image frames displayed on the spatial screen.

[0159] Therefore, through the above implementation method, the image rendering engine can be used to accurately and effectively render and display the position icon of the external positioning device 102 based on the deflection information.

[0160] In some optional embodiments, the display method in this application further includes: obtaining the depth information of the interface element where the location icon is located. Optionally, when rendering and displaying the location icon, the location icon can be rendered based on the deflection information and depth information to send the location icon to the spatial screen for display.

[0161] Each interface element in the spatial screen can possess depth information. Optionally, the depth of the spatial screen may vary depending on the scenario. Therefore, this application obtains the depth information of the interface element containing the location icon, and then renders the location icon based on the deflection and depth information. This effectively ensures the accuracy of the location icon of the external positioning device 102 in the rendered image, enabling the location icon to be used to accurately trigger or manipulate interface elements. This benefits the user experience by meeting various scenario requirements for manipulating interface elements on the spatial screen using a head-mounted display device.

[0162] Optionally, when rendering the location icon based on the deflection information and depth information, the depth information of the location icon of the external positioning device 102 can be set to be the same as the depth information of the obtained interface element. This allows the rendered location icon to trigger the interface element based on the user's operation of the external positioning device 102.

[0163] Optionally, based on the deflection and depth information, an image rendering thread can be used to render the position icon to the current frame rendered image, and the current frame rendered image containing the position icon can be displayed on the spatial screen of the head-mounted display device.

[0164] Optionally, in step S208 above, when rendering the location icon (e.g., using an image rendering thread), the location icon can be rendered using an image rendering thread based on deflection information, key press information, and target movement information. This allows the rendered location icon displayed on the spatial screen of the head-mounted display device to be controlled by the corresponding key press information. This enables the user to accurately control the movement and deflection of the location icon on the spatial screen via the external positioning device 102, which helps meet the user's control needs when using the head-mounted display device 106 and effectively improves the user experience.

[0165] It is understood that the above description is only some optional embodiments of the display method of this application and is not intended to limit the embodiments of this application.

[0166] According to another aspect of the embodiments of the application, a display device for a head-mounted display device is provided. (Refer to...) Figure 7 As shown, the display device 700 for a head-mounted display device includes:

[0167] The acquisition module 702 is used to acquire the current movement information of the external positioning device;

[0168] Module 704 is invoked to call the preset image rendering thread and store the current movement information in local storage;

[0169] The determination module 706 is used to determine the target movement information of the external positioning device based on the current movement information and the historical movement information if the local storage includes at least one historical movement information of the external positioning device.

[0170] Rendering module 708 is used for target movement information and uses an image rendering thread to render the location icon of the external positioning device.

[0171] Display module 710 is used to display the rendered location icon on the spatial screen of the head-mounted display device.

[0172] In some optional embodiments, the rendering module 708 is specifically used to: construct a spatial screen for the head-mounted display device and set a screen identifier for the spatial screen; and render a location icon using an image rendering thread based on the target movement information; the display module 710 is specifically used to: display the rendered location icon on the spatial screen based on the screen identifier.

[0173] In some optional embodiments, the movement amount information includes a first movement amount along a first direction and a second movement amount along a second direction; the determining module 706 is specifically used to: superimpose the first movement amount in the current movement amount information and the first movement amount in the historical movement amount information to obtain a first target movement amount; superimpose the second movement amount in the current movement amount information and the second movement amount in the historical movement amount information to obtain a second target movement amount; and determine the first target movement amount and the second target movement amount as target movement amount information.

[0174] In some optional embodiments, after the determining module 706 determines the target movement information of the external positioning device based on the current movement information and the historical movement information, the display device 700 is further configured to: clear the movement information stored in the local storage.

[0175] In some optional embodiments, the acquisition module 702 is specifically used for: dispatching a movement event of the external positioning device in response to determining that movement has occurred in the external positioning device; and acquiring current movement information based on the movement event.

[0176] In some optional embodiments, the local storage also includes key information of the external positioning device; the display device 700 is further configured to: retrieve key information of the external positioning device from the local storage; execute key commands indicated by the key information for the position icons of the interface displayed on the spatial screen of the head-mounted display device; and update the content displayed on the spatial screen of the head-mounted display device using an image rendering thread.

[0177] In some optional embodiments, the external positioning device is communicatively connected to the host device, and the display device 700 is further configured to: after the external positioning device is communicatively connected to the host device, control the system screen of the host device to enter a screen lock state, and hide the pointer icon on the system screen corresponding to the external positioning device; wherein, if the system screen enters the screen lock state, at least one interface element on the system screen is in a state that cannot be triggered by the external positioning device.

[0178] In some optional embodiments, the display device 700 is further configured to: render a touch panel on a system screen, the touch panel being used to control the display of a head-mounted display device; and, in response to receiving an interface lock operation from a user, control the touch panel interface to enter an interface lock state to lock the touch panel interface on the system screen.

[0179] The display device 700 for a head-mounted display device provided in this application embodiment is based on the same inventive concept as the aforementioned display method for a head-mounted display device. It corresponds to the respective display methods for a head-mounted display device in the foregoing multiple method embodiments and has the beneficial effects of the corresponding display method embodiments for a head-mounted display device; therefore, it will not be described again here. Furthermore, the implementation of each module in the display device 700 for a head-mounted display device in this application embodiment can refer to the description of the corresponding parts in the foregoing display method embodiments for a head-mounted display device, and will not be described again here.

[0180] According to another aspect of the embodiments of this application, an electronic device is provided, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store a computer program; the processor is used to execute the display method for a head-mounted display device described above by running the computer program stored in the memory.

[0181] Reference Figure 8 This illustration shows a structural schematic diagram of an electronic device according to an embodiment of this application. The embodiments of this application do not limit the specific implementation of the electronic device. Figure 8 As shown, the electronic device 800 may include: a processor 802, a communications interface 804, a memory 806, and a communications bus 808.

[0182] in:

[0183] The processor 802, communication interface 804, and memory 806 communicate with each other through the communication bus 808.

[0184] Communication interface 804 is used to communicate with other electronic devices or servers.

[0185] The processor 802 is used to execute the computer program 810, specifically the relevant steps in the above-described embodiment of the display method for a head-mounted display device.

[0186] Specifically, computer program 810 may include program code that includes computer operation instructions.

[0187] The processor 802 may be a CPU, a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0188] Memory 806 is used to store computer program 810. Memory 806 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0189] The computer program 810 may include multiple computer instructions. Specifically, the computer program 810 may use multiple computer instructions to cause the processor 802 to perform the operation corresponding to the display method for the head-mounted display device described in any of the foregoing multiple method embodiments.

[0190] The specific implementation of each step in computer program 810 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0191] According to another aspect of the embodiments of this application, an embodiment of this application also provides a computer storage medium storing a computer program thereon. When executed by a processor, the computer program implements the display method for a head-mounted display device described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk, etc.

[0192] According to another aspect of the embodiments of this application, the embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the display method for a head-mounted display device described in any of the foregoing method embodiments.

[0193] The display device 700 / display system 100 / electronic device 800 / computer storage medium / computer program product embodiment for head-mounted display device in this application has been described in detail in the foregoing display method embodiment for head-mounted display device. Therefore, its related content and beneficial effects can be understood by referring to the above method embodiment, and will not be repeated here.

[0194] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0195] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application. It should be understood that the various technical features in the technical solutions of the embodiments of this application can be combined in any suitable manner.

[0196] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0197] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0198] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

[0199] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". It should be noted that the concepts of "first", "second", etc., mentioned in the embodiments of this application are only used to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules, or units or their interdependencies. It should be noted that the modifications of "a" and "a plurality" mentioned in the embodiments of this application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of this application, and are not intended to limit them; although the embodiments of this application have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A display method for a head-mounted display device, comprising: Obtain the current movement information of the external positioning device; The preset image rendering thread is invoked to store the current movement information in local storage; If the local storage includes at least one historical movement information of the external positioning device, the target movement information of the external positioning device is determined based on the current movement information and the historical movement information. Based on the target movement information, the image rendering thread renders the location icon of the external positioning device; The rendered location icon is displayed on the spatial screen of the head-mounted display device.

2. The method according to claim 1, wherein, The step of rendering the location icon of the external positioning device using the image rendering thread based on the target movement information includes: Construct a spatial screen for the head-mounted display device, and set a screen identifier for the spatial screen; Based on the target movement information, the location icon is rendered using the image rendering thread; The step of displaying the rendered location icon on the spatial screen of the head-mounted display device includes: displaying the rendered location icon on the spatial screen based on the screen identifier.

3. The method according to claim 1, wherein, The movement information includes a first movement along a first direction and a second movement along a second direction; Determining the target mobility information of the external positioning device based on the current mobility information and the historical mobility information includes: The first movement amount in the current movement amount information and the first movement amount in the historical movement amount information are superimposed to obtain the first target movement amount; The second movement amount in the current movement amount information and the second movement amount in the historical movement amount information are superimposed to obtain the second target movement amount; The first target movement amount and the second target movement amount are determined as the target movement amount information.

4. The method according to any one of claims 1-3, wherein, After determining the target motion information of the external positioning device based on the current motion information and the historical motion information, the method further includes: Clear the movement information stored in the local storage.

5. The method according to any one of claims 1-3, wherein, The acquisition of the current movement information of the external positioning device includes: In response to determining that movement has occurred in the external positioning device, a movement event of the external positioning device is dispatched; Based on the movement event, the current movement amount information is obtained.

6. The method according to any one of claims 1-3, wherein, The local storage also includes the button information of the external positioning device; the method further includes: Retrieve the button information of the external positioning device from the local storage; For the position icon of the interface displayed on the spatial screen of the head-mounted display device, execute the button command indicated by the button information; The image rendering thread is used to update the content displayed on the spatial screen of the head-mounted display device.

7. The method according to any one of claims 1-3, wherein, The external positioning device is communicatively connected to the host computer, and the method further includes: After the external positioning device establishes a communication connection with the host, the system screen of the host is locked, and the pointer icon corresponding to the external positioning device on the system screen is hidden. If the system screen enters a screen lock state, at least one interface element on the system screen is in a state that cannot be triggered by the external positioning device.

8. The method according to claim 7, wherein, The method further includes: A touch panel is rendered on the system screen, and the touch panel is used to control the display of the head-mounted display device; In response to receiving a user's interface lock operation, the touch panel interface is controlled to enter an interface lock state to lock the touch panel interface on the system screen.

9. A display device for a head-mounted display device, comprising: The acquisition module is used to acquire the current movement information of the external positioning device; The calling module is used to call a preset image rendering thread to store the current movement information in local storage; The determination module is configured to determine the target movement information of the external positioning device based on the current movement information and the historical movement information if the local storage includes at least one historical movement information of the external positioning device; The rendering module is used to render the location icon of the external positioning device using the image rendering thread, based on the target movement information. The display module is used to display the rendered location icon on the spatial screen of the head-mounted display device.

10. A display system for a head-mounted display device, comprising: An external positioning device is configured to move in response to user actions; The controller is configured to acquire the current movement information of the external positioning device; A preset image rendering thread is invoked to store the current movement information in local storage; if the local storage includes at least one historical movement information of the external positioning device, the target movement information of the external positioning device is determined based on the current movement information and the historical movement information; based on the target movement information, the location icon of the external positioning device is rendered using the image rendering thread. The head-mounted display device is configured to display rendered location icons on a spatial screen.

11. The system according to claim 10, wherein, The controller is further configured to: In response to changes in the spatial distribution of the controller, the control cursor on the spatial screen of the head-mounted display device is moved; and / or, In response to detecting a preset operation by the user on the controller, the display interface of the space screen is updated.

12. An electronic device, comprising: The processor, the communication interface, the memory, and the communication bus are provided, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The memory is used to store computer programs; The processor is configured to perform the method of any one of claims 1-8 by running the computer program stored in the memory.

13. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-8.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.

Images