Information display method and device, head-mounted display equipment and storage medium

Abstract

The embodiment of the invention provides an information display method and device, head-mounted display equipment and a storage medium, and belongs to the field of information display. The method comprises the following steps: acquiring three-dimensional position information of input equipment relative to head-mounted display equipment; according to the three-dimensional position information, determining a target display area of an information input window of the head-mounted display device in a display screen of the head-mounted display device, and displaying the information input window in the target display area; and in response to an information input operation sent by the input device, displaying input information corresponding to the information input operation in the information input window. According to the technical scheme provided by the embodiment of the invention, the information input convenience is improved.

Description

Technical Field

[0001] The present invention relates to the field of information display technology, and in particular to an information display method, device, head-mounted display device and storage medium. Background Art

[0002] Head-mounted display devices can realize functions such as augmented reality (AR) or mixed reality (MR), which can bring users a better visual experience. However, existing head-mounted display devices are not convenient for users to input information, and the user experience is not good. Summary of the invention

[0003] Embodiments of the present invention provide an information display method, an apparatus, a head-mounted display device, and a storage medium, aiming to improve the convenience of information input.

[0004] In a first aspect, an embodiment of the present invention provides an information display method, which is applied to a head mounted display device, wherein the head mounted display device is communicatively connected to an input device, and the method comprises:

[0005] Acquire three-dimensional position information of the input device relative to the head mounted display device;

[0006] Determine, according to the three-dimensional position information, a target display area of ​​the information input window of the head-mounted display device within the display screen of the head-mounted display device, and display the information input window within the target display area;

[0007] In response to the information input operation sent by the input device, input information corresponding to the information input operation is displayed in the information input window.

[0008] In a second aspect, an embodiment of the present invention further provides an information display device, which is applied to a head mounted display device, wherein the head mounted display device is communicatively connected to an input device, and the information display device comprises:

[0009] A position acquisition module, configured to acquire three-dimensional position information of the input device relative to the head mounted display device;

[0010] an area determination module, configured to determine a target display area of ​​the information input window of the head mounted display device within the display screen of the head mounted display device according to the three-dimensional position information;

[0011] A window display module, configured to display the information input window in the target display area;

[0012] The operation response module is configured to respond to the information input operation sent by the input device and display the input information corresponding to the information input operation in the information input window.

[0013] In a third aspect, an embodiment of the present invention further provides a head-mounted display device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the information display method as described in the first aspect is realized.

[0014] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the information display method as described in the first aspect.

[0015] An embodiment of the present invention provides an information display method, an apparatus, a head-mounted display device and a storage medium. The method determines a target display area of ​​an information input window of a head-mounted display device within a display screen of the head-mounted display device through three-dimensional position information of an input device relative to the head-mounted display device, displays the information input window within the target display area, and then responds to an information input operation sent by the input device to display input information corresponding to the information input operation in the information input window. In this way, it can be ensured that the information input window displayed by the head-mounted display device and the input device are both within the user's viewing angle, which facilitates the user to input information and greatly improves the convenience of information input and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is a schematic diagram of a scenario for implementing the information display method provided by an embodiment of the present invention;

[0018] Figure 2 is a flow chart of an information display method provided by an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of a three-dimensional coordinate system in an embodiment of the present invention;

[0020] Figure 4 yes Figure 2 A schematic flow chart of sub-steps of the information display method;

[0021] Figure 5 is a schematic diagram of a scene of a target display area in an embodiment of the present invention;

[0022] Figure 6 yes Figure 2 A schematic flow chart of another sub-step of the information display method;

[0023] Figure 7 is another schematic diagram of a target display area in an embodiment of the present invention;

[0024] Figure 8 is another schematic diagram of a scene of a target display area in an embodiment of the present invention;

[0025] Fig. 9 is a schematic structural block diagram of an information display device provided by an embodiment of the present invention;

[0026] Fig.10 The present invention is a schematic structural block diagram of a head mounted display device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0029] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0030] Head-mounted display devices can realize functions such as augmented reality (AR) or mixed reality (MR), which can bring users a better visual experience. However, existing head-mounted display devices are not convenient for users to input information, and the user experience is not good.

[0031] To solve the above problems, embodiments of the present invention provide an information display method, apparatus, head-mounted display device and storage medium. The method determines the target display area of ​​the information input window of the head-mounted display device within the display screen of the head-mounted display device through the three-dimensional position information of the input device relative to the head-mounted display device, and displays the information input window in the target display area. Then, in response to the information input operation sent by the input device, the input information corresponding to the information input operation is displayed in the information input window. In this way, it can be ensured that the information input window displayed by the head-mounted display device and the input device are both in the user's viewing angle, which is convenient for the user to input information, thereby greatly improving the convenience of information input and user experience.

[0032] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0033] See also Figure 1 , Figure 1 It is a schematic diagram of a scenario for implementing the information display method provided by an embodiment of the present invention.

[0034] like Figure 1 As shown, the head mounted display device 100 is connected to the input device 200 for communication, and the head mounted display device 100 is worn on the user's head 11, and the head mounted display device 100 includes a display screen 101. When executing the information display method provided in the embodiment of the present invention, the head mounted display device 100 obtains the three-dimensional position information of the input device 200 relative to the head mounted display device 100; according to the three-dimensional position information, the target display area of ​​the information input window of the head mounted display device 100 in the display screen 101 is determined, and the information input window is displayed in the target display area; in response to the information input operation sent by the input device 200, the input information corresponding to the information input operation is displayed in the information input window. Among them, the display screen 101 can be a transparent display screen.

[0035] In some embodiments, the head mounted display device 100 and the input device 200 may establish a communication connection via Bluetooth or mobile wireless interconnection (wireless fidelity, wifi), etc. The head mounted display device 100 may include augmented reality (AR) glasses, AR helmets, mixed reality (MR) glasses or MR helmets, etc. The input device 200 may include a smart phone, a tablet computer, a laptop computer, a wired keyboard or a wireless keyboard, etc.

[0036] In some embodiments, the head mounted display device 100 includes a camera ( Figure 1(not shown), the shooting device is used to shoot the input device 200 to obtain corresponding image data, so that the head mounted display device 100 can use the preset position estimation algorithm and the image data obtained by shooting the input device 200 by the shooting device to determine the three-dimensional position information of the input device 200 relative to the head mounted display device 100. Among them, the shooting device can be a depth camera or a binocular vision module, etc.

[0037] In some embodiments, the head mounted display device 100 includes a sensor ( Figure 1 (not shown). For example, the sensor for collecting the posture data of the head mounted display device 100 may include an inertial measurement unit (IMU), and the inertial measurement unit may include an accelerometer, a gyroscope and / or a magnetometer. For example, the inertial measurement unit includes a three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer.

[0038] The following will be combined Figure 1 The information display method provided by the embodiment of the present invention is described in detail in the scenario in FIG. Figure 1 The scenario is only used to explain the information display method provided by the embodiment of the present invention, but does not constitute a limitation on the application scenario of the information display method provided by the embodiment of the present invention.

[0039] See also Figure 2 , Figure 2 It is a flowchart of an information display method provided by an embodiment of the present invention.

[0040] like Figure 2 As shown, the information display method includes steps S101 to S103.

[0041] Step S101: Acquire three-dimensional position information of an input device relative to a head mounted display device.

[0042] In this embodiment, the three-dimensional position information of the input device relative to the head-mounted display device includes the three-dimensional coordinates, direction, speed, depth (the distance of the input device relative to the head-mounted display device) and size of multiple corner points of the input device. The three-dimensional coordinates of the multiple corner points are determined based on the local coordinate system (also called the model coordinate system) of the input device. The local coordinate system of the input device is a spatial coordinate system based on the center of the input device.

[0043] In some embodiments, after the input device is successfully connected to the head-mounted display device, the three-dimensional position information of the input device relative to the head-mounted display device is obtained at preset intervals. Alternatively, after the input device is successfully connected to the head-mounted display device, if an information input event sent by the input device is obtained, the three-dimensional position information of the input device relative to the head-mounted display device is obtained at preset intervals. The preset interval can be set based on actual conditions, and the embodiment of the present invention does not specifically limit this. In this embodiment, by periodically obtaining the three-dimensional position information of the input device relative to the head-mounted display device, the position of the information input window displayed by the head-mounted display device can be periodically adjusted according to the three-dimensional position information to ensure that the information input window displayed by the head-mounted display device and the input device are both in the user's perspective.

[0044] In some embodiments, the posture data of the head-mounted display device is obtained, and the head movement of the user wearing the head-mounted display device is determined based on the posture data; if the head movement is determined to be a preset head movement, the cross-device input function of the head-mounted display device is enabled; after the cross-device input function is enabled, the three-dimensional position information of the input device relative to the head-mounted display device is obtained. In this embodiment, the enabling of the cross-device input function of the head-mounted display device can be controlled by head movement, which greatly improves the control convenience of the head-mounted display device.

[0045] The preset head action can be set based on actual conditions, and the embodiments of the present invention do not specifically limit this. For example, the preset head action is nodding. The posture data of the head-mounted display device can be collected by the inertial measurement unit of the head-mounted display device itself, and the posture data of the head-mounted display device may include the acceleration, angular velocity and orientation information of the head-mounted display device. For example, Figure 3 As shown, a three-axis coordinate system is constructed with the user's head 11 as the coordinate origin, and the three-axis coordinate system takes the positive direction of the Y axis directly in front of the user's head 11, the positive direction of the Z axis directly above the user's head 11, and the positive direction of the X axis directly to the right of the user's head 11. The acceleration of the head-mounted display device 100 may include the acceleration of the head-mounted display device 100 in the X-axis direction, the acceleration in the Y-axis direction, and / or the acceleration in the Z-axis direction, and the angular velocity of the head-mounted display device 100 may include the angular velocity of the head-mounted display device 100 around the X-axis, Y-axis, and / or Z-axis.

[0046] In some embodiments, the head-mounted display device further includes a photographing device, and obtaining the three-dimensional position information of the input device relative to the head-mounted display device may include: obtaining image data obtained by photographing the input device by the photographing device; and determining the three-dimensional position information of the input device relative to the head-mounted display device according to a preset position estimation algorithm and the image data. The preset position estimation algorithm may be set according to actual conditions, and the embodiments of the present invention do not specifically limit this. For example, the preset position estimation algorithm may be a position estimation algorithm based on deep learning.

[0047] Step S102: Determine, based on the three-dimensional position information, a target display area within the display screen of the head mounted display device where the information input window of the head mounted display device is located, and display the information input window within the target display area.

[0048] In this embodiment, the display screen of the head-mounted display device may be a transparent display screen. In this embodiment, the information input window of the head-mounted display device may be determined to be in a target display area within the transparent display screen of the head-mounted display device through the three-dimensional position information of the input device relative to the head-mounted display device, and the information input window may be displayed within the target display area, so that the information input window displayed by the head-mounted display device and the input device are both within the user's viewing angle, so that the user can see the input device in the real world through the transparent display screen, and can also see the information input by the user through the information input window displayed on the transparent display screen, which greatly improves the user experience.

[0049] In some embodiments, Figure 4 As shown, step S102 includes: sub-steps S1021 to S1023.

[0050] Sub-step S1021, acquiring the projection matrix, view matrix and model matrix associated with the input device according to the three-dimensional position information.

[0051] In this embodiment, the model matrix (Model matrix) associated with the input device is used to define the position, rotation and scaling of the input device in the local coordinate system of the input device (also referred to as the model coordinate system) to describe the conversion relationship between the local space (Local Space) of the input device itself and the world space (World Space). The view matrix (View matrix) associated with the input device is used to define the position and direction of the observer relative to the input device in the world coordinate system to describe the conversion relationship between the world space (World Space) and the visual space. The projection matrix (Projection matrix) associated with the input device is used to define the type and range of the projection of the input device in the projection coordinate system to describe the conversion relationship between the visual space and the clipping space (Clip Space).

[0052] Sub-step S1022: determining a projection position of the input device within the display screen of the head mounted display device according to the projection matrix, the view matrix and the model matrix.

[0053] In this embodiment, the projection matrix, the view matrix and the model matrix are multiplied to obtain the position information of multiple vertices of the input device in the clipping space, and then the position information of the multiple vertices of the input device in the clipping space is converted into the position in the screen coordinate system to obtain the projection position of the input device in the display screen of the head-mounted display device, that is, the projection position of the input device in the display screen of the head-mounted display device includes the screen coordinates of the multiple vertices of the input device in the screen coordinate system.

[0054] Sub-step S1023: determining a target display area of ​​the information input window of the head mounted display device within the display screen of the head mounted display device according to the projection position.

[0055] In this embodiment, the target display area can be a partial area within the display area corresponding to the projection position in the display screen of the head-mounted display device, or it can be a partial area within the display area in the display screen of the head-mounted display device excluding the display area corresponding to the projection position. The embodiment of the present invention does not specifically limit this.

[0056] In some embodiments, according to the projection position, the display screen of the head-mounted display device is divided into a first display area and a second display area, the first display area being the display area corresponding to the projection position in the display screen of the head-mounted display device, and the second display area being the display area other than the first display area in the display screen of the head-mounted display device; a target display area is determined in the second display area, and the target display area is located in a preset direction of the first display area. In this embodiment, the target display area of ​​the information input window of the head-mounted display device is determined in the display area other than the projection area corresponding to the input device in the display screen of the head-mounted display device, so that the information input window displayed in the target display area does not block the input device, and ensures that the input device and the information input window are in the same viewing angle of the user, so that the user can use the input device to input information across devices.

[0057] The preset direction can be set based on actual conditions, and the embodiment of the present invention does not specifically limit this. For example, the preset direction is above, below, left or right of the first display area. Figure 5As shown, the user can see the wireless keyboard 210 through the display screen 101 of the head-mounted display device, and the display area corresponding to the projection position of the wireless keyboard 210 in the display screen 101 is the first display area 21, the display area other than the first display area 21 in the display screen 101 is the second display area 22, the target display area 23 is a partial area within the second display area 22, and the target display area 23 is located above the first display area 21. Since the information input window is displayed in the target display area 23, the displayed information input window will not block the wireless keyboard 210 in the real world in the display screen 101, so that the user can input information through the wireless keyboard 210.

[0058] In some embodiments, determining the target display area in the second display area may include: obtaining a first size of the input device, and obtaining a second size matching the first size; determining a target display area of ​​the second size in the second display area, wherein the target display area is in contact with the first display area or the target display area is separated from the first display area by a preset distance. The first size is the size of the input device, and the second size is the size of the display area. A mapping relationship between the size of the input device and the size of the display area may be established in advance, and the size of the target display area may be determined later according to the size of the input device actually used by the user. The preset distance may be set according to actual conditions, and the embodiments of the present invention do not specifically limit this.

[0059] In some embodiments, Figure 6 As shown, step S102 includes: sub-steps S1024 to S1026.

[0060] Sub-step S1024: dividing the display screen of the head mounted display device into a first display area and a second display area according to the projection position, and acquiring a device type of the input device.

[0061] In this embodiment, the device type of the input device may include a first device type or a second device type, the first device type is different from the second device type, the first device type is used to describe that the input device includes a display screen, and the second device type is used to describe that the input device does not include a display screen. For example, if a smartphone includes a display screen, the device type of the smartphone is the first device type, and if a keyboard does not include a display screen, the device type of the keyboard is the second device type.

[0062] Sub-step S1025: when the device type is the first device type, determining a target display area within the first display area.

[0063] In this embodiment, when the device type is the first device type, it can be determined that the input device connected to the head-mounted display device is an input device with a display screen. The input device with a display screen will display a virtual keyboard, and the target display area determined in this embodiment can not block the virtual keyboard displayed by the input device, so that the information input window displayed in the target display area will not block the virtual keyboard, which is convenient for users to input information.

[0064] In some embodiments, determining the target display area in the first display area may include: obtaining an input box position sent by an input device, the input box position being the position of an input box in a selected state in the input device in the display screen of the input device; determining the target position of the input box in the first display area according to the input box position; determining the target display area in the first display area according to the target position, the display area corresponding to the target position in the first display area being a subset of the target display area. In this embodiment, the position of the input box in the display screen of the head-mounted display device can be determined through the input box position sent by the input device, and the target display area including the display area corresponding to the input box can be determined through the position, so that the information input window displayed in the target display area can overlap with the input box displayed by the input device, and the virtual keyboard will not be blocked, which is convenient for users to input information.

[0065] For example, Figure 7 As shown, the user can see the smartphone 220 through the display screen 101 of the head-mounted display device, and the display area corresponding to the projection position of the smartphone 220 in the display screen 101 is the first display area 21, the display area other than the first display area 21 in the display screen 101 is the second display area 22, and the projection area of ​​the input box in the smartphone 220 in the first display area 21 is the third display area 25. The target display area 23 includes the third display area 25, and the target display area 23 will not block the virtual keyboard 24 in the smartphone 220. Since the information input window is displayed in the target display area 23, the displayed information input window will not block the virtual keyboard 24 in the smartphone 220 in the display screen 101, so that the user can input information through the virtual keyboard 24.

[0066] In some embodiments, the input box position includes the first coordinates, second coordinates, third coordinates and fourth coordinates of the upper right corner, lower right corner, upper left corner and lower left corner of the input box in the display screen of the input device, respectively. According to the input box position, determining the target position of the input box in the first display area can be: determining a coordinate conversion relationship according to the size of the display screen of the input device and the size of the first display area; according to the coordinate conversion relationship, converting the first coordinate, the second coordinate, the third coordinate and the fourth coordinate to obtain the first target coordinates, second target coordinates, third target coordinates and fourth target coordinates of the upper right corner, lower right corner, upper left corner and lower left corner of the input box in the first display area, respectively; determining the target position of the input box in the first display area according to the first target coordinates, the second target coordinates, the third target coordinates and the fourth target coordinates.

[0067] In some embodiments, determining the target display area in the first display area may include: obtaining the virtual keyboard position sent by the input device, the virtual keyboard position being the position of the virtual keyboard in the input device in the display screen of the input device; determining the projection area of ​​the virtual keyboard in the first display area according to the virtual keyboard position; determining the display area other than the projection area in the first display area as the candidate display area, the candidate display area being located above the projection area; determining part of the display area in the candidate display area as the target display area, the size of the part of the display area being a preset size. Among them, the preset size can be set based on actual conditions, and the embodiment of the present invention does not specifically limit this. In this embodiment, the projection area of ​​the virtual keyboard position in the display screen of the head-mounted display device can be determined through the virtual keyboard position sent by the input device, and the target display area is determined in the display area other than the projection area, so that the information input window displayed in the target display area can not block the input device, which is convenient for users to input information.

[0068] For example, Figure 8As shown, the user can see the smartphone 220 through the display screen 101 of the head mounted display device, and the display area corresponding to the projection position of the smartphone 220 in the display screen 101 is the first display area 21, the display area other than the first display area 21 in the display screen 101 is the second display area 22, the projection area of ​​the virtual keyboard 24 in the smartphone 220 in the first display area 21 is the fourth display area 26, the display area other than the fourth display area 26 in the first display area 21 is the candidate display area 27, the target display area 23 is a partial display area in the candidate display area 27, and the target display area 23 will not block the virtual keyboard 24 in the smartphone 220. Since the information input window is displayed in the target display area 23, the displayed information input window will not block the virtual keyboard 24 in the smartphone 220 in the display screen 101, so that the user can input information through the virtual keyboard 24.

[0069] In some embodiments, the virtual keyboard position includes the fifth coordinates, sixth coordinates, seventh coordinates and eighth coordinates of the upper right corner, lower right corner, upper left corner and lower left corner of the virtual keyboard in the display screen of the input device, respectively. According to the virtual keyboard position, determining the projection area of ​​the virtual keyboard in the first display area may include: determining a coordinate conversion relationship according to the size of the display screen of the input device and the size of the first display area; according to the coordinate conversion relationship, converting the fifth coordinate, sixth coordinate, seventh coordinate and eighth coordinate to obtain the fifth target coordinates, sixth target coordinates, seventh target coordinates and eighth target coordinates of the upper right corner, lower right corner, upper left corner and lower left corner of the virtual keyboard in the first display area, respectively; determining the projection area of ​​the virtual keyboard in the first display area according to the fifth target coordinates, sixth target coordinates, seventh target coordinates and eighth target coordinates.

[0070] Sub-step S1026: When the device type is the second device type, determine a target display area within the second display area.

[0071] In this embodiment, when the device type is the second device type, it can be determined that the input device connected to the head-mounted display device is an input device without a display screen. At this time, the target display area is determined in the second display area, which can ensure that the target display area will not block the input device, and the information input window displayed in the target display area will not block the virtual keyboard, thereby facilitating the user to input information.

[0072] Step S103: respond to the information input operation sent by the input device, and display input information corresponding to the information input operation in the information input window.

[0073] In this embodiment, the size of the information input window is the same as the size of the target display area, or the size of the information input window is slightly smaller than the size of the target display area. After the head mounted display device displays the information input window, the head mounted display device can respond to the information input operation sent by the input device and display the input information corresponding to the information input operation in the information input window, so that the user can see the information input by himself, which is convenient for the user to input information.

[0074] See also Fig. 9 , Fig. 9 It is a schematic block diagram of the structure of an information display device provided by an embodiment of the present invention.

[0075] like Fig. 9 As shown, the information display device 300 includes:

[0076] A position acquisition module 310 is configured to acquire three-dimensional position information of the input device relative to the head mounted display device;

[0077] The area determination module 320 is configured to determine a target display area of ​​the information input window of the head mounted display device within the display screen of the head mounted display device according to the three-dimensional position information;

[0078] The window display module 330 is configured to display the information input window in the target display area;

[0079] The operation response module 340 is configured to respond to the information input operation sent by the input device and display the input information corresponding to the information input operation in the information input window.

[0080] In some embodiments, the region determination module 320 includes:

[0081] a matrix acquisition submodule, configured to acquire a projection matrix, a view matrix and a model matrix associated with the input device according to the three-dimensional position information;

[0082] a projection determination submodule, configured to determine a projection position of the input device within a display screen of the head mounted display device according to the projection matrix, the view matrix and the model matrix;

[0083] The area determination submodule is configured to determine a target display area of ​​the information input window of the head mounted display device within the display screen of the head mounted display device according to the projection position.

[0084] In some embodiments, the region determination submodule is further configured to:

[0085] According to the projection position, dividing the display screen of the head mounted display device into a first display area and a second display area, wherein the first display area is a display area corresponding to the projection position in the display screen, and the second display area is a display area in the display screen other than the first display area;

[0086] The target display area is determined in the second display area, and the target display area is located in a preset direction of the first display area.

[0087] In some embodiments, the region determination submodule is further configured to:

[0088] Dividing the display screen of the head mounted display device into a first display area and a second display area according to the projection position, and acquiring a device type of the input device;

[0089] In a case where the device type is a first device type, determining the target display area in the first display area, the first device type being used to describe that the input device includes a display screen, and the target display area is capable of not blocking a virtual keyboard displayed by the input device;

[0090] In a case where the device type is a second device type, the target display area is determined within the second display area, and the second device type is used to describe that the input device does not include a display screen.

[0091] In some embodiments, the region determination submodule is further configured to:

[0092] Acquire an input box position sent by the input device, where the input box position is a position of an input box in a selected state in the input device on a display screen of the input device;

[0093] Determining a target position of the input box in the first display area according to the position of the input box;

[0094] The target display area is determined in the first display area according to the target position, and the display area corresponding to the target position in the first display area is a subset of the target display area.

[0095] In some embodiments, the region determination submodule is further configured to:

[0096] Acquire a virtual keyboard position sent by the input device, where the virtual keyboard position is a position of a virtual keyboard in the input device on a display screen of the input device;

[0097] Determining a projection area of ​​the virtual keyboard within the first display area according to the position of the virtual keyboard;

[0098] Determine a display area other than the projection area in the first display area as a candidate display area, wherein the candidate display area is located above the projection area;

[0099] A partial display area in the candidate display area is determined as the target display area, and a size of the partial display area is a preset size.

[0100] In some embodiments, the location acquisition module 310 is further configured to:

[0101] Acquire image data obtained by photographing the input device by the photographing device;

[0102] The three-dimensional position information of the input device relative to the head mounted display device is determined according to a preset position estimation algorithm and the image data.

[0103] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the information display device described above can refer to the corresponding process in the aforementioned information display method embodiment, and will not be repeated here.

[0104] See also Fig.10 , Fig.10 It is a schematic block diagram of the structure of a head mounted display device provided by an embodiment of the present invention.

[0105] like Fig.10 As shown, the head mounted display device 100 includes a processor 110 and a memory 120, and the processor 110 and the memory 120 are connected via a bus 130, such as an I2C (Inter-integrated Circuit) bus.

[0106] Specifically, the processor 110 is used to provide computing and control capabilities to support the operation of the entire head-mounted display device. The processor 110 can be a central processing unit (CPU), and the processor 110 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0107] Specifically, the memory 120 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0108] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the head-mounted display device to which the embodiment of the present invention is applied. The specific head-mounted display device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0109] The processor 110 is used to run the computer program stored in the memory 120, and implement any one of the information display methods provided by the embodiments of the present invention when executing the computer program.

[0110] In some embodiments, the processor 110 is used to run a computer program stored in the memory, and implement the following steps when executing the computer program:

[0111] Acquire three-dimensional position information of the input device relative to the head mounted display device;

[0112] Determine, according to the three-dimensional position information, a target display area of ​​the information input window of the head-mounted display device within the display screen of the head-mounted display device, and display the information input window within the target display area;

[0113] In response to the information input operation sent by the input device, input information corresponding to the information input operation is displayed in the information input window.

[0114] In some embodiments, when determining the target display area of ​​the information input window of the head mounted display device within the display screen of the head mounted display device according to the three-dimensional position information, the processor 110 is configured to implement:

[0115] According to the three-dimensional position information, obtaining a projection matrix, a view matrix and a model matrix associated with the input device;

[0116] Determining a projection position of the input device within a display screen of the head mounted display device according to the projection matrix, the view matrix and the model matrix;

[0117] According to the projection position, a target display area of ​​the information input window of the head mounted display device within the display screen of the head mounted display device is determined.

[0118] In some embodiments, when determining the target display area of ​​the information input window of the head mounted display device within the display screen of the head mounted display device according to the projection position, the processor 110 is configured to implement:

[0119] According to the projection position, dividing the display screen of the head mounted display device into a first display area and a second display area, wherein the first display area is a display area corresponding to the projection position in the display screen, and the second display area is a display area in the display screen other than the first display area;

[0120] The target display area is determined in the second display area, and the target display area is located in a preset direction of the first display area.

[0121] In some embodiments, when determining the target display area of ​​the information input window of the head mounted display device within the display screen of the head mounted display device according to the projection position, the processor 110 is configured to implement:

[0122] Dividing the display screen of the head mounted display device into a first display area and a second display area according to the projection position, and acquiring a device type of the input device;

[0123] In a case where the device type is a first device type, determining the target display area in the first display area, the first device type being used to describe that the input device includes a display screen, and the target display area is capable of not blocking a virtual keyboard displayed by the input device;

[0124] In a case where the device type is a second device type, the target display area is determined within the second display area, and the second device type is used to describe that the input device does not include a display screen.

[0125] In some embodiments, when determining the target display area within the first display area, the processor 110 is configured to implement:

[0126] Acquire an input box position sent by the input device, where the input box position is a position of an input box in a selected state in the input device on a display screen of the input device;

[0127] Determining a target position of the input box in the first display area according to the position of the input box;

[0128] The target display area is determined in the first display area according to the target position, and the display area corresponding to the target position in the first display area is a subset of the target display area.

[0129] In some embodiments, when determining the target display area within the first display area, the processor 110 is configured to implement:

[0130] Acquire a virtual keyboard position sent by the input device, where the virtual keyboard position is a position of a virtual keyboard in the input device on a display screen of the input device;

[0131] Determining a projection area of ​​the virtual keyboard within the first display area according to the position of the virtual keyboard;

[0132] Determine a display area other than the projection area in the first display area as a candidate display area, wherein the candidate display area is located above the projection area;

[0133] A partial display area in the candidate display area is determined as the target display area, and a size of the partial display area is a preset size.

[0134] In some embodiments, the head mounted display device includes a photographing device, and the processor 110, when acquiring the three-dimensional position information of the input device relative to the head mounted display device, is used to implement:

[0135] Acquire image data obtained by photographing the input device by the photographing device;

[0136] The three-dimensional position information of the input device relative to the head mounted display device is determined according to a preset position estimation algorithm and the image data.

[0137] It should be noted that technicians in the relevant field can clearly understand that for the convenience and brevity of description, the specific working process of the head-mounted display device described above can refer to the corresponding process in the aforementioned information display method embodiment, and will not be repeated here.

[0138] An embodiment of the present invention further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any information display method provided in the description of the embodiment of the present invention.

[0139] The storage medium may be an internal storage unit of the head mounted display device described in the foregoing embodiment, such as a hard disk or memory of the head mounted display device. The storage medium may also be an external storage device of the head mounted display device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., provided on the head mounted display device.

[0140] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0141] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0142] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An information display method, characterized in that: Applied to a head mounted display device, the head mounted display device is communicatively connected with an input device, the method comprising: Acquire three-dimensional position information of the input device relative to the head mounted display device; Determine, according to the three-dimensional position information, a target display area of ​​the information input window of the head-mounted display device within the display screen of the head-mounted display device, and display the information input window within the target display area; In response to the information input operation sent by the input device, input information corresponding to the information input operation is displayed in the information input window.

2. The information display method according to claim 1, characterized in that: The step of determining a target display area of ​​the information input window of the head mounted display device within the display screen of the head mounted display device according to the three-dimensional position information comprises: According to the three-dimensional position information, obtaining a projection matrix, a view matrix and a model matrix associated with the input device; Determining a projection position of the input device within a display screen of the head mounted display device according to the projection matrix, the view matrix and the model matrix; According to the projection position, a target display area of ​​the information input window of the head mounted display device within the display screen of the head mounted display device is determined.

3. The information display method according to claim 2, characterized in that: The step of determining a target display area of ​​the information input window of the head mounted display device within the display screen of the head mounted display device according to the projection position comprises: Dividing the display screen of the head mounted display device into a first display area and a second display area according to the projection position, wherein the first display area is a display area corresponding to the projection position in the display screen, and the second display area is a display area in the display screen other than the first display area; The target display area is determined in the second display area, and the target display area is located in a preset direction of the first display area.

4. The information display method according to claim 2, characterized in that: The step of determining a target display area of ​​the information input window of the head mounted display device within the display screen of the head mounted display device according to the projection position comprises: Dividing the display screen of the head mounted display device into a first display area and a second display area according to the projection position, and acquiring a device type of the input device; In a case where the device type is a first device type, determining the target display area in the first display area, the first device type being used to describe that the input device includes a display screen, and the target display area is capable of not blocking a virtual keyboard displayed by the input device; In a case where the device type is a second device type, the target display area is determined within the second display area, and the second device type is used to describe that the input device does not include a display screen.

5. The information display method according to claim 4, characterized in that: The determining the target display area within the first display area includes: Acquire an input box position sent by the input device, where the input box position is a position of an input box in a selected state in the input device on a display screen of the input device; Determining a target position of the input box in the first display area according to the position of the input box; The target display area is determined in the first display area according to the target position, and the display area corresponding to the target position in the first display area is a subset of the target display area.

6. The information display method according to claim 4, characterized in that: The determining the target display area within the first display area includes: Acquire a virtual keyboard position sent by the input device, where the virtual keyboard position is a position of a virtual keyboard in the input device on a display screen of the input device; Determining a projection area of ​​the virtual keyboard within the first display area according to the position of the virtual keyboard; Determine a display area other than the projection area in the first display area as a candidate display area, wherein the candidate display area is located above the projection area; A partial display area in the candidate display area is determined as the target display area, and a size of the partial display area is a preset size.

7. The information display method according to any one of claims 1 to 6, characterized in that: The head mounted display device includes a photographing device, and the step of obtaining the three-dimensional position information of the input device relative to the head mounted display device includes: Acquire image data obtained by photographing the input device by the photographing device; The three-dimensional position information of the input device relative to the head mounted display device is determined according to a preset position estimation algorithm and the image data.

8. An information display device, characterized in that: Applied to a head mounted display device, the head mounted display device is communicatively connected with an input device, and the information display device comprises: A position acquisition module, configured to acquire three-dimensional position information of the input device relative to the head mounted display device; an area determination module, configured to determine a target display area of ​​the information input window of the head mounted display device within the display screen of the head mounted display device according to the three-dimensional position information; A window display module, configured to display the information input window in the target display area; The operation response module is configured to respond to the information input operation sent by the input device and display the input information corresponding to the information input operation in the information input window.

9. A head mounted display device, characterized in that: The head-mounted display device includes a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the information display method according to any one of claims 1 to 7 is realized.

10. A storage medium for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the information display method according to any one of claims 1 to 7.

Images