Method and apparatus for displaying information

Abstract

The embodiments of the present application disclose a method and device for displaying information. A specific implementation of the method includes: obtaining a display request of a head-mounted electronic device, displaying a first button and a second button on the screen of a terminal device; in response to detecting a trigger event of the first button, entering a projection display mode, obtaining data of an image to be displayed, rendering the data of the image to be displayed to generate a projection image, and displaying the projection image and the image to be displayed on the head-mounted electronic device and the terminal device, respectively; in response to detecting a trigger event of the second button, entering a mixed reality display mode, rendering a mixed reality operation image based on the posture information of the head-mounted electronic device, and controlling the head-mounted electronic device to display the mixed reality operation image, wherein the mixed reality operation image is updated based on the user's operation on the mixed reality operation image. This implementation allows users to quickly and conveniently enter the projection display mode or mixed reality display mode of a head-mounted electronic device.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and more particularly to a method and device for displaying information. Background Art

[0002] With the development of technology, more and more display devices appear in people's daily life and work, becoming an indispensable part of people's life, such as head-mounted electronic devices. Generally, head-mounted electronic devices used for display can achieve different effects such as virtual reality, augmented reality and mixed reality.

[0003] In the related art, users may have different needs when using head-mounted electronic devices. For example, users can use head-mounted electronic devices to present the content displayed by other devices, that is, use head-mounted electronic devices to project screens to other devices. Alternatively, users may not only need to use head-mounted electronic devices to present virtual content for viewing, but also need to interact with the objects displayed by the head-mounted electronic devices in real time in a visual environment, that is, to achieve mixed reality display through head-mounted electronic devices. Summary of the invention

[0004] The embodiments of the present application provide a method and device for displaying information.

[0005] In a first aspect, an embodiment of the present application provides a method for displaying information, the method comprising: obtaining a display request of a head-mounted electronic device, displaying a first button and a second button on a screen of a terminal device, wherein the first button is used to select a projection display mode and the second button is used to select a mixed reality display mode; in response to detecting a trigger event of the first button, entering the projection display mode, obtaining data of an image to be displayed, rendering the data of the image to be displayed to generate a projection image, and displaying the projection image and the image to be displayed on the head-mounted electronic device and the terminal device, respectively; in response to detecting a trigger event of the second button, entering the mixed reality display mode, rendering a mixed reality operation image based on posture information of the head-mounted electronic device, and controlling the head-mounted electronic device to display the mixed reality operation image, wherein the mixed reality operation image is updated based on the user's operation on the mixed reality operation image.

[0006] In a second aspect, an embodiment of the present application provides a device for displaying information, the device comprising: an acquisition unit, configured to acquire a display request of a head-mounted electronic device, and display a first button and a second button on the screen of the terminal device, wherein the first button is used to select a projection display mode, and the second button is used to select a mixed reality display mode; a first display unit, configured to enter the projection display mode in response to detecting a trigger event of the first button, acquire data of an image to be displayed, render the data of the image to be displayed to generate a projection image, and display the projection image and the image to be displayed on the head-mounted electronic device and the terminal device, respectively; a second display unit, configured to enter the mixed reality display mode in response to detecting a trigger event of the second button, render a mixed reality operation image based on posture information of the head-mounted electronic device, and control the head-mounted electronic device to display the mixed reality operation image, wherein the mixed reality operation image is updated based on the user's operation on the mixed reality operation image.

[0007] In a third aspect, an embodiment of the present application provides a terminal device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the one or more processors implement the above method.

[0008] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described above in the claims.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, wherein the computer program implements the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0011] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;

[0012] Figure 2 is a flow chart of an embodiment of a method for displaying information according to the present application;

[0013] Figure 3 is a flow chart of another embodiment of a method for displaying information according to the present application;

[0014] Figure 4 is a schematic diagram of an application scenario of the method for displaying information according to the present application;

[0015] Figure 5 is a schematic structural diagram of an embodiment of a device for displaying information according to the present application;

[0016] Figure 6 It is a structural diagram of a computer system of a terminal device suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0017] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0018] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the method for displaying information or the apparatus for displaying information of the present application can be applied.

[0020] like Figure 1 As shown, the system architecture 100 may include a terminal device 101 and a head-mounted electronic device 102. The terminal device 101 and the head-mounted electronic device 102 may be connected via a network, and the network connection may include multiple ways of connection, such as wired, wireless communication links or optical fiber cables, etc. The terminal device 101 and the head-mounted electronic device 102 may interact via the network to send or receive data information, etc.

[0021] The user can use the terminal device 101 to interact with the head-mounted electronic device 102 through the network to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc. The head-mounted electronic device 102 can be an electronic device with an image display function, including but not limited to AR smart glasses and VR smart glasses.

[0022] The terminal device 101 can be various electronic devices with a display screen, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop computers and desktop computers, etc. Of course, the above-mentioned terminal device 101 can also be software. When the terminal device 101 is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0023] The terminal device 101 may also provide various services, for example, providing support for the projection image or mixed reality operation image displayed on the head-mounted electronic device 102. The terminal device may analyze and process the acquired display request and other data, and send the processing result (such as the projection image or mixed reality operation image) to the head-mounted electronic device for display.

[0024] It should be noted that the method for displaying information provided in the embodiment of the present application is generally executed by the terminal device 101, and accordingly, the device for displaying information is generally set in the terminal device 101.

[0025] It should be understood that Figure 1 The number of terminal devices 101 and head-mounted electronic devices 102 in the embodiment is only for illustration. Any number of terminal devices 101 and head-mounted electronic devices 102 may be provided as required. For example, Figure 1 The system may include two terminal devices 101, one of which may analyze and process the acquired display request and other data, and send the processing results (such as the rendered projection image corresponding to the terminal device 101 and the projection image corresponding to the other terminal device 101) to the head mounted electronic device 102 for display. For another example, Figure 1 The system may also include two head-mounted electronic devices 102. The terminal device 101 may analyze and process the acquired display request and other data, and send the processing results (such as projection images or mixed reality operation images) to the two head-mounted electronic devices 101 for display.

[0026] It should also be pointed out that although the solution disclosed in the present application can be applied to the terminal device 101, this does not exclude that the solution can also be applied to the head-mounted electronic device 102. In the case where the solution disclosed in the present application is applied to the head-mounted electronic device 102, the head-mounted electronic device 102 can also obtain a display request and control the display of the first button and the second button on the screen of the terminal device 101. In response to the trigger event of the first button detected by the terminal device 101, the head-mounted electronic device 102 can enter the projection display mode, obtain the data of the image to be displayed, render the data of the image to be displayed to generate a projection image, and display the projection image and the image to be displayed on the head-mounted electronic device and the terminal device, respectively. Similarly, the head-mounted electronic device 102 can also enter the mixed reality display mode and display the mixed reality operation image. In this case, the method for displaying information can be executed by the head-mounted electronic device 102, and accordingly, the device for displaying information can also be set in the head-mounted electronic device 102.

[0027] Continue to refer Figure 2 , shows a process 200 of an embodiment of a method for displaying information according to the present application. The method for displaying information comprises the following steps:

[0028] Step 202: Obtain a display request of the head mounted electronic device, and display a first button and a second button on the screen of the terminal device.

[0029] In this embodiment, the execution subject of the method for displaying information (eg Figure 1 The terminal device shown in the figure) can obtain the display request of the head-mounted electronic device in various ways. Here, the display request can be used to request image display on the head-mounted electronic device. Then the above-mentioned execution subject can display the first button and the second button on the screen. The user can select the projection display mode through the first button, and can select the mixed reality display mode through the second button. Among them, the projection mode can be understood as the head-mounted electronic device serving as the projection device of the terminal device, and the screens of the head-mounted electronic device and the terminal device display the same content. The above-mentioned mixed reality display mode can be understood as a mode in which the head-mounted electronic device and the terminal device are combined to provide a virtual image that can interact and feedback with the user in real time in a real environment.

[0030] In some optional implementations of this embodiment, before obtaining the display request of the head-mounted electronic device, the above-mentioned execution subject may determine whether the execution subject has established a communication connection with the head-mounted electronic device. If the above-mentioned execution subject establishes a communication connection with the head-mounted electronic device, a display request may be directly generated so that the above-mentioned execution subject can obtain the display request. As an example, the data cable of the head-mounted electronic device can be directly connected to the terminal device, and the terminal device can directly obtain the generated display request. This implementation method can quickly display the first button and the second button on the screen, simplify the selection operation of the display mode, and further improve the efficiency of the user selecting the projection display mode and the mixed reality display mode. It can be understood that the above-mentioned execution subject can also obtain the display request in other ways. For example, after the head-mounted electronic device establishes a communication connection with the terminal device, the user can input the display request by clicking a preset button, etc., and the terminal device can obtain the display request input by the user.

[0031] In some optional implementations of this embodiment, in response to determining that a communication connection is established between the terminal device and the head-mounted electronic device, before generating a display request, the execution subject may also prompt the user to establish a communication connection between the terminal device and the head-mounted electronic device. Specifically, the execution subject may receive an opening instruction from the user for a pre-installed target application, run the target application, and display a first prompt message on the screen to prompt the user to establish a communication connection between the terminal device and the head-mounted electronic device; then, the user may connect the head-mounted electronic device to the terminal device through the prompt message. In this implementation, the execution subject may send the first prompt message through the target application to prompt the user to establish a communication connection between the terminal device and the head-mounted electronic device, so that the user can quickly enter the display mode selection without learning, thereby reducing the learning cost.

[0032] Typically, a head-mounted electronic device and a terminal device can be used in conjunction with each other. For example, the head-mounted electronic device and the terminal device can be used as split devices. In this case, the user can use the head-mounted electronic device to project the screen to the terminal device (projection display mode), and can also use the terminal device to interact with the head-mounted electronic device in mixed reality (mixed reality display mode). Therefore, when a user uses a head-mounted electronic device and a terminal device, they often consider issues such as what display mode to enter and how to enter a specified display mode for good display. The solution disclosed in the embodiments of the present application can provide an effective way to use a head-mounted electronic device for display when a user uses a head-mounted electronic device and a terminal device.

[0033] Step 204, in response to detecting the triggering event of the first button, enter the projection display mode, obtain data of the image to be displayed, render the data of the image to be displayed to generate a projection image, and display the projection image and the image to be displayed on the head-mounted electronic device and the terminal device respectively.

[0034] In this embodiment, based on the first button and the second button displayed in step 202, the execution subject can detect the triggering events of the first button and the second button. It can be understood that, depending on the actual situation, the user can have a variety of operations to generate corresponding triggering events. For example, the first button and the second button can be touch buttons, and the user's touch on the first button and the second button can generate corresponding button triggering events.

[0035] If the above-mentioned execution subject detects the triggering event of the first button, it can be determined that the user has selected the projection display mode. At this time, the above-mentioned execution subject can obtain the data of the image to be displayed on the screen, and the data to be displayed on the screen can be the image to be displayed. Then the above-mentioned execution subject can render the acquired data to generate a projection image. Here, the above-mentioned execution subject can render the acquired data through a rendering engine such as unity, so as to obtain a rendered image, which is the projection image. Finally, the above-mentioned execution subject can send the projection image to the head-mounted electronic device, so that the head-mounted electronic device can display the projection image and complete the projection display of the head-mounted electronic device. And, the above-mentioned execution subject can also display the image to be displayed on the screen of the terminal device. It can be understood that the content of the projection image is the same as the image currently displayed on the screen, so the display of the head-mounted electronic device can be equivalent to a mirror display of the screen of the above-mentioned execution subject.

[0036] It is understandable that the image displayed on the screen may be various images, and there is no single limitation here. As an example, the terminal device may be a mobile phone, in which case the image displayed on the screen of the terminal device may include only one interface such as a desktop. Alternatively, the terminal device may also be a computer, in which case the image displayed on the screen of the terminal device may include multiple window interfaces opened on the desktop.

[0037] In some optional implementations of this embodiment, after entering the projection display mode, the above-mentioned execution subject can return to the desktop of the terminal device regardless of the interface the screen was in before. Then, the above-mentioned execution subject can obtain the data of the image corresponding to the desktop of the terminal device, and the image can be the image to be displayed, so that the above-mentioned execution subject can render the projection image corresponding to the desktop image, and display the projection image on the head-mounted electronic device. It is understandable that the screen of the above-mentioned terminal device can also display the corresponding desktop. This implementation method allows the user to directly return to the desktop when using the projection display mode, without the user manually returning to the desktop, further improving the efficiency of the user using the projection display mode for display. It is understandable that after entering the projection display mode, the above-mentioned execution subject can directly obtain the data of the image to be displayed by the terminal device, and render the projection image corresponding to the image, so that the above-mentioned head-mounted electronic device can display the projection image.

[0038] Optionally, in the screen projection mode, the above-mentioned execution subject can update the image displayed on the screen when receiving the user's operation on the screen. Specifically, the above-mentioned execution subject can determine the image to be displayed displayed by the terminal device as the current display image of the screen, and then update the data of the image to be displayed in response to receiving the user's operation on the current display image displayed on the screen; finally, based on the updated data of the image to be displayed, the projection image can be updated, and the updated projection image and the image to be displayed are displayed on the head-mounted electronic device and the terminal device, respectively. This implementation method can make the projection image displayed in the head-mounted electronic device change in real time according to the image on the terminal device screen, ensuring that the head-mounted electronic device is synchronized with the terminal device data when projecting.

[0039] Step 206, in response to detecting a trigger event of the second button, entering a mixed reality display mode, rendering a mixed reality operation image based on the posture information of the head-mounted electronic device, and controlling the head-mounted electronic device to display the mixed reality operation image.

[0040] In this embodiment, if the above-mentioned execution subject detects the trigger event of the second button, it can be determined that the user selects the mixed reality display mode. At this time, the above-mentioned execution subject can analyze the posture information of the head-mounted electronic device, etc., so as to render a mixed reality operation image, and send the mixed reality operation image to the head-mounted electronic device, so that the head-mounted electronic device can display the mixed reality operation image. The above-mentioned mixed reality operation image can update the display content based on the user's direct or indirect operation of the mixed reality operation image, thereby realizing the interaction of mixed reality. As an example, the above-mentioned head-mounted electronic device can support gesture recognition, and the user can directly operate the mixed reality operation image displayed in the real environment through gestures, so that the mixed reality operation image displayed by the head-mounted electronic device can be updated.

[0041] In some optional implementations of this embodiment, after entering the mixed reality display mode, the above-mentioned execution subject can also render a mixed reality interface and display the mixed reality interface on the screen. In this case, the user can use the above-mentioned execution subject as a control handle of the head-mounted electronic device, and can indirectly operate the mixed reality operation interface displayed by the head-mounted electronic device through the mixed reality interface displayed on the screen. It can be understood that when the terminal device is used as a handle, the mixed reality interface displayed on the screen is usually different from the mixed reality operation interface displayed on the head-mounted electronic device. The solution provided by this implementation method can display the mixed reality interface on the screen of the terminal device, so that the user can use the terminal device as a handle to control the display of the head-mounted electronic device, making full use of the function of the terminal device, and controlling the display of the head-mounted electronic device by gesture recognition and other methods can give the user a new experience.

[0042] Optionally, the mixed reality interface may further include a mixed reality display setting button, which allows the user to set parameters of the mixed reality display mode (e.g., sleep mode, left-hand operation, etc.). Therefore, when the execution subject receives a second preset operation such as a single click by the user on the mixed reality display setting button, it may configure the parameters of the mixed reality display mode, thereby improving the user operation experience in the mixed reality display mode.

[0043] As an example, the terminal device may be a mobile phone, which can be used as a handle of a head-mounted electronic device after displaying a mixed reality interface. The user's operation on the mixed reality interface can change the position of the operation point in the mixed reality operation interface displayed by the head-mounted electronic device. For example, the user can control the operation point of the mixed reality operation interface to move to the location of application A through the mixed reality interface, and then the user can open the window of application A by double-clicking the application through the mixed reality interface, that is, the mixed reality operation interface displayed by the head-mounted electronic device can be updated to the application A window, or the mixed reality operation interface displayed by the head-mounted electronic device can be updated to an interface in which the application A window is added to the original display content.

[0044] In some optional implementations of this embodiment, the above-mentioned execution subject can render a mixed reality operation image in the following manner: obtain the posture information of the head-mounted electronic device and the reality data to be mixed, wherein the posture information is used to characterize the position and posture of the head-mounted electronic device; according to the posture information, adjust the position and posture of the virtual camera; use the adjusted virtual camera to render the reality data to be mixed to generate a mixed reality operation image. It can be understood that the above-mentioned reality data to be mixed can refer to data used to generate a mixed reality operation image, and the above-mentioned execution subject can render the mixed reality operation image to be displayed by the head-mounted electronic device through unity, etc. Therefore, before rendering the mixed reality operation image, the posture information of the head-mounted electronic device can be determined first, so as to adjust the position and posture of the virtual camera in unity, etc. according to the posture information, and render the mixed reality data according to the adjusted virtual camera, so as to generate a mixed reality operation image with a suitable perspective. The scheme disclosed in this implementation enables the head-mounted electronic device to quickly display a mixed reality operation interface suitable for the user's perspective. It can be understood that the above-mentioned execution subject can render a mixed reality operation image in other ways, and there is no unique limitation here. For example, according to the posture information of the head-mounted electronic device, an optical module composed of an image source and an optical lens in the head-mounted electronic device is adjusted so that the optical module can project a mixed reality operation image with a suitable viewing angle.

[0045] Optionally, when the above-mentioned execution entity uses Unity or the like to render the mixed reality operation image to be displayed by the head-mounted electronic device, in the above-mentioned mixed reality display mode, other images to be displayed by the head-mounted electronic device may also be generated using Unity or the like.

[0046] In some optional implementations of this embodiment, the above-mentioned control of the head-mounted electronic device to display the mixed reality operation image may include: for the target object in the mixed reality operation image displayed by the head-mounted electronic device, in response to receiving the user's operation on the target object, the control instruction for the target object can be called and executed to update the mixed reality operation image; the updated mixed reality operation image is sent to the head-mounted electronic device so that the head-mounted electronic device displays the updated mixed reality operation image. As an example, the mixed reality operation image displayed by the head-mounted electronic device can be a desktop formed by applications selected from the terminal device based on a white list, etc. For applications such as Google in the desktop, if a double-click operation of Google is received by the user, an opening instruction of applications such as Google can be called and executed, and the desktop is updated to an interface such as Google; then, the above-mentioned execution subject can send an interface such as Google to the head-mounted electronic device so that the head-mounted electronic device can display an updated interface such as Google. In this implementation, the mixed reality operation image displayed by the head-mounted electronic device can be quickly updated according to the user's operation on it. Of course, it is understandable that this implementation only provides a feasible operation method for the mixed reality display mode. There may also be other operation methods, such as updating the mixed reality operation image displayed by the head-mounted electronic device through the handle, etc. There is no unique limitation here.

[0047] The method for displaying information provided by the above-mentioned embodiments of the present application can obtain a display request and display a first button and a second button on the screen when the terminal device and the head-mounted electronic device are used in conjunction, and then, after determining that a trigger event of the first button is detected, enter the projection mode, display the content currently displayed on the screen of the terminal device in the head-mounted electronic device, and enter the mixed reality display mode after determining that a trigger event of the second button is detected. The user can operate the mixed reality operation image displayed by the head-mounted electronic device through the terminal device to realize the interaction of mixed reality. The scheme disclosed in the above-mentioned embodiments of the present application can accurately and quickly enter the selected mode for projection display or mixed reality display when the user initiates a display request of the head-mounted electronic device, without the need for user learning, thereby improving the operating efficiency of the head-mounted electronic device and reducing the difficulty of operating the head-mounted electronic device.

[0048] Further references Figure 3 , which shows a process 300 of another embodiment of a method for displaying information. The process 300 of the method for displaying information includes the following steps:

[0049] Step 302: Obtain a display request of the head mounted electronic device, and display a first button and a second button on the screen of the terminal device.

[0050] In this embodiment, step 302 is substantially the same as step 202 in the above embodiment, and will not be described in detail here.

[0051] Step 304, in response to detecting a trigger event of the second button, entering a mixed reality display mode, rendering a mixed reality operation image based on the posture information of the head-mounted electronic device, and controlling the head-mounted electronic device to display the mixed reality operation image.

[0052] In this embodiment, step 304 is substantially the same as step 206 in the above embodiment, and will not be described in detail here.

[0053] Step 306, in response to detecting the triggering event of the first button, entering the screen projection display mode, and displaying the third button and the fourth button on the screen.

[0054] In this embodiment, the above-mentioned execution subject can enter the projection display mode when the trigger event of the first button is detected. Then, the above-mentioned execution subject can display the third button and the fourth button on the screen in various ways in the projection display mode. It should be noted that the above-mentioned third button can be used for the user to select the first projection display mode or the second projection display mode in the projection display mode. Here, the size of the projection image in the first projection display mode can be larger than the size of the projection image in the second projection display mode, that is, the first projection display mode and the second projection display mode can be understood as a large projection display mode and a small projection display mode, respectively. Furthermore, the above-mentioned fourth button can be used for the user to switch from the current projection display mode to the mixed reality display mode.

[0055] In some optional implementations of this embodiment, displaying the third button and the fourth button on the screen may include the following steps: receiving a first preset operation of the user on the screen, and displaying a window including the third button and the fourth button on the screen. Specifically, after entering the projection display mode, the execution subject may receive a first preset operation of the user on the screen, such as a drop-down window, so that a drop-down window, etc., may be displayed on the screen, and the third button and the fourth button may be displayed in the drop-down window, etc. It is understandable that after entering the projection display mode, the user can directly perform the first preset operation such as the drop-down window. Alternatively, after entering the projection display mode, the execution subject may also first display a prompt information on the screen to prompt the user to perform a first preset operation such as a drop-down window to select a display mode, so that the user can open a window including the third button and the fourth button according to the prompt information. In this implementation, the execution subject displays the window including the third button and the fourth button only after receiving the first preset operation of the user, so that the user can independently choose whether to display the window including the third button and the fourth button, thereby improving the user experience. Of course, the user can also directly enter the first projection display mode, such as the first projection display mode, by triggering the first button, and there is no unique limitation here.

[0056] In some optional implementations of the present embodiment, displaying the third button and the fourth button on the screen may include the following steps: calling and executing a pop-up display instruction to control the screen to display a pop-up window including the third button and the fourth button within a preset duration. Specifically, after entering the projection mode, the above-mentioned execution subject may directly call the pop-up display instruction, and after executing the instruction, a pop-up window may be displayed on the screen, and the pop-up window may include the third button and the fourth button. Furthermore, the above-mentioned execution subject may control the pop-up window to display a preset duration, and may directly close the pop-up window after exceeding the preset duration. In this implementation, after entering the projection display mode, the terminal device may directly display the third button and the fourth button in a pop-up window, without the need for additional user operation, which is convenient and quick.

[0057] Step 308, in response to detecting the first trigger event of the third button, enter the first projection display mode, obtain data of the image to be displayed, render the data of the image to be displayed to generate a projection image, and display the projection image and the image to be displayed on the head-mounted electronic device and the terminal device respectively.

[0058] In this embodiment, the above-mentioned execution subject can enter the first projection display mode when the first trigger event of the third button is detected. The above-mentioned execution subject can obtain the data of the image to be displayed on the screen (the image to be displayed on the screen can be the above-mentioned image to be displayed), and render the acquired data, so as to generate a projection image of the size corresponding to the first projection display mode. Then, the above-mentioned execution subject can send the generated projection image to the head-mounted electronic device so that the head-mounted electronic device can display the generated projection image. That is, the head-mounted electronic device can perform projection display in the large projection display mode. It can be understood that the above-mentioned image to be displayed can also be displayed on the screen of the above-mentioned terminal device, that is, the screen of the terminal device and the image displayed by the head-mounted electronic device include the same content.

[0059] Step 310, in response to detecting a second trigger event of the third button, enter the second projection display mode, obtain data of the image to be displayed, render the data of the image to be displayed to generate a projection image, and display the projection image and the image to be displayed on the head-mounted electronic device and the terminal device respectively.

[0060] In this embodiment, the above-mentioned execution subject can enter the second projection display mode when the second trigger event of the third button is detected. The above-mentioned execution subject can obtain the data of the image to be displayed on the screen, and the image to be displayed on the screen is the image to be displayed. The acquired data is then rendered to generate a projection image of a size corresponding to the second projection display mode. Finally, the above-mentioned execution subject can send the generated projection image to the head-mounted electronic device so that the head-mounted electronic device can display the generated projection image. That is, the head-mounted electronic device can perform projection display in the small projection display mode. It can be understood that the above-mentioned image to be displayed can also be displayed on the screen of the above-mentioned terminal device, that is, the screen of the terminal device and the image displayed by the head-mounted electronic device include the same content.

[0061] Here, the execution subject may generate the first trigger event and the second trigger event in various ways. For example, the third button may be divided into a first area and a second area. The execution subject may generate the first trigger event when receiving a single click operation of the user on the first area of ​​the third button, and may generate the second trigger event when receiving a single click operation of the user on the second area of ​​the third button. Alternatively, the execution subject may generate the first trigger event when receiving a single click operation of the user on the third button, and may generate the second trigger event when receiving a double click operation of the user on the third button.

[0062] Step 312, in response to detecting a trigger event of the fourth button, switching from the projection display mode to the mixed reality display mode, rendering a mixed reality operation image based on the posture information of the head-mounted electronic device, and controlling the head-mounted electronic device to display the mixed reality operation image.

[0063] In this embodiment, the above-mentioned execution subject can switch from the projection display mode to the mixed reality display mode when the trigger event of the above-mentioned fourth button is detected. At this time, the above-mentioned execution subject can render a mixed reality operation image based on the posture information of the head-mounted electronic device, and send the mixed reality operation image to the head-mounted electronic device, so that the head-mounted electronic device can display the mixed reality operation image. This embodiment allows the user to switch directly from the projection display mode to the mixed reality display mode, without having to return to the interface displaying the first button and the second button to enter the mixed reality display mode, and the operation is simpler. It can be understood that the above-mentioned execution subject can adopt various methods to realize the rendering of mixed reality operation images based on the posture information of the head-mounted electronic device. There is no unique limitation here, see the above-mentioned embodiment.

[0064] In some optional implementations of this embodiment, when entering the mixed reality display mode, a switch button may be provided on the desktop of the head-mounted electronic device, and the switch button may switch the mixed reality display mode to the projection display mode.

[0065] from Figure 3 It can be seen that Figure 2 Compared with the corresponding embodiment, the process 300 of the method for displaying information in this embodiment highlights that when entering the projection display mode, you can also choose to enter the first projection display mode or the second projection display mode, and you can directly switch to the mixed reality display mode when entering the projection display mode. Therefore, the solution described in this embodiment allows the user to quickly enter the large projection display mode or the small projection display mode, and can also quickly switch to the mixed reality display mode in the projection display mode, so as to further improve the operation speed of the user's mode selection when using the head-mounted electronic device and reduce the difficulty of operation.

[0066] Continue to see Figure 4 , Figure 4 FIG. 1 is a schematic diagram of an application scenario of the method for displaying information according to this embodiment. Figure 4 In the application scenario, the user first initiates a display request of the head-mounted electronic device 401 by inserting the data cable of the head-mounted electronic device 401 into the terminal device, so that the first button and the second button can be displayed on the screen of the terminal device, such as Figure 4 Afterwards, if the terminal device detects a triggering event of the first button, it can enter the projection display mode, such as Figure 4As shown, the terminal device can obtain an image to be displayed (e.g., a desktop to be displayed on a screen), and the terminal device can render the obtained data to generate, for example, a projection image 402 of the desktop, and the head-mounted electronic device 401 can display the projection image 402, so as to project the projection image 402 into space, such as Figure 4 As shown, the terminal device can display the desktop. Optionally, when entering the screen projection display mode, the screen of the terminal device can also first display the third button and the fourth button, such as Figure 4 As shown, when the terminal device detects a first trigger event of the third button (for example, the user clicks the left area of ​​the third button), it can enter the first projection display mode, and the head-mounted electronic device 401 displays the projection image 402 in the space in a large projection display manner. Alternatively, when the terminal device detects a second trigger event of the third button (for example, the user clicks the right area of ​​the third button), it can enter the second projection display mode, and the head-mounted electronic device 401 displays the projection image 402 in the space in a small projection display manner. Furthermore, if the terminal device detects a trigger event of the second button, it can enter the augmented reality display mode. At this time, the terminal device can render the mixed reality operation image 403 based on the posture information of the head-mounted electronic device, and control the head-mounted electronic device 401 to display the mixed reality operation image 403, as shown in FIG. Figure 4 As shown, the user can operate the mixed display operation image 403 to update the displayed image. Optionally, after entering the mixed reality display mode, the terminal device can be used as an operating handle of the head-mounted electronic device 401, and the handle can be as follows: Figure 4 As shown, the user can use the handle to control the mixed reality operation image 403 displayed by the head-mounted electronic device 401, thereby updating the image displayed by the head-mounted electronic device 401. Optionally, for the fourth button, in the projection display mode, if the user clicks the fourth button, the projection display mode can be switched to the mixed reality display mode, such as Figure 4 shown.

[0067] Further references Figure 5 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a device for displaying information. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0068] like Figure 5As shown, the device 500 for displaying information in this embodiment includes: an acquisition unit 501, a first display unit 502, and a second display unit 503. The acquisition unit 501 is configured to acquire a display request of a head-mounted electronic device, and display a first button and a second button on the screen of the terminal device, wherein the first button is used to select a projection display mode, and the second button is used to select a mixed reality display mode; the first display unit 502 is configured to enter the projection display mode in response to detecting a trigger event of the first button, acquire data of an image to be displayed, render the data of the image to be displayed to generate a projection image, and display the projection image and the image to be displayed on the head-mounted electronic device and the terminal device respectively; the second display unit 503 is configured to enter the mixed reality display mode in response to detecting a trigger event of the second button, render a mixed reality operation image based on the posture information of the head-mounted electronic device, and control the head-mounted electronic device to display the mixed reality operation image, wherein the mixed reality operation image is updated based on the user's operation on the mixed reality operation image.

[0069] In some optional implementations of this embodiment, the first display unit 502 is further configured to: render a mixed reality interface; display the mixed reality interface on the screen so that the user can operate the mixed reality operation image through the mixed reality interface to update the mixed reality operation image displayed by the head-mounted electronic device.

[0070] In some optional implementations of this embodiment, the apparatus 500 further includes a generating unit configured to generate a display request in response to determining that a communication connection is established between the terminal device and the head mounted electronic device.

[0071] In some optional implementations of this embodiment, the device 500 also includes an establishing unit: configured to receive an opening instruction from a user for a pre-installed target application, run the target application, and display a first prompt message on the screen, wherein the first prompt message is used to prompt the establishment of a communication connection between the terminal device and the head-mounted electronic device; and establish a communication connection between the terminal device and the head-mounted electronic device.

[0072] In some optional implementations of this embodiment, the first display unit is further configured to: acquire desktop data of the terminal device; and determine an image formed by the desktop as the image to be displayed.

[0073] In some optional implementations of this embodiment, the first display unit 502 includes: a display module, configured to display a third button and a fourth button on the screen, wherein the third button is used to select a first projection display mode or a second projection display mode, the size of the projection image in the first projection display mode is larger than the size of the projection image in the second projection display mode, and the fourth button is used to switch from the projection display mode to the mixed reality display mode; in response to detecting a first trigger event of the third button, entering the first projection display mode; in response to detecting a second trigger event of the third button, entering the second projection display mode; in response to detecting a trigger event of the fourth button, switching from the projection display mode to the mixed reality display mode.

[0074] In some optional implementations of this embodiment, the display module is further configured to: receive a first preset operation of the user on the screen, and display a window including a third button and a fourth button on the screen.

[0075] In some optional implementations of this embodiment, the display module is further configured to: call and execute a pop-up window display instruction to control the screen to display a pop-up window including a third button and a fourth button within a preset time length.

[0076] In some optional implementations of this embodiment, the first display unit 502 also includes: a first update module, configured to determine the image to be displayed displayed by the terminal device as the current display image of the screen; in response to receiving a user operation on the current display image displayed on the screen, update the data of the image to be displayed; based on the updated data of the image to be displayed, update the projection image, and display the updated projection image and the image to be displayed on the head-mounted electronic device and the terminal device respectively. Determine the image to be displayed displayed by the terminal device as the current display image of the screen; in response to receiving a user operation on the current display image displayed on the screen, update the data of the image to be displayed; based on the updated data of the image to be displayed, update the projection image, and display the updated projection image and the image to be displayed on the head-mounted electronic device and the terminal device respectively.

[0077] In some optional implementations of this embodiment, the second display unit 503 is further configured to: obtain posture information of the head-mounted electronic device and the reality data to be mixed, wherein the posture information is used to characterize the position and posture of the head-mounted electronic device; adjust the position and posture of the virtual camera in the preset rendering engine according to the posture information; and render the reality data to be mixed based on the adjusted virtual camera to generate a mixed reality operation image.

[0078] In some optional implementations of this embodiment, the second display unit 503 is further configured to: for a target object in a mixed reality operation image displayed by a head-mounted electronic device, in response to receiving an operation of the user on the target object, call and execute a control instruction for the target object to update the mixed reality operation image; and send the updated mixed reality operation image to the head-mounted electronic device so that the head-mounted electronic device displays the updated mixed reality operation image.

[0079] In some optional implementations of this embodiment, the mixed reality interface includes a mixed reality display setting button; the first display unit 502 is further configured to: configure parameters of the mixed reality display mode in response to receiving a second preset operation of the user for the mixed reality display setting button.

[0080] The units described in the device 500 and the reference Figure 2 and Figure 3 Therefore, the operations and features described above for the method are also applicable to the device 500 and the units included therein, and will not be described in detail here.

[0081] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present application. Figure 6 The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0082] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0083] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0084] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0085] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0086] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0087] The units involved in the embodiments described in the present application may be implemented by software or by hardware. The described units may also be set in a processor. For example, they may be described as: a processor includes an acquisition unit, a first display unit, and a second display unit. The names of these units do not constitute a limitation on the units themselves in some cases. For example, the acquisition unit may also be described as "a unit that obtains a display request of a head-mounted electronic device and displays a first button and a second button on the screen of a terminal device."

[0088] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by the device, the device: obtains a display request of a head-mounted electronic device, displays a first button and a second button on the screen of the terminal device, wherein the first button is used to select a projection display mode, and the second button is used to select a mixed reality display mode; in response to detecting a trigger event of the first button, enters the projection display mode, obtains data of an image to be displayed, renders the data of the image to be displayed to generate a projection image, and displays the projection image and the image to be displayed on the head-mounted electronic device and the terminal device respectively; in response to detecting a trigger event of the second button, enters the mixed reality display mode, renders a mixed reality operation image based on the posture information of the head-mounted electronic device, and controls the head-mounted electronic device to display the mixed reality operation image, wherein the mixed reality operation image is updated based on the user's operation on the mixed reality operation image.

[0089] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for displaying information, applied to a terminal device, the method comprising: Obtaining a display request of a head-mounted electronic device, and displaying a first button and a second button on a screen of the terminal device, wherein the first button is used to select a projection display mode, and the second button is used to select a mixed reality display mode; In response to detecting a trigger event of the first button, entering the projection display mode, acquiring data of an image to be displayed, rendering the data of the image to be displayed to generate a projection image, and displaying the projection image and the image to be displayed on the head-mounted electronic device and the terminal device, respectively; after entering the projection display mode, in response to receiving a first preset operation of the user on the screen, displaying a third button and a fourth button on the screen, the third button is used to select a first projection display mode or a second projection display mode, the size of the projection image in the first projection display mode is larger than the size of the projection image in the second projection display mode, the first projection display mode is entered when a first trigger event of the third button is detected, and the second projection display mode is entered when a second trigger event of the third button is detected, and the fourth button is used to switch from the projection display mode to the mixed reality display mode, so that the user can directly switch from the projection display mode to the mixed reality display mode without returning to the interface displaying the first button and the second button to enter the mixed reality display mode again; In response to detecting a trigger event of the second button, entering the mixed reality display mode, rendering a mixed reality operation image based on the posture information of the head-mounted electronic device, and controlling the head-mounted electronic device to display the mixed reality operation image, wherein the mixed reality display mode is characterized by a combination of a head-mounted electronic device and a terminal device to provide a virtual image that can interact and feedback with the user in real time in a real environment, and the mixed reality operation image is updated based on the user's operation of the mixed reality operation image. The head-mounted electronic device displays a switch button, and the switch button is used to switch from the mixed reality display mode to the projection display mode, so that the user can switch from the mixed reality display mode to the projection display mode without having to return to the interface displaying the first button and the second button to enter the projection display mode again.

2. The method according to claim 1, wherein: After entering the mixed reality display mode, the method includes: Rendering mixed reality interfaces; A mixed reality interface is displayed on the screen, so that the user operates the mixed reality operation image through the mixed reality interface to update the mixed reality operation image displayed by the head-mounted electronic device.

3. The method according to claim 1, wherein: Before obtaining the display request of the head mounted electronic device, the method further includes: In response to determining that the terminal device has established a communication connection with the head mounted electronic device, the display request is generated.

4. The method according to claim 3, wherein: In response to determining that the terminal device and the head mounted electronic device have established a communication connection, before generating the display request, the method further includes: receiving an opening instruction from a user for a pre-installed target application, running the target application, and displaying first prompt information on the screen, wherein the first prompt information is used to prompt the establishment of a communication connection between the terminal device and the head-mounted electronic device; Establishing a communication connection between the terminal device and the head mounted electronic device.

5. The method according to claim 1, wherein: After entering the screen projection display mode, the method further includes: Acquire desktop data of the terminal device; The image formed by the desktop is determined as the image to be displayed.

6. The method according to claim 1, wherein: After entering the screen projection display mode, the method further includes: In response to detecting a first trigger event of the third button, entering the first screen projection display mode; In response to detecting a second trigger event of the third button, entering the second screen projection display mode; In response to detecting a trigger event of the fourth button, switching from the projection display mode to the mixed reality display mode.

7. The method according to claim 6, wherein: The displaying the third button and the fourth button on the screen includes: Call and execute a pop-up window display instruction to control the screen to display a pop-up window including the third button and the fourth button within a preset time length.

8. The method according to claim 1, wherein: After entering the screen projection display mode, the method further includes: Determining the image to be displayed displayed by the terminal device as the current display image of the screen; In response to receiving a user operation on a current display image displayed on the screen, updating data of the image to be displayed; Based on the updated data of the image to be displayed, the projection image is updated, and the updated projection image and the image to be displayed are displayed on the head-mounted electronic device and the terminal device respectively.

9. The method according to claim 1, wherein: The rendering of a mixed reality operation image based on the posture information of the head mounted electronic device includes: Acquire posture information of the head-mounted electronic device and the reality data to be mixed, wherein the posture information is used to characterize the position and posture of the head-mounted electronic device; According to the posture information, adjusting the position and posture of a virtual camera in a preset rendering engine; The mixed reality operation image is generated by rendering the reality data to be mixed based on the adjusted virtual camera.

10. The method according to claim 1, wherein: The controlling the head mounted electronic device to display a mixed reality operation image comprises: For a target object in a mixed reality operation image displayed by the head-mounted electronic device, in response to receiving an operation of the target object by a user, calling and executing a control instruction for the target object to update the mixed reality operation image; Sending an updated mixed reality operation image to the head-mounted electronic device so that the head-mounted electronic device displays the updated mixed reality operation image.

11. The method according to claim 2, wherein: The mixed reality interface includes a mixed reality display setting button; The method further comprises: In response to receiving a second preset operation of the user on the mixed reality display setting button, parameters of the mixed reality display mode are configured.

12. A device for displaying information, applied to a terminal device, the device comprising: an acquisition unit, configured to acquire a display request of the head-mounted electronic device, and display a first button and a second button on the screen of the terminal device, wherein the first button is used to select a projection display mode, and the second button is used to select a mixed reality display mode; The first display unit is configured to enter the projection display mode in response to detecting a trigger event of the first button, obtain data of the image to be displayed, render the data of the image to be displayed to generate a projection image, and display the projection image and the image to be displayed on the head-mounted electronic device and the terminal device respectively; the first display unit also includes a display module, configured to display a third button and a fourth button on the screen in response to receiving a first preset operation of the user on the screen, the third button is used to select the first projection display mode or the second projection display mode, the size of the projection image in the first projection display mode is larger than the size of the projection image in the second projection display mode, the first projection display mode is entered when a first trigger event of the third button is detected, and the second projection display mode is entered when a second trigger event of the third button is detected, and the fourth button is used to switch from the projection display mode to the mixed reality display mode, so that the user can directly switch from the projection display mode to the mixed reality display mode without returning to the interface displaying the first button and the second button to enter the mixed reality display mode again; The second display unit is configured to enter the mixed reality display mode in response to detecting a trigger event of the second button, render a mixed reality operation image based on the posture information of the head-mounted electronic device, and control the head-mounted electronic device to display the mixed reality operation image, wherein the mixed reality display mode is characterized by a combination of a head-mounted electronic device and a terminal device to provide a virtual image that can interact and feedback with the user in real time in a real environment, and the mixed reality operation image is updated based on the user's operation of the mixed reality operation image. The head-mounted electronic device displays a switch button, and the switch button is used to switch from the mixed reality display mode to the projection display mode, so that the user can switch from the mixed reality display mode to the projection display mode without having to return to the interface displaying the first button and the second button to enter the projection display mode again.

13. A terminal device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images