Picture display method and device, equipment and medium

Abstract

The invention provides a method and device for displaying a picture, equipment and a medium. According to the specific implementation scheme, display information of a to-be-displayed picture is obtained, and the display information is used for representing whether the to-be-displayed picture comprises a virtual scene image or not; in response to the virtual scene image included in the to-be-displayed picture, constructing a virtual scene, rendering an original map of the to-be-displayed picture into the virtual scene, and generating a rendering result of the to-be-displayed picture; calling an image rendering driving component to render the rendering result again to obtain a to-be-displayed picture; in response to the situation that the to-be-displayed picture does not comprise the virtual scene image, calling an image rendering driving component to render an original map of the to-be-displayed picture to obtain the to-be-displayed picture; and sending the to-be-displayed picture to the head-mounted display device for display.

Description

Technical Field

[0001] The present disclosure relates to the technical field of scene rendering, and in particular, to a method, device, equipment, and medium for displaying a picture. Background Art

[0002] In related technologies, a head-mounted display device can achieve different effects such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) by sending light to the eyes of the wearing user. When the head-mounted display device is displaying, it needs to drive multiple modules such as hardware components, algorithms, and communication, and usually has a high energy consumption. Therefore, how to reduce the energy consumption of the head-mounted display device has become an urgent problem to be solved in the field. Summary of the Invention

[0003] In a first aspect, an embodiment of the present disclosure provides a method for displaying a picture, including: obtaining display information of a picture to be displayed, where the display information is used to represent whether the picture to be displayed includes a virtual scene image; in response to the picture to be displayed including a virtual scene image, constructing a virtual scene, rendering an original texture map of the picture to be displayed into the virtual scene, and generating a rendering result of the picture to be displayed; calling an image rendering driver component to render the rendering result again to obtain the picture to be displayed; in response to the picture to be displayed not including a virtual scene image, calling the image rendering driver component to render the original texture map of the picture to be displayed to obtain the picture to be displayed; and sending the picture to be displayed to a head-mounted display device for display.

[0004] In a second aspect, an embodiment of the present disclosure provides a device for displaying a picture, including: an obtaining module, configured to obtain display information of a picture to be displayed, where the display information is used to represent whether the picture to be displayed includes a virtual scene image; a first processing module, configured to, in response to the picture to be displayed including a virtual scene image, construct a virtual scene, render an original texture map of the picture to be displayed into the virtual scene, and generate a rendering result of the picture to be displayed; call an image rendering driver component to render the rendering result again to obtain the picture to be displayed; a second processing module, configured to, in response to the picture to be displayed not including a virtual scene image, call the image rendering driver component to render the original texture map of the picture to be displayed to obtain the picture to be displayed; and a sending module, configured to send the picture to be displayed to a head-mounted display device for display.

[0005] In a third aspect, an embodiment of the present disclosure provides a computer-readable storage medium storing a computer program for executing the method for displaying a picture provided in any one of the above embodiments of the present disclosure.

[0006] Fourth aspect, embodiments of the present disclosure provide an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for displaying a screen provided in any of the above embodiments of the present disclosure.

[0007] Fifth aspect, embodiments of the present disclosure provide a head-mounted display device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for displaying a screen provided in any of the above embodiments of the present disclosure.

[0008] Sixth aspect, embodiments of the present disclosure provide a computer program product, including computer program instructions, which when run by a processor, cause the processor to execute the method for displaying a screen provided in any of the above embodiments.

[0009] The technical solutions of the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Description of the Drawings

[0010] By describing the embodiments of the present disclosure in more detail with reference to the accompanying drawings, the above and other objects, features, and advantages of the present disclosure will become more obvious. The drawings are used to provide a further understanding of the embodiments of the present disclosure, and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation to the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 is an exemplary system architecture that can be applied to embodiments of the method or device for generating a display screen of the present disclosure;

[0012] Figure 2 is a flowchart of the method for displaying a screen provided in some exemplary embodiments of the present disclosure;

[0013] Figure 3 is a flowchart of the process for generating a rendering result of a screen to be displayed provided in some exemplary embodiments of the present disclosure;

[0014] Figure 4 is a flowchart of the process for re-rendering the rendering result provided in some exemplary embodiments of the present disclosure;

[0015] Figure 5 is a flowchart of the process for re-rendering the rendering result provided in some other exemplary embodiments of the present disclosure;

[0016] Figure 6It is a schematic diagram of the rendering process of the screen to be displayed of non-virtual scene images provided by some exemplary embodiments of the present disclosure;

[0017] Figure 7 It is a flowchart of a method for displaying a screen provided by some exemplary embodiments of the present disclosure;

[0018] Figure 8 It is a schematic structural diagram of a device for displaying a screen provided by some exemplary embodiments of the present disclosure;

[0019] Figure 9 It is a schematic structural diagram of some application embodiments of an electronic device of the present disclosure. Detailed implementation manners

[0020] Next, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. It should be understood that the present disclosure is not limited by the exemplary embodiments described herein.

[0021] It should be noted that: Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and values set forth in these embodiments do not limit the scope of the present disclosure.

[0022] Those skilled in the art can understand that terms such as "first", "second", etc. in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate an inevitable logical order between them.

[0023] It should also be understood that in the embodiments of the present disclosure, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0024] It should also be understood that the present disclosure emphasizes the differences between the various embodiments. The similarities or similarities between them can be referred to each other. For the sake of brevity, they will not be described one by one.

[0025] Embodiments of the present disclosure can be applied to electronic devices such as head-mounted display devices, terminal devices, servers, etc., which can operate together with many other general-purpose or special-purpose computing system environments or configurations. Electronic devices such as head-mounted display devices, terminal devices, servers, etc. can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, target programs, components, logics, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, where tasks are executed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0026] The following describes in detail the specific embodiments of the present disclosure with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining and illustrating the present disclosure, and are not used to limit the present disclosure.

[0027] Figure 1 is an exemplary system architecture of an embodiment of a method or apparatus for generating a display screen that can be applied to the present disclosure. The system architecture can include a head-mounted display device 1, a network 2, and a terminal device 3. The network 2 can be a medium providing a communication link between the head-mounted display device 1 and the terminal device 3. The network 2 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0028] The above-mentioned head-mounted display device 1 can be an electronic device with an image display function and an audio playback function. Users can view flat or three-dimensional pictures or videos through the head-mounted display device. The head-mounted display device includes, but is not limited to, augmented reality (AR) devices, virtual reality (VR) devices, mixed reality (MR) devices, etc.

[0029] The head-mounted display device 1 generally can include an optical imaging system, a frame for carrying the optical imaging system, etc. Through the optical imaging system, users can view an image with a suitable size. Through the frame, users can wear the head-mounted display device 1 on the head to view the image through the optical imaging system. The form of the frame can be in the form of glasses, headbands, helmets, etc.

[0030] In some alternative embodiments, the head-mounted display device 1 may be an all-in-one device. The all-in-one head-mounted display device 1 may integrate the function of a computing unit, and the computing unit may provide the head-mounted display device 1 with task processing capabilities such as algorithm operation. Various client applications installed on the head-mounted display device 1 are actually installed in the computing unit. Optionally, the head-mounted display device 1 may also be a split device. The split head-mounted display device 1 may include a head-mounted display and a computing unit that is separately arranged from the head-mounted display. The display screen of the head-mounted display is used to display the image processed by the computing unit. The computing unit in the split head-mounted display device 1 may be a terminal device, such as Figure 1 as shown, the terminal device may include, but is not limited to, a screen mirroring device, a smart phone, a tablet computer, an e-book reader, a laptop computer, a desktop computer, and so on. For the case where the execution subject is the head-mounted display device 1, those skilled in the art should understand that for a split head-mounted display device, the execution subject may be the computing unit.

[0031] The terminal device 3 may be an electronic device that provides various services. The terminal device 3 may be a mobile phone, a tablet, a computer, etc. Figure 1 Taking the screen mirroring device as an example, the screen mirroring device may obtain the display information of the to-be-displayed picture. If the to-be-displayed picture is a virtual scene image, a virtual scene may be constructed, the original texture map of the to-be-displayed picture may be rendered into the virtual scene, a rendering result of the to-be-displayed picture may be generated, and the image rendering driver component may be called to render the rendering result again to obtain the to-be-displayed picture. If the to-be-displayed picture is a non-virtual scene image, the image rendering driver component may be called to render the original texture map of the to-be-displayed picture to obtain the to-be-displayed picture, and finally the to-be-displayed picture may be sent to the head-mounted display device 1 for display. Correspondingly, the device for displaying the picture may be arranged in the terminal device 3.

[0032] It should also be noted that the method for displaying a picture provided in the present disclosure may also be applied to the head-mounted display device 1. When the method of the present disclosure is applied to the head-mounted display device 1, the head-mounted display device 1 obtains the display information of the to-be-displayed picture. If the to-be-displayed picture is a virtual scene image, a virtual scene is constructed, the original texture map of the to-be-displayed picture is rendered into the virtual scene, a rendering result of the to-be-displayed picture is generated, and the image rendering driver component is called to render the rendering result again to obtain the to-be-displayed picture; if the to-be-displayed picture is a non-virtual scene image, the image rendering driver component is called to render the original texture map of the to-be-displayed picture to obtain the to-be-displayed picture, and finally the to-be-displayed picture is sent to the display screen of the head-mounted display device 1 for display. In this case, the above system architecture may not include the terminal device 3. Correspondingly, the device for displaying the picture may be arranged in the head-mounted display device 1.

[0033] In addition to the head-mounted display device 1 and the terminal device 3, the method for displaying a picture can also be applied to a server (not shown in the figure), and the server can be a background server. Correspondingly, the device for displaying a picture can be arranged in the server. There is no unique limitation here.

[0034] Exemplary method

[0035] Figure 2 It is a schematic flowchart of a method for displaying a picture provided by some exemplary embodiments of the present disclosure. The embodiments of the present disclosure can be applied to an electronic device, such as Figure 2 As shown, the method includes the following steps:

[0036] Step 210, obtain the display information of the picture to be displayed.

[0037] In some optional embodiments of the present disclosure, the display information is used to characterize whether the picture to be displayed includes a virtual scene image. The execution subject of the method for displaying a picture (for example, Figure 1 the screen mirroring device shown) can obtain the display information of the picture to be displayed.

[0038] In some optional embodiments of the present disclosure, the display information of the above-mentioned picture to be displayed can be generated by any implementable method. For example, the display information of the picture to be displayed can be generated by any method such as the interaction state of the head-mounted display device, the type of application program corresponding to the picture to be displayed, and the user's independent selection. The specific content of the display information can be set according to actual needs so as to determine whether the picture to be displayed includes a virtual scene image according to the display information. Optionally, the virtual scene image can be understood as an image obtained by generating virtual information for the original texture map and performing scene rendering on the generated virtual information. As an example, for the original texture map, an image rendering engine such as unity3D is usually used to obtain the corresponding virtual information and perform virtual scene rendering, so as to obtain a virtual scene image.

[0039] In some optional embodiments of the present disclosure, the virtual scene image can refer to an image that needs to perform virtual scene rendering on the original image to be displayed. For example, in the scenario where the user interacts with the picture to be displayed through the head-mounted display device, the user needs to adjust the picture brightness, adjust the sound, etc., and an interaction interface for adjusting the brightness and adjusting the sound needs to be drawn on the original image to be displayed. Therefore, virtual scene rendering is required. For another example, for a three-dimensional application program (3D APP), in order to enable the user to perceive the three-dimensional visual effect, virtual scene rendering can be performed. The virtual scene is a scene of a constructed virtual three-dimensional space. The three-dimensional objects in the virtual three-dimensional space can be observed through a virtual camera to obtain a virtual scene image.

[0040] It can be understood that through this step, it can be determined whether it is necessary to perform virtual scene rendering on the original map of the screen to be displayed. Alternatively, through this step, it can be determined whether it is necessary to call an image rendering engine such as unity3D to perform virtual scene rendering on the original map.

[0041] Step 220, in response to the screen to be displayed including a virtual scene image, construct a virtual scene, render the original map of the screen to be displayed into the virtual scene, generate a rendering result of the screen to be displayed; call an image rendering driver component to perform secondary rendering on the rendering result to obtain the screen to be displayed.

[0042] In some optional embodiments of the present disclosure, if the screen to be displayed includes a virtual scene image, it means that the screen to be displayed needs to be rendered with a virtual scene. First, a virtual scene is constructed using an image rendering engine or the like, and then the original map of the screen to be displayed is rendered into the virtual scene, so that a rendering result of the screen to be displayed can be generated.

[0043] In some optional embodiments of the present disclosure, the original map of the screen to be displayed may include the original picture to be displayed, the original video frame image, etc. Optionally, the original map of the screen to be displayed may also include the map of the interaction interface for adjusting brightness, adjusting sound, etc., and there is no unique limitation here.

[0044] In some optional embodiments of the present disclosure, after obtaining the rendering result of the screen to be displayed through virtual scene rendering, an image rendering driver component (which can be referred to as an SDK) can be called to perform secondary rendering on the rendering result to obtain the screen to be displayed. The secondary rendering may include, for example, performing time warping, space warping, etc. on the rendering result to improve the effect of the screen to be displayed in terms of real-time performance and display effect.

[0045] Generally, the image rendering driver component can support the implementation of virtual reality or augmented reality, etc., so that the image can be displayed on a head-mounted display device. The image rendering driver component may include various application components such as content rendering and scene recognition. After content rendering, the image rendering driver component can display digital information and / or 3D objects as images. Therefore, the secondary rendering of the above image rendering driver component can render the rendering result such as the 3D object obtained by rendering in the virtual scene as the content to be re-rendered and then display it.

[0046] Step 230, in response to the screen to be displayed not including a virtual scene image, call an image rendering driver component to render the original map of the screen to be displayed to obtain the screen to be displayed.

[0047] In some alternative embodiments of the present disclosure, if the to-be-displayed screen does not include a virtual scene image, it can be understood that the to-be-displayed screen is a non-virtual scene image. In such a case, for the original texture map of the to-be-displayed screen, there is no need for virtual scene processing, and the digital information of the original texture map can be directly rendered to obtain the image to be displayed on the head-mounted display device.

[0048] In some alternative embodiments of the present disclosure, if the to-be-displayed screen is a non-virtual scene image, the image rendering driver component can be called to directly display the image on the specified spatial plane without virtual scene rendering. For example, when a user views a picture, a video, etc. through the head-mounted display device, the picture, the video, etc. can be rendered to the specified spatial plane through the image rendering driver component, and the user can view the content displayed on the specified spatial plane through the head-mounted display device.

[0049] Step 240, send the to-be-displayed screen to the head-mounted display device for display.

[0050] In some alternative embodiments of the present disclosure, after obtaining the to-be-displayed screen, the to-be-displayed screen can be sent to the head-mounted display device for display on the display screen of the head-mounted display device.

[0051] The method for displaying a screen provided by the embodiments of the present disclosure can determine whether the to-be-displayed screen includes a virtual scene image by obtaining the display information of the to-be-displayed screen. For a virtual scene image, a virtual scene can be constructed, and the original texture map of the to-be-displayed screen can be rendered to the virtual scene to generate a rendering result of the to-be-displayed screen. Then, the obtained rendering result can be re-rendered by calling the image rendering driver component to obtain the to-be-displayed screen. For a non-virtual scene image, the image rendering driver component is directly called to render the original texture map of the to-be-displayed screen to obtain the to-be-displayed screen. The method of the embodiments of the present disclosure differentiates between virtual scene images and non-virtual scene images for the to-be-displayed screen. For non-virtual scene images, virtual scene rendering is skipped, which effectively reduces the rendering time of the to-be-displayed screen while balancing the display effect, helps save power consumption, and can improve the real-time performance of the screen.

[0052] In some alternative embodiments of the present disclosure, the display information of the to-be-displayed screen can be obtained through the following steps: determining the interaction state of the head-mounted display device, where the interaction state includes either interaction or no interaction; in response to the interaction state being no interaction, obtaining the display information of the to-be-displayed screen, where the display information is used to represent that the to-be-displayed screen does not include a virtual scene image; in response to the interaction state being interaction, obtaining the display information of the to-be-displayed screen, where the display information is used to represent that the to-be-displayed screen includes a virtual scene image.

[0053] In some alternative embodiments of the present disclosure, the interaction state of the head-mounted display device can be determined based on whether an interaction component is invoked. For example, when the user adjusts the picture sound, adjusts the picture brightness, etc. through interaction operations, the interaction components for adjusting the sound, adjusting the brightness, etc. can be invoked. At this time, it can be determined that the head-mounted display device is in an interactive state. It can be understood that any implementable method can be specifically adopted to determine the interaction state of the head-mounted display device, such as whether the above-mentioned execution entity receives operations such as clicks from the user.

[0054] In some alternative embodiments of the present disclosure, in response to the interaction state of the head-mounted display device being non-interactive, the display information of the display screen is obtained, and the display information is used to characterize that the to-be-displayed screen does not include a virtual scene image. In response to the interaction state of the head-mounted display device being interactive, the display information of the display screen is obtained, and the display information is used to characterize that the to-be-displayed screen includes a virtual scene image.

[0055] It can be understood that in an interactive scenario, the to-be-displayed screen displayed on the head-mounted display device can include the original image and the interaction components for the user to adjust the device attributes. In this case, it is necessary to add interaction components to the original image through the rendering of the virtual scene. Therefore, the original texture map of the to-be-displayed screen is first rendered with the virtual scene to obtain a rendering result including the interaction components, and then the rendering result is rendered again, so that the to-be-displayed image that can be displayed on the head-mounted display device can be obtained. In a non-interactive scenario, the content of the to-be-displayed screen displayed on the head-mounted display device is the same as the content of the original image. In this case, the image rendering driver component can be called to directly render the original texture map into the to-be-displayed image that can be displayed on the head-mounted display device.

[0056] In this embodiment, by determining the interaction state of the head-mounted display device to determine whether the to-be-displayed screen includes a virtual scene image, it helps to obtain the to-be-displayed screen through virtual scene rendering combined with calling the image rendering driver component in an interactive scenario, ensuring the display effect of the to-be-displayed screen. In a non-interactive scenario, virtual scene rendering is skipped, reducing the rendering calculation amount, improving the rendering efficiency, and the impact of skipping virtual scene rendering on the display effect of the to-be-displayed screen in a non-interactive scenario is relatively small, thereby achieving power saving in rendering and improving the real-time performance of the screen while balancing the display effect.

[0057] In some alternative embodiments of the present disclosure, the display information of the to-be-displayed screen can be obtained through the following steps: determining the application program type corresponding to the to-be-displayed screen. In response to the application program type being a preset type, obtaining the display information of the to-be-displayed screen, where the display information is used to characterize that the to-be-displayed screen includes a virtual scene image. In response to the application program type not being a preset type, obtaining the display information of the to-be-displayed screen, where the display information is used to characterize that the to-be-displayed screen does not include a virtual scene image.

[0058] In some alternative embodiments of the present disclosure, the application types may include two-dimensional applications (2D APPs), three-dimensional applications (3D APPs), etc. The preset type may be an application with a stereoscopic display effect such as a three-dimensional application. It can be understood that those skilled in the art can set the application of the preset type according to actual needs.

[0059] In some alternative embodiments of the present disclosure, the type of the application corresponding to the to-be-displayed screen can be determined according to the specific application selected by the user. For example, if the user selects to open Application A, the application type corresponding to the to-be-displayed screen is the type of Application A.

[0060] Optionally, the type of the application opened by the user can be determined by means of a preset whitelist, etc. As an example, the 3D APP installed on the head-mounted display device can be set in the whitelist. When the user opens an application, it can be determined whether the application is in the whitelist. If so, it means that the application opened by the user is a preset type of application.

[0061] In some alternative embodiments of the present disclosure, if the type of the application corresponding to the to-be-displayed screen is the preset type, it can be indicated that the to-be-displayed screen includes a virtual scene image. Therefore, the display information of the to-be-displayed screen obtained by the above execution entity can be the display information indicating that the to-be-displayed screen includes a virtual scene image. In this case, the to-be-displayed screen on the head-mounted display device needs to be rendered for the virtual scene. For example, the to-be-displayed screen can be indicated as a virtual scene image and a non-virtual scene image respectively through display information of different contents.

[0062] In this embodiment, whether the to-be-displayed screen includes a virtual scene image is determined by the application type of the to-be-displayed screen, so that the virtual scene of the preset type of application can be rendered and combined with the call of the image rendering driver component to obtain the to-be-displayed screen, ensuring the display effect of the to-be-displayed screen of the preset type of application. For non-preset type applications, virtual scene rendering can be skipped, reducing the rendering calculation amount and improving the rendering efficiency. The solution disclosed in this embodiment can save rendering power consumption and improve the real-time performance of the screen while ensuring the display effects of different types of applications.

[0063] In some alternative embodiments of the present disclosure, the display information of the to-be-displayed screen can be obtained through the following steps: determining the rendering mode selected by the user; in response to the rendering mode being the preset mode, obtaining the display information of the selected to-be-displayed screen, where the display information is used to indicate that the to-be-displayed screen does not include a virtual scene image; in response to the selected rendering mode not being the preset mode, obtaining the display information of the to-be-displayed screen, where the display information is used to indicate that the to-be-displayed screen includes a virtual scene image.

[0064] In some alternative embodiments of the present disclosure, the rendering mode may include a two-layer rendering mode and a one-layer rendering mode. Here, the two-layer rendering mode may include the above-mentioned virtual scene rendering (the first layer of rendering) and the re-rendering (the second layer of rendering) by invoking the image rendering driver component. For example, the two-layer rendering may be unity3D (virtual scene rendering) + re-rendering of the SDK. The one-layer rendering mode is the mode of directly invoking the image rendering driver component to render the original texture map of the to-be-displayed screen. For example, the one-layer rendering mode may be the SDK rendering mode.

[0065] In some alternative embodiments, the above preset mode may be the one-layer rendering mode. In response to the rendering mode selected by the user being the preset mode, the display information of the to-be-displayed screen obtained is information indicating that the to-be-displayed screen does not include virtual scene images. In response to the rendering mode selected by the user not being the preset mode, the display information of the to-be-displayed screen obtained is information indicating that the to-be-displayed screen includes virtual scene images.

[0066] In the solution disclosed in this embodiment, the user can independently select the mode for rendering the to-be-displayed screen, so as to determine whether the to-be-displayed screen includes virtual scene images. This solution helps to perform rendering using the corresponding rendering mode according to the user's needs to obtain the to-be-displayed screen. When the user pays attention to power consumption, the virtual scene rendering can be skipped, reducing the rendering calculation amount and improving the rendering efficiency. When the user pays attention to the display effect, the two-layer rendering mode can be used for rendering to enhance the display effect, thus meeting different needs of the user and improving the user experience.

[0067] Figure 3 It is a schematic flowchart of generating the rendering result of the to-be-displayed screen provided by some exemplary embodiments of the present disclosure.

[0068] In some alternative embodiments of the present disclosure, as Figure 3 shown, the rendering result of the to-be-displayed screen can be generated through the following steps:

[0069] Step 22110, determine the first device pose information of the head-mounted display device.

[0070] In some alternative embodiments of the present disclosure, the first device pose information of the head-mounted display device may include at least one of the position information and the attitude information of the head-mounted display device in the world coordinate system. The above-mentioned execution subject may position the head-mounted display device based on the data collected by the sensors on the head-mounted display device to obtain the first device pose information of the head-mounted display device. For example, the first device pose information of the head-mounted display device may be determined by means of Simultaneous Localization and Mapping (SLAM). The first device pose information of the head-mounted display device may also be determined by other means.

[0071] Step 22120: Use a preset image rendering engine to construct a virtual scene.

[0072] In some alternative embodiments of the present disclosure, the image rendering engine may be any implementable rendering engine, such as the unity3D engine or other rendering engines. The above-mentioned execution subject may construct a virtual scene through the image rendering engine.

[0073] Step 22130: Determine the first camera pose information of the first virtual camera in the virtual scene according to the first device pose information.

[0074] In some alternative embodiments of the present disclosure, the first virtual camera may be a structure used for rendering in the virtual scene, and may be used to observe an object in the virtual scene and render the object.

[0075] In some alternative embodiments of the present disclosure, the above-mentioned first device pose information may be the pose information in the world coordinate system, and there is a correlation between the world coordinate system and the virtual space coordinate system of the virtual scene. Therefore, the pose information of the head-mounted display device in the virtual space coordinate system can be determined according to the first device pose information of the head-mounted display device. The pose information of the head-mounted display device in the virtual space coordinate system may include at least one of the position information and the attitude information of the head-mounted display device in the virtual space coordinate system. The first virtual camera has a relatively fixed association relationship with the head-mounted display device. Therefore, according to the pose information of the head-mounted display device in the virtual space coordinate system, the first camera pose information of the first virtual camera in the virtual scene can be determined.

[0076] Step 22140: Determine the pose information of the virtual plane in the virtual space coordinate system according to the first camera pose information of the first virtual camera.

[0077] In some alternative embodiments of the present disclosure, the virtual plane may be a plane or layer for displaying a to-be-displayed picture in a virtual scene. For example, the virtual plane may be a plane at a preset distance from the first virtual camera. After determining the first camera pose information of the first virtual camera in the virtual scene, the pose information of the virtual plane in the virtual space coordinate system may be calculated according to the relative position relationship between the virtual plane and the first virtual camera.

[0078] Step 22150: Render the original texture map of the to-be-displayed picture onto the virtual plane to obtain the rendering result of the to-be-displayed picture.

[0079] In some alternative embodiments of the present disclosure, the original texture map of the to-be-displayed picture may be mapped to various positions on the virtual plane, so that a picture corresponding to the original texture map of the to-be-displayed picture is displayed on the virtual plane, and the rendering result of the to-be-displayed picture is obtained. The rendering result of the to-be-displayed picture may be a rendered virtual scene image. That is, the scene image seen by observing the virtual scene through the first virtual camera.

[0080] In some alternative embodiments, the original texture map of the to-be-displayed picture may include an original picture to be displayed, an original video frame image, etc. For an interactive scene, in the constructed virtual scene, the texture map of the interactive component may be superimposed on the original texture map. For example, according to the occlusion relationship of the texture map of the interactive component in the to-be-displayed picture, the original texture map of the to-be-displayed picture may be rendered onto the virtual plane, and the texture map of the interactive component may be rendered onto the layer on the virtual plane of the original texture map.

[0081] In this embodiment, the pose information of the virtual plane in the virtual space coordinate system is determined through the first camera pose information of the first virtual camera in the virtual scene, and the original texture map of the to-be-displayed picture is rendered onto the virtual plane to obtain the rendering result of the to-be-displayed picture. In the virtual scene rendering, only one texture (buffer) of the first virtual camera is rendered. Compared with rendering two textures in the virtual scene in the related art, the embodiments of the present disclosure can effectively reduce the computational amount and power consumption of the virtual scene rendering.

[0082] Figure 4 It is a schematic flowchart of re-rendering the rendering result provided by some exemplary embodiments of the present disclosure.

[0083] In some alternative embodiments of the present disclosure, as Figure 4 shown, the re-rendering of the rendering result may be performed through the following steps:

[0084] Step 22210: Determine the second device pose information of the head-mounted display device.

[0085] In some alternative embodiments of the present disclosure, the method for obtaining the second device pose information of the head-mounted display device is the same as that for obtaining the first device pose information, which will not be elaborated here.

[0086] In some alternative embodiments of the present disclosure, the acquisition time of the second device pose information may be later than that of the first device pose information.

[0087] For example, in the case of considering time warp (such as asynchronous time warp (ATW)), the acquisition time of the second device pose information is later than that of the first device pose information. That is, after obtaining the rendering result of the virtual scene and before the display screen of the to-be-displayed image is turned on, and close to the time of turning on the screen, the pose information of the head-mounted display device is determined as the second device pose information.

[0088] In some alternative embodiments of the present disclosure, if time warp is not considered, the second device pose information may be obtained at the same time as the first device pose information. That is, the second device pose information may be the first device pose information.

[0089] Step 22220: According to the first device pose information and the second device pose information, call the image rendering driver component to perform time warp on the rendering result of the generated to-be-displayed image, so as to obtain the to-be-displayed image.

[0090] In some alternative embodiments of the present disclosure, time warp may refer to, for a rendered frame of an image (i.e., the rendering result of the to-be-displayed image), correcting the rendering result according to the angle by which the user's head rotates after the virtual scene is rendered, so as to obtain a frame of image that is more accurate relative to the current position of the user's head. Time warp can reduce the latency of the user's viewing of the image. Therefore, according to the pose change information of the second device pose information relative to the first device pose information, while reducing the power consumption of the image display, the rendering result of the generated to-be-displayed image can be corrected, and the display effect of the to-be-displayed image can be improved.

[0091] For the rendering result of the virtual scene, the solution disclosed in this embodiment can call the image rendering driver component to perform time warp on the rendering result, so as to obtain the to-be-displayed image. That is, after rendering a texture in the virtual scene, time warp is performed during the rendering by the image rendering driver component to obtain a to-be-displayed image. The embodiment of the present disclosure helps to further reduce the rendering calculation amount and further save the rendering power consumption while ensuring the display effect of the to-be-displayed image.

[0092] Figure 5 It is a schematic flowchart of re-rendering the rendering result provided by some other exemplary embodiments of the present disclosure.

[0093] In some alternative embodiments of the present disclosure, re-rendering the rendering result by invoking the image rendering driver component may be performed through the following steps:

[0094] Step 22230: Based on the parallax information and the first camera pose information of the first virtual camera, determine the second camera pose information of the second virtual camera and the third camera pose information of the third virtual camera.

[0095] In some alternative embodiments of the present disclosure, the second virtual camera may be a camera corresponding to the first screen of the head-mounted display device. The third virtual camera may be a camera corresponding to the second screen of the head-mounted display device.

[0096] In some alternative embodiments of the present disclosure, the parallax information may refer to the horizontal pixel displacement difference information between corresponding points on the imaging planes of two cameras. The parallax information has an associated relationship with the depth information. In order to enable the user to view a picture with a sense of depth through the first screen and the second screen of the head-mounted display device, the planar image rendered in the virtual scene with the first virtual camera as the center camera can be used as the image to be rendered by the image rendering driver component. The stereo image with depth information is obtained by simulating the parallax principle of human binoculars viewing the same object through the second virtual camera and the third virtual camera. Here, based on the parallax information and the first camera pose information of the first virtual camera, the second camera pose information of the second virtual camera corresponding to the first screen and the third camera pose information of the third virtual camera corresponding to the second screen can be determined. The parallax information may be the parallax information generated by the second virtual camera corresponding to the first screen and the third virtual camera corresponding to the second screen at a certain depth (for example, the depth of the virtual plane relative to the camera). By observing the virtual plane through the second virtual camera and the third virtual camera, the parallax of the human eye viewing the first screen and the second screen can be simulated, thereby achieving a three-dimensional visual effect.

[0097] Step 22240: Invoke the image rendering driver component and move the rendering result based on the second camera pose information and the third camera pose information.

[0098] In some alternative embodiments of the present disclosure, after determining the second camera pose information of the second virtual camera and the third camera pose information of the third virtual camera, an image rendering driver component may be invoked to move the rendering results according to the second camera pose information and the third camera pose information, obtaining the first moved rendering result and the second moved rendering result. The process of obtaining the first moved rendering result and the second moved rendering result by translating the rendering result of the virtual scene rendering may be referred to as spatial warping (or SpaceWarp). Spatial warping can be understood as spatially warping (or warping) an image. Since there is parallax information in the imaging planes of the second virtual camera and the third virtual camera, the positions of the same point on the rendering result are different in the first moved rendering result and the second moved rendering result after movement, thereby generating parallax on the first screen and the second screen, enabling the user to view a picture with a depth effect.

[0099] In some alternative embodiments of the present disclosure, the movement of the rendering result may be achieved by respectively converting the rendering result from the camera coordinate system of the first virtual camera to the camera coordinate systems of the second virtual camera and the third virtual camera.

[0100] Step 22250, rendering the moved rendering result again to obtain the first picture to be displayed and the second picture to be displayed.

[0101] In some alternative embodiments of the present disclosure, when considering temporal warping, rendering the moved rendering result again may include: respectively performing temporal warping on the first moved rendering result and the second moved rendering result according to the second device pose information of the head-mounted display device, obtaining the first picture to be displayed and the second picture to be displayed. The first picture to be displayed is for display on the first screen of the head-mounted display device, and the second picture to be displayed is for display on the second screen.

[0102] In some alternative embodiments of the present disclosure, when not considering temporal warping, the first moved rendering result may be used as the first picture to be displayed, and the second moved rendering result may be used as the second picture to be displayed.

[0103] In this embodiment, the second camera pose information of the second virtual camera and the third camera pose information of the third virtual camera are determined based on the parallax information and the first camera pose information of the first virtual camera, which are used to translate the rendering result obtained from the first-layer rendering, realizing the spatial distortion of the rendering result, obtaining two translated rendering results with parallax, and further obtaining the first display screen corresponding to the first screen and the second display screen corresponding to the second screen, and there is a certain parallax between the two, so as to provide the user with a display screen with a three-dimensional effect and achieve the balance between power consumption saving and visual effect. The solution disclosed in this embodiment can divide an image rendered in a virtual scene into two display screens that can achieve a three-dimensional stereoscopic effect in an image rendering driver component, further improving the stereoscopic effect of the head-mounted display device while saving power consumption.

[0104] It can be understood that the solution disclosed in this embodiment can distort the first virtual camera pose according to the parallax information to the second virtual camera and third virtual camera poses in the image rendering driver component, so that the image rendering driver component renders with the parallax information superimposed, and a display screen with the depth of a three-dimensional scene can be obtained. The distortion of the virtual camera pose in this solution is independent of time and does not require image interpolation. It is the distortion of spatial information.

[0105] Figure 6 It is a schematic diagram of the rendering process of the display screen of a non-virtual scene image provided by some exemplary embodiments of the present disclosure.

[0106] In some alternative embodiments of the present disclosure, calling the image rendering driver component to render the original texture map of the display screen to obtain the display screen may include:

[0107] Step 2310, determine the attribute information of the spatial plane according to the first device pose information of the head-mounted display device.

[0108] In some alternative embodiments of the present disclosure, the attribute information may include the position information and size information of the spatial plane, etc.

[0109] In some alternative embodiments of the present disclosure, the spatial plane can be understood as a specified plane in the three-dimensional space viewed through the head-mounted display device. The position information of the spatial plane may include at least one of the position information and attitude of the spatial plane in the three-dimensional space. The size information of the spatial plane may include the height information and width information of the spatial plane, etc.

[0110] In some alternative embodiments of the present disclosure, the attribute information of the spatial plane can be set according to actual needs. For example, the spatial plane can be a plane with a specified size at a preset distance in front of the head-mounted display device, a specified size plane in the upper left corner within the viewing range of the head-mounted display device, etc.

[0111] Step 2320: Invoke the image rendering driver component to obtain the original texture map of the picture to be displayed.

[0112] In some optional embodiments of the present disclosure, for the case where the virtual scene image is not included, the original texture map of the picture to be displayed may be the original picture to be displayed, the original video frame image, etc. The image rendering driver component may obtain the original texture map of the picture to be displayed from the image rendering engine. The image rendering engine may call the decoding interface of the system (i.e., the system supporting the functions of the head-mounted display device) to decode the encoded data, so as to obtain the original texture map of the picture to be displayed. The image rendering engine may directly transmit the original texture map of the picture to be displayed to the image rendering driver component.

[0113] Step 2330: Render the original texture map of the picture to be displayed onto the spatial plane to obtain the picture to be displayed.

[0114] In some optional embodiments of the present disclosure, according to the attribute information of the spatial plane, each pixel of the original texture map of the picture to be displayed may be mapped to each plane point on the spatial plane to establish the mapping relationship between the original texture map of the picture to be displayed and the spatial plane, so that the original texture map of the picture to be displayed can be displayed on the spatial plane to obtain the picture to be displayed.

[0115] In this embodiment, for the case of non-virtual scene images, according to the first device pose information of the head-mounted display device, the attribute information of the spatial plane is determined, and the original texture map of the picture to be displayed is directly rendered onto the spatial plane to obtain the picture to be displayed. That is, in the case of non-virtual scene images, the first layer of rendering is omitted, and the image rendering driver component directly performs one-time rendering to obtain the picture to be displayed, and the picture to be displayed is displayed on the first screen and the second screen of the head-mounted display device, further reducing the rendering calculation amount and saving the rendering power consumption.

[0116] In some optional embodiments of the present disclosure, rendering the original texture map of the picture to be displayed onto the spatial plane in step 2330 to obtain the picture to be displayed includes: based on the parallax information, rendering the original texture map of the picture to be displayed onto the spatial plane to obtain a first picture to be displayed and a second picture to be displayed.

[0117] In some optional embodiments of the present disclosure, the first picture to be displayed and the second picture to be displayed may be respectively displayed on the first screen and the second screen of the head-mounted display device.

[0118] In some optional embodiments of the present disclosure, the parallax information is used to simulate the parallax of the user's left and right eyes. Based on the parallax information, the original texture map of the picture to be displayed is rendered onto the spatial plane to obtain the first picture to be displayed and the second picture to be displayed with parallax, so that the user can view a three-dimensional stereoscopic picture effect, improving the image display effect on the basis of saving power consumption.

[0119] In some alternative embodiments of the present disclosure, Figure 7 is a flowchart of a method for displaying a screen provided by some exemplary embodiments of the present disclosure. As Figure 7 shown, when a screen needs to be displayed, it can be first determined whether the screen to be displayed includes a virtual scene image. If the screen to be displayed includes a virtual scene image, an image rendering engine is called to generate a rendering result of the screen to be displayed, and then an image rendering driver component is called to render the rendering result again to obtain a first screen to be displayed and a second screen to be displayed. The first layer of rendering completes the rendering of one texture, and the second layer of rendering completes the rendering of two textures, with a total of three textures being rendered. Compared with the related art, the embodiments of the present disclosure can reduce the rendering of one texture in the first layer of rendering (rendering of images in the virtual scene). Since the first layer of rendering has a relatively large amount of computation, reducing one texture can significantly save power consumption. Moreover, the second layer of rendering (rendering of images by the image rendering driver component) can compensate for parallax information and perform spatial distortion using the parallax information, so that users can effectively perceive depth and provide a three-dimensional stereoscopic visual experience for users, while ensuring the image display effect while reducing power consumption.

[0120] If the screen to be displayed does not include a virtual scene image, the first layer of rendering is omitted, and the image rendering driver component is directly called to render based on the parallax information to obtain a first screen to be displayed and a second screen to be displayed with parallax. A total of two textures are rendered, which generally has no visual impact on non-virtual scene images (such as images of non-interactive scenes). And finally, the two different screens to be displayed are presented on the first screen and the second screen of the head-mounted display device, with three-dimensional parallax information, enabling users to perceive depth and experience a three-dimensional stereoscopic visual effect. Therefore, there is no loss in any display effect.

[0121] In some alternative embodiments of the present disclosure, according to the above embodiments, the rendering modes of the embodiments of the present disclosure can be divided into four types. Among them, for the virtual scene image scenario, there are two corresponding rendering modes: Rendering Mode 1: The image rendering engine performs the first-layer rendering to render a texture, and the image rendering driver component performs the second-layer rendering to render a display screen. Finally, the first screen and the second screen of the head-mounted display device display the same display screen, which can effectively save power consumption. Rendering Mode 2: The image rendering engine performs the first-layer rendering to render a rendering result, and the image rendering driver component performs the second-layer rendering to render two display screens. Finally, the first screen and the second screen of the head-mounted display device display two display screens with parallax. This solution can compensate for parallax information and improve the user viewing experience. When there is no virtual scene image, there are two corresponding rendering modes: Rendering Mode 3: The image rendering driver component can be directly called to render a display screen, omitting the first-layer rendering. Finally, the first screen and the second screen of the head-mounted display device display the same display screen, which can effectively save power consumption. Rendering Mode 4: Based on the parallax information on the basis of Rendering Mode 3, the image rendering driver component is called to render two display screens with parallax, which can compensate for parallax information and further improve the user viewing experience while saving power consumption.

[0122] In this solution, the user can select any one of the above Rendering Modes 1-4 of the embodiments of the present disclosure according to actual needs to render the display screen. And during the application process, the rendering mode can be switched in real time according to needs. For example, according to the processing power, energy consumption, remaining battery power, battery life, etc. of the terminal device used for rendering, as well as the application type, interaction status, user selection, etc., the rendering mode is switched to improve the flexibility of rendering.

[0123] The above embodiments or alternative examples of the present disclosure can be implemented separately or in any combination without conflict, which can be specifically set according to actual needs, and the present disclosure does not make any limitations.

[0124] Any method for displaying a screen provided by the embodiments of the present disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers, etc. Or, any method for displaying a screen provided by the embodiments of the present disclosure can be executed by a processor. For example, the processor executes any method for displaying a screen mentioned in the embodiments of the present disclosure by calling the corresponding instructions stored in the memory. This will not be elaborated further below.

[0125] Exemplary apparatus

[0126] Figure 8It is a schematic structural diagram of a device for displaying a picture provided by some exemplary embodiments of the present disclosure. The device for displaying a picture in this embodiment can be used to implement the corresponding method embodiment for displaying a picture of the present disclosure, such as Figure 8 The device shown includes: an acquisition module 310, a first processing module 320, a second processing module 330, and a sending module 340.

[0127] The acquisition module 310 is configured to acquire display information of a picture to be displayed, where the display information is used to characterize whether the picture to be displayed includes a virtual scene image.

[0128] The first processing module 320 is configured to, in response to the picture to be displayed including a virtual scene image, construct a virtual scene, render the original texture map of the picture to be displayed into the virtual scene, generate a rendering result of the picture to be displayed, and call an image rendering driver component to render the rendering result again to obtain the picture to be displayed.

[0129] The second processing module 330 is configured to, in response to the picture to be displayed not including a virtual scene image, call an image rendering driver component to render the original texture map of the picture to be displayed to obtain the picture to be displayed.

[0130] The sending module 340 is configured to send the picture to be displayed to a head-mounted display device for display.

[0131] In some optional embodiments of the present disclosure, the acquisition module 310 is specifically configured to: determine an interaction state of the head-mounted display device, where the interaction state includes any one of having interaction and having no interaction. In response to the interaction state being having no interaction, acquire display information of the picture to be displayed, where the display information is used to characterize that the picture to be displayed does not include a virtual scene image. In response to the interaction state being having interaction, acquire display information of the picture to be displayed, where the display information is used to characterize that the picture to be displayed includes a virtual scene image.

[0132] In some optional embodiments of the present disclosure, the acquisition module 310 is specifically configured to: determine an application program type corresponding to the picture to be displayed. In response to the application program type being a preset type, acquire display information of the picture to be displayed, where the display information is used to characterize that the picture to be displayed includes a virtual scene image. In response to the application program type not being a preset type, acquire display information of the picture to be displayed, where the display information is used to characterize that the picture to be displayed does not include a virtual scene image.

[0133] In some alternative embodiments of the present disclosure, the obtaining module 310 is specifically configured to: determine the rendering mode selected by the user. In response to the selected rendering mode being a preset mode, obtain the display information of the to-be-displayed picture, where the display information is used to characterize that the to-be-displayed picture does not include a virtual scene image. In response to the selected rendering mode not being a preset mode, obtain the display information of the to-be-displayed picture, where the display information is used to characterize that the to-be-displayed picture includes a virtual scene image.

[0134] In some alternative embodiments of the present disclosure, the first processing module 320 is specifically configured to: determine the first device pose information of the head-mounted display device. Use a preset image rendering engine to construct a virtual scene. According to the first device pose information, determine the first camera pose information of the first virtual camera in the virtual scene. According to the first camera pose information of the first virtual camera, determine the pose information of the virtual plane in the virtual space coordinate system. Render the original texture map of the to-be-displayed picture onto the virtual plane to obtain the rendering result of the to-be-displayed picture.

[0135] In some alternative embodiments of the present disclosure, the first processing module 320 is specifically configured to: determine the second device pose information of the head-mounted display device. Wherein, the acquisition time of the second device pose information is later than the acquisition time of the first device pose information. According to the first device pose information and the second device pose information, call the image rendering driver component to perform time warping on the rendering result of the generated to-be-displayed picture to obtain the to-be-displayed picture.

[0136] In some alternative embodiments of the present disclosure, the first processing module 320 is specifically configured to: based on the parallax information and the first camera pose information of the first virtual camera, determine the second camera pose information of the second virtual camera and the third camera pose information of the third virtual camera. Wherein, the second virtual camera is the camera corresponding to the first screen of the head-mounted display device. The third virtual camera is the camera corresponding to the second screen of the head-mounted display device. Call the image rendering driver component to move the rendering result based on the second camera pose information and the third camera pose information. Render the moved rendering result again to obtain the first to-be-displayed picture and the second to-be-displayed picture.

[0137] In some alternative embodiments of the present disclosure, the second processing module 330 is specifically configured to: according to the first device pose information of the head-mounted display device, determine the attribute information of the spatial plane. Wherein, the attribute information includes the position information and the size information of the spatial plane. Call the image rendering driver component to obtain the original texture map of the to-be-displayed picture. Render the original texture map of the to-be-displayed picture onto the spatial plane to obtain the to-be-displayed picture.

[0138] In some alternative embodiments of the present disclosure, the second processing module 330 is specifically configured to: based on the parallax information, render the original texture map of the to-be-displayed picture onto the spatial plane to obtain a first to-be-displayed picture and a second to-be-displayed picture.

[0139] In some alternative embodiments of the present disclosure, the first to-be-displayed picture and the second to-be-displayed picture are respectively displayed on the first screen and the second screen of the head-mounted display device.

[0140] It should be noted that the specific implementation manner of the device for displaying pictures in the embodiments of the present disclosure is similar to the specific implementation manner of the method for displaying pictures in the embodiments of the present disclosure. For details, please refer to the part about the method for displaying pictures. To reduce redundancy, it will not be elaborated here.

[0141] Exemplary electronic device

[0142] Embodiments of the present disclosure further provide an electronic device, including: a processor and a memory for storing processor-executable instructions.

[0143] The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method for displaying pictures described in any of the above embodiments of the present disclosure.

[0144] Exemplary head-mounted display device

[0145] Embodiments of the present disclosure further provide an electronic device, including: a processor and a memory for storing processor-executable instructions.

[0146] The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method for displaying pictures described in any of the above embodiments of the present disclosure.

[0147] Figure 9 It is a schematic structural diagram of some application embodiments of the electronic device of the present disclosure. The structure of the head-mounted display device can be referred to the structure of this electronic device. In this embodiment, the electronic device 100 includes one or more processors 110 and a memory 120.

[0148] The processor 110 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to execute desired functions.

[0149] The memory 120 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage media, and the processor 110 may run the program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage media.

[0150] In one example, the electronic device 100 may further include: an input device 130 and an output device 140, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0151] For example, the input device 130 may be the above-mentioned microphone or microphone array for capturing the input signal of the sound source.

[0152] In addition, the input device 130 may further include, for example, a keyboard, a mouse, etc.

[0153] The output device 140 may output various information to the outside, including the determined distance information, direction information, etc. The output device 140 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0154] Of course, for simplicity, Figure 9 only some of the components related to the present disclosure in the electronic device 100 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device 100 may further include any other appropriate components.

[0155] Exemplary computer program product and computer-readable storage medium

[0156] In addition to the above methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, and when the computer program instructions are run by a processor, the processor is caused to execute the steps in the methods according to the various embodiments of the present disclosure described in the "Exemplary Method" section above of this specification.

[0157] The computer program product may be written in any combination of one or more programming languages for executing the program code of the operations of the embodiments of the present disclosure. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, executed as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0158] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are run by a processor, the processor is caused to execute the steps in the methods according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0159] The computer-readable storage medium may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0160] It should also be noted that in the devices, apparatuses, and methods of the present disclosure, each component or each step may be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present disclosure.

[0161] The above description has been given for purposes of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for displaying a picture, comprising: Obtaining display information of the picture to be displayed, where the display information is used to characterize whether the picture to be displayed includes a virtual scene image; In response to the picture to be displayed including a virtual scene image, constructing a virtual scene, rendering the original texture map of the picture to be displayed into the virtual scene, generating a rendering result of the picture to be displayed; calling an image rendering driver component to perform secondary rendering on the rendering result to obtain the picture to be displayed; In response to the picture to be displayed not including a virtual scene image, calling the image rendering driver component to render the original texture map of the picture to be displayed to obtain the picture to be displayed; Sending the picture to be displayed to a head-mounted display device for display.

2. The method according to claim 1, wherein The obtaining of the display information of the picture to be displayed includes: Determining the interaction state of the head-mounted display device, where the interaction state includes either having interaction or not having interaction; In response to the interaction state being not having interaction, obtaining the display information of the picture to be displayed, where the display information is used to characterize that the picture to be displayed does not include a virtual scene image; In response to the interaction state being having interaction, obtaining the display information of the picture to be displayed, where the display information is used to characterize that the picture to be displayed includes a virtual scene image.

3. The method according to claim 1, wherein, The obtaining of the display information of the picture to be displayed includes: Determining the application program type corresponding to the picture to be displayed; In response to the application program type being a preset type, obtaining the display information of the picture to be displayed, where the display information is used to characterize that the picture to be displayed includes a virtual scene image; In response to the application program type not being the preset type, obtaining the display information of the picture to be displayed, where the display information is used to characterize that the picture to be displayed does not include a virtual scene image.

4. The method according to claim 1, wherein, The obtaining of the display information of the picture to be displayed includes: Determining the rendering mode selected by the user; In response to the selected rendering mode being a preset mode, obtaining the display information of the picture to be displayed, where the display information is used to characterize that the picture to be displayed does not include a virtual scene image; In response to the selected rendering mode not being the preset mode, obtaining the display information of the picture to be displayed, where the display information is used to characterize that the picture to be displayed includes a virtual scene image.

5. According to the method as claimed in any one of claims 1-4, wherein, The constructing of the virtual scene, rendering the original texture map of the picture to be displayed into the virtual scene, and generating the rendering result of the picture to be displayed includes: Determining the first device pose information of the head-mounted display device; Using a preset image rendering engine to construct the virtual scene; According to the first device pose information, determining the first camera pose information of the first virtual camera in the virtual scene; According to the first camera pose information of the first virtual camera, determining the pose information of the virtual plane in the virtual space coordinate system; Rendering the original texture map of the picture to be displayed onto the virtual plane to obtain the rendering result of the picture to be displayed.

6. The method according to claim 5, wherein, The calling of the image rendering driver component to perform secondary rendering on the rendering result to obtain the picture to be displayed includes: Determine the second device pose information of the head-mounted display device; the acquisition time of the second device pose information is later than the acquisition time of the first device pose information; According to the first device pose information and the second device pose information, call the image rendering driver component to perform temporal warping on the rendering result of the generated displayable frame to obtain the displayable frame.

7. The method according to claim 5 or 6, wherein The calling the image rendering driver component to re-render the rendering result to obtain the displayable frame includes: Based on the parallax information and the first camera pose information of the first virtual camera, determine the second camera pose information of the second virtual camera and the third camera pose information of the third virtual camera, where the second virtual camera is the camera corresponding to the first screen of the head-mounted display device; the third virtual camera is the camera corresponding to the second screen of the head-mounted display device; Call the image rendering driver component to move the rendering result based on the second camera pose information and the third camera pose information; Re-render the moved rendering result to obtain a first displayable frame and a second displayable frame.

8. According to the method described in any one of claims 1-4, wherein The calling the image rendering driver component to render the original texture map of the displayable frame to obtain the displayable frame includes: According to the first device pose information of the head-mounted display device, determine the attribute information of the spatial plane; the attribute information includes the position information and size information of the spatial plane; Call the image rendering driver component to obtain the original texture map of the displayable frame; Render the original texture map of the displayable frame to the spatial plane to obtain the displayable frame.

9. The method according to claim 8, wherein The rendering the original texture map of the displayable frame to the spatial plane to obtain the displayable frame includes: Based on the parallax information, render the original texture map of the displayable frame to the spatial plane to obtain a first displayable frame and a second displayable frame, where the first displayable frame and the second displayable frame are respectively displayed on the first screen and the second screen of the head-mounted display device.

10. A device for displaying a frame, comprising: An acquisition module, configured to acquire display information of a displayable frame, where the display information is used to characterize whether the displayable frame includes a virtual scene image; A first processing module, configured to, in response to the displayable frame including a virtual scene image, construct a virtual scene, render the original texture map of the displayable frame into the virtual scene to generate a rendering result of the displayable frame; call an image rendering driver component to re-render the rendering result to obtain the displayable frame; A second processing module, configured to, in response to the displayable frame not including a virtual scene image, call the image rendering driver component to render the original texture map of the displayable frame to obtain the displayable frame; A sending module, configured to send the displayable frame to a head-mounted display device for display.

11. A computer-readable storage medium storing a computer program for executing the method for displaying a frame according to any one of claims 1-9 above.

12. An electronic device, the electronic device comprising: A processor; A memory for storing executable instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for displaying a screen according to any one of claims 1-9 above.

13. A head-mounted display device, the head-mounted display device comprising: A processor; A memory for storing executable instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for displaying a screen according to any one of claims 1-9 above.

Images