Scene display method and device, equipment and medium

By determining the target scene in the virtual reality device and rendering it with pre-generated video, the delay and performance problems of virtual reality devices when rendering virtual scenes are solved, and efficient and real-time virtual scene display is achieved, improving the user experience.

CN119052440APending Publication Date: 2024-11-29BEIJING QIYI CENTURY SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410998690.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Existing virtual reality devices are prone to delay problems when rendering virtual scenes, and due to high network requirements or limited device performance, the scene effect is poor.

Method used

By determining the first target scene and the second target scene to be displayed in the virtual reality device, and rendering it with the pre-generated target video, a comprehensive scene picture is generated, which is completely implemented by the device itself, avoiding network factors and reducing device performance requirements. .

Benefits of technology

Real-time and low-latency virtual scene rendering is realized, improving user experience and ensuring the realistic and stable scene effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119052440A_ABST
    Figure CN119052440A_ABST
Patent Text Reader

Abstract

The embodiment of the invention relates to a scene display method and device, equipment and a medium. The method comprises the steps of determining a to-be-displayed first target scene and a to-be-displayed second target scene; calling a target video corresponding to the second target scene; wherein the target video is generated in advance; determining a to-be-rendered object in the first target scene; and rendering based on the to-be-rendered object and the target video to obtain a comprehensive scene picture, and displaying the comprehensive scene picture. According to the embodiment of the invention, a comprehensive scene picture combining different scenes can be efficiently and reliably presented for a user, the scene effect presented for the user is well guaranteed, the method is not limited by network factors, the method has high real-time performance, and the performance requirement for virtual reality equipment can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of multimedia technology, and in particular to a scene display method, device, equipment and medium. Background Art

[0002] Nowadays, virtual reality devices such as VR (Virtual Reality) headsets have been widely used in scenarios such as gaming, education, and social interaction. Virtual reality devices can present virtual scenes to users through display screens such as glasses, giving users an immersive experience.

[0003] Virtual scenes need to be rendered before being displayed to users. There are two existing ways to render virtual scenes. The first method requires the use of a server to complete the rendering of the virtual scene and transmit the rendered image to the virtual reality device in real time for display. This method is prone to delays and other problems and has high network requirements. The second method requires the virtual reality device to perform local real-time rendering, but it is limited by the performance of the virtual reality device and usually presents a poor scene effect to the user. In summary, the existing scene display method needs to be improved. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a scene display method, device, equipment and medium.

[0005] The disclosed embodiment provides a scene display method, which is applied to a virtual reality device, and includes: determining a first target scene and a second target scene to be displayed; retrieving a target video corresponding to the second target scene; wherein the target video is pre-generated; determining an object to be rendered in the first target scene; rendering a comprehensive scene picture based on the object to be rendered and the target video, and displaying the comprehensive scene picture.

[0006] Optionally, the first target scene is a scene containing near-field objects, and the second target scene is a scene containing far-field objects; and / or, the first target scene is a preset real scene or the first target scene is a virtual scene obtained by mapping the preset real scene, and the second target scene is a virtual scene; and / or, the first target scene is a scene containing interactive objects, and the second target scene is a scene containing non-interactive objects.

[0007] Optionally, determining the object to be rendered in the first target scene includes: determining the object to be rendered in the first target scene based on a current viewing angle and / or a current position of the virtual reality device.

[0008] Optionally, the first target scene is a virtual scene obtained by mapping based on a preset real scene, and determining the object to be rendered in the first target scene based on the current viewing angle and / or current position of the virtual reality device includes: determining the target physical object in the preset real scene based on the current viewing angle and / or current position of the virtual reality device in the preset real scene; wherein the target physical object is located within the visible range of the virtual reality device, and / or the distance between the target physical object and the virtual reality device is within a preset distance range; based on a preset mapping relationship between physical objects and virtual objects, determining a target virtual object corresponding to the target physical object in the first target scene; and using the target virtual object as the object to be rendered in the first target scene.

[0009] Optionally, the rendering based on the object to be rendered and the target video obtains a comprehensive scene picture, including: rendering the object to be rendered based on a pre-established 3D model of the object to be rendered to obtain a first scene picture; and obtaining a second scene picture corresponding to the target video at the current moment; and generating a comprehensive scene picture based on the first scene picture and the second scene picture.

[0010] Optionally, the rendering of the object to be rendered based on the pre-established 3D model of the object to be rendered to obtain a first scene picture includes: determining basic rendering information corresponding to the object to be rendered based on the pre-established 3D model of the object to be rendered and the current viewing angle of the virtual reality device; determining additional rendering information corresponding to the object to be rendered based on picture content in the second scene picture; and rendering the object to be rendered based on the basic rendering information and the additional rendering information to obtain the first scene picture.

[0011] Optionally, generating a comprehensive scene picture based on the first scene picture and the second scene picture includes: based on the position of the object to be rendered, superimposing the first scene picture and the second scene picture to obtain a comprehensive scene picture.

[0012] Optionally, based on the position of the object to be rendered, the first scene picture and the second scene picture are superimposed to obtain a comprehensive scene picture, including: obtaining a special effects picture rendered in advance; based on the position of the object to be rendered, the first scene picture, the second scene picture and the special effects picture are superimposed to obtain a comprehensive scene picture.

[0013] Optionally, the target video corresponding to the second target scene is pre-generated based on the following steps: based on a preset 3D model of the second target scene and preset scene animation sequence information, the second target scene is rendered to obtain multiple rendered frame images; the formats of the multiple rendered frame images are converted into an equirectangular projection format; a panoramic video is obtained by synthesizing the multiple rendered frame images after format conversion, and the panoramic video is used as the target video corresponding to the second target scene.

[0014] Optionally, rendering the second target scene includes: performing offline rendering on the second target scene using a preset rendering tool.

[0015] Optionally, the first target scene is determined based on a current position of the virtual reality device, and the second target scene is determined based on scene information sent by a server.

[0016] The disclosed embodiment also provides a scene display device, including: applied to a virtual reality device, including: a scene determination module, used to determine a first target scene and a second target scene to be displayed; a video retrieval module, used to retrieve a target video corresponding to the second target scene; wherein the target video is pre-generated; an object determination module, used to determine an object to be rendered in the first target scene; a scene display module, used to render a comprehensive scene picture based on the object to be rendered and the target video, and display the comprehensive scene picture.

[0017] The embodiment of the present disclosure also provides a virtual reality device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the scene display method provided by the embodiment of the present disclosure.

[0018] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the scene display method provided by the embodiment of the present disclosure.

[0019] The above technical solution provided by the embodiment of the present disclosure can determine the first target scene and the second target scene to be displayed, determine the object to be rendered in the first target scene, and retrieve the target video of the pre-generated second target scene, so as to render the object to be rendered and the target video to obtain a comprehensive scene picture for display. The above method can be completely implemented by the virtual reality device itself, without the help of a server, so it will not be limited by network factors, and can effectively avoid problems such as network delays, and has strong real-time performance. In addition, the above method can pre-generate corresponding videos for some scenes (that is, the second target scene), which can be directly retrieved in actual application, and then combined with the objects to be rendered in other scenes (that is, the first target scene) for rendering to obtain a comprehensive scene picture, which can not only reduce the performance requirements for virtual reality devices, but also can efficiently and reliably present a comprehensive scene picture combining different scenes to users, and better guarantee the scene effect presented to users.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 A schematic diagram of a flow chart of a scene display method provided in an embodiment of the present disclosure;

[0024] Figure 2 A schematic diagram of a scenario provided for an embodiment of the present disclosure;

[0025] Figure 3 A schematic diagram of a scene display process provided by an embodiment of the present disclosure;

[0026] Figure 4 A schematic diagram of a flow chart of a scene display method provided in an embodiment of the present disclosure;

[0027] Figure 5 A schematic diagram of the structure of a scene display device provided in an embodiment of the present disclosure;

[0028] Figure 6A schematic diagram of the structure of a virtual reality device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0031] Figure 1 The present invention provides a flow chart of a scene display method provided in an embodiment of the present invention. The method can be executed by a scene display device, wherein the device can be implemented by software and / or hardware and can generally be integrated in a virtual reality device. Figure 1 As shown, the method mainly includes the following steps S102 to S108:

[0032] Step S102, determine the first target scene and the second target scene to be displayed. The first target scene and the second target scene can be regarded as different local scenes in the comprehensive scene displayed to the user. The embodiment of the present disclosure does not limit the specific scene content contained in the first target scene and the second target scene.

[0033] In practical applications, different scenes can be flexibly divided according to needs. In some implementation examples, the first target scene and the second target scene can be one of the following three division methods, or a combination of two or three of the following three division methods. Specifically: Method 1: The scene is divided according to the distance of the objects in the comprehensive scene presented to the user. At this time, the first target scene is a scene containing near-view objects, and the second target scene is a scene containing far-view objects; Method 2: The scene is divided according to whether the objects in the comprehensive scene presented to the user are real objects or whether they are mapped based on real objects. At this time, the first target scene is a preset real scene or the first target scene is a virtual scene obtained by mapping based on the preset real scene, and the second target scene is a virtual scene; Method 3: The scene is divided according to whether the objects in the comprehensive scene presented to the user are interactive. At this time, the first target scene is a scene containing interactive objects, and the second target scene is a scene containing non-interactive objects. It should be noted that the relationship between the above three methods is and / or, that is, each method can be used alone, or two or three methods can be used in combination, which is not limited here.

[0034] In some implementation examples, the first target scene is determined based on the current position of the virtual reality device, and the second target scene is determined based on the scene information sent by the server. In practical applications, the virtual reality device can be a VR head-mounted device, and the position of the virtual reality device is also the current position of the user of the virtual reality device (the wearer of the VR head-mounted device). Based on this position, the real scene where the user of the virtual reality device is currently located can be known. In addition, in the embodiment of the present disclosure, the target video of the second target scene can be pre-rendered and generated. The virtual reality device stores one or more target videos. The scene information sent by the server can include the video identification information of the second target scene to be displayed. The server can determine the second target scene to be displayed based on one or more factors of the current time, game progress and the current position of the virtual reality device, and send a display instruction carrying the video identification information of the second target scene to the virtual reality device. In the above manner, the virtual reality device can accurately know the first target scene and the second target scene to be displayed.

[0035] Step S104, retrieve a target video corresponding to the second target scene; wherein the target video is generated in advance.

[0036] In order to reduce the real-time rendering pressure of the virtual reality device on the basis of ensuring the scene display effect and reduce the performance requirements for the virtual reality device, the embodiment of the present disclosure can render and generate the target video of the second target scene in advance. The embodiment of the present disclosure does not limit the format of the target video. In order to further ensure that the user has a good visual experience, the target video can be a panoramic video, which is also a 360-degree panoramic video and can be generated using panoramic video technology. The pre-generated target video can be stored in the virtual reality device for retrieval when needed. It should be noted that in actual applications, the target video can be generated by other devices besides the virtual reality scene, such as using a device with strong performance to generate a target video with rich details offline, and then providing the target video to the virtual reality device for storage. The specific settings can be flexibly set and are not limited here.

[0037] Step S106: determining the objects to be rendered in the first target scene.

[0038] In practical applications, the objects to be rendered in the first target scene may be virtual objects, physical objects, or virtual objects associated with physical objects. In some embodiments, all objects contained in the first target scene may be used as objects to be rendered. In other embodiments, the objects to be rendered in the first target scene may be determined based on the current viewing angle and / or current position of the virtual reality device. Specifically, the current viewing angle and / or current position of the virtual reality device is the current viewing angle and / or current position of the user of the virtual reality device. The user's head posture information, eyeball information, and other information may be collected and analyzed by one or more sensors such as cameras and position sensors configured on the virtual reality device, thereby determining the current viewing angle and / or current position of the virtual reality device. In practical applications, based on the current viewing angle of the virtual reality device, objects in the first target scene that are within the sight range of the virtual reality device may be used as objects to be rendered. Based on the current position of the virtual reality device, objects in the first target scene that are within a preset distance from the virtual reality device may be used as objects to be rendered. In practical applications, the objects to be rendered in the first target scene may also be determined in combination with the viewing angle and current position of the virtual reality device. The specific setting may be flexible and is not limited here. Through the above method, the processing resources of the virtual reality device can be effectively saved without affecting the user's viewing experience, and the rendering efficiency of the virtual reality device can be improved.

[0039] In actual applications, for multiplayer game scenarios, each user wears a corresponding virtual reality device, and each virtual reality device can receive the video identification information of the second target scene sent by the server in parallel, so that the second target scene (such as a virtual distant view) viewed by each user is the same. However, due to the different positions / perspectives of each user, the content of the first target scene that they can view may be different, that is, the objects to be rendered in the first target scene corresponding to different virtual reality devices may be different.

[0040] Step S108, obtaining a comprehensive scene picture based on the object to be rendered and the target video, and displaying the comprehensive scene picture.

[0041] In practical applications, real-time rendering can be performed on the object to be rendered, and then combined with the target video obtained by pre-rendering. By combining real-time rendering with pre-rendering, a comprehensive scene picture that simultaneously presents different scenes can be obtained. The comprehensive scene picture contains both information about the first target scene and information about the second target scene, giving users a rich visual experience.

[0042] The above technical solution provided by the embodiment of the present disclosure determines the first target scene and the second target scene to be displayed, determines the object to be rendered in the first target scene, and retrieves the target video of the pre-generated second target scene, so as to render the object to be rendered and the target video to obtain a comprehensive scene picture for display. The above method can be completely implemented by the virtual reality device itself, without the help of a server, so it will not be limited by network factors, and can effectively avoid problems such as network delays, and has strong real-time performance. In addition, the above method can pre-generate corresponding panoramic videos for some scenes (that is, the second target scene), which can be directly retrieved in actual application, and then combined with the objects to be rendered in other scenes (that is, the first target scene) for rendering to obtain a comprehensive scene picture, which can not only reduce the performance requirements for virtual reality devices, but also can efficiently and reliably present a comprehensive scene picture combining different scenes to users, and better guarantee the scene effect presented to users.

[0043] In some embodiments, the first target scene is a virtual scene obtained by mapping a preset real scene. When determining the object to be rendered in the first target scene in the aforementioned step S106, the object to be rendered in the first target scene may be determined based on the current viewing angle and / or current position of the virtual reality device. Specifically, the following steps 1 to 3 may be referred to for execution:

[0044] Step 1: Based on the current viewing angle and / or current position of the virtual reality device in the preset real scene, determine the target entity object in the preset real scene; wherein, the target entity object is within the visible range of the virtual reality device, and / or the distance between the target entity object and the virtual reality device is within the preset distance range. The preset distance range can be flexibly set according to the needs, and is not limited here. In practical applications, the user of the virtual reality device is in the real scene. The disclosed embodiment does not limit the type of entity objects in the preset real scene. Any contactable physical objects such as floors, handrails, tables and chairs are entity objects. In practical applications, all entity objects in the first target scene can be used as target entity objects. In other embodiments, some entity objects in the first target scene can be used as target entity objects. Specifically, the target entity objects in the first target scene can be determined based on the current viewing angle and / or current position of the virtual reality device, so that rendering resources can be effectively saved and rendering efficiency can be improved on the basis of ensuring user perception.

[0045] Step 2: Based on a preset mapping relationship between physical objects and virtual objects, determine a target virtual object corresponding to the target physical object in the first target scene.

[0046] In order to enable users to have a more exciting and interesting immersive experience, the disclosed embodiment may not render the real form of the physical object, but map the physical object with the virtual object. Specifically, the physical object with a mapping relationship has the same or similar form, touch, posture and other one or more features as the virtual object, and the position of the physical object and the virtual object has a corresponding relationship. The relative position relationship between different virtual objects is also consistent with the relative position relationship between different physical objects. The position of the physical object in the real scene corresponds to the position of the virtual object with a mapping relationship with the physical object viewed by the user through the virtual reality device. For example, assuming that modern tables and chairs and carpets are placed in the real experience scene, a mapping relationship can be established between the modern tables and chairs and the ancient tables and chairs, and a mapping relationship can be established between the carpet and the lawn. Taking the virtual reality device as a VR head-mounted device as an example, the user can view the effect of placing ancient tables and chairs on the lawn through the glasses screen of the VR head-mounted device, and the user can actually touch the real tables and chairs. The carpet can also provide the user with a realistic feeling of walking on the lawn. In addition, the relative position relationship between the carpet and the modern tables and chairs placed in the real scene is consistent with the relative position relationship between the lawn and the ancient tables and chairs viewed by the user through the glasses screen. The above are all exemplary descriptions. In actual applications, the mapping relationship between the physical objects in the first target scene and the virtual objects that you want to present to the user can be flexibly set according to needs, such as mapping a shaking wooden board in the scene to a virtual ship, mapping an empty picture frame to a virtual picture frame containing specific content, etc.

[0047] Step three: Use the target virtual object as the object to be rendered in the first target scene.

[0048] Through the above steps one to three, the user can be in a real scene, and the target physical object in the real scene can provide the user with a tangible real physical sensation. Rendering the target virtual object associated with the target physical object can provide the user with a virtual perception, which comprehensively provides the user with an interesting experience combining physical objects with virtual objects, and can also better enhance the user's sense of scene involvement.

[0049] In some implementations, the step of obtaining a comprehensive scene image based on the object to be rendered and the target video in the above step S108 can be performed with reference to the following steps A to B:

[0050] Step A, based on the pre-established 3D model of the object to be rendered, render the object to be rendered to obtain the first scene picture; and obtain the second scene picture corresponding to the target video at the current moment. In order to ensure the user's visual perception, in some specific implementation examples, the target video is a panoramic video, and the second scene picture is carried on the surface of a preset spherical model. In practical applications, a 3D model of the object to be rendered can be pre-constructed so that the object to be rendered can be rendered directly based on the 3D model. The position of the 3D model matches the position of the physical object in the first target scene, thereby achieving the effect of mapping the real object space to the virtual experience space. In addition, a spherical model can also be constructed and rendered for the spherical model. The second scene picture can be attached to the surface of the spherical model, and the panoramic video can be played through the spherical model. In this way, the user can be presented with a viewing effect of viewing the virtual distant view from any angle from top to bottom, left to right.

[0051] In order to achieve a realistic rendering effect of the object to be rendered, in some specific implementation examples, based on a pre-established 3D model of the object to be rendered, the object to be rendered is rendered, and the step of obtaining the first scene image can be performed with reference to the following steps (1) to (3):

[0052] Step (1) determines basic rendering information corresponding to the object to be rendered based on a pre-established 3D model of the object to be rendered and the current viewing angle of the virtual reality device; wherein the current viewing angle of the virtual reality device is also the current viewing angle of the user of the virtual reality device. It is understandable that the user of the virtual reality device sees different images of the object to be rendered due to different viewing angles of the virtual reality device, and therefore the rendering angle of the 3D model of the object to be rendered can be determined based on the current viewing angle of the virtual reality device, and the basic rendering information includes information such as the rendering angle of the object to be rendered.

[0053] Step (2), based on the picture content in the second scene picture, determine the additional rendering information corresponding to the object to be rendered. In the embodiment of the present disclosure, the object to be rendered is not rendered separately, but the visual effect of the second scene picture on the object to be rendered will be fully considered when rendering the object to be rendered. In order to highlight this effect, the additional rendering information of the object to be rendered can be determined based on the picture content. The additional rendering information is the information that needs to be added during the rendering process of the object to be rendered due to the picture content in the second scene picture. For example, the virtual distant view presents the effect of fireworks blooming in the night sky. Based on the position of the fireworks, the color of the fireworks and other contents presented in the second scene picture, the additional rendering information such as the color adjustment information corresponding to the object to be rendered can be determined so that the object to be rendered can appear to be illuminated by fireworks.

[0054] Step (3) renders the object to be rendered based on the basic rendering information and the additional rendering information to obtain a first scene image. Specifically, a 3D engine may be used to render the object to be rendered based on the basic rendering information and the additional rendering information, which will not be described in detail here. In this way, the first scene image can present a realistic perception of the real scene being affected by the virtual distant view, further improving the user experience.

[0055] Step B, generating a comprehensive scene picture based on the first scene picture and the second scene picture. Exemplarily, the first scene picture and the second scene picture can be superimposed based on the position of the object to be rendered to obtain a comprehensive scene picture. In the superposition process, it will be fully considered that the second scene picture may be blocked by the first scene picture, so it is necessary to determine the blocked content of the second scene picture based on the position of the object to be rendered. In other words, the content that can be displayed in the second scene picture is determined based on the position of the object to be rendered. For example, the object to be rendered is a virtual wall with a window, and part of the content of the second scene picture corresponding to the panoramic video can be viewed through the window. The comprehensive scene picture can present the picture content corresponding to the object to be rendered and the content in the second scene picture that can be presented through the window to the user.

[0056] In order to further provide users with a rich visual experience and save processing resources of virtual reality devices on this basis and improve the rendering efficiency of virtual reality devices, in some specific implementation examples, the step of superimposing the first scene picture and the second scene picture based on the position of the target virtual object to obtain a comprehensive scene picture can be performed with reference to the following steps (1) to (2):

[0057] Step (1), obtaining a pre-rendered special effect picture. In practical applications, a plane model can be constructed and rendered for the plane model. The special effect picture can be attached to the surface of the plane model and displayed through the plane model. The special effect picture can be a special effect with high performance consumption, such as a loop display special effect and a projectile special effect. In the above manner, there is no need for the virtual reality device to render the special effect in real time. It is only necessary to call the pre-rendered special effect picture, which can effectively save the processing resources of the virtual reality device and improve the rendering efficiency.

[0058] Step (2), based on the position of the target virtual object, the first scene picture, the second scene picture and the special effect picture are superimposed to obtain a comprehensive scene picture. Based on the position of the target virtual object, the positional relationship of the target virtual object relative to the second scene picture and the special effect picture can be determined, and the picture content that needs to be displayed in the second scene picture and the special effect picture can be obtained, so as to superimpose the first scene picture, the second scene picture and the special effect picture to obtain a comprehensive scene picture, presenting a rich visual experience of the integration of real close-up, virtual long-range and special effects to the user.

[0059] For ease of understanding, the disclosed embodiment further provides a method for generating a target video corresponding to the second target scene, that is, the target video corresponding to the second target scene is pre-generated based on the following steps a to c:

[0060] Step a, based on the preset 3D model of the second target scene and the preset scene animation sequence information, the second target scene is rendered to obtain multiple rendered frame images. In order to make the target video of the second target scene more realistic, the embodiment of the present disclosure can pre-set the 3D model of the second target scene, and render the 3D model, so as to better ensure the realism and exquisiteness of the distant rendering result. In addition, the embodiment of the present disclosure also sets the scene animation sequence information, which can indicate the scene action information, such as the movement information of the moon, the information of the geese flying in the sky, the information of the dock personnel on the distant shore, etc. Exemplarily, the scene animation sequence information can indicate the action information of the key frames of the animation sequence in the second target scene. Through the above method, the second target scene can be rendered to obtain multiple rendered frame images, and the rendered frame images can be used as video frame images to synthesize the target video.

[0061] For ease of processing, when rendering the second target scene, the second target scene can be rendered offline using a preset rendering tool. The preset rendering tool may be, for example, an advanced Monte Carlo ray tracing renderer, a movie-level rendering system that allows the scene to be rendered into a high-quality animation sequence, etc., which are not limited here. In addition, by using offline rendering, high-quality images can be generated using stronger computing resources. Since it is not subject to the time limit of real-time rendering, details such as complex lighting effects and shadows can be beautifully rendered, which can fully guarantee the rendering effect of the second target scene and present users with a beautiful and realistic virtual vision.

[0062] Step b: converting the formats of the multiple rendered frame images into an equirectangular projection format. In order to display the rendered frame images in the form of a 360-degree panoramic video, the formats of the rendered frame images can be converted into an equirectangular projection format in advance, which can be achieved by using relevant format conversion tools.

[0063] Step c, synthesizing a panoramic video based on the multiple rendered frame images after format conversion, and using the panoramic video as the target video corresponding to the second target scene. In practical applications, an existing game engine such as Unreal Engine can be used to generate a panoramic video, and the panoramic video can be played on a spherical model, which can also be understood as a spherical mesh shell. The position of the spherical model can be determined based on the position of the virtual reality device (that is, the user position of the virtual reality device), such as achieving the effect that the user is in the spherical shell.

[0064] For easier understanding, please refer to Figure 2 A schematic diagram of a scene is shown, in which the spherical model carrying the panoramic video is referred to as the panoramic video spherical shell, which is used to play the panoramic video corresponding to the virtual distant view. Taking the virtual reality device as a VR head-mounted device as an example, the user of the virtual reality device, that is, the wearer of the VR head-mounted device, is in the first target scene, Figure 2 The first target scene includes a floor and a touchable physical table and chair as an example to briefly illustrate the physical objects, which should not be regarded as a limitation. In actual applications, the physical objects that users can touch or interact with can be flexibly set in the real scene according to needs. Based on the user's perspective and position, the virtual reality device can present a comprehensive scene image that integrates the near view and the far view to the user through the display screen (the glasses of the VR headset). It should be noted that Figure 2 It is only for illustrative purposes, and the relative position relationship among the user, the physical object and the spherical shell is clearly presented. A comprehensive scene image can be rendered based on the relative position relationship and the user's perspective. It does not mean that the user and the physical object are actually located inside the spherical shell. In fact, the spherical shell is only a virtual model.

[0065] See also Figure 3 A schematic diagram of a scene display process is shown, which illustrates real-time rendering based on touchable near-field objects (corresponding to the physical objects in the aforementioned first target scene) to obtain a first scene image. In actual applications, the rendering can be directly based on the physical object itself to present the visual effect of the physical object to the user, or it can be based on the virtual object associated with the physical object to present the visual effect of the virtual object to the user. There is no limitation here. Pre-offline rendering is performed based on the 3D model and scene animation sequence in the second target scene to obtain a 360 panoramic video, thereby obtaining the second scene image at the current moment. The first scene image is superimposed on the second scene image to obtain and display the comprehensive scene image. Through the above-mentioned near-field real-time rendering and long-field pre-rendering method, the performance requirements for virtual reality equipment can be reduced on the basis of ensuring a better scene effect presented to the user, and it helps to improve rendering efficiency.

[0066] For easier understanding, see Figure 4 The flowchart of a scene display method shown in FIG. 4 is a schematic diagram, and the method mainly includes the following steps S402 to S420:

[0067] Step S402, determining a first target scene and a second target scene to be displayed; wherein the first target scene is a virtual near view obtained based on a preset real scene mapping, and the second target scene is a virtual far view.

[0068] Step S404, retrieve the panoramic video corresponding to the second target scene; wherein the panoramic video is generated in advance.

[0069] Step S406: determining a target entity object in the preset real scene based on the current viewing angle and / or current position of the virtual reality device in the preset real scene.

[0070] Step S408: Based on the preset mapping relationship between the physical object and the virtual object, determine the target virtual object corresponding to the target physical object in the first target scene, and use the target virtual object as the object to be rendered in the first target scene. In this way, both a tactile real body feeling and a virtual experience space that can map the real object space can be presented to the user, so as to achieve a real experience combining physical objects with virtual objects.

[0071] Step S410, obtaining a second scene image corresponding to the panoramic video at the current moment.

[0072] Step S412: determining basic rendering information corresponding to the object to be rendered based on a pre-established 3D model of the object to be rendered and the current viewing angle of the virtual reality device.

[0073] Step S414: determining additional rendering information corresponding to the object to be rendered based on the picture content in the second scene picture.

[0074] Step S416, based on the basic rendering information and the additional rendering information, the object to be rendered is rendered to obtain a first scene image. In this way, the first scene image can present a realistic feeling of the real scene being affected by the distant view, further improving the user experience.

[0075] Step S418: Based on the position of the object to be rendered, the first scene picture and the second scene picture are superimposed to obtain a comprehensive scene picture.

[0076] Step S420: display the comprehensive scene image through the display screen of the virtual reality device.

[0077] The specific implementation method of the above steps can refer to the relevant technology and is not limited here. Through the above method, the virtual distant view is presented through the panoramic video generated offline in advance, and the influence of the virtual distant view on the near view is fully considered. The near view is rendered in real time in combination with the virtual distant view, and finally a realistic scene picture that integrates the near view and the distant view is presented to the user, providing the user with a rich visual experience. The combination of real-time rendering of the near view and pre-rendering of the virtual distant view can effectively reduce the performance requirements of the virtual reality device while ensuring the scene effect, and improve the scene rendering and display efficiency.

[0078] Corresponding to the aforementioned scene display method, the embodiment of the present disclosure further provides a scene display device, Figure 5FIG. 1 is a schematic diagram of a scene display device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in a virtual reality device, such as Figure 5 As shown, the scene display device includes:

[0079] The scene determination module 502 is used to determine a first target scene and a second target scene to be displayed.

[0080] The video retrieval module 504 is used to retrieve the target video corresponding to the second target scene; wherein the target video is generated in advance.

[0081] The object determination module 506 is used to determine the object to be rendered in the first target scene.

[0082] The scene display module 508 is used to render a comprehensive scene picture based on the object to be rendered and the target video, and display the comprehensive scene picture.

[0083] The above-mentioned device provided by the embodiment of the present disclosure can determine the first target scene and the second target scene to be displayed, and determine the object to be rendered in the first target scene and call the target video of the pre-generated second target scene, so as to render the object to be rendered and the target video to obtain a comprehensive scene picture for display. The above-mentioned device can be completely implemented by the virtual reality device itself, without the help of a server, so it will not be limited by network factors, and can effectively avoid problems such as network delays, and has strong real-time performance. In addition, the above-mentioned device can pre-generate corresponding videos for some scenes (that is, the second target scene), which can be directly called in actual application, and then combined with the objects to be rendered in other scenes (that is, the first target scene) for rendering to obtain a comprehensive scene picture, which can not only reduce the performance requirements for virtual reality devices, but also can efficiently and reliably present a comprehensive scene picture combining different scenes to users, and better guarantee the scene effect presented to users.

[0084] In some embodiments, the first target scene is a scene containing near-view objects, and the second target scene is a scene containing far-view objects; and / or, the first target scene is a preset real scene or the first target scene is a virtual scene obtained by mapping based on the preset real scene, and the second target scene is a virtual scene; and / or, the first target scene is a scene containing interactive objects, and the second target scene is a scene containing non-interactive objects.

[0085] In some implementations, the object determination module 506 is specifically configured to determine the object to be rendered in the first target scene based on the current viewing angle and / or current position of the virtual reality device.

[0086] In some embodiments, the first target scene is a virtual scene obtained by mapping based on a preset real scene, and the object determination module 506 is specifically used to: determine the target physical object in the preset real scene based on the current viewing angle and / or current position of the virtual reality device in the preset real scene; wherein the target physical object is located within the visible range of the virtual reality device, and / or the distance between the target physical object and the virtual reality device is within a preset distance range; based on the mapping relationship between preset physical objects and virtual objects, determine the target virtual object corresponding to the target physical object in the first target scene; and use the target virtual object as the object to be rendered in the first target scene.

[0087] In some embodiments, the scene display module 508 is specifically used to: render the object to be rendered based on a pre-established 3D model of the object to be rendered to obtain a first scene picture; and obtain a second scene picture corresponding to the target video at the current moment; and generate a comprehensive scene picture based on the first scene picture and the second scene picture.

[0088] In some embodiments, the scene display module 508 is specifically used to: determine the basic rendering information corresponding to the object to be rendered based on a pre-established 3D model of the object to be rendered and the current viewing angle of the virtual reality device; determine the additional rendering information corresponding to the object to be rendered based on the picture content in the second scene picture; render the object to be rendered based on the basic rendering information and the additional rendering information to obtain a first scene picture.

[0089] In some implementations, the scene display module 508 is specifically configured to: based on the position of the object to be rendered, superimpose the first scene picture and the second scene picture to obtain a comprehensive scene picture.

[0090] In some embodiments, the scene display module 508 is specifically used to: obtain a pre-rendered special effect picture; based on the position of the object to be rendered, superimpose the first scene picture, the second scene picture and the special effect picture to obtain a comprehensive scene picture.

[0091] In some embodiments, the device also includes a video pre-generation module, which is specifically used to: render the second target scene based on a preset 3D model of the second target scene and preset scene animation sequence information to obtain multiple rendered frame images; convert the formats of the multiple rendered frame images into an equirectangular projection format; synthesize the multiple rendered frame images after format conversion to obtain a panoramic video, and use the panoramic video as the target video corresponding to the second target scene.

[0092] In some implementations, the scene display module 508 is specifically configured to: use a preset rendering tool to perform offline rendering on the second target scene.

[0093] In some implementations, the first target scene is determined based on the current location of the virtual reality device, and the second target scene is determined based on scene information sent by the server.

[0094] The scene display device provided in the embodiments of the present disclosure can execute the scene display method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0095] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0096] An embodiment of the present disclosure provides a virtual reality device, which includes: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of any method in the present disclosure.

[0097] Reference below Figure 6 , which shows a schematic diagram of the structure of a virtual reality device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The virtual reality device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0098] like Figure 6 As shown, the virtual reality device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the virtual reality device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0099] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the virtual reality device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The virtual reality device 600 having various devices is shown, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0100] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0101] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the image processing method provided by the embodiments of the present disclosure. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, the programming languages ​​including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0102] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the scene display method provided by the embodiment of the present disclosure.

[0103] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0104] The embodiment of the present disclosure also provides a computer program product, including a computer program / instruction, which implements the scene display method in the embodiment of the present disclosure when the computer program / instruction is executed by a processor.

[0105] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0106] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0107] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0108] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0109] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0110] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A scene display method, characterized in that: Applied to virtual reality devices, including: Determining a first target scene and a second target scene to be displayed; Retrieving a target video corresponding to the second target scene; wherein the target video is generated in advance; Determining an object to be rendered in the first target scene; A comprehensive scene picture is obtained based on the object to be rendered and the target video rendering, and the comprehensive scene picture is displayed.

2. The method according to claim 1, characterized in that The first target scene is a scene containing a near-view object, and the second target scene is a scene containing a far-view object; and / or, The first target scene is a preset real scene or the first target scene is a virtual scene mapped based on the preset real scene, and the second target scene is a virtual scene; and / or, The first target scene is a scene containing interactive objects, and the second target scene is a scene containing non-interactive objects.

3. The method according to claim 1, characterized in that The determining of the to-be-rendered object in the first target scene includes: Based on the current viewing angle and / or current position of the virtual reality device, an object to be rendered in the first target scene is determined.

4. The method according to claim 3, characterized in that The first target scene is a virtual scene obtained by mapping a preset real scene, and determining the object to be rendered in the first target scene based on the current viewing angle and / or current position of the virtual reality device includes: Determine a target entity object in the preset real scene based on a current viewing angle and / or a current position of the virtual reality device in the preset real scene; wherein the target entity object is within a visible range of the virtual reality device, and / or a distance between the target entity object and the virtual reality device is within a preset distance range; Based on a preset mapping relationship between physical objects and virtual objects, determining a target virtual object corresponding to the target physical object in the first target scene; The target virtual object is used as an object to be rendered in the first target scene.

5. The method according to claim 1, characterized in that The rendering based on the object to be rendered and the target video to obtain a comprehensive scene picture includes: Based on the pre-established 3D model of the object to be rendered, the object to be rendered is rendered to obtain a first scene picture; and a second scene picture corresponding to the target video at the current moment is obtained; Based on the first scene picture and the second scene picture, a comprehensive scene picture is generated.

6. The method according to claim 5, characterized in that The step of rendering the object to be rendered based on the pre-established 3D model of the object to be rendered to obtain a first scene picture includes: Determining basic rendering information corresponding to the object to be rendered based on a pre-established 3D model of the object to be rendered and a current viewing angle of the virtual reality device; Determining additional rendering information corresponding to the object to be rendered based on the picture content in the second scene picture; Based on the basic rendering information and the additional rendering information, the object to be rendered is rendered to obtain a first scene picture.

7. The method according to claim 5, characterized in that The generating a comprehensive scene picture based on the first scene picture and the second scene picture includes: Based on the position of the object to be rendered, the first scene picture and the second scene picture are superimposed to obtain a comprehensive scene picture.

8. The method according to claim 7, characterized in that The step of superimposing the first scene picture and the second scene picture based on the position of the object to be rendered to obtain a comprehensive scene picture includes: Get the pre-rendered special effects picture; Based on the position of the object to be rendered, the first scene picture, the second scene picture and the special effect picture are superimposed to obtain a comprehensive scene picture.

9. The method according to any one of claims 1 to 8, characterized in that: The target video corresponding to the second target scene is pre-generated based on the following steps: Based on a preset 3D model of a second target scene and preset scene animation sequence information, the second target scene is rendered to obtain a plurality of rendered frame images; Convert the formats of the plurality of rendered frame images into an equirectangular projection format; A panoramic video is synthesized based on the multiple rendered frame images after format conversion, and the panoramic video is used as a target video corresponding to the second target scene.

10. The method according to claim 9, characterized in that The rendering of the second target scene includes: The second target scene is rendered offline using a preset rendering tool.

11. The method according to claim 1, characterized in that: The first target scene is determined based on the current position of the virtual reality device, and the second target scene is determined based on the scene information sent by the server.

12. A scene display device, characterized in that: Applied to virtual reality devices, including: A scene determination module, used to determine a first target scene and a second target scene to be displayed; A video retrieval module, used to retrieve a target video corresponding to the second target scene; wherein the target video is generated in advance; An object determination module, used to determine an object to be rendered in the first target scene; The scene display module is used to render a comprehensive scene picture based on the object to be rendered and the target video, and display the comprehensive scene picture.

13. A virtual reality device, characterized in that: The virtual reality device comprises: a storage device having a computer program stored thereon; A processing device, used to execute the computer program in the storage device to implement the steps of the scene display method described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the scene display method described in any one of claims 1 to 11.