Virtual reality live broadcast method and device, head-mounted display equipment, storage medium and program product

Abstract

The embodiment of the invention provides a virtual reality live broadcast method and device, head-mounted display equipment, a computer readable storage medium and a computer program product, and relates to the technical field of virtual reality. The virtual reality live broadcast method comprises the following steps: after an audience user enters a live broadcast room through head display equipment, obtaining a panoramic picture; configuring the panoramic picture as a virtual background in a virtual reality environment; rendering the virtual reality environment to generate a virtual reality environment picture; and displaying the virtual reality environment picture on a screen of the head-mounted display device. According to the invention, the audience user can watch the live video in an immersive manner, and the live watching experience of the audience user is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of virtual reality technology, and in particular to a virtual reality live broadcast method, apparatus, head-mounted display device, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid popularization of Internet technology, online live broadcasting is being accepted and loved by more and more people. When broadcasting live, the anchor will generally arrange the live broadcast scene to provide the audience with a better viewing experience. However, due to the limitations of the audience's terminal devices, no matter how the anchor arranges the live broadcast scene, the audience cannot achieve an immersive live viewing experience.

[0003] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0004] The embodiments of the present application provide a virtual reality live broadcast method, apparatus, head-mounted display device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0005] One aspect of an embodiment of the present application provides a virtual reality live broadcast method, which is applied to a head-mounted display device, and the method includes: After the audience user enters the live broadcast room through the head display device, a panoramic picture is obtained; configuring the panoramic image as a virtual background in a virtual reality environment; Rendering the virtual reality environment to generate a virtual reality environment picture; The virtual reality environment image is displayed on the screen of the head display device.

[0006] Optionally, configuring the panoramic picture as a virtual background in a virtual reality environment includes: Performing correction processing on the panoramic picture to obtain a corrected panoramic picture; The corrected panoramic picture is configured as a virtual background in a virtual reality environment.

[0007] Optionally, configuring the panoramic picture as a virtual background in the virtual reality environment further includes: Extracting the depth value of the panoramic image using a pre-trained depth estimation model; The depth value is configured as a depth value of the virtual background in the virtual reality environment.

[0008] Optionally, the correction processing includes color correction processing, and performing the correction processing on the panoramic image to obtain a corrected panoramic image includes: Obtaining color information of the live broadcast room; Color correction is performed on the panoramic picture according to the color information to obtain a corrected panoramic picture.

[0009] Optionally, the method further comprises: The panoramic picture is subjected to format verification processing, and after the panoramic picture passes the format verification processing, the panoramic picture is subjected to correction processing to obtain a corrected panoramic picture.

[0010] Optionally, when there are multiple panoramic pictures, the performing format check processing on the panoramic pictures includes: Perform stitching processing on multiple panoramic pictures to obtain a stitched panoramic picture; Perform format verification processing on the stitched panoramic picture.

[0011] Optionally, the method further comprises: In response to a scene content adjustment instruction for the virtual reality environment screen, adjusting configuration information of the virtual background based on the scene content, the configuration information being used to determine a range and depth information of a panoramic picture to be rendered; Re-rendering the virtual reality environment based on the configuration information to generate an adjusted virtual reality environment image; The adjusted virtual reality environment image is displayed on the screen of the head display device.

[0012] Another aspect of the embodiments of the present application provides a virtual reality live broadcast device, which is applied to a head-mounted display device, and the device includes: An acquisition module, used to acquire a panoramic picture after the audience user enters the live broadcast room through the head display device; A configuration module, used to configure the panoramic image as a virtual background in a virtual reality environment; A rendering module, used to render the virtual reality environment to generate a virtual reality environment picture; A display module is used to display the virtual reality environment image on the screen of the head display device.

[0013] Another aspect of an embodiment of the present application provides a head mounted display device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0014] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0015] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned method when executed by a processor.

[0016] The embodiment of the present application adopts the above technical solution and may include the following advantages: after the audience user enters the live broadcast room using the head-mounted display device, the embodiment of the present application obtains the panoramic picture uploaded by the anchor in advance, and then provides the obtained panoramic picture to the hardware system of the head-mounted display device. The hardware system of the head-mounted display device configures the panoramic picture as the virtual background of the virtual reality environment, and then renders the virtual reality environment to generate a virtual reality environment picture. Finally, the virtual reality environment picture is displayed on the screen of the head-mounted display device. In this way, the audience user can enter a virtual environment live broadcast room with a panoramic picture as the background through the head-mounted display device to watch, so as to obtain an immersive live viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 The operating environment diagram of the virtual reality live broadcast method according to the first embodiment of the present application is schematically shown.

[0019] Figure 2 The flowchart of the virtual reality live broadcast method according to the first embodiment of the present application is schematically shown.

[0020] Figure 3 Schematically shows Figure 2 Detailed diagram of the step flow of step S202.

[0021] Figure 4 Schematically shows Figure 2 Detailed diagram of the step flow of step S202.

[0022] Figure 5 Schematically shows Figure 3 Detailed diagram of the step flow of step S300.

[0023] Figure 6A block diagram of a virtual reality live broadcast device according to Embodiment 2 of the present application is schematically shown.

[0024] Figure 7 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0026] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0027] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0028] First, the following terms are explained: Head-mounted display device: also known as head-mounted display device (HMD), is a device that provides an immersive visual experience by placing the display screen close to the viewer's eyes.

[0029] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: When broadcasting live, anchors usually use curtains or real objects to set up the live scene. However, due to the limitations of the audience's terminal devices, no matter how the anchor arranges the live scene, the audience users cannot achieve an immersive live viewing experience. 。

[0030] To this end, an embodiment of the present application provides a virtual reality live broadcast technical solution. In this technical solution, after the audience user uses a head-mounted display device to enter the live broadcast room, the panoramic picture uploaded in advance by the anchor is obtained, and then the obtained panoramic picture is provided to the hardware system of the head-mounted display device. The hardware system of the head-mounted display device will configure the panoramic picture as the virtual background of the virtual reality environment, and then render the virtual reality environment to generate a virtual reality environment picture. Finally, the virtual reality environment picture will be displayed through the screen of the head-mounted display device. In this way, the audience user can enter a virtual environment live broadcast room with a panoramic picture as the background through the head-mounted display device to watch, so as to get an immersive live viewing experience. See below for details.

[0031] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0032] Figure 1 The following schematically shows an environmental application diagram according to an embodiment of the present application.

[0033] The environment diagram may include a service platform 2, anchor terminals (4A, 4B, ..., 4M), and audience terminals (6A, 6B, ..., 6N). In a live broadcast scenario, the anchor terminal (4A, 4B, ..., 4M) logs in to the service platform 2, and the service platform 2 pushes the live broadcast data to the audience terminal (6A, 6B, ..., 6N) in real time.

[0034] Service platform 2 can provide live broadcast service, which can be a single server, a server cluster or a cloud computing service center.

[0035] The anchor terminal (4A, 4B, ..., 4M) is used to generate live broadcast data in real time and perform streaming operations on the live broadcast data. The live broadcast data may include audio data or video data. The anchor terminal may be an electronic device such as a smart phone, a tablet computer, a head-mounted display device, etc. Of course, the anchor terminal may be a virtual computing instance in the service platform 2.

[0036] The audience terminal (6A, 6B, ..., 6N) may be configured to receive the live broadcast data of the anchor terminal in real time. The audience terminal (6A, 6B, ..., 6N) may be any type of computing device, such as a smart phone, a tablet device, a laptop computer, a smart TV, a vehicle-mounted terminal, etc. The audience terminal (6A, 6B, ..., 6N) may have a built-in browser or a special program, and receive the live broadcast data through the browser or the special program to output content to the audience user. The content may include video, audio, comments, text data and / or the like.

[0037] The audience terminal (6A, 6B, ..., 6N) may include a player. The player outputs (e.g., displays, presents) content to the audience user. The content may include video, audio, comments, text data and / or the like. The audience terminal (6A, 6B, ..., 6N) may include an interface, which may include an input element (touch screen). For example, the input element may be configured to receive audience user instructions, and the audience user instructions may enable the audience terminal (6A, 6B, ..., 6N) to perform various operations, such as sending bullet comments, entering comments, giving gifts, etc.

[0038] The anchor terminals (4A, 4B, ..., 4M), the audience terminals (6A, 6B, ..., 6N) and the service platform 2 may be connected via a network. The network may include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls and / or proxy devices, etc. The network may include physical links, such as coaxial cable links, twisted pair cable links, optical fiber links, combinations thereof and / or the like. The network may include wireless links, such as cellular links, satellite links, Wi-Fi links and / or the like.

[0039] It should be noted that the number of the host terminals and the audience terminals in the figure is only for illustration and is not used to limit the scope of patent protection of this application. According to actual conditions, there can be any number of host terminals and audience terminals.

[0040] The technical solutions of the present application are described below through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described here.

[0041] Embodiment 1 Figure 2 The flowchart of the virtual reality live broadcast method according to the first embodiment of the present application is schematically shown. The virtual reality live broadcast method is applied to a head display device.

[0042] like Figure 2 As shown, the virtual reality live broadcast method may include steps S200 to S206, wherein: Step S200, after the audience user enters the live broadcast room through the head display device, a panoramic picture is obtained.

[0043] Step S202: configuring the panoramic image as a virtual background in a virtual reality environment.

[0044] Step S204: Render the virtual reality environment to generate a virtual reality environment screen.

[0045] Step S206: display the virtual reality environment image on the screen of the head display device.

[0046] The virtual reality live broadcast method provided in this embodiment obtains the panoramic picture uploaded by the anchor in advance after the audience user enters the live broadcast room using the head display device, and then provides the obtained panoramic picture to the hardware system of the head display device. The hardware system of the head display device configures the panoramic picture as the virtual background of the virtual reality environment, and then renders the virtual reality environment to generate a virtual reality environment picture. Finally, the virtual reality environment picture is displayed on the screen of the head display device. In this way, the audience user can enter a virtual environment live broadcast room with a panoramic picture as the background through the head display device to watch, so as to obtain an immersive live viewing experience.

[0047] The following combination Figure 2 , each step in steps S200~S206 and other optional steps are described in detail.

[0048] Step S200 After the audience user enters the live broadcast room through the head display device, a panoramic picture is obtained.

[0049] After the audience enters the live broadcast room through the head-mounted display device, the head-mounted display device can display the panoramic pictures uploaded in advance by the anchor from the local storage or the cloud.

[0050] In one embodiment, when there are multiple panoramic pictures in the cloud or local storage, a selection interface can be provided for the viewer to select the currently required panoramic picture. In another embodiment, when there are multiple panoramic pictures in the cloud or local storage, the multiple panoramic pictures can also be directly combined into one panoramic picture.

[0051] In this embodiment, the format of the panoramic image may be JPEG, PNG, HDR, etc.

[0052] Step S202 , configuring the panoramic image as a virtual background in a virtual reality environment.

[0053] In a specific implementation, a game engine such as Unity3D or Unreal Engine may be used to configure the panoramic image as a virtual background in a virtual reality environment.

[0054] Among them, Unity3D is a cross-platform game development engine that allows developers to create three-dimensional (3D) and two-dimensional (2D) games, as well as interactive content such as simulations and visualizations.

[0055] Unreal Engine is a powerful game engine developed by Epic Games. It is widely used to develop various types of video games, visualization, architectural visualization, virtual production, VR and AR applications, etc.

[0056] It is understandable that when the panoramic picture is configured as a virtual background in a virtual reality environment, the live broadcast screen of the live broadcast room can be configured as a scene content screen in the virtual reality environment at the same time.

[0057] In an optional embodiment, if Figure 3 As shown, step S202 includes: S300, performing correction processing on the panoramic image to obtain a corrected panoramic image.

[0058] S302: configuring the corrected panoramic image as a virtual background in a virtual reality environment.

[0059] In specific implementation, the correction process may include image distortion correction process, image color correction process, etc. The correction process may be implemented using libraries such as OpenCV, TensorFlow, or PyTorch.

[0060] Image distortion correction refers to the use of image processing technology to correct image distortion caused by the optical characteristics of the camera lens. This distortion usually causes straight lines at the edge of the image to bend, or certain parts of the image to appear stretched or compressed.

[0061] Image distortion is mainly divided into two types: radial distortion and tangential distortion. Among them, radial distortion is mainly caused by the lens elements inside the lens. In radial distortion, the farther the point is from the center of the image, the greater its position deviation. Tangential distortion is mainly caused by the position error of the lens elements of the lens relative to the optical axis. Tangential distortion manifests itself as objects near the center of the image not being completely parallel to the optical axis of the lens, resulting in the radial distortion of the image not being completely corrected.

[0062] Image color correction processing refers to adjusting the color of an image, which may include white balance adjustment, color cast correction adjustment, contrast adjustment, etc.

[0063] In this embodiment, the panoramic picture uploaded by the anchor may have some defects, for example, the panoramic picture may have some small distortion, or the color of the panoramic picture may not match the color of the current live broadcast room screen, etc. In order to allow the audience to have a better live viewing experience, this embodiment corrects the defects in the panoramic picture by first correcting the panoramic picture before configuring the panoramic picture as the virtual background in the virtual reality environment, and then configures the corrected panoramic picture as the virtual background in the virtual reality environment.

[0064] In an optional embodiment, if Figure 4 As shown, step S202 also includes: Step S400: extracting the depth value of the panoramic image using a pre-trained depth estimation model.

[0065] Step S402: configuring the depth value as the depth value of the virtual background in the virtual reality environment.

[0066] In a specific implementation, the depth estimation model is used to extract depth values ​​from a panoramic image. The depth estimation model can be obtained through supervised training or semi-supervised training of a large number of panoramic images.

[0067] In this embodiment, when configuring the background image in the virtual reality environment, the real depth value of the panoramic image is set as the depth value of the virtual background in the virtual reality environment, instead of the head display device estimating the depth value according to a preset algorithm. This allows a more realistic scene image to be obtained after the virtual reality environment is subsequently rendered, and the audience user will have a higher level of immersion after entering the virtual reality scene.

[0068] In an optional embodiment, the correction process includes a color correction process, such as Figure 5 As shown, step S300 includes: Step S500, obtaining color information of the live broadcast room.

[0069] Step S502: performing color correction processing on the panoramic image according to the color information to obtain a corrected panoramic image.

[0070] In a specific implementation, the color information of the current live broadcast room is used as a reference for color correction of the panoramic image, and the panoramic image is subjected to color correction processing, so that the color of the panoramic image matches the color information.

[0071] The color information can be obtained through the following steps: First, capture the live video stream using AVFoundation or a similar library.

[0072] Next, extract the frames from the video stream, usually extracting a few frames per second; Finally, color analysis is performed on each frame of the video to obtain color intensity, which can be used as color information of the live broadcast room.

[0073] It should be noted that when performing color analysis, CIImage of UIImage can be used for analysis and processing to obtain color intensity, wherein the color intensity can be represented by an average color value (such as RGB average value) or a histogram.

[0074] In this embodiment, by performing color correction processing on the panoramic image, the virtual reality environment picture obtained by subsequent rendering is made to better match the current live broadcast room, so that the audience users will have a better immersive experience.

[0075] Step S204 , rendering the virtual reality environment to generate a virtual reality environment picture.

[0076] In a specific implementation, the rendering engine will render the virtual reality environment according to the configured rendering parameters to generate a virtual reality environment picture. The rendering parameters include the virtual background, texture data, light source data, camera position data, etc.

[0077] In this embodiment, in order to ensure the real-time rendering, the graphics processing unit (GPU) configured in the head display device can be used for rendering. In addition, during the rendering process, in order to obtain low-latency and high-frame rate images, the graphics rendering pipeline can be configured and advanced anti-aliasing technology and rendering optimization strategies (such as Level of Detail, LOD) can be used to balance image quality and performance.

[0078] In one embodiment, during the rendering process, in order to facilitate the audience user to perform subsequent interactive operations in the virtual reality environment screen, interface elements for interacting with the audience user can also be presented during rendering, such as an immersion adjustment slider, elements for adjusting scene content, etc.

[0079] Step S206 , displaying the virtual reality environment image on the screen of the head display device.

[0080] In a specific implementation, after the rendering engine completes the rendering of the virtual reality environment picture, it will directly present the picture on the screen of the head-mounted display device, so that the audience user can watch the virtual reality environment picture through the screen of the head-mounted display device, thereby allowing the audience user to obtain an immersive live broadcast experience.

[0081] In an optional embodiment, the method further comprises: Perform a format check process on the panoramic picture. After the panoramic picture passes the format check process, execute step S300.

[0082] In specific implementation, since the panoramic pictures uploaded by the anchor in advance may not necessarily match the head-mounted display device used by the current audience user, the head-mounted display device will first perform format verification processing on the panoramic pictures after obtaining the panoramic pictures uploaded by the anchor, so as to verify whether the format of the panoramic pictures meets the format requirements of the head-mounted display device. If the format verification processing is passed, the head-mounted display device will continue to perform correction processing on the panoramic pictures to obtain the corrected panoramic pictures; if the format verification processing is not passed, the head-mounted display device can feedback prompt information to prompt the anchor that the format of the current panoramic pictures does not meet the requirements, so that the anchor can upload qualified panoramic pictures in time.

[0083] In this embodiment, after the panoramic picture is acquired, a format check process is first performed so that the panoramic picture that does not conform to the format can be discovered in time.

[0084] In an optional embodiment, when there are multiple panoramic pictures, the format verification processing of the panoramic pictures includes: stitching the multiple panoramic pictures to obtain a stitched panoramic picture; and format verification processing of the stitched panoramic picture.

[0085] In a specific implementation, when uploading a panoramic picture, the host can upload multiple panoramic pictures at one time. When the host uploads multiple panoramic pictures at one time, it indicates that the host hopes to use the stitched pictures of these pictures as a complete panoramic picture. Based on this situation, when the head display device receives multiple panoramic pictures, it will stitch the multiple panoramic pictures to synthesize a panoramic picture. After obtaining the stitched panoramic picture, the format check process will continue to be performed on the stitched panoramic picture, so there is no need to perform a format check process on each panoramic picture in turn.

[0086] In this embodiment, the panoramic pictures are first stitched together and then formatted and verified. This allows the verification operation to be completed at one time, saving verification time.

[0087] In an optional embodiment, the method further comprises: In response to a scene content adjustment instruction for the virtual reality environment picture, adjusting configuration information of the virtual background based on the scene content, the configuration information being used to determine a range and depth information of a panoramic picture to be rendered; re-rendering the virtual reality environment based on the configuration information to generate an adjusted virtual reality environment picture; and displaying the adjusted virtual reality environment picture on a screen of the head-mounted display device.

[0088] In a specific implementation, the scene content adjustment instruction may include a viewing angle range adjustment instruction, a scene focus adjustment instruction, etc.

[0089] The viewing angle range adjustment instruction is used to adjust the viewing angle range of the viewer user in the virtual reality environment. When the viewer user needs a higher degree of immersion, the viewing angle range of the viewer user in the virtual reality environment can be set to a 360-degree viewing angle, and when the viewer user needs a lower degree of immersion, the viewing angle range of the viewer user in the virtual reality environment can be set to a certain direction.

[0090] The scene focus adjustment command is used to adjust the scene content within the viewer user's sight focus. When the viewer user triggers the scene focus adjustment command, certain parts of the panoramic image can be dynamically selected through a preset algorithm for highlighting or introducing animation elements for display.

[0091] In addition, in order to obtain better display effects, when rendering the virtual reality environment, PBR (Physically Based Rendering) technology can be used to dynamically adjust the ambient lighting to ensure the realistic light and shadow effects in the scene. Ambient lighting includes parameters such as light source, shadow, and reflection.

[0092] In this embodiment, the virtual reality scene is adjusted by responding to the scene content adjustment instruction, so that the scene can be dynamically configured according to the needs of the audience user, thereby enhancing the audience user experience.

[0093] Embodiment 2 Figure 6 The block diagram of the virtual reality live broadcast device 600 according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 6 As shown, the device 600 may include: an acquisition module 601, a configuration module 602, a rendering module 603 and a display module 604, wherein: The acquisition module 601 is used to acquire a panoramic picture after the audience user enters the live broadcast room through the head display device.

[0094] The configuration module 602 is used to configure the panoramic image as a virtual background in a virtual reality environment.

[0095] The rendering module 603 is used to render the virtual reality environment to generate a virtual reality environment picture.

[0096] The display module 604 is used to display the virtual reality environment image on the screen of the head display device.

[0097] As an optional embodiment, the configuration module 602 is further used to: perform correction processing on the panoramic image to obtain a corrected panoramic image; and configure the corrected panoramic image as a virtual background in a virtual reality environment.

[0098] As an optional embodiment, the configuration module 602 is further used to: extract the depth value of the panoramic image using a pre-trained depth estimation model; and configure the depth value as the depth value of the virtual background in the virtual reality environment.

[0099] As an optional embodiment, the correction processing includes color correction processing, and the configuration module 602 is also used to: obtain color information of the live broadcast room; perform color correction processing on the panoramic picture according to the color information to obtain a corrected panoramic picture.

[0100] As an optional embodiment, the device 600 is also used for: performing format verification processing on the panoramic picture, and after the panoramic picture passes the format verification processing, performing correction processing on the panoramic picture to obtain a corrected panoramic picture.

[0101] As an optional embodiment, when there are multiple panoramic pictures, the device 600 is further used to: stitch the multiple panoramic pictures to obtain a stitched panoramic picture; and perform format verification on the stitched panoramic picture.

[0102] As an optional embodiment, the device 600 is also used to: respond to a scene content adjustment instruction for the virtual reality environment picture, adjust configuration information of the virtual background based on the scene content, and the configuration information is used to determine the range and depth information of the panoramic picture to be rendered; re-render the virtual reality environment based on the configuration information to generate an adjusted virtual reality environment picture; and display the adjusted virtual reality environment picture on the screen of the head display device.

[0103] Embodiment 3 Figure 7 The schematic diagram of the hardware architecture of a head display device 700 suitable for implementing a virtual reality live broadcast method according to the third embodiment of the present application is schematically shown. In some embodiments, the head display device 700 may be VR glasses, VR helmets, etc. Figure 7 As shown, the head display device 700 includes but is not limited to: a memory 701, a processor 702, and a network interface 703 that can communicate with each other through a system bus. The memory 701 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 701 can be an internal storage module of the head display device 700, such as a hard disk or memory of the head display device 700. In other embodiments, the memory 701 can also be an external storage device of the head display device 700, such as a plug-in hard disk equipped on the head display device 700, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 701 can also include both the internal storage module of the head display device 700 and its external storage device. In this embodiment, the memory 701 is generally used to store the operating system and various application software installed on the head display device 700, such as the program code of the virtual reality live broadcast method, etc. In addition, the memory 701 can also be used to temporarily store various types of data that have been output or are to be output.

[0104] In some embodiments, the processor 702 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 702 is generally used to control the overall operation of the head display device 700, such as performing control and processing related to data interaction or communication with the head display device 700. In this embodiment, the processor 702 is used to run the program code stored in the memory 701 or process data.

[0105] The network interface 703 may include a wireless network interface or a wired network interface, and the network interface 703 is generally used to establish a communication link between the head display device 700 and other computer devices. For example, the network interface 703 is used to connect the head display device 700 to an external terminal through a network, and to establish a data transmission channel and a communication link between the head display device 700 and the external terminal. The network can be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0106] It should be pointed out that Figure 7 Only a computer device having components 701 - 703 is shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0107] In this embodiment, the virtual reality live broadcast method stored in the memory 701 can also be divided into one or more program modules and executed by one or more processors (such as processor 702) to complete the embodiment of the present application.

[0108] Embodiment 4 An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the virtual reality live broadcast method in the embodiment are implemented.

[0109] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the computer-readable storage medium may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the virtual reality live broadcast method in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.

[0110] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0111] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0112] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A virtual reality live broadcast method, applied to a head-mounted display device, characterized in that: The method comprises: After the audience user enters the live broadcast room through the head display device, a panoramic picture is obtained; configuring the panoramic image as a virtual background in a virtual reality environment; Rendering the virtual reality environment to generate a virtual reality environment picture; The virtual reality environment image is displayed on the screen of the head display device.

2. The method according to claim 1, characterized in that The configuring the panoramic picture as a virtual background in the virtual reality environment comprises: Performing correction processing on the panoramic picture to obtain a corrected panoramic picture; The corrected panoramic picture is configured as a virtual background in a virtual reality environment.

3. The method according to claim 2, characterized in that The configuring the panoramic picture as a virtual background in the virtual reality environment further comprises: Extracting the depth value of the panoramic image using a pre-trained depth estimation model; The depth value is configured as a depth value of the virtual background in the virtual reality environment.

4. The method according to claim 2, characterized in that: The correction processing includes color correction processing, and the correction processing of the panoramic picture to obtain a corrected panoramic picture includes: Obtaining color information of the live broadcast room; Color correction is performed on the panoramic picture according to the color information to obtain a corrected panoramic picture.

5. The method according to claim 2, characterized in that: The method further comprises: The panoramic picture is subjected to format verification processing, and after the panoramic picture passes the format verification processing, the panoramic picture is subjected to correction processing to obtain a corrected panoramic picture.

6. The method according to claim 5, characterized in that In the case that there are multiple panoramic pictures, the format checking process for the panoramic pictures includes: Perform stitching processing on multiple panoramic pictures to obtain a stitched panoramic picture; Perform format verification processing on the stitched panoramic picture.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: In response to a scene content adjustment instruction for the virtual reality environment screen, adjusting configuration information of the virtual background based on the scene content, the configuration information being used to determine a range and depth information of a panoramic picture to be rendered; Re-rendering the virtual reality environment based on the configuration information to generate an adjusted virtual reality environment image; The adjusted virtual reality environment image is displayed on the screen of the head display device.

8. A virtual reality live broadcast device, applied to a head-mounted display device, characterized in that: The device comprises: An acquisition module, used to acquire a panoramic picture after the audience user enters the live broadcast room through the head display device; A configuration module, used to configure the panoramic image as a virtual background in a virtual reality environment; A rendering module, used to render the virtual reality environment to generate a virtual reality environment picture; A display module is used to display the virtual reality environment image on the screen of the head display device.

9. A head mounted display device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 7 are implemented.

Images